Could-not-load-Backtest-Data-in-this-Scale-Range-8

I have a similar but not the same issue with Norgate data that might be related to this. It happens almost every day if I run a dozens of backtests with a few strategies. The only thing I can do then is to restart WL and it will definingly go away until it happens again. I'm not sure how to reproduce this exactly, but usually it happens with the follow sequence of symptoms.

1. A strategy that uses a Norgate dataset backtests just fine.
2. Somehow when I backtest the strategy again with the same dataset, only a very small number of symbols are loaded (In the screenshot i'm using N100 current & past which has 196 symbols if loaded correctly. Here it only shows 13 symbols are loaded when the problem happens).
3. When the data is reloaded and strategy is rerun, the "can not load dataset in this scale/range" happens. At which point I have to restart WL if i want to use the same dataset in any strategies.

2 does not happen right after 1 and usually there are other operations such as running other strategies, etc. This is the part that seems to be random to me so I do not know the exact steps to reproduce this;
3 usually happens right after 2.

Any thoughts about this cause? It seems to be happening on more and more of my new and modified strategies.

It seems to be something wrong with the Norgate extension, so could also write their support to have a look at this thread.

I don't use Norgate. My problem is with data that is strictly ASCII.

One bit of info...

If I copy the strategy into a new C# template with all of the same strategy parameters that sometimes solves the problem.

@Glitch I'll write an email to Norgate.

Does WL cache data in memory to use across different strategies? If possible could WL have a button to clear cached data and force reload all data from source? Since restart WL solve the issue for me, I feel that force reload all data might help in this case. This happened for me more often recently so I had to restart multiple times each day sometime.

You can force a reload by clicking the message at the bottom of the Strategy window that says "N Symbols Loaded". That will change the message to a red "Need to Load Data" and it will do a fresh reload the next time you run the Strategy.

Thanks @Glitch. I actually use that button a lot if i want to reload data for a single strategy. However once the problem I described here happens that won't work and I keep getting the same error with reload.

My last post was based on my guess that there might be a global cache in WL for datasets for all strategies (e.g. two strategies under backtesting use the same dataset), so whether it's possible to invalidate all data in that cache and reload from disk. I might be totally wrong about WL here.

QUOTE:
... once the problem I described here happens that won't work and I keep getting the same error with reload....

We are talking about two different caches. There's a cache created by Data Manager, and there's a cache created by the strategy itself. You can reload the latter one, but you really can't reload the former one without going into Data Manager and selecting "Delete Local Files" for that particular data provider. But if you do that, then that provider has to re-download all the data for all your stocks, which you don't want to do unless those files (which are saved to disk) got corrupted somehow like during a power failure or an OS lockup.

But what you're saying is that there's something wrong with the Data Manager data provider. So are you using Norgate for your data provider?

@superticker
Yes that's what I meant. Now to force reload data from data providers one has to delete local data files. It would be great if one could reset/clear WL in-memory cached data for all data providers with a click of a button without deleting and downloading the data files from source. Then if sometime a data provider is misbehaving due to bugs, like what I'm experiencing now, one can just click on the button to reload all data.

For data source, I use WD for most live strategies if i can use it, but I do have two strategies for R1000 that I use Norgate. And for strategy development I use both.

No, there's no such global cache, each Strategy maintains its own cache.

QUOTE:
It would be great if one could reset/clear WL in-memory cached data for all data providers with a click of a button without deleting and downloading the data files from source.

Well, what's in RAM memory is cached by the strategy itself, and you can reload that, but you're saying that doesn't fix the problem.

What's cached by the Data Manager is cached on disk--not RAM memory. But restarting WL isn't going to fix what's cached on disk. And the discussion is saying restarting WL fixes the problem initially, so what's cached on disk is okay.

So are we saying the process of fetching the data (by the Norgate provider) from disk to strategy is broken? If so, then why does the strategy execute the first time okay but not subsequent times? And if the strategy is corrupting its own copy of the RAM cached data, then why doesn't reloading that data (from the provider's disk cache) fix it?

More importantly, why am I--and everyone else--not seeing this problem? Are you suggesting this is a memory management problem unique to your hardware? If so, have you flashed your BIOS lately? Remember, the BIOS steers the MMU (Memory Management Unit) of your processor. I think your BIOS is throwing a MMU mapping register in the large memory model. (Applications with smaller memory footprints would remain unaffected.)