For-my-custom-provider-Data-Manager-DataSets-table-shows-incorrect-data

It's determined by issuing a call to the standard WL8 GetHistories method, and examining the resulting BarHistories returned. The request is made in an "offline mode" status so only existing cached data is returned. Click the "Update DataSet" button there to force a data update.

If the DataSet is unlinked, WL8 will attempt to get a symbol's data from the Historical Providers you have enabled in the Data Manager, Historical Providers tab, in the order that is set up there.

If the DataSet is linked to a Provider, it will attempt to get the data from the linked Provider first before going through any other Providers.

You can examine the source column, the last column in the table, to see which Provider returned data for any symbol.

I'm removing the bug tag since, while there might be a bug in your custom Provider you're developing, there doesn't appear to be a bug in WL8 around this, the DataSets tab is operating correctly as far as I can see.

I think I see what is going on here.

My Historical data Provider gets its data from .zip files stored on the local disk.
It uses the Persistent Storage Mechanism of WL8 to avoid unzipping and parsing these files again and again.

Currently this Mechanism creates Files in
<User Data Folder>\finantic.HDP\1 Minute\EURUSD.QX

This very same location is used for all DataSets created by finantic.HDP, which means the data of different DataSets create/override Files at the same location.

Is there a (simple) way to use WL's Persistent Storage Mechanism but use different Folders for the QX files (depending on the DataSet name or DSString etc)?

I Figured I could simply add these lines to my Historical Data Provider class:

CODE:
       
public override BarHistory LoadFromStorage(string symbol, HistoryScale scale, int maxBars = 0)
{
            BarHistory bh = new BarHistory(symbol, scale);
            string fileName = GetStorageFileName(symbol, scale);
            bh.ReadFromBinaryFile(fileName, DateTime.MinValue, DateTime.MaxValue, maxBars);
            return bh;
}

public override void SaveToStorage(BarHistory bh)
{
            // Create Unique FileName
            string fileName = GetStorageFileName(bh.Symbol, bh.Scale);
            bh.WriteToBinaryFile(fileName);
}

But these two functions get never called(!)

Do I miss something?

Hmm, it looks like the data loader is not calling those methods at all of the places that it could (and maybe should) be. It's bypassing them. I can make some changes for Build 52 to make sure these methods get called and that should resolves this issue for you.

It looks like we'll also have to change the signature of LoadFromStorage to accept the start and end date parameters.

While we're at it we can also make ConstructSymbolFileName virtual, so it might be better for you to intercept it there once Build 52 is out.

I have a Data Set which contains 1 million bars per symbol.
However, if I show Data Manager->DataSets the Symbol table shows something different:


Even if I do "Update DataSet" the information shown is still not accurate.

When I click one of the symbols in WL's DataSets tree, a Chart comes up with the complete data (all bars available in the DataSet for this symbol).

After this action the Symbol table shows the correct information for this clicked symbol.

I'd expect the symbol table showing all *available* data. i.e. the same number of bars as a new chart window (with Settings "All Data").

I'm not able to duplicate. Can you show the complete table including the Source column at the end?

I also thought along the same lines as Dion: if your provider fails to unpack those ZIP files or something, the next data provider (with a 100K bars limit) checked on the Historical Providers tab may be picked up by WL to deliver the data.

Ok, let's wait for build 52 and see if it improves the situation.

I am happy!
With the new

CODE:
virtual string ConstructSymbolFileName()

things look much better now. The binary QX files find their way to separated folders for each DataSet. Great!

Now Data manger behaves as it should!

I no longer need to override LoadFromStorage() and SaveToStorage(BarHistory bh).

Just as a side note: Even if I override these two routines, SaveToStorage() is never called in my HDP.

Yes, I see a few spots where it should be called but isn't. Changing that for B53.

Just one remark:
With file sizes/history lengths like this:

Things get really slooooow....

This is obviously caused by reading all the data into a BarHistory object (as explained above) just to show the statistics in the Symbol table.

I'd #FeatureRequest to us a special access to the QX files here and read the required data in some direct access method.

It took several minutes before the symbol table was displayed - with just 8 forex pairs.
Imagine I create a DataSet with 100 pairs...

BTW: The same happens with 1-Minute intraday data from Tiingo. (before you accuse my new Historical Data Provider)

Yes but the issue is that the data might not be coming from QX files. We'd need a new virtual method to return the metadata. Would be a good optimization here, agreed.