The number of bars stored in my datasets seems to vary a ton after "Update DataSet", and I'm wondering if there's some sort of bug. In the image below, notice that the ETH symbol has 1.7 million bars, but BTC only has 380 thousand. What could be the cause of this?
Rename
I don't know what might have caused the issue in your case but the number of added bars for all symbols looks unexpectedly big.
Highlight just ETHUSD, enter Data Truncation, choose "For 1 symbol selected - Delete all data". Then you can update the DataSet. Let us know if the issue repeats.
Highlight just ETHUSD, enter Data Truncation, choose "For 1 symbol selected - Delete all data". Then you can update the DataSet. Let us know if the issue repeats.
Thank you for the suggestion, Eugene. Deleting and redownloading the data seems to have corrected the issue.
Having said that, I'm pretty certain there's a fairly critical bug taking place here. Now that I know what to look for, I've noticed that it's happened in other datasets as well. I've only been using Tiingo recently, so I don't know if the problem is specific to that data provider. It's not limited to just Crypto symbols, though.
The issue seems to be that sometimes when you press the "Update DataSet" button in the "Data Manager" on an existing data set, it will download far more rows than it should. Perhaps they are some sort of duplicate, but they don't seem to be flagged by the "Data Integrity" check. But these extra rows drastically affect the results of the backtest, so I'm finding that any strategies that were optimized or backtested on that data are pretty much worthless.
If you are able, could you please flag my original post as a bug report? It looks like I am no longer able to do so, and I wasn't sure if it was actually a bug or not when I first posted.
Having said that, I'm pretty certain there's a fairly critical bug taking place here. Now that I know what to look for, I've noticed that it's happened in other datasets as well. I've only been using Tiingo recently, so I don't know if the problem is specific to that data provider. It's not limited to just Crypto symbols, though.
The issue seems to be that sometimes when you press the "Update DataSet" button in the "Data Manager" on an existing data set, it will download far more rows than it should. Perhaps they are some sort of duplicate, but they don't seem to be flagged by the "Data Integrity" check. But these extra rows drastically affect the results of the backtest, so I'm finding that any strategies that were optimized or backtested on that data are pretty much worthless.
If you are able, could you please flag my original post as a bug report? It looks like I am no longer able to do so, and I wasn't sure if it was actually a bug or not when I first posted.
One possible cause just occurred to me: sometimes I run two Windows of WL7, so that one window can be optimizing a strategy, while I design strategies in a separate WL7 Window (to avoid input response lag from the optimization). What happens if both WL7 Windows attempt to "Update DataSet" at various times throughout the day?
QUOTE:
What happens if both WL7 Windows attempt to "Update DataSet" at various times throughout the day?
Mike, to clear your concern I started a simultaneous update of an incomplete Tiingo intraday DataSet in two Wealth-Lab 7 windows. Shortly they both appear updated having the same bar count. Although I wouldn't recommend doing this.
QUOTE:
The issue seems to be that sometimes when you press the "Update DataSet" button in the "Data Manager" on an existing data set, it will download far more rows than it should.
It looks like we have a bug in the Tiingo provider. Due to subscription quota it has to download the intraday history in chunks. This is where I suppose the bug may be but to fix it we'd need a step by step procedure to reproduce.
Your guess sounds correct to me. I don't know exact reproduction steps yet, but this is the behavior I generally noticed:
1) I would click "Update DataSet", usually for 15 minute or 1 minute scale. My largest DataSet has about 25 symbols in it.
2) I would notice that the number of rows or the "Last Date" appeared incomplete
3) I would continue pressing "Update DataSet" at various times throughout the day, and sometimes it would say "No bars added", and other times it would download more rows
After several days of repeating the above steps, I started to notice that the total number of bars for certain symbols had become much larger than other comparable symbols.
1) I would click "Update DataSet", usually for 15 minute or 1 minute scale. My largest DataSet has about 25 symbols in it.
2) I would notice that the number of rows or the "Last Date" appeared incomplete
3) I would continue pressing "Update DataSet" at various times throughout the day, and sometimes it would say "No bars added", and other times it would download more rows
After several days of repeating the above steps, I started to notice that the total number of bars for certain symbols had become much larger than other comparable symbols.
Thanks for the follow-up Mike, I'll try to reproduce.
Adding some safeguards against inserting such duplicate bars by Tiingo provider in DataExtensions B17.
Thank you much, installing the update now! Is it possible for the data integrity tool to identify this issue, so that I can see which of my existing data sets need to be deleted and redownloaded?
You're welcome. Although the data integrity tool checks for synthetic bars (i.e. adjacent duplicate bars), this issue cannot be identified. Here you're dealing with blocks of duplicate bars. I suggest you reload previously accumulated Tiingo intraday data by using either truncation or "Delete local files".
Your Response
Post
Edit Post
Login is required