Data Update after running Strategy Monitor
Author: Shaaker
Creation Date: 7/27/2010 7:04 PM
profile picture

Shaaker

#1
How can I prevent Strategy Monitor from updating datasets?
I activate SM on 1min datasets before the market open (around 9:20am). As I only need price action starting 9:30am I select the minimum Data Range of 2Bars. But it seems that SM updates the dataset files up to the EOD of the day before even when both Update Data on Demand under File menu and in Data Manager are unchecked.

Also, running the Bad History Data Check strategy shows that some of thes files now have bad bars (like data gaps) going back months, however they had 0 bad bars before being update by SM. Shouldn't updating only add data to the end of the file and check for corrections of the last update without changing older data?



profile picture

Cone

#2
QUOTE:
How can I prevent Strategy Monitor from updating datasets?
Easy answer: don't activate any Strategies that point to EOD DataSets. When you say EOD, we think "Daily" scale.

The S. Monitor will not update Fidelity static data files for intraday intervals 10 minutes or less (other Providers may do it). Nonetheless, if you use a 2-bar range and start monitoring at or before 0930, the first 2 bars have to come from the previous day because that's what you've requested.

QUOTE:
Shouldn't updating only add data to the end of the file and check for corrections of the last update without changing older data?
Intraday corrections can be applied to the last 3 days stored in the history, but sure, it shouldn't create gaps. But look, be specific about symbols, scales, dates, and your procedure to make this happen. As I said, the S. Monitor should not be updating Fidelity static intraday data for scales under 10 minutes, so it's hard for me to understand what your seeing or how it could even occur.
profile picture

Shaaker

#3


First, I should mention that by "EOD" I was referring to end of the day for 1min scale Dataset that were being used in S. Monitor. Based on your recommendation this morning I activated the Strategies only after 9:31 when selecting "Most Recent 1Days" as Data Range. But it's still updating the data files.

On the second problem here is what happened:
During the weekend I updated and checked all 1min data files by running the Bad History Data Check strategy and reloaded those with bad bars (like gaps, etc) and checked again making sure they were flawless, and I was planing to update only over the weekends after correcteions by Fidelity. Tuesday night I noticed that all the files had been updated upto the end of Monday and the modification time was Tuesday morning when I had activated S. Monitor. Just as an example: CSCO data now has a gap of 2043 days at 7/15/2010 beside some other bad bars. However I'm sure it had 0 bad bars and it's one of the many files that have been altered.
profile picture

Cone

#4
Let me back up. The Fidelity DataStore for 10-Min intervals and below will initially update all data files on the first request in the S. Monitor, but thereafter the data files will not updated with each subsequent request. That's because rewriting huge data files to disk every minute or so takes a long time and would slow things down to nearly a halt.

QUOTE:
2043 days at 7/15/2010
That doesn't sound right. Maybe you meant "bars" because 2043 days is a little over 8 years of data - more history than any intraday symbol should even contain.

I'll take a look now to see if I can duplicate what you're seeing with the gaps in the updates.
profile picture

Shaaker

#5
1. As you said S. Monitor does not update with subsequent requests, but then can we say there is no way to control updating upto the time of initial update regardless of time of activation or selection of Data Range?

2. 2043 days is very strage to me too, but when I check the chart going backwards there is an abrupt change from 7/15/2010 to 12/10/2004 (1min scale chart). I'm sure if I reload the chart or the data file bad bars would be corrected as I did to many data files over the weekend, but the strange part is how these file got corrupted again after being updated for only 1day of data?

It seems other users have experienced similar problem:
/Forum/Posts/Checking-data-for-out-of-sequence-Dates-30651
profile picture

Eugene

#6
Re: #2. What Cone was trying to say is that there are about 1500 trading sessions between 2004 and 2010. So it's simply not possible to have 2043 days skipped on an intraday Fidelity chart (as implied) as their historical data only goes back to 2004.

Question is, how did you determine that number? And how does your affected chart look like?
profile picture

Shaaker

#7
Eugene,

That number comes from Bad History Data Check strategy. As I said the chart has a sudden change of date and a price gap up. I can open a ticket and sent you a picture of the chart or even the datafile. But main question is how could possibly a once sound data file become so corrupted after update for only 1day of data?
profile picture

Cone

#8
I was able to duplicate a gap problem in several symbols after updating a Nasdaq 100 1-Minute bar DataSet using the Strategy Monitor as described. While I only concentrated on checking the most-recent year of data, what strikes me as very odd is that the data are being lost (5-day gaps showing, 9 days if you include the weekend span) 2 months before the last bar in the previus update.

Thanks for bringing this to our attention, I'd appreciate it if you could create a Support Ticket and we'll follow up with Fidelity to investigate the cause.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).