Extra bars appear in my Fidelity static data
Author: KGo
Creation Date: 11/28/2016 7:49 PM
profile picture

KGo

#1
Friday intraday Fidelity data has extra bars at EOD Friday. SPY bars at 1:01, 1:11 and 1:13 pm should not be there. All higher intraday time frames have extra bars consistent with that 1 min data. I have reported this to Fidelity so hopefully someday a refresh will be correct.

In WLP 6:9:16 I have tried to delete the extra bars on a 5 min chart unsuccessfully. Did below after searching the forum/WIKI:
1. Turned off on demand updates.
2. Opened a single new strategy in static mode
3. Refreshed all SPY Data into today
4. Double clicked the extra bars and chose Remove
5. Clicked another symbol and back to SPY to see edits were still good
6. Opened a 5 min workspace with both static and streaming windows to find edits were gone and in fact a fourth extra bar was added after 1:35 pm with 1:05 pm time and data.

How to I keep the edits when WLP reopens the workspace?
profile picture

Eugene

#2
Cone should know better but it seems that the Fidelity provider is programmed to download corrections on backfill which may include last N sessions.
profile picture

KGo

#3
Thanks, I'll wait for Cone. Knowing how may sessions are corrected for intraday data will help.

Meanwhile this error hit virtually all Fidelity ETF and stocks. So AAPL, IBM, QQQ, GOOGL and all others have typically at least 1 extra bar. Users should contact Fidelity to improve chances of getting it fixed.
profile picture

KGo

#4
Today, on Tuesday 35 min into the session, Friday data was corrected again and the changes did not hold.

Notes
1. The daily download skipped SPY as already up to date.
2. At 5 min scale, after deleting SPY 1:05, 1:15, and 1:35 pm bars upon opening a new workspace those bars were added back PLUS a new 1:05 pm bar was added. So now it's worse. There are in order 1:05, 1:15, 1:35 and 1:05 pm bars in that order.

So, after data edits downloaded corrections on backfill of N sessions does not match a data refresh.

profile picture

Cone

#5
QUOTE:
How to I keep the edits when WLP reopens the workspace?
Nothing is required. Changes for specific dates are saved with the data file and should not be overwritten on updates. However, apparently this isn't working properly for deletions. (Note that the Reload Chart History action will refresh all chart history from the server.)

How to proceed?
Intraday data are updated with a 2-day overlap. Consequently, after today's session, you can delete bars from 11/25 and they should not be touched again.

Also, these records need to be added to the Market.xml file located in C:\Program Files\Fidelity Investments\Wealth-Lab Pro 6\Data\Markets.xml You'll need to uncheck the read-only file property, and then open it in Notepad using Administrator mode to add and save these changes.

CODE:
Please log in to see this code.


Or, just replace the file with the one in the zip file attached here.
profile picture

KGo

#6
I replaced the Markets.xml file and restarted WLP. Will this remove extra bars in future reloads?

Live the bars keep coming back. Hopefully as you say this will be fixable tomorrow, Thursday.

Note there was one scenario that allowed streaming data with the corrected bars. When data was corrected between sessions and the streaming charts were opened before the new session the corrections persisted in the live streaming windows. That was the best workaround.

Unfortunately one cannot open a streaming chart during the session with corrected data. And a static chart used during the session reloaded the error bars but had no effect on the streaming windows.

So, the error bars were Friday and the earliest true correction is Thursday. That's Error day+4. Too long to be running with bad data. Perhaps another method to immediately correct intraday data errors can be created.
profile picture

KGo

#7
The changes persisted this Thursday morning. Manual bar edits of Friday data held as Cone explained. Thank you!
profile picture

Cone

#8
QUOTE:
Perhaps another method to immediately correct intraday data errors can be created.
I don't think so, but the manual deletions should not be overwritten. That's a bug that could be fixed.

QUOTE:
I replaced the Markets.xml file and restarted WLP. Will this remove extra bars in future reloads?
Unfortunately that doesn't appear to be the case. I had hoped so, and that's why I looked there and noticed the missing data.
profile picture

KGo

#9
I've had multiple instances in the last 1-2 months where extra bars were added at 4pm and for multiple days. Sometimes there have been a dozen new bars each day. This is a separate problem from the one discussed above which originated with incorrect Fidelity server data but the symptom is very similar in that extra bars are added after 4pm. The bars are only added during the prior few days that are in the Fidelity update overlap window.

I suspect the bars were added by WLP when a static strategy was executed coincident with a streaming update. That would make it a WLP error. But that's just a guess. Unfortunately I cannot replicate the behavior at will and the daily data download does not correct it.

To fix a copy of SPY from an second streaming system is used. I have a SPY.WL file demonstrating the extra bars. However it cannot be attached here.
profile picture

Cone

#10
QUOTE:
I suspect the bars were added by WLP when a static strategy was executed coincident with a streaming update.
It's an interesting observation if it could be duplicated, but probably that can't happen since the static provider doesn't add streaming bars to its history cache. New streaming bars are added to the chart instance at the end of each interval, but the static provider is "done" after the initial data fill.

p.s. If required, you can zip a non-image or text file type to attach.
profile picture

KGo

#11
The streaming provider clearly adds data to the .WL each new bar. The streaming updates attached show 48 bytes are added every 5 min to SPY.WL.

The ERR_OK files show the condition of SPY (_OK) on an undisturbed streaming system vs the condition of SPY(_ERR) with extra bars on a system that had other work in progress. Note extra bars times and values.

My thought was that the coincident static execution during the trading day caused a data malfunction. The static checks Fidelity to see if data is up to date and finds the last bar is needed. Meanwhile a streaming interval occurs also to add the last bar and the data being updated from 2 processes gets corrupted.

i hope that is incorrect. However my point is that this data malfunction occurs however rarely and I document it here in case others see the same behavior. Perhaps it can be understood and prevented.

All attached are 5 min bars. One Zip to verify updates every 5 min and one with _OK _ERR files showing extra bars.

Zipped SPY.wl is 4.7MB so cannot share.

Eugene, Could you remove the date from this topic title. I just corrected extra bars that were added sometime today.

Please confirm:
A) streaming strategy symbol .wl files are written to disk every bar interval.
B) a non-streaming strategy thru current day will write to disk all completed bars at execution time.
profile picture

Cone

#12
A) We can't confirm that for the Fidelity provider, but if that's your observation, then it must be true. I'm at least 99% certain that the Strategy Monitor won't update data files for less than 10 minute intervals (possibly inclusive) because of the overhead of writing enormous amounts of data to disk.

B) Only if you've selected to update data on-demand (File menu, or Data Manager > Update Data > On Demand checkbox (lower right corner)
profile picture

KGo

#13
Thanks, I can't speak to Strategy Monitor updates, but using Strategy Window, 5 min scale files are written every bar. The 4pm bar however is not written at 4pm but will wait for the overnight update. A static execution after 4:05pm will also write the 4pm bar.

In the attached images extra bars are inserted after the 3:55pm bar and before the 4pm bar. The extra bars have been inserted for 4 days and are data from the following day. All extra bars have following day date with 12:00 time. Prices match bars at 10 min intervals the following day. So the 9:40, 9:50, 10:00 etc bars on 1/5 were inserted before 4pm on 1/4. Visually it's easy to see. I've seen this alternate bar behavior a few times now though only a partial day may be inserted.
profile picture

KGo

#14
Third file did not show up. Here it is again.
profile picture

KGo

#15
Today's occurrence. Just noticed 1 bar has been added 1/6 and 1/9. Different style from above.Just 1 tall bar per day. See the 2 highest volume bars in attached.
profile picture

KGo

#16
The problem is nearly repeatable.

Extra bars were inserted again this morning. The workspace containing 10 static and streaming strategy windows opened successfully with no extra data present. Streaming bars were updating normally. When the end date for one of the static strategies was changed from 1/6 to 1/12 (today) an extra bar was inserted in 1/9 and 1/10. The session had been open 20 minutes.

Fixed SPY. Then closed workspace and reopened it without error. Upon changing a static strategy date from 1/6 to 1/12 lots of bars were added. See attached. (Chart is right click copy not full screen shot which may be reason for low quality but that's all I have now)

Replicated this behavior multiple times after reopening workspace or WLP.

When the workspace static end date was set to 1/3 the extra bars did not appear when updated to 1/12. Saved the workspace with earlier static start date and hope behavior does not return. Time will tell.

This is clearly a WLP data update error.
profile picture

Cone

#17
I appreciate the follow up and documentation. It's on the bug list.

Do you know if this occurs for regular stocks? or just for ETFs?
profile picture

KGo

#18
My setup is for ETF's so can't say with stocks.

All was working perfectly for over a year until the first problem report above. Now with the new year it seems to have become routine in the style reported yesterday. Yes, it happened again this AM first time static strategy was put on current date.