Fidelity's Dirty Intraday Data
Author: jonludwig
Creation Date: 2/18/2009 5:37 PM
profile picture

jonludwig

#1
Any reason why Fidelity is publishing intraday data with so many bad Highs & Lows?
Take a look at SPY on a 1 minute chart. This could easily be cleaned up with a bad tick filter.
profile picture

Cone

#2
Unfortunately, those ticks actually occur, and it seems there's nothing practical that can be done about it with the current Fidelity provider. For now, use the bad tick filter - it works well in 5.3.

Also, in 5.3, you can double-click a bar and correct it manually in a static chart. Once that's done, those corrections will also be used in streaming charts.

For the record, Fidelity's 1-minute chart of SPY yesterday looks identical to eSignal's chart except for the large spike on the 10:12 bar.
profile picture

jonludwig

#3
The trades don't occur, just the bad ticks. They are usually the result of a very late posted trade.
Will the bad tick filter work for historical data?
profile picture

TrendCatcher

#4
But the bad ticks play huge bad tricks. They make the script not display proper results. I have few scripts display information based on the bad tick data. Makes a ton of difference in making a choice. I think it could be part of the manipulation around.
profile picture

Cone

#5
The bad tick filter only filters streaming data. That's the only opportunity Wealth-Lab has to see the ticks and compare if 2 consecutive ticks are x% apart.
profile picture

TrendCatcher

#6
Cone,
How come we don't have streaming charts during extended hrs in WL 5.3 unlike in WL 4.5?
profile picture

Cone

#7
WLP 4.5 was somehow able to get snapshots of post market data in a streaming chart - it wasn't streaming, and it was a mistake.

Although Wealth-Lab can help with discretionary trading, that's not its main purpose, and I assume that's why Fidelity doesn't provide pre/post market data in Wealth-Lab. There could be other reasons.
profile picture

hb

#8
I'm just starting to work with WLP 5 and haven't done anything in build 4.x in many years.

I thought FIDCO does a review every night correcting bad data. I would think that FIDCO would use the corrected data in their historical data files. No?

I'm a bit confused about the relationship between Streaming data used for back testing and that used for trading. Say I have script that is supposed to trigger a sell when a stock trades 10% below its moving average. When I'm back testing it will of course act on any bad data that is there and sell on inaccurate information. However when I am actually trading with this script and there is a bad tick will it also trigger a sell? What would be the execution price for a market order? The bad tick price or the actual market? I assume it would trigger the sell and it would trade at the actual market in which case the back testing is more or less useless. Am I wrong?
profile picture

Cone

#9
Corrections are applied, but most ticks that occur away from the inside market are actually valid - it's just that you and I don't want to see them in our data. On the other hand, those prints may mean something to somebody using a more discretionary strategy.

QUOTE:
I'm a bit confused...
This subject is treated in the User Guide, Orders > Portfolio Synch > Theoretical vs. Actual.
profile picture

redliontrader

#10
I like the title, I think you should change it to "Fidelity's dirty little secret" - lol. (don't you just hate people that laugh at their own jokes)

Because the data is sooooooooooo bad from Fidelity, it prevents me from auto-trading. I find that I have to trade discretionary because I have to decide before each trade if the data is any good.

Here is a picture of today (March 4th's) 1 minute data:



And for comparison the same data from TradeStation:


I don't have e-signal so I can not vouch for that, but I do have StockFinder and Etrade as well as TradeStation and none of these have such dirty data. I have complained for years, but because of the crazy arrangement in the relationship between Fidelity and Wealth-Lab nothing ever gets better. Just lots of finger pointing.

Unfortunately Fidelity does not provide us a place to bitch, so we dump our frustration here at the WLP site, because WL does foster a community. I am sure working the data divide is very frustrating for WL. I embrace everyone's frustration, all around.


btw: now that I know how to embed images one the the things that bugs me about the WLP site is the button below the post that I am currently editing. It looks like this:



The big green button says edit, but I am already editing, and I did not hit it to edit. (I hit a link).

Now that I am done I want to "Save" as opposed to Cancel. My point being, don't you think that green button should read "Save" instead of "Edit"?

profile picture

hb

#11
Thank you Cone for your reply. The information in the user guide will, I'm sure, be helpful.

I'm afraid I don't understand what you are saying in the first paragraph. What are
QUOTE:
ticks that occur away from the inside market?
Why don't we want to see them? what is
QUOTE:
a more discretionary strategy?
profile picture

jonludwig

#12
redlion,
I see those bad SPY ticks all the time on the historical intraday data. That is exactly what I am talking about. Fidelity never corrects them, and they are NOT real. Often times in stocks, wayward ticks are real. I know, because I am often on the other side of those trades. This cannot and will not happen in a highly arbed ETF like SPY or QQQQ, yet the bad ticks are all over the intraday historical data.
profile picture

Algotrader

#13

Fix bad data trick ?

So how can I purge my Fido dataset and use Yahoo for my database? I am guessing I would have to switch back and forth to do real time trading which incrediably has filtering....

Trade station has like 3 tiers of rules to allow trading like must hit NBBO two time or trade inside bid. Why is Fido so full of hoeuy?