Why Yahoo! data provider only download partial data for some tickers?
Author: wbzhang
Creation Date: 12/4/2010 10:02 PM
profile picture

wbzhang

#1
I use Yahoo! data provider to download S&P 500 stocks. However, for some tickers it only downloaded piece of data, not the full data. What's the reason?

For example, for AAPL, the data only started from 2010-11-23. I updated many times, the problem was still there.
But if I go to Yahoo!Finance URL, I can found data for the whole time range. Any solutions for that?
profile picture

Eugene

#2
Possibly because you specified 2010-11-23 as the starting date of the DataSet.
profile picture

wbzhang

#3
not really.

it worked well for other tickers, only a small portion of tickers with this problem.
profile picture

Eugene

#4
Right click on a chart of AAPL, select "Reload price history", did that help? If not, try creating a DataSet with these symbols starting at some distant past date, and updating it.
profile picture

wbzhang

#5
THanks! it seems "reload" work..
profile picture

wbzhang

#6
Hi Eugene, I still have the problem now :-( now even "reload" not working..

For example, I created new Dataset from Yahoo, starting from year 01/01/2008. But while updating the dateset, it always update staring from April 2009 (812 bars). I cannot get any data beyond Apr 2009.

I created another dateset, since 2000, it still shows me data since Apr 2009. With exporting these data, they shows data are since 2009-04-21.

Even for sys-defined indices or Dow 30, they are also from 2009-04-21, not earlier. Any problems?
profile picture

wbzhang

#7
I tried other data providers (Google provider), also provide data only after Apr 2009. This looks wield. Any solutions?
profile picture

Eugene

#8
First thing to check: reloading Y! data (only) will work properly on a chart with the most distant starting date. Otherwise you can shorten the data if you reload an "improper" DataSet i.e. with a closer starting date.

If that simple workaround doesn't help, then either your data loading range starts in Apr 2009, or the unknown symbol's data is actually starting from Apr 2009 on Y!/Google servers.

The Google provider has no connection to Yahoo (except for querying company names from Y! missing in G data but that's completely unrelated).

profile picture

wbzhang

#9
1) I created a new Yahoo dataset, tried a closer date, e.g., 2011-01-01.
while updating, it still shows:
[0] AAPL 2822 bars 03.22.2011 0 bars added ==> note: 2822 bars are since year 2000. Not sure why ...

With clicking "reload chart history", it still shows data from Apr 2009.

I updating the dataset again, it shows:
[0] AAPL 55 bars 03.22.2011 0 bars added
this is the right one, 55 bars means starting from 2011-01-01.

Then clicking "reload chart history", it shows data from 2011-01-01.

However, if I have 500 symobles, I need to do the same job for each one....


2) Now I create another new Yahoo dataset, tried a earlier date, eg. 2007-01-01.
while updating, it still shows:
[0] AAPL 2822 bars 03.22.2011 0 bars added ==> note: 2822 bars are since year 2000. Not sure why ...

but with clicking "reload chart history", it still shows data from Apr 2009, not 2007, not 2000 either.

This made me almost crazy...


profile picture

Cone

#10
This really is a lot simpler than you're making it out to be.

When creating a Y! DataSource, you shouldn't try to limit the data. Get as much history as you think you'll ever need and use the Data Loading control to specify the data required for the test.

When a symbol is created for a particular DataSet, that same data file is used for all DataSets that you create thereafter. You can't have a DataSet with an AAPL data file that has only 2 years of data and another AAPL file that has 2500 bars. Only one AAPL Daily file is created and used for all Y! DataSets that require AAPL.

It sounds like you're either looking at the Most recent 2 years, or 500 bars, or something of the sort. The big question is how do you have the Data Loading Control configured?
profile picture

wbzhang

#11
Yes, that makes sense. I created different datasets for different strategies. Probably I set the start date sometime ago, but now I need a bigger time range, how to update that? Now I cannot retrieve yahoo data beyond Apr 2009, after many "updating" operations.

Another question is, I create several ASCII datasets which also containing AAPL, and these ASCII datasets starting from 2008. Now even these local symbols show data only after Apr 2009. How to solve that? Thanks.
profile picture

wbzhang

#12
QUOTE:
Another question is, I create several ASCII datasets which also containing AAPL, and these ASCII datasets starting from 2008. Now even these local symbols show data only after Apr 2009. How to solve that? Thanks.


I tried different ticker names, and create ASCII datasets. WL always shows me the data is starting from Apr 2009. Crazy.. (when I was creating dataset based on ASCII files, I can still see the completed dataset in preview, but the dataseries are truncated in WL plot or my strategy programming...)
profile picture

wbzhang

#13
I finally noticed this is because of the global "data range", I set it to a certain number of bars.. sorry for that.

Sometimes a small negligence could cause big problems...
profile picture

Eugene

#14
QUOTE:
When creating a Y! DataSource, you shouldn't try to limit the data.

The Y! provider's starting date concept can be confusing and doesn't fit the idea of simplicity. Like the Reload delivering different data amounts depending on starting date. I don't see any practical reason for this option and will try to suggest the developer to eliminate it someday.
QUOTE:
I finally noticed this is because of the global "data range",

Just like I mentioned. ;)