Errors updating Yahoo Data
Author: ddaurelio
Creation Date: 5/24/2017 3:06 AM
profile picture

ddaurelio

#1
I've been having ttouble updating certain tickers in a Fidelity data set. I keep getting the error below:

SPXL  Error: The remote server returned an error: (401) Unauthorized.

I tried logging into Fidelity an rerunning the request. I also tried Yahoo, but that's even worse. It doesnt update any of the symbols in my data set. For each one, it displays an error like this:

[5] SPXL  Error: The remote server returned an error: (502) Bad Gateway.
profile picture

Eugene

#2
I'm sure that the source of both 401 and 502 errors is Yahoo, not Fidelity. It's very well documented in a hot thread under this General category:

Yahoo! Data Not Updating for Last Session

Please read from post #30 and to the end.

P.S. A search tip: try typing in the first letters of a keyword (like "Yah...") in forum's search box. It's nearly real-time.
profile picture

ddaurelio

#3
Thanks! Will check through the other posts as well
profile picture

Eugene

#4
We're pleased to announce that the Yahoo provider has been fully restored. Update the extension now!
profile picture

Eugene

#5
Attention: our Yahoo! provider has been updated to v.2017.07.

This is another MUST HAVE release. Last weekend Yahoo has made some breaking changes to:

1. the order of fields in the exported data (swapped Close and AdjustedClose)
2. split/dividend adjustment of the Close price
3. improve the way some historic split/dividend adjustments are handled

If you updated the data this week or downloaded symbols from scratch, it's highly recommended to truncate this week's bars using the Data Tool and to repeat update using v.2017.07.

On the bright side, the number of splits that you'd put on the "Ignore Selected Splits" list has visibly decreased.
profile picture

ewessely

#6
Hi Eugene,
I just updated to the Yahoo 2017.07 and used the datamanager to delete all data (of all yahoo datasets). I verified by creating a Chart that there are no data available. I updated my dataset, but the data seems to be corrupted in the past.
Reproduce for example AA: every data before 2016-11-01 seems not to be ok.

Thanks for checking
profile picture

ewessely

#7
Sample Screenshot
profile picture

Cone

#8
Right click and "Reload chart history". AA is now the "new AA". The data history for the former AA is now in symbol ARNC. More info: http://www.stockspinoffs.com/2016/11/03/alcoa-completes-spinoff-newly-renamed-arconic/
profile picture

ewessely

#9
Cone,
thanks for the prompt reply - my fault. "Reload chart history" does not make any change.
But can you also take a look at AAPL or CSCO for example? (attached) Also Looks very strange....
profile picture

Eugene

#10
Well, Yahoo has changed something again in their data for the 3nd time since May! I fixed the thing twice in May and June.

Please take a look at this in recent quotes of CSCO on their website (the order is OHLC + Adj.Close):

Jun 29, 2017 31.79 32.03 31.25 31.12 31.41
Jun 28, 2017 31.78 32.28 31.76 31.78 32.08

As can be seen, its Close is below the Low price. I think they have again changed the order of fields. This time we're probably going to wait until the dust settles and Yahoo fixes their data bugs and wildly changing order of fields.


So, if Wealth-Data doesn't have some symbols you're looking for, please choose the recently released Tiingo provider, it's a great drop-in replacement for Yahoo. Be sure to sign up at their website, though:

Tiingo Static, Real-Time and Fundamental
Online user guide
profile picture

ewessely

#11
Eugene,
thanks for analysing. I'll give Wealth-Data and Tiingo a try.
profile picture

innertrader

#12
@Eugene

QUOTE:
I think they have again changed the order of fields.


By downloading into spreadsheet from Yahoo site, I can confirm that data in the Close and Adj Close columns have been switched. Column header titles have not changed order, only the data.
profile picture

Eugene

#13
So I've unrolled the v2017.07 extension build. This should fix Yahoo data but I'm puzzled at how they let this erroneous order of fields exist in the data, fix it, and then break it again.

Please reinstall v2017.06 from our server if you're on v2017.07. This should fix Yahoo data until they make unannounced breaking change again.

As any data downloaded since June 29 can be incorrect, it's advised to check your Y! data using the Data Tool and, if errors were found, either truncate some last bars in symbols which existed before June 29 or reload from scratch the new symbols you added since June 29.
profile picture

Eugene

#14
The Close and Adj Close columns have again been switched in Yahoo data. Working on an automated solution to this problem.

Please update to v2017.07 if you're on v2017.06. This should work around Yahoo data until we make the fix.

As any data downloaded since ca. July 10 can be incorrect, it's advised to check your Y! data using the Data Tool and, if errors were found, either truncate some last bars in symbols which existed before ca. July 10 or reload from scratch the new symbols you added since ca. July 10.
profile picture

innertrader

#15
@Eugene

I am not sure if this is related to issues discussed above in this thread, but I recently noticed that Yahoo! does historical data splitting on Close prices, even if you uncheck the relevant box in data manager. If you do check the split box, it spits *twice* so for a 5:1 split; it divides historical Close prices by 25.

This may be something you are already aware of. If you are not, then take a look at what happens to symbol EEM before 5:1 split on 7/24/08. You can see that the Close data (only) is adjusted for splits by default. If you check the split box, then OHL data will be now split, and C data will be split a second time.

This behavior is not limited to this symbol, based on my brief research. Please inform whether this is a known issue and any fix or workaround.
profile picture

Cone

#16
The split behavior looks correct to me. By the way, the split on 7/24/2008 looks like a 3:1.

1. Which version of the Yahoo! Provider are you using? Check in Tools > Extensions > Providers

2. If you're using the current version 2017.08, try right-click the chart > Reload Chart History.
profile picture

Eugene

#17
@innertrader

1. Do not make changes to the Yahoo! options while you have an open chart. Reopen the chart or restart WLP. If it doesn't help, I second Cone's suggestion:
2. Reload Chart History. Yahoo has made a lot of conflicting changes which could have reflected in your existing data.



3. If still no go, close all chart windows and delete this file (how to enable hidden file visibility), then take step #2 again.

C:Users\Windows username\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\YahooFundamentalProvider\E\EEM.WLF
profile picture

innertrader

#18
@Cone
QUOTE:
The split behavior looks correct to me. By the way, the split on 7/24/2008 looks like a 3:1.

You are correct. I was confusing with another symbol I was looking at with 5:1, ILF.
QUOTE:
1. Which version of the Yahoo! Provider are you using? Check in Tools > Extensions > Providers

In the Extension Manager>Providers there are 3 listed:
1. ASCII Files Static (2017.07)
2. WL Files Static (1.2.5.0)
3. Yahoo! Finance Static, .... ( 2017.07)
I don't see any updates available. I've seen threads about updating to Yahoo! Provider 2017.07 but don't know how to get 2017.08.

@Eugene
Tried all of your suggestions above . attached is an image of what I'm seeing with no boxes checked on Yahoo! tab of data manager, as compared to the one you shared.

BTW, it looks like the data in your image is split 3:1. Prices should be ~$150 during the time period you are displaying, as you will see in attached image.
profile picture

Eugene

#19
It's my bad, I forgot to upload v2017.08 - was full of wishful thinking that Yahoo has settled down. Sorry.

I've downgraded to v2017.07 to see that Yahoo has changed again the order of fields so said version is now broken (and v2017.06 is probably working again but that doesn't matter). Cone's expressed suspicion that they must be doing it intentionally is now confirmed.

Fixed. Please update the extension to v2017.08 with the automatic detection.
profile picture

innertrader

#20
Voila! That did it.
profile picture

Eugene

#21
Once again, I'm getting a "(401) Unauthorized" error when trying to update Yahoo data. However, when I visited their website and tried to download data in the browser I got this response:
QUOTE:
{
"finance": {
"error": {
"code": "Unauthorized",
"description": "Invalid cookie"
}
}
}

This indicates that it's an issue on Yahoo's end. It's not a Wealth-Lab problem. You have to wait until's it's resolved by Y!
profile picture

ReneW

#22
Short question:

How do I get rid of the wrong company names ?
Even after reloading symbol details with the new Yahoo Data provider they are wrong.
profile picture

Cone

#23
When you update the symbols in the Data Manager, it should refresh the names. Give us some examples of symbols with incorrect company names so that we can check.
profile picture

ReneW

#24
I reloaded my SP600 dataset and the symbol names changed for many, but not all, from one wrong name to another wrong name.
APA is my standard example, which should be "Apache", now its "Air Products And Chemicals ,Inc").


I reloaded Symbol Details and restarted WL to avoid caching issues.


See picture.
profile picture

Eugene

#25
QUOTE:
How do I get rid of the wrong company names ?

1. Set up your Extension Manager to check for extension updates on startup.
2. When an update to the Yahoo provider is detected, update it and restart WLD.
3. In the same Extension Manager window, check change logs of the extension that causes you trouble. This would let you be informed about what has recently happened to Yahoo streaming data and company names, for example:

Yahoo provider change logs

Re: APA

APA works fine here:



P.S. If there's something that I cannot reproduce, chances are that Yahoo is returning bad output to break the automated downloaders. Like they already have been periodically randomizing the order of OHLC in the exported data. This happens with free feeds, that's life.
profile picture

ReneW

#26
During startup my WL checks for extension updates. The Installed Yahoo component is 2017.12

I attached a picture of the latest updatet dataset. The first companies have the correct name,
then all other companie names are wrong.

What do you advise is the best solution for my problem ?
profile picture

Eugene

#27
Rene,

Solution is easy:

1. In the update log, note down any symbols that return a "(404) Not found" error and weed them out of your DataSet.
2. Update the DataSet to reload its symbol names.

To be fixed in upcoming Y! provider build. Thanks for the heads-up.
profile picture

ReneW

#28
Thank you, Eugene. I noticed eventually, that small datasets, containing only valid symbols are updated correctly.
profile picture

Eugene

#29
You're welcome. While waiting for the fix any DataSet (small or big) will do. Simply make sure it doesn't contain any invalid symbols.