Yahoo! Data Not Updating for Last Session
Author: Cone
Creation Date: 3/21/2014 3:31 AM
profile picture

Cone

#1
Since this happens several times a year, I'll try to update this topic each time it occurs.

There are still 6 hours before the next session opens, but data for March 20 is still not available from Y! Finance. Get your backup provider ready. For another free provider, try Morningstar.
profile picture

kyokushin

#2
How Do I add Morningstar to the update list?
profile picture

Eugene

#3
Question is ambiguous:

* On how to install/uninstall/update Wealth-Lab extensions, review this illustrated tutorial and/or check out the Wealth-Lab User Guide > Extension Manager.

* On how to update data, see the User Guide: Data > Data Manager > Update Data.
profile picture

kyokushin

#4
Thanks, Eugene.

I installed Morningstar Extension but there are no bars added with EOD data.

Here's a fragment of the latest data update log of Morningstar symbols:

Updating Provider Morningstar Static ...
Provider update complete for Morningstar Static
----------------

Updating Provider Morningstar Fundamental Data ...
^AEX 83 fundamental items updated
...
ZMH 83 fundamental items updated
Provider update complete for Morningstar Fundamental Data
----------------

Update completed (1 min 34.6 sec)
----------------
profile picture

Eugene

#5
You've updated the fundamental data by checking "Morningstar Fundamental data" among "Fundamental data providers". The checkbox for static data is above, among "Historical data providers", and is called "Morningstar Static".
profile picture

kyokushin

#6
I updated Morningstar Static too. This was the outcome:

Updating Provider Morningstar Static ...
Provider update complete for Morningstar Static
----------------

Update completed (0.15 sec)




No bar added...
profile picture

Eugene

#7
For some reason, update all data may not work immediately for Morningstar data when you haven't used it before. Try highlighting your Morningstar DataSet and click "Update DataSet (Price)". This should kickstart it, and then data update will start working as expected.
profile picture

Cone

#8
It was a good run for the last 6 months, but Yahoo! is not updating again today for the 17 Sep 2014 session.
profile picture

cicerotullius

#9
any news here? I just tried it again its now after the closing of the session of the 18th of Sept and still there is no data for the 17th?

Might there be some next change in their API / Format as such things tend to occur some times? Ok it seems that on Yahoo finance page its not updated as well.
profile picture

Eugene

#10
For news you might want to contact Yahoo support. There are many backup data feeds if you need the data now.
profile picture

Cone

#11
As usual, it was just a temporary condition. Y! is updating to the prior session (18 Sept.) now.
profile picture

tallvira

#12
You can find some other sort of technical support services from yahoo as I got from ADVERTISING LINK REMOVED
One of the best free web mail service provider, well known to the world with the name of Yahoo! everyone is using yahoomail as with their communication medium

ACCOUNT BANNED FOR ADVERTISING
profile picture

Harapa

#13
This yahoo URL allows downloading last OHLCV data (as EOD, 15 min delayed) any time of the day.
http://quote.yahoo.com/d/quotes.csv?s=@@@&f=sd1ohgl1v&e=.csv
"@@@@" = Symbol/Ticker
Using this as a standby Yahoo provider will help to circumvent the "missing last EOD data". Off course one would need a provider "Add on" to use it in WL
profile picture

Cone

#14
In that case, open the Data Manager (Ctrl+M), go to the Yahoo! Data tab, and put a check next to "Always return data with partial bar". It works! Good show Harapa ;)
profile picture

Harapa

#15
Yes, indeed it works (except for Mutual funds...minor issue for most folks).
Glad I could help!
profile picture

Harapa

#16
Cone,
With the above URL one can download last data for symbols without history (for example ^VXX-IV). However, I get no data with Yahoo DL in WL (Partial bar checked). Wonder if this functionality can be Incorporated in future Yahoo provider. If there is a trick to get this even with the existing version, llet me know.
profile picture

Cone

#17
Yahoo! is updating.. .to 12/30/2015 without partial bar checked and to 12/31/2015 with it. Are you having trouble with a specific symbol? Please be more specific.
profile picture

Harapa

#18
Yes, there isn't any issue with yahoo updating except for symbols for which there is no history, like the one I mentioned.
I am interested in fetching data for this class (Intraday/EOD data yes, History No)..

Symbol examples
^VXX-IV
^XIV-IV
^NUGT-IV
SPXH16.CME
profile picture

Eugene

#19
There is nothing to fix or request here. All these symbols except the futures are happily streaming in a Quotes window. As per the FAQ, it is not possible to get the futures data from Yahoo. Neither is possible to obtain their historical data because Yahoo doesn't have it but it's not a problem for other providers.

Shouldn't Fidelity have symbols like .VXX in their data??
profile picture

Harapa

#20
QUOTE:
There is nothing to fix or request here.


My query highlighted the "possibility "of using the above URL to fetch these in WL. At the moment, I do this in another program but then I have to make those accessible in WL (lots of maneuvering...). So I asked...



QUOTE:
Shouldn't Fidelity have symbols like .VXX in their data??


No, checked already...

Actually, most are intraday NAV... examples tickers are for yahoo...other sources may use different ticker

profile picture

Eugene

#21
Re: lots of maneuvering

I don't see a good reason to include this URL in the Yahoo! provider as there's no history for those symbols anyway. If you need this for some purpose, you're better off creating a simple script for fetching the data on-the-fly. I already showed you how it can be done, so there's really no need for maneuvering or Excel:

Extracting Shares Related Info for Proshare ETF/Ns
WealthScript Techniques | Access Data from the Internet on-the-fly

/Should you have interest in discussing it please do so outside this thread that deals with "Yahoo! Data Not Updating for Last Session"./
profile picture

Eugene

#22
Many symbols are not up to date (missing July 13th prices). The Yahoo! Finance site seems to have had an overhaul recently. Maybe it's connected.
profile picture

Eugene

#23
Neopolix wrote in a duplicate thread:

QUOTE:
Hi,
i' ve got no Data today from Yahoo. Is that a General Problem?
profile picture

Eugene

#24
This is a problem at Yahoo's side. You might want to contact them (indicating that the "Download Data" link on the Historical Data page doesn't contain the data since Nov 7th) or use one of the many backup providers:

Report a data problem to Yahoo
profile picture

Neopolix

#25
Thanks Eugene!
profile picture

Eugene

#26
ATTENTION

Note that Yahoo does not seem to be returning the Dec 7 bar. Suggestion: use any backup data providers.
profile picture

Cone

#27
If you trade only Nasdaq 100 and S&P 100 stocks, give Wealth-Data a try!

Wealth-Lab Developer customers should already have the Wealth-Data extension installed with 6.9.15, so just sign up at the Wealth-Data site for access.
profile picture

Harapa

#28
Even since I activated "Always return data with partial bar"(Data Manger/Yahoo Data), Yahoo provider fetches data anytime of the day without any issue. Never a single missing day.
profile picture

Eugene

#29
Harapa, that's good advice. However, after not being able to update the data the traditional way today (last date is still 05/12/2017), I've used the partial bar technique and it couldn't update a few symbols:

QUOTE:
...
Security Names updated.
[1] AXP 11339 bars 15.05.2017 1 bars added
[3] CAT 13937 bars 12.05.2017 0 bars added
[6] DD 13937 bars 15.05.2017 1 bars added


So I had to update it again and this time it worked:
...
QUOTE:
Security Names updated.
[1] AXP 11339 bars 15.05.2017 0 bars added
[3] CAT 13938 bars 15.05.2017 1 bars added
[5] CVX 11951 bars 15.05.2017 0 bars added
profile picture

Harapa

#30
Today, in WL, I am not getting any data as well. This must be something to do with yahoo URL.
I am getting bad gateway error instead of no bars added
"CSCO Error: The remote server returned an error: (502) Bad Gateway"

as this works (on a different platform, though)
http://download.finance.yahoo.com/d/quotes.csv?s=@@@&f=sd1ohgl1v&e=.csv

but this worked this morning but stopped working EOD
http://chartapi.finance.yahoo.com/instrument/1.0/@@@/chartdata;type=quote;range=15d/csv
profile picture

Eugene

#31
QUOTE:
as this works (on a different platform, though)

This would work in Wealth-Lab too if we kept on using HTTP as in your link (we've switched to HTTPS). However, it's just the current day's snapshot quote - not very useful for historic backfills. Currently, I get a '504 Gateway Timeout Error' for all data update requests. Could be a glitch, could be depreciation of the old API. Let's keep an eye on this:

Yahoo Historical Data - Did they change the URL recently?
Is Yahoo! Finance API broken?
has yahoo stopped history data api service?

QUOTE:
but this worked this morning but stopped working EOD

We weren't using this RESTful API which reminds me of Google API.

Just noticed that the CSV download link on the Y! Finance website has changed to:
CODE:
Please log in to see this code.

The bad thing is this there's more under the hood than this vanilla URL. Now it requires cookie authorization and a "crumb" (the last param) which is sort of CSRF token.

Although there's a way to request the data without cookies it consumes too much network traffic and therefore may not be suitable for all.
profile picture

Neopolix

#32
Hi there,

what is your opinion. When does it work again?

profile picture

Eugene

#33
Hi,

It might work again some day when there's more clarity on what's happened. The outcome depends very much on what it is:

1) an outage? = wait for a few days until it starts working
2) old API decommissioned, API switchover? = fixing it may take a long time

For now it seems like the worst has happened.

At any rate, there's no shortage of alternative data providers. As here we need to focus strictly on the technical aspects of Yahoo APIs, you're welcome to discuss alternatives in other forum threads. Thanks.
profile picture

Harapa

#34
Yahoo Provider:
Working URL's for data download/retrieval as of 5/17/2017 AM. No cookies involved...
//@@@ or @@@%%% ...fills in the symbol

Last Trading only. Reports last price (intraday or EOD after market close))
http://download.finance.yahoo.com/d/quotes.csv?s=@@@&f=sd1ohgl1v&e=.csv

EOD...Fetches History
Defined Interval: (Need to define beginning and end dates...e.g 1/1/1950 to today). Note UNIX time stamp after"="
https://finance.yahoo.com/quote/@@@%%%/history?period1=-630961200&period2=1494993600&interval=1d&filter=history&frequency=1d

profile picture

Eugene

#35
Thanks for trying to help.

QUOTE:
EOD...Fetches History

I knew that but this is not suitable as every page is a 0.5MB request with tons of supporting stuff packed in whereas a correctly formed Y! request consumes 2 orders of magnitude less traffic (only a handful KB).

QUOTE:
Last Trading only. Reports last price (intraday or EOD after market close))

Question is for how long will it last.

On the bright side, the so called protection (cookie + crumb) is a joke compared to more sophisticated obstacles which we've overcome in some other projects. The strikingly bad news is what I've just read on ET board:

1. Previously, OHLC were unadjusted and now OHL are split-adjusted
2. Sudden injections of random false data - I can confirm e.g. a couple of "null" bars in AAPL
3. Gaps of data (blocks or certain dates) - e.g. a hole of 143 bars in TRV

Time will tell whether this is intentional or just the effects of transition.
profile picture

Vadim_Kaplya

#36
Yahoo provider still does not have data (since 12th). It looks they could have changed the links for data fetching, please check it. WLD prints 504 error into the update log.Thank you
profile picture

Eugene

#37
Vadim, thank you for letting us know. Please scroll this page up to my post #29. In the conversation that followed you might find some new insight into the problem. Thanks again.
profile picture

Vadim_Kaplya

#38
Eugene, thank you. I apologise, It does not work, - I used to load a bunch of stocks every day from indices to check the candidates, now none of the universe is loading even partial bar.
profile picture

Eugene

#39
Chisholm's Third Law says, If you explain so clearly that nobody can misunderstand, somebody will. ;)

Vadim, the idea was that if you read the conversation that followed up to post #35 and not just #29, you'd know the verdict: old Yahoo data as we knew it is now broken for everyone in the world. The old API is discontinued. There can be no workaround. Now it's a major effort to adjust to the new API considering all its drawbacks known and unknown (see #35).
profile picture

Eugene

#40
An updated Yahoo provider is already being tested. We'll try to release it after we iron out some glitches.
profile picture

Eugene

#41
We're pleased to announce that the Yahoo provider has been fully restored. Update the extension now!
profile picture

innertrader

#42
Eugene - Thank you. Yes, it appears to be working.

Were you able to implement without cookie/crumb method? If so, are you able to share the approach you used to eliminate the need for C&C?

Also, I see that Yahoo historical data is not currently calculating prices adjusted for dividends (recent problem). However, it looks like WL is correctly adjusting historical Yahoo data for dividends. Did you find a different place to download adjusted data or are you calculating adjusted data within the program or on your server (i.e., post Yahoo download)?
profile picture

Eugene

#43
QUOTE:
Were you able to implement without cookie/crumb method? If so, are you able to share the approach you used to eliminate the need for C&C?

I went the cookie/crumb path as the method with faster downloads and minimum network traffic.

QUOTE:
Also, I see that Yahoo historical data is not currently calculating prices adjusted for dividends (recent problem).

Provider now unadjusts the data before storing it in raw format and then adjusts it for splits/dividends on demand. Before the old API got decomissioned it had been easier.

P.S. To the list of issues in post #35 I can add another:

4. For a number of symbols, the data is returned raw (split-unadjusted). This creates a false gap. Compare for yourself:

* An example of how it should be: recent NVUS 1/9 split. Data is adjusted on the website. Provider works correctly.

* A 2/1 split in BLL which isn't reflected in the data. Provider cannot handle this properly.
* Now, see how a historic split in the same stock is reflected correctly on the website: BLL 2/1 split @ 02/16/2011
profile picture

innertrader

#44
QUOTE:
Provider now unadjusts the data before storing it in raw format and then adjusts it for splits/dividends on demand. Before the old API got decomissioned it had been easier.

Interesting. Is there an option in the URL string that enables the download of dividend-adjusted data? Or by "on demand" did you mean at the time you download it? It is not dividend-adjusted for symbols I've run history for on their Website. It "magically" got adjusted properly (dividends at least) when downloaded into WL history.

Sounds like this is a widely-known problem by now, so I'm curious how you got it to work.
https://www.bogleheads.org/forum/viewtopic.php?t=219605
profile picture

Eugene

#45
QUOTE:
Or by "on demand" did you mean at the time you download it?

Yes. Let's rephrase:

1. The Y! provider has always been storing the raw DTOHLCV. Previously the data was returned raw.

2. Now after the recent Y! website changes which resulted in forced OHLC split-adjustment, the provider has to unadjust the data before saving. Which means to remove the split adjustment from an OHLC.

3. Adjustment for splits and/or dividends is enabled by default and is applied on-the-fly in the Data Manager (Yahoo! Data) as the provider keeps a separate database of corporate events (splits and dividend) for all downloaded symbols. You can disable it and see the raw price/volume data after reopening the chart (or F5).

I hope it makes sense.
profile picture

innertrader

#46
Eugene - in this explanation, who is the "provider"? Yahoo or a 3rd party that processes Yahoo data for WL? Or are you referring to a software module?

QUOTE:
Adjustment for splits and/or dividends is enabled by default


Enabled where? What software or service are you referring to here? Since these are checkbox options (not default) in the Data Manager for Yahoo tab I am confused about what "default" means in this context.
profile picture

Eugene

#47
There's no "3rd party that processes Yahoo data for WL". The Yahoo! (data) provider is the Wealth-Lab extension created by us and built into WLP and WLD. That's where the processing of data takes place:

Yahoo! Finance Static, Real-Time and Fundamental

QUOTE:
Did you find a different place to download adjusted data or are you calculating adjusted data within the program or on your server (i.e., post Yahoo download)?

It's important to stress that the data providers created by us (MS123 LLC) do not redistribute Yahoo or other data. It's not processed on our servers and then returned to you. It's always your own PC that initiates connections to their servers and grabs the data. No redistribution takes place, and there isn't a middleman.
profile picture

innertrader

#48
It's clear now. Thank you.
profile picture

Eugene

#49
ATTENTION: The data for CAT is currently missing at Yahoo! and you will get an "Error 404" in the update log. When you get an "Error 404" for any stock, check up "Historical Data" on their website's and (optionally) ask Yahoo support.
profile picture

Cone

#50
The Yahoo! data for CAT appear to have returned...

CODE:
Please log in to see this code.
profile picture

ReneW

#51
Today I am getting an authorization error (401). Is Yahoo broken again ?
( By the way, with python scripts outside Wealthlab I checked the new Yahoo API and it
was not very reliable. In 3 out of 10 cases I received an empty data message without error indication.
I used the pandas and its new fix_yahoo_finance package )



Updating DataSet Dow 30 ...
Preparing requests ...
Requests are ready to go.
Yahoo session cookie cached
Updating Security Names for 31 symbols...
Security Names updated.
[3] BA Error: Der Remoteserver hat einen Fehler zurückgegeben: (401) Nicht autorisiert.
[0] AA Error: Der Remoteserver hat einen Fehler zurückgegeben: (401) Nicht autorisiert.
...
[0] DIS Error: Der Remoteserver hat einen Fehler zurückgegeben: (401) Nicht autorisiert.
profile picture

Eugene

#52
Ouch, Error 401 is the last error code you'd ever want to encounter from the new Yahoo as it might indicate a protection change. But it's working for me right now. It also works on Wealth-Data website where we display Yahoo's recent quote to highlight data discrepancies in comparison to the clean Wealth-Data.

Please try to restart WLD. If it won't work, try to change your IP address by switching to your backup ISP. But if you're running WLD on a server (which you've been doing), who knows how does Yahoo react to it?

UPDATE: I restarted WLD, switched to German regional settings, and updated w/o a problem.

P.S. My log (running on a physical PC, not server):

Updating DataSet Dow 30 ...
Preparing requests ...
Requests are ready to go.
Yahoo session cookie cached
Updating Security Names for 30 symbols...
Security Names updated.
[4] CSCO 6889 bars 20.06.2017 2 bars added 1 bars corrected
...
[3] CAT 13963 bars 20.06.2017 2 bars added 1 bars corrected
[6] XOM 11976 bars 20.06.2017 2 bars added 1 bars corrected
Update completed (5.346 sec)
----------------
profile picture

ReneW

#53
Thank you for your effort, Eugene, it is working here again...
Next time I will be having a little more patience with this Yahoo...
profile picture

Eugene

#54
I'm getting a "(401) Unauthorized" error when trying to update Yahoo data. However, when I visited their website and tried to download data in the browser I got this response:
QUOTE:
{
"finance": {
"error": {
"code": "Unauthorized",
"description": "Invalid cookie"
}
}
}

This indicates that it's an issue on Yahoo's end. It's not a Wealth-Lab problem. You have to wait until's it's resolved by Y!
profile picture

mrsic

#55
Hi,

I get since today also the error message 401.
I have WL, PC and DSL restarted but the error is still there.
Has anybody an another idea.

greetings damir
profile picture

Eugene

#56
Damir, if you scroll the page up one post and read my post #24 from 2 hours ago would this explain everything?

UPDATE 06/25/2017: As "Error 401" is fixed at Yahoo's end, the provider is working again.

Thanks.
profile picture

innertrader

#57
@Eugene
RE: #45

---

Yahoo is finally correctly adjusting historical data both on their website and with downloads. I now find an adjusted discrepancy with historical Yahoo data in WL before a dividend payment date on an ETF. I found that I had to *unclick* the "Perform Dividend Adjustment" box in Data Manager > Yahoo! Data to make the data correct.

If I understood your previous explanation (#45), when the Yahoo! Data box is checked, WL will do internal dividend adjustment from *raw* data received from Yahoo. I suspect Yahoo is now (maybe after fixing their website behavior) shipping you adjusted data when you are expecting raw data. Now, with the box checked, dividend is subtracted *twice* from actual close data.

Both symbols ILF and VNQ had very recent dividend payments, so you should be able to easily see what I am referring to. Yahoo is driving all of us crazy with their surprises, so let me know if I got this one right.
profile picture

Eugene

#58
cicerotullius wrote in a duplicate thread:

QUOTE:
Was there another change at Yahoo Data provider, sind a few days I only receive from them:

Error: The remote server returned an error: (401) Unauthorized.

for all symbols


Please search the forum before posting. See posts #51 to #56 in this thread. Error can return at any time. As we're dealing with a free data feed, we get what we paid for.
profile picture

cicerotullius

#59
@Eugene: I definitely get the error since a couple of days, not change, what do you suggest, restart WL?
profile picture

Eugene

#60
@cicerotullius: As we are not related to Yahoo Finance anyhow, what do you expect me to suggest? Recent changes might have not been propagated everywhere. You're hitting a misbehaving server in their CDN/farm. Patience goes a long way. Or you could use Wealth-Data or alternative providers.

@innertrader: Thank you for the observations, I'll investigate. What's even more important is that Y! has changed the order of columns in downloads (Close and Adjusted Close). Reloading the data ruins it if there were splits or dividends. Fix is in the works.