Google Finance static/streaming intraday/daily data provider
Author: Eugene
Creation Date: 9/9/2009 5:48 AM
profile picture

Eugene

#1
Google Finance static data provider, which is a part of Community.Providers, has been released for download via the Extension Manager:

Google Finance Static Provider
profile picture

smiranda

#2
Hello Eugene,

This works great for US traded scripts. However if I look at foreign markets like the Bombay Stock Exchange, the data provider does not return any data. For example I wanted to look at
Welspun-Gujarat Stahl Rohren Limited traded on the Bombay stock exchange. I can see this on Google Finance via the symbol "BOM:532144"

When I try to get this in Wealthlab pro using Google as a data provider I get the message "The remote server returned an error (404) not found"

Any ideas what I am missing?
profile picture

Eugene

#3
Hello Sameer,

That's because Google doesn't support exporting this stock's data to a CSV for downloading.

Compare the 2 pages:

Alcoa Inc. historical prices

Do you see a widget called "Export" with this option: "Download to spreadsheet"? Yes.

Now, Welspun-Gujarat Stahl Rohren Limited historical prices.

Is there a download option? No. We can't download this data from Google Finance.
profile picture

aequatis

#4
Hi Eugene,

It seems that Welspun-Gujarat Stahl Rohren Limited historical prices link is working now. It displayed price data from Dec 2008. However, wealthLab Pro is not able to read it in correctly. I also found that WealthLab Pro was also not able to download Dow Jones Index: http://www.google.com/finance/historical?q=INDEXDJX:.DJUSIT using INDEXDJX:.DJUSIT as the symbol.
profile picture

Eugene

#5
Hi Sean,
QUOTE:
It seems that Welspun-Gujarat Stahl Rohren Limited historical prices link is working now.

No, it doesn't.

Re-read my reply above carefully. Open the Alcoa page, notice there a widget on the right called "Export - Download to spreadsheet".

So, is there a download option for Welspun-Gujarat? No.

And the same for INDEXDJX:.DJUSIT. This data can't be downloaded from Google Finance. I described how you can check is the historical data available or not on your own.
profile picture

Eugene

#6
While Google's not making any progress in providing the Welspun India historical data, :) with the BBFree data provider it's become possible to download two Welspun stocks, WLCO:IN an WLSI:IN. Check it out.
profile picture

Eugene

#7
Google Finance provider updated to v2010.06.2:

Added: support for Strategy Monitor
Added: support for bar data editing feature (click on chart to remove/edit a bar)
profile picture

Eugene

#8
Google Finance static data provider updated to 2010.09.

Summary of changes:

Added: multi-threaded data download and update (now 3x faster)
Change: possible to install in Wealth-Lab 6
Fixed: bar data editing fix

Wealth-Lab 6 customers may install it from Extensions. Updated project source code is available from the Wealth-Lab Wiki for download.
profile picture

Eugene

#9
Google Finance static data provider updated to 2010.11.

Summary of changes:

Fixed: potential network issues and crash due to invalid data

Wealth-Lab 6 customers may install it from Extensions. Updated project source code is available from the Wealth-Lab Wiki for download.
profile picture

Eugene

#10
Google Finance data provider updated to 2011.01.

This is a milestone release adding many new features (and some bug fixes):

* free intraday historical data backfill (1..5 minute, up to 2 months of 5-min data),
* free intraday and daily Streaming data provider (snapshot quotes updating every 10 seconds),
* Wealth-Lab 6.1 compatibility (returns trading session's open price - supports the new GetSessionOpen method of WL 6.1 - check out the QuickRef for details and code example)

As always, Wealth-Lab customers can install it from Extensions and get the updated project source code from the Wealth-Lab Wiki. Updating Wealth-Lab to 6.1 is required to be able to use the provider.
profile picture

csn

#11
What ist the syntax for the dow jones industrial average for the google streaming provider?
I tried .DJI but it does not work.
profile picture

Eugene

#12
Look it up on finance.google.com. Start typing the first letters of your instrument, and Google Finance will autocomplete it. It's the best practice when dealing with symbology by different providers.
profile picture

Eugene

#13
Google Finance data provider updated to 2011.09.

Change summary:

* Added: Market Manager support
profile picture

abegy

#14
I'm unable to download historical data for symbol codes like EPA:SPI, EPA:AC, EPA:BNP or EPA:AF. All the time, I have the following message and nothing more.

Updating Provider Google Finance ...
Provider update complete for Google Finance
----------------

Update completed (0.20 sec)
----------------

Normally, data exists in the Google website (sse http://www.google.com/finance/historical?q=EPA:SPI#).
profile picture

Eugene

#15
See my reply from 11/24/2009 3:25 AM in this thread (post #3).
profile picture

Eugene

#16
Google Finance static/streaming data provider updated to 2011.11.

Highlights:

* Added: support for Change in streaming Quotes window
* Fixed: optimized static data update requests, potentially improving Strategy Monitor data update performance on large DataSets
profile picture

Eugene

#17
Google Finance static/streaming data provider updated to 2012.01.

Maintenance release, no highlights.

* Fixed: symbol description routine crashes WL if network connection goes down abruptly
profile picture

kas

#18
Is it possible to add 60 minute timeframe to Google data provider? I understand that downloading 5-min we can then convert them to hours. But only couple months available for download. If I need a longer history what can I do?
profile picture

Eugene

#19
QUOTE:
Is it possible to add 60 minute timeframe to Google data provider?

Sorry, it's impossible. With this provider, you already have everything that Google intended to make available: 1-, 2-, and 5-minute data going back as deep as Google wants it to be. So, ask Google - not us. :)
QUOTE:
If I need a longer history what can I do?

If you need a longer history, consider purchasing an IQFeed subscription.
profile picture

abegy

#20
Is it possible to update this extension to allow download quotes from symbols which have not the option "Download to spreadsheet" ?

I ask you that because only US symbols have this option.
profile picture

Eugene

#21
Technically it's possible but for any practical purposes, it is unlikely that we're going to consider it. First, there's not enough "bang for the buck". Second, concerns about overloading a robust, fast solution with an "uncharted" (unknown) feature.

The solution is open source, so feel free to experiment and try adding this feature if you like to. On the surface, here's a list of challenges you'll face:

1. Pagination.
a. The web-based quote tables deliver only 200 rows at a time.
b. When to stop?
c. When updating a symbol with the normal "download to spreadsheet" feature disabled, your code should care about going back N pages depending on the last update date to avoid possible gaps in the middle (would apply if the symbol wasn't updated for a long time, but still).

2. Download speed. It will be considerably slower than it is now because web-based HTML quote tables have a larger size per se and also come in packs. Consequently, you'll want to consider (more) multi-threading to grab the whole data range.

3. How to determine when grabbing the HTML tables is appropriate for a symbol?

4. Parsing HTML tables (this part is by far the easiest by virtue of HTML Agility Pack).
profile picture

kelvinyip

#22
Hi

I downloaded this google streamer, intraday/EOD module but when I tried to update for EOD and intraday and it returned right away no error messages but no data downloaded either. I am on 6.3

thx
profile picture

Eugene

#23
Please be more specific. Which bar scale(s) and symbols?
profile picture

kelvinyip

#24
simple SPY on daily or 5mins
profile picture

Eugene

#25
Works perfectly for me, double checked on different internet connections. As always, more details are welcome (sequence of actions, error/status messages, reported bars count, data loading range etc.)
profile picture

kelvinyip

#26
If I use data manager and hit update, it does not seem to update the quotes. If I just fetch it one symbol at a time, it seems to work fine. I put in sp500 symbols. I don't think the datamanager let me choose the daily data range. I am on the 64 bit version.

Where do we store the historical data on disk ? Perhaps I can just see if they are actually downloaded
profile picture

Eugene

#27
First, what does "seem" mean? And what is the bars count and the last update date for the symbol that seems not updating for you? The details please.
profile picture

kelvinyip

#28
I see, I get it. symbol details tells me how many bars. No there are no bars... There are loads of bars for yahoo data but not for google data.
profile picture

Eugene

#29
For that Google DataSet, how does the update log message look like? Please copy/paste when replying.
profile picture

kelvinyip

#30
it does not say that it updated


Updating Provider Google Finance ...
Provider update complete for Google Finance
----------------

Update completed (0.2 sec)
----------------

Updating Provider Google Finance ...
Provider update complete for Google Finance
----------------

Update completed (0.5 sec)
----------------
profile picture

kelvinyip

#31
did it from home as well, same no quotes downloaded so it is not a firewall issue
profile picture

kelvinyip

#32
ok, so I gave it one more try and uninstalled the addin and restalled it... Now it works...

thx
profile picture

Eugene

#33
No, definitely not a firewall issue. Similar problem happened before with the Fidelity static provider. However, I don't believe I experienced it in WLD with Google provider.
profile picture

abegy

#34
According to this article http://www.codeproject.com/Articles/221952/Simple-Csharp-DLL-to-download-data-from-Google-Fin, it's possible to download from the Google intraday API all symbol quotes in a daily and intraday based.

Is it possible for you to integrate this new specification in your data provider extension ?
profile picture

Eugene

#35
We're already using this route for the intraday and streaming quotes.

As for the Daily data, following the author's URI returns approximately 60 days of historical data for a symbol that you can't download using the regular method. Such as BNP:EPA or SPI:EPA from one of your previous replies:
QUOTE:
http://www.google.com/finance/getprices?q=BNP&ex=EPA&i=86400&p=1000d&f=d,c,h,l,o,v
http://www.google.com/finance/getprices?q=SPI&ex=EPA&i=86400&p=1000d&f=d,c,h,l,o,v


Good idea, but unfortunately, 60 bars of Daily data just does not qualify. If you like, you can modify the solution on your own: it'd take merely half a dozen code lines of the GoogleClient class - in DownloadHistory and ParseHistory*. Note that the existing parsing operations themselves are absolutely not affected by the change. The idea is to replace the regular Daily data request with the already existing intraday data call. I'm leaving this excercise to you.
profile picture

abegy

#36
You are right that the historical data can be limited to 60 days. But it's not the case for all types of markets. If I take a currency, I have more than 2400 of quotes.

CODE:
Please log in to see this code.
profile picture

Eugene

#37
Still, isn't worth the trouble. We already have enough providers supporting currency data from 1-minute to Daily: Forexite (1-min), Dukascopy (10-, 60-min, Daily), Finam (Tick to Monthly), QuoteMedia (Daily). Heck, even MSN and Yahoo might support Forex (haven't checked in a while).

Keep in mind that we have a little "army" of Google provider users out there, so any changes to the core functionality should be considered only if absolutely necessary.
profile picture

veganath

#38
In evaluating Google finance as an intraday source I noticed that there are significantly large 'Volume' discrepancies for ETFs(Stocks are OK), e.g. SPY, DIA, etc... Am I missing something here??

e.g. SPY 5min bars most recent day Jun06, 2012

Compare with Yahoo or your broker.. roughly an order of magnitude of 10 difference

profile picture

Eugene

#39
Thank you for your observation. On data issues, please consider contacting Google support.
profile picture

Eugene

#40
Google Finance static data provider updated to 2012.09.

Summary of changes:

* Fix: "Google Domestic Trends" symbols weren't downloaded/parsed (non-trading indices based on search sentiment rank provided by Google)
* Fix: (62778) Shifted DataSeries break execution in Strategy Monitor on non-native scale

For more on non price-based sentiment rank called "Google Domestic Trends", check out the provider's page in the Wealth-Lab Wiki.
profile picture

Eugene

#41
Google Finance static data provider updated to 2012.11.

Summary of changes:

* Change: reduced volume of queries to Google server in Streaming mode by caching supplemental requests
* Change: Wealth-Lab 6.4+ required to install/update (.NET 4.0 required)
profile picture

Eugene

#42
richard1000 reports:

When I tried to update Daily scale data using Google Streaming provider, I received an error. To show the error, I did the following:

-set Preferences -> Streaming Data -> Google Streaming Provider.
-create a new DataSet of Dow 30 stocks with Daily bar scale.
-update DataSet(Pricing).
-message "Wealth-Lab has encountered a problem and needs to close. ..."
profile picture

Eugene

#43
Thank you for the heads-up. An error in Google data is being detected by the provider and an exception message about the data error is being thrown. (At least 4 of Dow 30 symbols suddenly started demonstrating data flaws: missing element of OHLC bar.) Unfortunately, it crashes Wealth-Lab. I suspect it's a leftover used by me for debugging purposes.

I'll suspend generation of that superfluous error message, work on a fix and upload it in a matter of days.
profile picture

Eugene

#44
Google Finance static data provider has been updated to 2012.12.

Summary of changes:

* Fix: crash when updating Daily data
* Fix: changed approach of handling data errors

Previously, a bad bar in Google data should have abruptly stopped processing all further historical quotes for that symbol. Additionally, an unlucky exception message window - a leftover from a debugging session - caused crash on EOD data because of suddenly appearing errors in Google data.

From now on, bad bars will be skipped and no exception message will be generated.
profile picture

kelvinyip

#45
Hi

I want to get sp 500 equal weight index. On google, I can type INDEXSP:SP500EW and get historical quote for this index. On the google data provider, I tried INDEXSP:SP500EW , SP500EW but cannot get the historical data back. Any idea ?

This is no quote from yahoo and I don't find sp500 equal weight index on fidelity either.

thx
profile picture

Eugene

#46
See my post #3 from 11/24/2009 3:25 AM.
profile picture

kelvinyip

#47
indeed, I don't see the download button for that either.
profile picture

Eugene

#48
tradercn wrote in a deleted duplicate thread:

Tried to see if google extension works. Input the symbol "000001.sz", it shows No Data Available. But on google website, this symbol works. http://www.google.com.hk/finance?q=000001.sz&ei=zoOrUqP0N8TYqAGDsAE
Could you please help? Thanks.
profile picture

Eugene

#49
No, this symbol will not work. See my posts #3 and #5 above on how to determine if the data for any given symbol will be available in Wealth-Lab or not.
profile picture

tradercn

#50
Got it. Tks.
profile picture

Eugene

#51
Harapa asked:

I can get only intraday data but not EOD (for any symbol) from this provider. I did create two different data sets ;one for intraday and other for EOD.
Here are few (of many ) symbols, for example, that I have tried
SPY, IWM, QQQ

Update Long

Updating Provider Google Finance ...
30 Minute data is being updated...

Up-to-date symbols: 0, Update required for: 0, New symbols: 0

5 Minute data is being updated...

Up-to-date symbols: 0, Update required for: 19, New symbols: 5
Symbol: RVX, Bars: 243, Last date: 7/2/2015
Symbol: VIF, Bars: 243, Last date: 7/2/2015
Symbol: VIN, Bars: 243, Last date: 7/2/2015
Symbol: VIX, Bars: 243, Last date: 7/2/2015
Symbol: VIXAUG, Bars: 244, Last date: 7/2/2015
Symbol: VIXDEC, Bars: 244, Last date: 7/2/2015
Symbol: VIXJAN, Bars: 244, Last date: 7/2/2015
Symbol: VIXJUL, Bars: 244, Last date: 7/2/2015
Symbol: VIXNOV, Bars: 244, Last date: 7/2/2015
Symbol: VIXOCT, Bars: 244, Last date: 7/2/2015
Symbol: VIXSEP, Bars: 244, Last date: 7/2/2015
Symbol: VXAPL, Bars: 234, Last date: 7/2/2015
Symbol: VXD, Bars: 244, Last date: 7/2/2015
Symbol: VXGDX, Bars: 234, Last date: 7/2/2015
Symbol: VXMT, Bars: 243, Last date: 7/2/2015
Symbol: VXN, Bars: 243, Last date: 7/2/2015
Symbol: VXO, Bars: 234, Last date: 7/2/2015
Symbol: VXST, Bars: 243, Last date: 7/2/2015
Symbol: VXV, Bars: 243, Last date: 7/2/2015
Updating new symbols...
Symbol: VIXAPR, Bars: 0
Symbol: VIXFEB, Bars: 0
Symbol: VIXJUN, Bars: 0
Symbol: VIXMAR, Bars: 0
Symbol: VIXMAY, Bars: 0

60 Minute data is being updated...

Up-to-date symbols: 0, Update required for: 0, New symbols: 0

Daily data is being updated...

Up-to-date symbols: 0, Update required for: 0, New symbols: 24
Updating new symbols...
Symbol: RVX, Bars: 0
Symbol: VIF, Bars: 0
Symbol: VIN, Bars: 0
Symbol: VIX, Bars: 0
Symbol: VIXAPR, Bars: 0
Symbol: VIXAUG, Bars: 0
Symbol: VIXDEC, Bars: 0
Symbol: VIXFEB, Bars: 0
Symbol: VIXJAN, Bars: 0
Symbol: VIXJUL, Bars: 0
Symbol: VIXJUN, Bars: 0
Symbol: VIXMAR, Bars: 0
Symbol: VIXMAY, Bars: 0
Symbol: VIXNOV, Bars: 0
Symbol: VIXOCT, Bars: 0
Symbol: VIXSEP, Bars: 0
Symbol: VXAPL, Bars: 0
Symbol: VXD, Bars: 0
Symbol: VXGDX, Bars: 0
Symbol: VXMT, Bars: 0
Symbol: VXN, Bars: 0
Symbol: VXO, Bars: 0
Symbol: VXST, Bars: 0
Symbol: VXV, Bars: 0

Provider update complete for Google Finance
----------------

Update completed (7.585 sec)
----------------
profile picture

Eugene

#52
1. Everything is running perfectly with the latest version under Win7 x64. As you can see, the data is fetched up to the ultimate trading session:

QUOTE:
Updating DataSet Google (Daily) ...
Up-to-date symbols: 0, Update required for: 0, New symbols: 3
Updating new symbols...
Symbol: IWM, Bars: 3778, Last date: 02.07.2015
Symbol: QQQ, Bars: 3646, Last date: 02.07.2015
Symbol: SPY, Bars: 3981, Last date: 02.07.2015
Update completed (1.153 sec)
----------------


Sometimes the feed can have issues (but not today) e.g. Google Data feed stopped at 22 May?

2. If some symbols like VXV deliver 0 bars, this is not an issue. If you wonder why, please read my explanation in posts #3 and #5:

Google Finance static/streaming intraday/daily data provider

I'm going to add this to the Wiki FAQ right now to avoid having to answer this very question over and over.
profile picture

Harapa

#53
I now understand the issue. Thanks for you help.
profile picture

Eugene

#54
Glad to have helped. And FYI, symbols like VIX, VXV could be downloaded through a dedicated provider: CBOE Static Provider

QUOTE:
I'm going to add this to the Wiki FAQ right now to avoid having to answer this very question over and over.

Done: FAQ > Google data provider
profile picture

Harapa

#55
For this provider all intraday data for CBOE products (VIX, VXV ,VXMT, VIF,VIN, etc.) is being reported using CST while all other symbols (trading at NYSE/NASDAQ...SPY, IWM, QQQ etc..) are reported using EST.
Co-mingling these symbols in a strategy creates an artificial shift to the right by an hour.
Also, VIX related data have 2-3 bars past 3PM CST (= 4:00 EST) which are now not filterable by selecting trading hours in Market Manager.
If one applies Synchronization the system eliminates pre- 9:30 EST bars if chart is for a regular stock) and adds duplicate bars using 3:15 CST bar til 4PM EST (as you would expect).
Wonder if someone else noticed this and have come up with a work around.
I know many VIX related symbols are available via Fidelity but Google has many more (e.g VXMT, VIX Futures etc).
profile picture

Eugene

#56
Tell you more: time zone conversion was implemented in the Market Manager from the beginning but remains hidden all the way. Unfortunately, the feature never came to fruition. There are some compatibility issues and enhancements to be made to WL itself before we're able to enable it.

Possible alternatives may include:

1. You might want to consider subscribing to IQFeed. Additional CBOE delayed data package costs $1 a month (RT - $6). I've not explored this option altogether so I don't know how is their CBOE data timestamped, are all required indices available in the subscription etc. IQFeed has free trial available.

2. Come up with a script that processes intraday bars for a CBOE product bar by bar, shifts the time, and exports the data to CSV file (to be read back by ASCII provider). [Further discussion, if required, is outside the scope of this thread.]
profile picture

Harapa

#57
Thanks.
profile picture

Panache

#58
Please correct me if I'm wrong, but I believe stock splits cause problems for Google data. It appears that Google adjusts its data for stock splits, so when you first download historical data from Google, you are getting split adjusted data. However, it does not appear that when Wealth-Lab updates Google data, it does anything with the existing data. Therefore, unless you either manually perform every split since you first loaded your data or reload the data every time there is a stock split, you end up with a combination of adjusted and non-adjusted data.

I believe NASDAQ data suffers from the same problem.

If you had CPST loaded before the reverse split on 10/9/2015, you'll see the problem.

In contrast, if I understand correctly, QuoteMedia is always non-adjusted data. If you want adjusted data, you have to manually perform every split, but at least what you have is consistent. Personally, I can't see a reason why anyone would want to use non-adjusted data for back testing, but I'm always happy to learn something new.

Yahoo is non-adjusted data, but by checking Perform Split Adjustment in DataManager, you get adjusted data. Fidelity is automatically adjusted for splits as new data is downloaded, so it is always adjusted data.

To summarize, it appears that choosing different data providers can have a significant impact on what you're going to see in a Chart window or have happen in your Strategy! If I am correct, if you use Google or NASDAQ as a data provider, you better be prepared to spend a lot of time keeping your data up to date, not just relying on DataManager.
profile picture

Eugene

#59
Kurt, your understanding is correct. Unlike Yahoo!, all these feeds do not have the means for split- and/or dividend-adjustment. Usually, it's not present in their data. So, to maintain the data in good shape you either have to perform splits manually or reload the data (for which the Data Tool comes handy as it allows to perform batch reload of the DataSet). A note has been added to each affected provider's Wiki page.
profile picture

richard1000

#60
Problems in Google Finance provider:

Updating DataSet Google (Daily) Dow 30 ...
Up-to-date symbols: 0, Update required for: 0, New symbols: 30
Updating new symbols...
Error setting min/max download threads
Update completed (0.46 sec)


I think other data providers are also having the same error and needs to be checked.

I just checked PiTrading provider with the same error. But since data isn't updating, you may or may not fix this error.
profile picture

Eugene

#61
This bug occurs under very rare conditions. I'll upload an updated extension shortly. Thank you for the heads-up.

P.S. We should probably take the PiTrading and TradingBlox providers down rather than update them.
profile picture

Eugene

#62
What's new in Google provider's recent version 2016.05?

* Fix: "Error setting min/max download threads" message prevents data update
* Fix: removed artificial limit (.NET default) in New DataSet Wizard preventing from creating DataSets which contain more than ~6500 symbols (32767 chars)
* Change: .NET 4.5 and Wealth-Lab 6.8+ are required to install/update extension
* Change: (internal) updated browser agent version
* Change: (internal) increased thread download timeout to prevent failed requests
* Change: (internal) multithreaded download routine rewritten. Should improve work on modern multi-core CPUs (e.g. servers)
profile picture

Eugene

#63
What's new in Google provider's recent version 2016.06?

* New: support for streaming data updates in Accounts window for Paper Trading
profile picture

Eugene

#64
What's new in version 2017.04:

* Fix: "Update all data" doesn't update any data during first-time update
* Change: Wealth-Lab 6.9+ required to install/update
profile picture

Eugene

#65
What's new in version 2017.08:

* Change: more clear data update message saying "Bars added"
profile picture

Eugene

#66
What's new in version 2017.10:

* New: support for automatic bar data corrections
profile picture

Glitch

#67
Is it me, or is Google only returning 1 year of historical daily data now?
profile picture

Eugene

#68
It's just you Dion. I just checked and 4,000 bars is the upper limit for Google EOD data.
profile picture

Glitch

#69
Thanks Eugene, it's working fine now, not sure what was happening!
profile picture

Eugene

#70
It appears that a change to the intraday data format has taken place at Google Finance lately. I'm not alone, somebody else have also discovered this: https://stackoverflow.com/a/46309633. Here's what I found:

1. The intraday timestamps are wrong (a date in July, time is after market)
2. Intraday charts now return only 50 bars (but only 1 bar is shown since the Market Manager filters out the after market bars)
3. There's no difference between 1, 2 and 5 minute charts: they all return last 50 5-minute bars.

There is no ETA for a fix currently.
profile picture

Eugene

#71
Good news. Look for a fix for broken intraday data in the upcoming v2017.11 of the provider.
profile picture

Eugene

#72
What's new in version 2017.11:

* Fix: intraday data was broken after a URI change at Google
* Change (internal): switched requests to HTTPS from HTTP
* Change (internal): upgraded browser user-agent
profile picture

Eugene

#73
What's new in 2017.12?

* Fix: company name not appended when requesting data on demand for new symbols
* Fix: company name broken due to deprecation of dependent service at Yahoo (replaced with Google Finance)
* Change: company names are now cached for current WL session to speed up subsequent requests