Google Trends in Wealth-Lab
Author: Eugene
Creation Date: 2/2/2012 7:22 AM
profile picture

Eugene

#1
Starting from v2012.02, Community Components hosts a new class for downloading and caching a novel sentiment indicator:

* Google Trends in the Wiki

Your questions, ideas and experience are welcome here.
profile picture

gbullr

#2
This is AWESOME...


For novices to this (Like Me)

If you have two step verification on google you will get an error. You have to generate a Application Specific Password and input that password.

http://support.google.com/accounts/bin/answer.py?hl=en&answer=185833

No need to leave spaces.

profile picture

Eugene

#3
Later this year or maybe next year, we're going to publish a Trading System Lab article (ActiveTrader Magazine) devoted to various possible ways of using Google Trends in trading strategies.
profile picture

gbullr

#4
I think this goes back to my post a couple years back regarding counting the number of news stories (which I think is still a good idea)

profile picture

Eugene

#5
Thanks for the information, I've added it to the online help page:

Google Trends > Notes

Maybe some day we (Robert?) can do another AT Mag article counting the number of news stories, now that we have two historical news providers (Google and Reuters).

Community members - please share your thoughts.
profile picture

gbullr

#6
Is there a way to block certain symbols from running in a backtest other than eliminating from dataset. Problem arises in googletrends b/c symbols from FIDO are not the same as for googlefinance and so I would like to block 3 symbols from running in backtest. The symbols have this / in common. BRK/A VIA/B and something else.


Thanks.

profile picture

Cone

#7
CODE:
Please log in to see this code.

Or, just make the substitution for the right character, and don't skip them!
CODE:
Please log in to see this code.
profile picture

Eugene

#8
QUOTE:
Later this year or maybe next year, we're going to publish a Trading System Lab article (ActiveTrader Magazine) devoted to various possible ways of using Google Trends in trading strategies.

This has happened:

ActiveTrader 2013-01 | Google Trend Sentiment Systems

Note: please carefully read the REQUIREMENTS section before asking questions!
profile picture

falcon82

#9
Hello,

I think that the idea of Google Trends in WL is pretty cool. I tried the example from the Wealth-Lab Wiki:
http://www2.wealth-lab.com/WL5Wiki/GoogleTrends.ashx

I used my Gmail and I typed in my password. With Google I have the two way authentication so I had to generate a application specific password. But I still get the following runtime error:

The index is out of range. It should not be smaller or negative to the listing.
Wealth.Lab.Strategies.GoogleTrendsDemo.Execute()
WealthLab.WealthScripts.Synchronize(DataSeries source)
WealthLab.BarScaleConverter.Synchronize(DataSeries source.Bars bars)
System.Collections.Generic.List 1.get_Intem(Int32 index)

I used the Yahoo! Symbol MSFT like in the example and I tried a few other symbols, too. I had the Data Range selected with -All Data-. I think it has something to do with the Google Trends data that these are not synchronized properly?

P.S.: I get the same error with the ActiveTrader 2013-01 Google Trends Strategies.

Could you please help me with that error?
Thanks for your help!
profile picture

Eugene

#10
Hi Andreas,

The code does not work any longer because it appears that recently, Google has made some changes to the Google Trends API and Google authentication. Also, Google Trends has been switched to a different scale, making any downloaded results differ from the previous numbers (e.g. the original Trading System Lab`s numbers).

We will have to fix this, but here come the bad news. With the new website, we lost backward Extension download compatibility, so any Extension updates can only be propagated after the WL 6.5 release. We`re several months from the desired fix to Community Components. Sorry.
profile picture

Eugene

#11
ATTENTION

We already fixed it in v2013.05 but in October'13, Google's made some changes to their login form, breaking our Google Trends code again. A new fix is required. An update to Community Components library is to follow; ETA = early December. Please stay tuned.
profile picture

mikesblack

#12
Hi Eugene,

Have you bin able to make the needed changes to trade from Google Trends? The error message I am getting for Google Trends(individual stocks)[rev.A] is "WealthLab GoogleTrendsStrategies2 GetSearchTerm(WealthLab Bars) not all code paths return a value. I also have 2 of the same warnings, The predefined type 'System Tasks Task' is defined in multiple assemblies to the global alias: using definition from c:\Windows.Net\Framework64v

Mike
profile picture

Eugene

#13
Hi Mike,

By occasion I've verified earlier today that Google Trends still works. There's a demo strategy in the Wiki that you can play with. Don't forget to read the Notes and enter your Gmail credentials in the code.

Re: 'System Tasks Task' is defined in multiple assemblies - delete System.Threading.dll from Wealth-Lab's main folder.
profile picture

gbullr

#14
Re Started two step verification

got a new code for wealth lab

Copied the code from the wiki (above)

put in my gmail and the 16 digit code excluding the spaces and click on a symbol

and get the following error.

QUOTE:
Runtime Error Index was out of range must be non negative and less than the size of the collection
at wealthlab strategies google trends demo execute()
at wealthlab wealthscript synchronize (Data series source)
at wealthlab bar scale converter synchronize ( data series source bars bars )
at System Collections Generic.list. get item (int 32 index)
at system throwhelper throw argument out of range exception



I shortened up the data range field and still got the problem.

Help would be appreciated.

Thanks.

profile picture

Eugene

#15
Fortunately, can not reproduce in any timeframe from an intraday to monthly scale. Troubleshooting and fixing Google Trends happened to be a quite painful process in the past.

Speaking of the Wiki example code, how does your "Microsoft.csv" file look like? You can find it there assuming you've enabled hidden file visibility:

c:\Users\Windows username\AppData\Roaming\Fidelity Investments\WealthLabPro\1.0.0.0\Data\GoogleTrendsCache\

I've attached mine for your comparison.
profile picture

gbullr

#16
Where is yours?

Here is mine.

<html><head><title>Redirecting</title>

<script type="text/javascript" language="javascript">
// Accessing window.external members can cause IE to throw exceptions.
// Any code that acesses window.external members must be try/catch wrapped
/** @preserveTry */
try {
if (top == self) {
if (window.gtbExternal) {
window.gtbExternal.setM();
} else {
window.external.setM();
}
}
}
catch(err) {
}
</script>

<meta http-equiv="refresh" content="0; url=&#39;http://www.google.com/trends#q=Microsoft&amp;hl=en-US&amp;content=1&#39;"></head>
<body bgcolor="#ffffff" text="#000000" link="#0000cc" vlink="#551a8b" alink="#ff0000"><script type="text/javascript" language="javascript">
location.replace("http://www.google.com/trends#q\x3dMicrosoft\x26hl\x3den-US\x26content\x3d1")
</script></body></html>
profile picture

Eugene

#17
Here's mine to prove that it still works. Rename Microsoft.txt to *.csv.

1. Update C.Components to its latest version 2016.05, restart WLP. Does it work now?
2. If still no go, disable two-step verification in your Google account and retry with your actual password in the Strategy code. This works for me.


profile picture

gbullr

#18
Finally got it to work.

Got rid of two step verification and went in w/ my normal password.

Thanks.
profile picture

Eugene

#19
FYI

Something has changed in the Google authentication form and it's not evident to me how to pass it now. It's a prerequisite to be able to request the Google Trends data without authentication. (Google wants to know what you're requesting.)

Another way to grab that data even without Google account credentials that I researched is using the JSON objects which power up their web charts. However, the way it's tokenized for security might make it slower and less convenient to use in code (if it's possible at all).

Also I face an error claiming I've sent too many search requests to Google Trends but that's not true. Way too much trouble with the data that should be free and easy to request - it's not worth it to continue fighting with breaking changes.

No solution so far. I've taken down the Google Trends strategies and the Wiki page. This is where it ends.
profile picture

Eugene

#20
Good news: Google Trends has been partially restored!

While both approaches above were tough and not feasible to implement, a way around was found. It's less precise as the data is only Monthly (Google typically returns Weekly data for Google Trends). But it does not require your Gmail account and password any longer! To make it work:

1. Update or install Community Components v2017.09 or newer. Restart WLP/D.
2. Download updated Google strategies from Wealth-Lab's "Open Strategy" dialog (click "Download").

Old strategies and your own works should function, too.
profile picture

Eugene

#21
Bad news: Google Trends has broken again

The convenient Monthly source of Trends data at Google has been depreciated permanently. We're taking taking down the published Strategies again. Please don't ask for solution or ETA, thank you.