Yahoo Dataset Symbol Limit
Author: adrian50697
Creation Date: 5/12/2013 11:25 PM
profile picture

adrian50697

#1
I'm attempting to create a Yahoo dataset with somewhere between 9,000-10,000 symbols, however the Symbols text box in the Data Manager appears to max out it's character limit before I can enter all the stock codes. I've minimised the text volume by using the shortest stock reference codes possible and only using a single space between symbols and still cannot fit everything in. Is there a way around this?

Cheers,
Adrian
profile picture

Eugene

#2
There's no such thing as DataSet symbol limit. However, according to documenation, the number of characters that can be entered into a TextBox is 32767 by default. This is what happens: it hits this limit and maxes out.

Workaround (unsupported):

1. Create a Y! DataSet with a couple of symbols only: AAPL,GOOG,FB for example.
2. Close Wealth-Lab. Yes, WLD must be closed before moving on.
3. Open up the Wealth-Lab User Guide to find the location of the DataSets folder with regard to your operating system. See Data > Where Data are Stored. With hidden file/folder visibility enabled, find your ..Data\DataSets directory.
4. Once you're there, find your DataSet just created, and open it in your text/XML editor of choice (let's say it's Notepad++). There you'll find this line with your few symbols, comma-separated:

CODE:
Please log in to see this code.


5. Now, open a new blank document inside Notepad++, paste your gigantic list of symbols, and replace all single spaces with commas in your text editor (usually, the shortcut for Find/Replace dialog is Ctrl-H). Note: there should be no line breaks or something - this must be 1 line of comma-separated symbols. Copy the result to clipboard.

6. Back to your open DataSet's XML file. Replace all tickers in the string i.e. AAPL,GOOG,FB with your gigantic string of symbols in the clipboard. Make sure to preserve the XML tags.

Save the file. Start WL. Voila! And if it didn't work, you probably broke the XML structure e.g. by deleting a "{" or "}" inadvertently.
profile picture

Cone

#3
7. Make a mental note that you won't be able to edit that DataSet in the Data Manager.

The 32K character limit for a TextBox is surprising to me since a single string could be more than 2G (theoretically). 32K must be artificial, so let's look into changing that.
profile picture

Eugene

#4
Re: #7.

+1. Only manual editing of that DataSet would be possible, with Wealth-Lab closed.

Re: 32K.

Right, the MaxLength can be set to 0 to override that artificial default limit - considering that the real limit of a TextBox is 2 Gigabytes. Please create a support ticket. Many other data providers use TextBoxes for the same purpose, so we might want to change them as well.
profile picture

Eugene

#5
Better late than never. This limit will be removed in the upcoming build of Yahoo!, Google, Morningstar, and Quotemedia providers.

Possibly IQFeed if we get to release it soon. A few more affected providers (FRED Static, Nasdaq, Quandl and Stooq providers) may be updated eventually in the future.