Quandl static provider + Zacks fundamental provider
Author: Eugene
Creation Date: 11/30/2014 6:55 AM
profile picture

Eugene

#1
Just released: a suite comprised of Quandl static data provider and Zacks fundamental data provider. With its help, Wealth-Lab users can download countless datasets from Quandl website, such as:

* free and paid static EOD data for global instruments (futures, commodities, rates, stocks, indices, bonds, currencies...)
* COT data (in legacy and new formats)
* subscription-based fundamental data by Zacks (dividends, earnings esimates/surprises, analyst ratings)


The provider is available for installation from our website (Extensions section):

Quandl provider - installation link

Please read the online setup guide before using:

Quandl provider online notes in the Wealth-Lab Wiki


Currently, we support futures from OFDP and CHRIS groups, COT data, stocks (although you wouldn't want them as Quandl takes the data off of Yahoo and Google). If a symbol is unsupported or returns empty data, you've come to the right place.

Please suggest your symbols and report bugs here!
profile picture

Eugene

#2
Happy New Year!

Version 2015.01 is out with a fix which is a must have for most international customers: fixed incorrect data in some locales.
profile picture

shardis89

#3
Apologies if this is not the spot to post this, but in the Quandl provider online notes in the Wealth-Lab Wiki link from above, how would I arrive at defining the Bars.NamedSeries for a specific COT Data named series? Better yet, how could I create a custom dataseries from the COT Named Series data defined as Commercial Long - Commercial Short and then apply an indicator (e.g. RSI.Series(Close,2)) to the custom dataseries to normalize the custom index. Hence we are creating a Commitment of Traders Index indicator without coding a bunch of math, but rather utilizing the pre-programmed indicators to do the calculations on the custom dataseries.
profile picture

Eugene

#4
I added the code sample that you requested to the Wiki link above. Note that it can only be used with Quandl symbols containing legacy series i.e. ending with "_L_ALL" (L is for legacy):

Quandl provider suite > COT Data (scroll to the very bottom of the page)
profile picture

shardis89

#5
Awesome, thanks Eugene.

Since the idea is to use the data as an indicator and visualize how it relates to price, I added to it so that a person could pull in the Quandl Futures Price data for the related COT data. Since the COT data is compiled weekly, in Wealth Lab you have to set the scale to daily. So when pulling in the futures data you must set that data to synchronize to the weekly data manually in the code. This code also highlights the overbought and oversold conditions for all the visable panes. Below is the code I used.

CODE:
Please log in to see this code.
profile picture

lukeallen63

#6
Request for new data collection from Quandl...

I ran across a premium paid data collection called Steven's Continuous Futures (SCF). I haven't subscribed but would be interested in using the data since they cover a lot of futures and scrub the data for errors. I'm not sure something would need to be setup to use it or not though, if anyone knows please advise.

Thanks!
profile picture

lukeallen63

#7
Just noticed this on the ODFP section of Quandl...

QUOTE:
"OFDP is being deprecated as a database on Quandl in favour of more comprehensive databases. All the futures data, and more, under OFDP can be found for free under exchange databases, CME, ICE, LIFFE, EUREX, and continuous contracts can be found under CHRIS. We are sorry for any inconvenience this may cause. If you have any questions please don't hesitate to contact us at connect@quandl.com ."


I can also see provider's like the CME group, so maybe adding some of those as well will help users get futures data.

I personally would find this very helpful

Thanks again
profile picture

Eugene

#8
QUOTE:
Request for new data collection from Quandl...

Simply give it a try as explained in provider's online guide in the Wiki and you'll see that SCF is already supported (for trial users too). Don't forget to type in your API key. Most data sources, whether existing or new, are picked up automatically. See "How it works" for more details on automatic format detection. A request is needed only if something doesn't work.
profile picture

lukeallen63

#9
Thanks Eugene, I did try it and it worked. Thanks again, this is a great tool.
profile picture

sedelstein

#10
I read the Wiki http://www2.wealth-lab.com/WL5WIKI/ZacksFundamental.ashx

Right now, there are 4 data items available

QUOTE:
Out of fundamental items available on subscription basis, four are currently supported:

[z] Zacks_Analyst_Ratings
[z] Zacks_Dividend
[z] Zacks_Earnings_Estimates
[z] Zacks_Earnings_Surprises


Zacks/Quandl offer many more. Is there way to get them through WL if you are a Zacks subscriber?

Thanks for your help
profile picture

Eugene

#11
Many more is probably not the correct word. Zacks Street Earnings Estimates, Consensus Earnings Trends, Sales Surprises seem somewhat less popular to our audience to justify their possible development at the moment. The two remaining mainstream collections, Zacks Fundamentals Collection A and B, may be of potential interest for development due to returning 270+ fundamental items for $3,000/year. This task is on our product backlog (low priority). If there's enough community following, this can help prioritize it. For now, you can access a lot of fundamental items with a subscription to YCharts, for example.
profile picture

sedelstein

#12
Thanks Eugene

Fundamentals A&B were what I was thinking of.

YCharts Pro is $300/month so Zacks is marginally cheaper and looks to have orders of magnitude more fundamental items

Hopefully others read this post and chime in.
profile picture

Eugene

#13
Please note that a valid Quandl API token (customer, full access) will be required in order to develop the provider. The demo key allows to make only 100 requests which is not enough to backfill even a single symbol once. This is a must. Working on it with the Quandl team.

QUOTE:
YCharts Pro is $300/month so Zacks is marginally cheaper and looks to have orders of magnitude more fundamental items

It's off topic here but to close this discussion, both points seem arguable:

1 - There's a $40/month plan (however, seriously limited to a 25MB/month data export quota).
2 - It's possible to ask us to implement items available on their website but missing in the provider.
profile picture

mikesblack

#14
Question about Zachs Data. After update I am only finding one item, last date is the present date for Zachs analyst ratings and earning estimates. Earning surprises go back till 6 2013 and Dividends go back until 1 1992. I sampled a number of stocks with the same results. What will I need to do to acquire more data from Zachs?
profile picture

Eugene

#15
It's better if you inquire Quandl or Zacks support on this. If you are a subscriber, there's premium support. For demo users, serious restrictions apply - like making only 100 data requests.
profile picture

mikesblack

#16
Thanks Eugene. I'll check it out.

What does Quandl offer that Zach doesn't and visa versa?
profile picture

Eugene

#17
Quandl is an aggregator that provides access to various vendors, Zacks is one of them.
profile picture

mikesblack

#18
Thanks!

Eugene, Can you explain the process of how I can download the Quandl data? If I go to their site, there are many data sets, free and premium. How do I add them into the Data Manager?
profile picture

Eugene

#19
I realized that this question may be coming and this is why the very first post in this thread has a help link in bold. You may have noticed that we maintain documentation for most providers in the Wiki. Quandl is no exception. Scroll up and you shall find the online guide with your question answered.

P.S. Quandl is like a shopping mall with lots of independent boutiques. With the various types of data hosted by Quandl, obviously not every data feed is compatible but the provider is smart enough to figure out most types of data it is processing (like COT, stocks, futures, rates etc.)
profile picture

mikesblack

#20
Thanks Eugene. Very helpful. Have a great weekend.

Aside from what is already provided, if I purchase one of their services, for example Zack analyst rankings, how would I access that data and then display it on a chart,etc.?
profile picture

Eugene

#21
Mike, if I say something like "You may have noticed that we maintain documentation for most providers in the Wiki", you can trust me on that. It was only yesterday and nothing has changed:

Zacks fundamental provider

By the way, this link could be found in the first paragraph of the Quandl page I referred to in my penultimate reply.

Re: display it on the chart. Since the approach is universal for every fundamental provider, you can find the answer in the Wealth-Lab User Guide.
profile picture

mikesblack

#22
Sorry Eugene, I did read the Wiki page and have set up the Data Manager as illustrated. Just unclear how to make use of a payed service. Zacks didn't have a clue about WealthLab integration. Quandl have tons of cool packages, but I am still unclear how to access or update the data. I wouldn't want to pay for a service I can't put to immediate use.
profile picture

Eugene

#23
That's why there are tons of free data to play around with and get a feel of the Quandl provider, a (very limited) free trial of those cool packages (paid services), and two pages of documentation covering everything you need (hopefully).
profile picture

kbellare

#24
I'm getting 0 data results (scresnshot below) when downloading from Quandl - CHRIS dataset. I installed Quandl (with my API key), added 41 symbols to the dataset and tried downloading. Is there something i'm missing? How do i test that the Quandl service is sending data?

profile picture

Eugene

#25
All these symbols are invalid. To save yourself time, visit Quandl's website and verify that none of these tickers exist (I did). From a logical standpoint, this is what I would do with any given data provider. Now you've learnt this trick, too.

Bonus:

Add a "2" to each ticker and it will return data e.g. CHRIS/CME_AD2. (Don't ask me why, ask Quandl.)
profile picture

Dave B.

#26
Eugene:

As an early adopter of Quandl and having brought it to your attention, I thought you and the Wealth-Lab Community might be interested in this article just published:

https://www.quandl.com/blog/alternative-data

For the record, I have no connection to Quandl.
profile picture

Eugene

#27
What's new in version 2016.08:

* Fixed no data for CBOE/VIX symbol
* Removed artificial limit (by default) in New DataSet Wizard preventing from creating DataSets which contain more than ~6500 symbols (32767 chars)
profile picture

Eugene

#28
What's new in version 2016.09:

* Added support for CBOE/VVIX symbol (requested by ReneW)
profile picture

Panache

#29
I'm doing something wrong, but I don't know what it is.

My API Key is entered in the Quandl tab. I created a Quandl data set containing "WIKI/A". When I Update Data, I just get:

Updating Provider Quandl ...
Provider update complete for Quandl
----------------

Update completed (0.0 sec)
----------------

I can get the data in Python, so I know my API Key and symbol are correct. I can also see the data at https://www.quandl.com/data/WIKI/A-Agilent-Technologies-Inc-A-Prices-Dividends-Splits-and-Trading-Volume

The QuandlStaticProvider directory is empty except for QuandlSettings.xml (which does contain my API Key).

Any ideas?
profile picture

Eugene

#30
I think you ran into an intermittent first-time data update issue. As a workaround, I suggest that before trying to "Update All Data" after first-time DataSet creation by a new provider (e.g. Quandl), you'd either make a one-time update of the Quandl DataSet or restart WLP.

Why would Data Manager not download and update any data?

Please let me know if it's helpful or not.
profile picture

Panache

#31
Thanks Eugene.

Restarting Wealth-Lab Pro didn't help, but clicking "Update DataSet" solved the problem. After doing that, "Update all data for selected Providers" works as expected.

I just want to point out to anyone else reading this thread that all Quandl stock symbols must contain the prefix "WIKI/", which you must add to each of the exchange symbols, ie."A" is "WIKI/A". As a result, Wealth-Lab Pro puts all stock symbols in the Quandl/Daily/W directory. Therefore, I if your Strategy uses the exchange symbol, you have to add "WIKI/" to the symbol before calling methods such as GetExternalSymbol.
profile picture

Eugene

#32
QUOTE:
all Quandl stock symbols must contain the prefix "WIKI/"

That's correct as long as you're downloading free EOD stock data from the "Wiki EOD Stock Prices". For every other (currently only paid) databases there's a different prefix.

QUOTE:
Restarting Wealth-Lab Pro didn't help, but clicking "Update DataSet" solved the problem.

Thanks for the update Kurt. I'll revisit this issue.
profile picture

Eugene

#33
QUOTE:
Restarting Wealth-Lab Pro didn't help, but clicking "Update DataSet" solved the problem. After doing that, "Update all data for selected Providers" works as expected.

According to my findings, this is a limitation/issue in Wealth-Lab. Most providers using the BarDataStore are affected - in particular: FRED, Dukascopy, Quandl and Google providers. However, it's pretty easy to work around on our end. A fix is coming in next extension update (2017.04).
profile picture

Eugene

#34
@Panache: fixed, please update the extension and restart WLP.

What's new in version 2017.04:

* Fix: "Update all data" doesn't update any data during first-time update
* Change: Wealth-Lab 6.9+ required to install/update
profile picture

pnperez

#35
Hello

I used the Quandl data static provider for some time but since mid april the display of graphs in wealth lab is wrong
i use this database SCF continuous futures
the data i capture is correct but the graphs are wrong

https://www.quandl.com/data/SCF-Continuous-Futures
I use price data ending in FW

if i do copy to clipboard data, there is no doubled up price data, and there are none on the quandl website if i download manually
could you check if there is a bug in the graph display?

Thanks in advance
profile picture

pnperez

#36
Hello

I used the Quandl data static provider for some time but since mid april the display of graphs in wealth lab is wrong
i use this database SCF continuous futures
the data i capture is correct but the graphs are wrong

https://www.quandl.com/data/SCF-Continuous-Futures
I use price data ending in FW

if i do copy to clipboard data, there is no doubled up price data, and there are none on the quandl website if i download manually
could you check if there is a bug in the graph display?

I attach the jpg of the print screen

Thanks in advance
profile picture

Eugene

#37
Hello,

I don't think there's anything wrong with the provider as I use Quandl data too and would've noticed that. Probably your data is corrupt. Right click on the chart and select "Reload Chart History" to reload the symbol from scratch.

The free sample data (2517 bars ending in 2014) loads up correctly for me as can be seen (please use PNG instead of JPG for screen captures):

profile picture

pnperez

#38
Hi Eugene

Yes it works if I reload the chart

Thanks for your help

Pierre
profile picture

Eugene

#39
Pierre, that's good news to hear! Glad to help.
profile picture

Eugene

#40
What's new in version 2017.06:

* New: AAII sentiment data (requested by richard1000)
profile picture

Eugene

#41
What's new in version 2017.08:

* New: automatic recognition of any data having just 2 fields as Close-only data
profile picture

Eugene

#42
Changes in version 2017.09:

* Fix: increased stability by fixing startup crash on broken configuration file
profile picture

Eugene

#43
I was thinking about creating a data provider for Bitcoin data but stumbled onto a Quandl ticker for Bitcoin price at Mt.Gox. Simply add BCHAIN/MKPRU to your Quandl DataSet(s) to track Bitcoin in Wealth-Lab:

BCHAIN/MKPRU

Just a proof how comprehensive has Quandl become.

UPDATE 10/14/2017: Another bitcoin dataset discovered by a Wealth-Lab community user:

BCHARTS/BITSTAMPUSD
profile picture

Aldo

#44
Hi,

please is there a way to attribute a SecurityName to Quandl Symbols (or any other symbol missing a name)?
For example
"Gold" for CHRIS/CME_GC1
"Bund" for EUREX_FGBL1 (!)

"Symbol info manager" is the tool that I expected could be used for this, but there is no SecurityName field there.

It is apparently a minor problem, but apart from identifying odd symbols, a customizable name could also be useful to sort securities within datasets -- if of course datasets panel would display names instead of symbols.

Thanks in advance for the answer.

Kind regards
Aldo
profile picture

Eugene

#45
Hi Aldo,

There is no easy workaround to append a secuity name to existing data.

While some of Quandl's APIs may return company name, this isn't the case for the API our parser is based on (unfortunately). We have this enhancement idea in our product backlog as a low-priority task but I'm making no promises currently: it's not clear if it's feasible yet.
profile picture

Eugene

#46
Good news Aldo. Starting from v2017.11, the Quandl provider will append security names to new symbols. This is to minimize the number of data requests since we're going to add an extra request for security name. For the security name to appear in an existing symbol you'd have to remove all its data (e.g. using "Remove all data" feature of the Data Tool extension) and update the DataSet (or "Reload chart history" if you don't have that much symbols).
profile picture

Aldo

#47
Hi Eugene,

thanks, that´s great, I´ll wait for the new version to be released then.

profile picture

Eugene

#48
Aldo, please note however that there will be a considerable tradeoff behind getting those symbol names.

1. First, security names cannot be recognized for the stock symbols from WIKI/ or EOD/ groups.

2. The provider is built around CSV API which doesn't return company name. And it shouldn't - the structure of a CSV file doesn't play well with security name. Quandl's other APIs, JSON and XML, perfectly fit but conversion of our mature CSV client to another API is out of question since it's an involved task.

So we have to fire an extra request for symbol name whenever Wealth-Lab downloads the history for a symbol for the first time (only). Enter Quandl's fair usage policy. If the provider makes more than 2000 calls in a 10 minute slot or even Quandl thinks it's "speeding", your data requests are throttled. There are other limitations to avoid misuse but this is not the point. For you this means that during initial backfill of a XXX-symbol DataSet you may not get say some symbols and will have to re-request them by repeating the update. (It may be a tough call to introduce automatic retries for failed data updates at this stage).

That's the price of the request that initially seemed innocuous. What I've said applies to the current version as well. Of course the provider will do its best to adhere to Quandl's call frequency policy but it does not guarantee from not getting skipped symbols and/or missing company names.

Workaround:

o When doing an initial backfill, break up your big DataSet in parts and update them manually.
o If updating a big DataSet starts returning 0 bars, take a break and repeat later.
profile picture

Eugene

#49
What's new in version 2017.11:

* New: Meet Zacks Fundamentals A and B collections (paid). These are intended to replace YCharts as fundamental provider. Quandl should be much more robust and easier to support: it does not have download quotas and the guys at Quandl never do breaking changes.

* New: Option to prepend symbols with prefix if they belong to one group (e.g. WIKI for free U.S. stock EOD data, or EOD for paid). Don't add the "/" slash (it will be removed anyway).

* New: Provider now appends symbol's security name during initial data download for new symbols (after updating the data itself)

* Change: Data download requests are now throttled with regard to Quandl policy (2000 calls per 10 minutes etc.) This should avoid situations with no data returned due to the Quandl policy violation

* Fix: Volume not read for symbols in BCHAIN dataset

* New: Support for JOHNMATT dataset (requested by wealthpro25)
profile picture

Aldo

#50
Hi Eugene,

I see that as you promised the new version updates the security name, although very few symbols actually have a name. Thanks for that.

And yes, I have to relauch the update of the dataset a few times before all the symbols (they are just 40) update. And a couple of them will never download.
Do these limits apply only to the free database, if I switch to the premium Quandl service it will be fixed?

Kindest regards
Aldo
profile picture

Eugene

#51
Hi Aldo,

QUOTE:
I see that as you promised the new version updates the security name, although very few symbols actually have a name.

All of the datasets I tested with like AAII, COT, Bitcoin etc. do have the security name. The majority of stock symbol in the WIKI/ and EOD/ databases can not be parsed correctly because the security name there is in an irregular format (I've already put a note on the Wiki). I don't really care about these two datasets - the Tiingo provider is the preferred way to work with the Quandl stock data because it has support for splits and dividends which the Quandl provider will not have.

Besides them, can you voice the datasets that still don't have the security name? I'm willing to look into it for and fix the issue for you if it's feasible from a programming standpoint.

QUOTE:
Do these limits apply only to the free database, if I switch to the premium Quandl service it will be fixed?

Premium symbols do have a limitation too - it's 5000 calls in a 10-minute slot vs. 2000 for the free service. However, Quandl starts to cool you down way before you reach the upper limit. And this is undocumented. During the development it was very soon after sending more than merely half a dozen concurrent requests when I started getting that HTTP 429 (too many requests). For this reason I had to curb the number of update threads to 3 (AFAIR). Also note that there's always been a delay between requests.

So my suggestion is "please give it a try and let the forum know".
profile picture

Aldo

#52
Hi Eugene,

thanks, I thought you already answered, writing that in most of WIKI the security name is not parsed correctly.

Here are those without names:
CHRIS/CME_EC1, CHRIS/CME_GC1, CHRIS/CME_HG1, CHRIS/CME_NQ1, CHRIS/CME_PA1, CHRIS/CME_PL1, CHRIS/CME_RP1, CHRIS/CME_SF1, CHRIS/CME_SI1, CHRIS/CME_SP1, CHRIS/CME_TIO1, CHRIS/CME_YM1, CHRIS/EUREX_FDAX1, CHRIS/EUREX_FGBL1, CHRIS/EUREX_FSMI1, CHRIS/EUREX_FSTX1, CHRIS/ICE_B1, CHRIS/ICE_DX1, CHRIS/ICE_T1, CHRIS/ICE_TF1,
CHRIS/LIFFE_FCE1, CHRIS/LIFFE_Z1, CHRIS/SGX_TW1

Another problem with the Quandl provider is the following, the Data Manager will repeatedly download the last bar on a few symbols and for another couple will not be able to download anything (although in Quandl database data are shown - CHRIS/HKEX_HSI1 ):

Symbol: CHRIS/SGX_TW1, Bars: 1, Last date: 02/11/2017
Symbol: CHRIS/CBOE_VX1, Bars: 1, Last date: 06/11/2017
Symbol: CHRIS/SGX_CN1, Bars: 1, Last date: 03/11/2017
Updating security names...
Updating new symbols...
Symbol: CHRIS/HKEX_HSI1, Bars: 0
Symbol: CHRIS/HKEX_HSI2, Bars: 0
Updating security names...
profile picture

Eugene

#53
Answering to #2:

QUOTE:
Another problem with the Quandl provider is the following, the Data Manager will repeatedly download the last bar on a few symbols and for another couple will not be able to download anything (although in Quandl database data are shown

Maybe it looks like that there's a pattern but in fact it's not. Next time you see something like this you'll have an explanation by simply looking at the symbol's data on Quandl website. For the nuts and bolts please revisit the online guide, namely "How it works". This is why you do not get the data for CHRIS/HKEX_HSI1 and CHRIS/HKEX_HSI2:
QUOTE:
The data is so far from being uniform that groups of related instruments (let's say futures) might have different field sets!


So for example:

CHRIS/SGX_TW1 has fields which are easy to recognize since it's already "trained" for these field names:

Date, Open, High, Low, Close, Settle, Volume, Prev. Day Open Interest

Now take CHRIS/HKEX_HSI1:

Date, Open, Bid, Ask, Last Traded, High, Low, Volume, Prev. Day Settlement Price, Net Change, Prev. Day Open Interest

Yes, that's two Quandl symbols which belong to the same database yet have a different structure and field names. Fortunately, it's easy to "acquaint" the provider to CHRIS/HKEX_HSI... - I will add support for this symbol in v2017.12.
profile picture

Eugene

#54
...and now answering #1:

QUOTE:
Here are those without names:

They all work for me but I had to update them twice. Take a break after an update and retry "Update DataSet" shortly after to get the remainder of security names for new symbols (only):



The only reasonable explanation, with regard to my posts #48 and #51, is that Quandl still considers that the provider sends quite many requests too quickly (those separate security name requests count as well). Alas, this is a design limitation we will have to live with.
profile picture

Aldo

#55
all fixed, thanks!

Sorry, my misunderstanding, I reloaded chart history from chart window, not data from data manager.
Actually I also deleted the whole Dataset and recreated it, but probably I made some mistake there too.