Morningstar fundamental, static and streaming data provider
Author: abegy
Creation Date: 4/27/2012 4:33 AM
profile picture

abegy

#1
I see that in the last version of the MS123 Extra Fundamental/News providers extension, you have added the possibility to download dividend from Morningstar.

Is it possible for the next release to add more fundamentals data from this source ?

Morningstar is a worldwide database. Moreover, you can download fundamentals data directly on a csv file (see export feature on the following link)

http://financials.morningstar.com/ratios/r.html?t=BNP&region=FRA&culture=en-US
http://financials.morningstar.com/income-statement/is.html?t=BNP&region=FRA&culture=en-us

If you have the possibility to add also Morningstar has an historical/streaming Datasource extension, it would be nice for an international trader because :

- Quality of data is very good
- There is more than 10 years online of historical data
- 5 years of fundamentals data (more than 20 differents items
- There is realtime streaming for some markets
- it's free of charge

For an international trader like me, there is no others source at this time in WLD where I can match streaming/Historical/Fundamentals data because international symbol code are all the time differents depending of the source as I have explained several time here.

I know that I can develop my own dataprovider but what I would like is to spend my time is on trading strategy development and not on technical modules

WLD is a nice tool with a great support
Give for international traders a minimum of facility to trade on their local markets ;-)

Thanks for you understanding and I hope you will be able to do something :-)
profile picture

Eugene

#2
Good day,

Some background on why did I pick Morningstar's dividend feed. Mainly for four reasons:

1. relative simplicity of implementation (compared to other Morningstar data items).
2. its exclusive dividend data fields missing in Fidelity's data.
3. the historical data depth surpasses Fidelity's (in this particular case).
4. based on experience, it might take a lot of time to introduce a change to Fidelity data feed.

Unfortunately, the data items you're proposing is a different story. As you pointed out, symbol names are different across different data providers. Wealth-Lab wasn't designed with symbol aliases in mind, and this feature is not on our radar this year. I think you'd agree with me that this Morningstar fundamental data provider would hardly be useful for EU stocks without a companion static data provider. Not to mention that the same stock data can be obtained from Yahoo! (BNP.PA) with full precision: for unknown reason, Morningstar rounds volume in their data e.g. "32.07 Mil".

Besides duplicating our effort of development, these Key Ratios are a challenge to parse. At first glance, it seems that the exported data presentation may require a sizable change of the design pattern I've been successfully using for other fundamental providers. Lastly, it's annual data like the already existing MSN/Yahoo fundamental providers.

I haven't made a decision yet, but chances of implementing this feed look pretty remote as of today.
profile picture

Eugene

#3
If we focus only on getting a subset of fixed items from "Key Ratios", then I think it's doable (and even implementing the companion static provider). Please create a support ticket and refer to this thread so it's not forgotten.
profile picture

Eugene

#4
QUOTE:
- Quality of data is very good

I. Quality of their static data is rather poor. How about lots of fake bars like this one particular example?


Now compare this with Yahoo, no sign of the fake 1/16/2012 bar:



Notice the different open price. This ain't some pink sheets, that's AAPL Nasdaq.

Not to mention that their static data (like Google) does not contain the open price in the 1990s.

And the worst thing, you could not weed out the fake bars if their volume is "0.0 Mil" - just because lots of perfectly valid symbols do have this "fake zero volume" too even though they were trading. But since the folks at Morningstar decided to lose precision by virtue of rounding, a stock still has a "0.0 Mil" volume if say 50,000 shares changed hands.

II. Their fundamental data suffers from two connected issues:

* Loss of true announcement date. For instance, fiscal year ended at 23 September but the data are marked as September - w/o the date.
* No date by default means 1st of month which is dead wrong if the data became available around the end of month. Consequently, arising possibility of peeking into the future. This is a serious problem that will be handled exactly like we're used to do with YCharts.
profile picture

Eugene

#5
Their ASCII Export produces the most malformed CSV data I've ever seen in years:

* Systematic violation of CSV standard RFC 4180 e.g. guideline #2.4
* Bad formation of interim headers (sometimes split in two lines, somewhere not)
* Multiple currency separators in the file...
* ...with subsequent need to wrap some values in double quotes
* No uniform section headers
* No uniformity in dates in the headers. Could start/end as "TTM" or "Latest Qtr"
* Essentially duplicate fields: "Revenue"
* As the result, an irregular structure preventing from creating a robust parser. It could be done by making several assumptions about the particular sections, so end result can break at any time if data vendor considers a change.

Example
profile picture

Eugene

#6
Nevertheless, I'll build the static and fundamental providers.
profile picture

abegy

#7
Thanks a lot ! Are you going to build also the streaming ?
profile picture

Eugene

#8
No, streaming is not considered.
profile picture

abegy

#9
Why ? As you know, the problem with international stocks is the code translation.
If you don't consider it, we will have no possibility to scan in realtime orders.
It's petty because you have it with BBFree, yahoo and google extension. So, why not with Morningstar ?
Do not forget that Morningstar provides some realtime quotes depending of the market.
profile picture

Eugene

#10
Well, maybe I was looking in the wrong place but there doesn't seem to be any sign of free streaming data. Can you point me?
profile picture

abegy

#11
In the quote page, there is the following note :
QUOTE:
ll data supplied by Morningstar Real-Time Data. U.S. intraday real-time exchange quotes are sourced from BATS when available. End-of-day quotes for Nasdaq, NYSE, and Amex securities will appear 15 minutes after close. Graph times are Eastern Standard.
profile picture

Eugene

#12
If you want me to take a look, you have to point me to the place where it's streaming, and what is required to get there.
profile picture

abegy

#13
In the quote page, http://quote.morningstar.com/stock/s.aspx?t=XPAR%3aBNP, you can see that the last price is updated automatically every second.
profile picture

Eugene

#14
Yep, I see their streaming quotes now that European market is open. Wasn't streaming for me before because I must had been visiting the page outside of EU market hours. Thanks.

Wish my JavaScript reverse engineering skills were better as the ins and outs of quote processing seems to be obfuscated. Nonetheless, it's clear to me how it's made. Let's see if I can work it out...
profile picture

Eugene

#15
Here's good news for you abegy: Morningstar Streaming provider will also be built.
profile picture

abegy

#16
Thanks a lot ! :-)
profile picture

Eugene

#17
The Morningstar Static (EOD), Streaming and Fundamental data provider for U.S. and international securities has been released. Registered Wealth-Lab customers please download and install it from Extensions:

Morningstar data provider

This thread is the place to discuss the provider. Make sure you've read the online guide before asking questions or reporting errors:

Morningstar Static and Streaming provider
Morningstar Fundamental Provider
profile picture

abegy

#18
I try to download fundamentals data for XPAR:BNP. When I update data, I have no problem (see the log here).

CODE:
Please log in to see this code.


But when I try to put a Fundamental item on a chart (for example dividend), I see nothing in the specific chart pane.
Is anything wrong ?
Thanks for your help.
profile picture

Eugene

#19
Reason: the fundamental items from "Morningstar Fundamental Data" haven't downloaded. Server did not return any valid data. (The number of fundamental data items updated is not indicative in this case. By design.)

With Morningstar being a free data vendor, this can and will occasionally happen. I faced it a couple of times during the development. But don't give up after first try. Retry and the server will eventually return valid data.
profile picture

Eugene

#20
The fundamental data feed has slightly changed their output format, rendering a dozen of key fundamental items blank (e.g. "Earnings Per Share"). A fix is in the works.
profile picture

abegy

#21
Do you have an example of the change ? I ask you because I haven't seen any change
profile picture

Eugene

#22
They started attaching currency (EUR) and/or volume qualifier (e.g. Mil) to exported CSV files. For example:

Was: Revenue
Is: Revenue USD Mil

Consequently, the provider is no longer able to match the string against its list of fundamental items.
profile picture

Eugene

#23
The Morningstar provider has been upgraded to 2012.11.1

* Fix: (Fundamental) almost dozen of key Financial items stopped working (zero values) due to a format change at Morningstar
* Change: (Fundamental) "Shares Mil" item was renamed "Shares"
profile picture

abegy

#24
Eugene, I see in the fundamental items that there is 2 items for morningstar Dividend :
- one with the morningstar logo
- another one with "f" as the logo

The first item report differents date for dividend operations but 0 for the cash dividend
The second one report only one date (I don't know for which operation) and the correct cash dividend amount.

For example, try with XPAR:BNP to see that.
Is it normal ?
profile picture

Eugene

#25
Yes, it is normal. Reporting 0 is not normal though (investigating). Will post in the proper forum thread.

There are two independent Morningstar fundamental providers, and each one has a dividend item:

* Morningstar extended dividend data ("[ms] dividend") - part of MS123 Extra Fundamental/News Providers
* Morningstar static/streaming/fundamental provider ("[ms] Dividends", aka "the one with f") - this provider

p.s. The "f" is is Wealth-Lab's default fundamental item icon that appears because the obfuscator has swallowed the correct icon.
profile picture

Eugene

#26
The Morningstar provider has been upgraded to 2012.11.2

Another hotfix:

* Change: (Fundamental) chart popups of fundamental items will display their title (to be easily recognized on the chart among 82 items)
* Fix: (Fundamental) fundamental icons appear as "f" (missing glyph)
profile picture

abegy

#27
Hi Eugene,

Is it possible to add the "TTM" values for all fundamentals items as the last value of the current fundamentals items ?
For example, I would like to see for XPAR:AC, -2.40 for Earnings per share instead of 0.12 in the current version of the extension.

Thanks a lot.
profile picture

Eugene

#28
Alexandre, TTM values will be displayed in upcoming release of the provider (no ETA right now).
profile picture

Eugene

#29
The Morningstar provider has been upgraded to 2013.05

Added TTM values (trailing 12-month).
profile picture

abegy

#30
Hi Eugene,

When I try to update historical quotes from Morningstar extension to have the last up to date data, I see that not all symbols are updated the first time. I need to update 2/3 times more to have it all.

Is it normal ?
profile picture

Eugene

#31
I have no idea. Many factors may play a role, like network connectivity, or free data feed limitations (did you know that some of them don't like parallel or sequential downloads so much that in the past one could even get 'banned' at Yahoo?). It's a free data feed, and when you update thousands of symbols, anything can happen.
profile picture

abegy

#32
I try to update only 40 symbols. See the log. As you will see

QUOTE:
Daily data is being updated...

Updating Provider Morningstar Static ...

Up-to-date symbols: 0, Update required for: 42, New symbols: 0
Symbol: XPAR:ALU, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:BNP, Bars: 0
Symbol: XPAR:RNO, Bars: 0
Symbol: XPAR:SPI, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:UFF, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:AC, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:ACA, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:AI, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:ALO, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:BN, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:CA, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:CAP, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:CS, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:DG, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:EAD, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:EDF, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:EI, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:EN, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:FP, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:FTE, Bars: 0
Symbol: XPAR:GLE, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:GSZ, Bars: 1, Last date: 7/15/2013
Symbol: XPAR:LG, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:LR, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:MC, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:ML, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:MT, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:OR, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:PP, Bars: 0
Symbol: XPAR:PUB, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:RI, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:SAF, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:SAN, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:SGO, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:STM, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:SU, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:TEC, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:UG, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:UL, Bars: 0
Symbol: XPAR:VIE, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:VIV, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:VK, Bars: 2, Last date: 7/15/2013

Provider update complete for Morningstar Static
----------------


QUOTE:
Daily data is being updated...

Updating Provider Morningstar Static ...

Up-to-date symbols: 37, Update required for: 5, New symbols: 0
Symbols already up to date: XPAR:ALU,XPAR:SPI,XPAR:UFF,XPAR:AC,XPAR:ACA,XPAR:AI,XPAR:ALO,XPAR:BN,XPAR:CA,XPAR:CAP,XPAR:CS,XPAR:DG,XPAR:EAD,XPAR:EDF,XPAR:EI,XPAR:EN,XPAR:FP,XPAR:GLE,XPAR:GSZ,XPAR:LG,XPAR:LR,XPAR:MC,XPAR:ML,XPAR:MT,XPAR:OR,XPAR:PUB,XPAR:RI,XPAR:SAF,XPAR:SAN,XPAR:SGO,XPAR:STM,XPAR:SU,XPAR:TEC,XPAR:UG,XPAR:VIE,XPAR:VIV,XPAR:VK,
Symbol: XPAR:BNP, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:RNO, Bars: 2, Last date: 7/15/2013
Symbol: XPAR:FTE, Bars: 0
Symbol: XPAR:PP, Bars: 0
Symbol: XPAR:UL, Bars: 0

Provider update complete for Morningstar Static
----------------


Update completed (4.92 sec)
----------------


This is a nightmare to manage the data with this extension. For quote, it's simple to see if there is a problem of update. But this is impossible to see if the same problem is available with the fundamentals items.

Is it possible for you to check again this extension to try to find a fix ?
profile picture

Eugene

#33
Hi Alexandre,

Thanks for the update log. I was able to nail down the bug in my code. Testing shows that this DataSet can now be reliably updated multiple times w/o issues. The fix is to follow in next update.

P.S. Occasionally you might still see an uncommon "0 bars" when Morningstar actually returns an empty document. You can not combat this - it's a free feed.
profile picture

abegy

#34
Thanks a lot !
profile picture

abegy

#35
Eugene, is it possible to have the Morningstar extension compatible with paper accounts trading ?
profile picture

Eugene

#36
Yes, in upcoming update.
profile picture

Eugene

#37
The Morningstar provider has been upgraded to 2013.09

New: Paper Trading support (requested by abegy)
profile picture

Areality

#38
Does this provider work for anyone? I cannot get any data for AA or any other stock. Below is the log from the Data Manager. Print screen with an empty chart pane attached.

Updating DataSet Test ...
Preparing requests ...
Requests are ready to go.
Updating Security Names for 1 symbols...
Security Names updated.
[0] AA 11010 bars 08.21.2013 0 bars added
Updating Provider Morningstar Fundamental Data ...
AA 83 fundamental items updated
Provider update complete for Morningstar Fundamental Data
----------------

Update completed (4.189 sec)

----------------

profile picture

abegy

#39
You need to add the prefix exchange code before the symbol. For AA, it's XNYS:AA
profile picture

Eugene

#40
Actually, Morningstar Fundamental Data updates without any issues. The prefix is not required, as I seem to get the fundamental data using Yahoo notation (like you just did, because Morningstar provider does not do any "Updating Security Names"):



The issue might be with your system, if the FRED provider fails for you for no apparent reason when it works for everyone else.
profile picture

Eugene

#41
Indeed, the provider has a problem with non-U.S. delimiters. The bug will be fixed in an upcoming release. Thank you for spotting it.

UPDATE

Fixed in v.2013.09.01.
profile picture

Eugene

#42
Areality asked:

QUOTE:
Please add fundamental fields which were presented exclusively in MSN and can be found from Morningstar. Being selfish I need a substitute for "[msn] depreciation" field. It can be found from Morningstar. Here is IBM example http://financials.morningstar.com/cash-flow/cf.html?t=IBM&region=USA&culture=en-US . The third row from top. "Depreciation & amortiz...". Your work is greatly appreciated.
profile picture

Eugene

#43
Thanks for your request. I'll take a look later.

There's twice less data (5 years) for the Cash Flow (Balance Sheet, Income Statement) items than the usual 10 for the "Key Ratio" items, though.
profile picture

Eugene

#44
Areality,
QUOTE:
Being selfish I need a substitute for "[msn] depreciation" field.

I determined that the task of adding is too complex (requires significant effort), and the data is very limited. Considering you already are a YCharts Pro user who have requested to add a depreciation item to that provider, it would be way easier to add a range of depreciation items to the YCharts provider early next month.
profile picture

Areality

#45
Just wondering if anyone at WealthLab looked at Morningstar Premium as an alternative to YCharts Pro. Morningstar data quality seems to be better and the price tag is only $199.00/year vs $1000-$2000 for Ycharts. Maybe it is also a more friendly animal for Wealth Lab developers? Does anyone has experience using Morningstar Premium? How did you like it? I can arrange access to Morningstar Premium for Wealth Lab developer interested in the project.
profile picture

Eugene

#46
The only difference between Free vs. Premium I've found is the data loading depth:

* Quarterly: 5 vs. 10 quarters
* Annual: 5 vs. 10 years

Is it worth it? Do 10 quarters (i.e. less than 3 years) make great difference when compared to the data loading depth at YCharts?
profile picture

abegy

#47
Hi Eugene,

On the Morningstar websute, there is a specific page where you can access a specific market indicator calculated by Moningstar. There are available on the following page (http://www.morningstar.com/market-valuation/market-fair-value-graph.aspx).

Is it possible for you to give us the possibility to download all of them inside the extension ?

Thanks a lot.
profile picture

Eugene

#48
Hi Alexandre,

Sorry, this looks neither promising nor feasible.
profile picture

gbullr

#49
Trying to use Morningstar to pull in revenues.


Used this code.

CODE:
Please log in to see this code.


Pulls in Fiscal Yearly Revenue from 2012.

What am I doing wrong.

Is provider no longer valid?

Thanks.

profile picture

Eugene

#50
The provider is working correctly. For all the Dow 30 symbols I've just tested it on, the last change happened in 2014. Note that the data is annual.

Make sure you're running the latest version 2013.09.01 (see Extension Manager > Others). What were the symbols?
profile picture

gbullr

#51
It was MSFT and it was pulling in


http://financials.morningstar.com/income-statement/is.html?t=MSFT&region=usa&culture=en-US

the 73723 revenue num for June 2012.
profile picture

Eugene

#52
Can not reproduce. MSFT or Dow 30, still works perfectly. Note the 86,833 number for 2014-06. Follow my advice on getting the last version.


profile picture

gbullr

#53
Will re install..

Thanks.

As an aside..

How did you get such a nice yearly chart above?

profile picture

Eugene

#54
QUOTE:
How did you get such a nice yearly chart above?


By switching to Monthly scale. There are two ways as per the User Guide:

+ Charting > Chart Window > "8. Chart Scaling"
+ Charting > Chart Control > Data Panel
profile picture

LenMoz

#55
The Wiki for this provider states, "•The last quarter or trailing twelve months value is dropped.". This does not appear to be true.. When I run the script below for Alcoa, "AA", it shows the last fundamental data item to be the TTM value on the Morningstar web site. It's dated 1/1/2015. I get...
CODE:
Please log in to see this code.
On the Morningstar web site, the 18.4 value is titled "TTM" , and 16.3 is titled "2013-12".


CODE:
Please log in to see this code.
profile picture

Eugene

#56
TTM are no longer being dropped based on customer's request (see messages #27 to #29 in this thread). The Wiki page has been corrected. Thanks for the heads-up.
profile picture

richard1000

#57
Problems in Morningstar Provider:

Updating DataSet Morningstar Dow 30 ...
Up-to-date symbols: 0, Update required for: 30, New symbols: 0
Error setting min/max download threads
Error setting min/max download threads
Update completed (0.125 sec)
profile picture

Eugene

#58
This bug occurs under very rare conditions. I'll upload an updated extension (ETA end of month). Thank you for the heads-up.
profile picture

Eugene

#59
What's new in Morningstar provider's recent version 2016.05?

* Fix: "Error setting min/max download threads" message prevents static provider data update
* Fix: removed artificial limit (.NET default) in New DataSet Wizard preventing from creating DataSets which contain more than ~6500 symbols (32767 chars)
* New: due to holiday bars in EOD data, added support for Market Manager in static data provider
* Change: .NET 4.5 and Wealth-Lab 6.8+ are required to install/update extension
* Change: (internal) updated browser agent version
* Change: (internal) increased thread download timeout to prevent failed requests
* Change: (internal) Multithreaded download routine rewritten. Should improve work on modern multi-core CPUs (e.g. servers)
profile picture

Andrii

#60
Morningstar extentionI I have noticed is using the latest annual financial reportings data versus trailing twelve month, probably I am using it wrong, let us know the way we can switch it to TTM or this is prebuild? Thank you very much in advance.
profile picture

Eugene

#61
The provider returns the data exactly as delivered by the free feed. But don't hesitate to visit the Wiki and search the forums: there's a good chance you'll find a workaround. For example, here's how to get TTM values:

Weath-Lab Wiki FAQ > How can I get TTM, MRQ and FTM values of estimated earnings?
profile picture

Eugene

#62
What's new in Morningstar provider's recent version 2016.11?

NOTE: This update is a must have. It's recommended that you reload all accumulated fundamentals by Morningstar.

* Fix: many values were recognized incorrectly (lower than they are e.g. 2,353 was presented as 2.353 rather than 2353)
* Fix: "Book value per share" is unrecognized due to item renaming in downloaded data
* New: added new items: Payout Ratio %, Interest Coverage, Free Cash Flow Per Share
profile picture

Eugene

#63
What's new in version 2017.04:

* Fix: "Update all data" doesn't update any data during first-time update
* Change: Wealth-Lab 6.9+ required to install/update
profile picture

Eugene

#64
What's new in version 2017.10:

* New: support for automatic bar data corrections
profile picture

tobiastempus

#65
hi,

i dont get fundamental data for european stocks. i read here that there are some Problems with different european ticker Symbols but i dont know how to fix it.

thank you


here is a picture
profile picture

Eugene

#66
Hi,

As for me, the 1st paragraph of the Wiki page you refer to tries to explain it pretty well that the Yahoo! static data for ADS.DE (like shown on your screenshot) cannot be used with the Morningstar fundamental data:

QUOTE:
Since most every data vendor has its own symbology, Morningstar is no different. This is not a big issue for U.S. symbols which are universal, but for overseas symbols it presents a problem. For example, the ticker for a European stock symbol will most likely differ from its counterpart at MSN, Yahoo or QuoteMedia. For example, the same BNP stock traded on XETRA is known as "BNP.F" at Yahoo but exists as "XETR:BNP" at Morningstar. Consequently, attempting to download the historical fundamental data for BNP.F from the vendor that knows the stock as XETR:BMP will fail because it's not recognized.


Solution is obvious. Please visit Morningstar's website and look up the correct ticker for Adidas in their notation (could be XETR:ADS, for example - I didn't check it). Create a DataSet with this symbol using the Morningstar static provider and it should be a good match to Morningstar's fundamental data.