MS123 Fundamental/News providers: Stock News, Analyst Ratings, Insider Transactions, Earnings, Economic Data
Author: Eugene
Creation Date: 9/1/2011 4:24 AM
profile picture

Eugene

#1
profile picture

ecorderob

#2
Hi Eugene,

I have one question related to this new extension (insiders).

I have some datasets from Yahoo and BBFree. I have realized that it only downloads some data for the yahoo symbols, not the BBFree. Is that right?

Another strange behaviour I have detected is that I only get values for a few symbols, indeed only 4 symbols (A, AA, AAPL and ABC) then no other symbol has insider info. Have you detected this kind of behaviour?
profile picture

Eugene

#3
QUOTE:
I have realized that it only downloads some data for the yahoo symbols, not the BBFree. Is that right?

If these BBFree symbols are defined like "AAPL:US" i.e. have a suffix, the symbol could not be found in the insider data source and no data can be downloaded. The symbol name has to be plain vanilla e.g. AAPL, without a suffix or other vendor specific additions.

A fundamental data provider does not know anything about the static data provider and it does not matter if it's asked to request AAPL while running on a Yahoo or Google DataSet, but the suffix makes the whole difference. (Stripping out suffix might be a good idea for provider enhancement.)

QUOTE:
Have you detected this kind of behaviour?

No, never noticed. What's the 5th symbol? Or could you zip and upload latest data update log somewhere?
profile picture

ecorderob

#4
The 5th symbol is : ABT

I will execute now the update again to check the log, but.. could it have some relation to the fact that I have many data from other providers with different preffix or suffix?
profile picture

Eugene

#5
Each symbol is processed individually, and neither valid symbols (like ABT) nor those with a prefix/suffix can't break the whole process. Most likely, a network/server glitch or something like that. ABT returns insider transactions just fine for me. In any case, the insider transactions portion of your recent data update log could help me shed some light on what's going on there.

QUOTE:
(Stripping out suffix might be a good idea for provider enhancement.)

Answering to myself: no, it's not as simple as I thought it to be. To avoid opening a can of worms, let's discard this idea. If required (e.g. AAPL:US), have the provider collect the transactions for AAPL in a Yahoo! DataSet first. When executing a Strategy that makes use of FundamentalDataItems on your favorite BBFree DataSet (if you have this preference for data), just strip out the suffix in the current Bars.Symbol. It's much easier/safer than to patch the provider!
profile picture

ecorderob

#6
Weird...

I have removed all the data from Yahoo and downloaded it again. Then I updated the insiders and.... now I don't even see the ones I was seeing before!

I have uploaded the log file. Nothing strange in there:

http://www.megaupload.com/?d=37160SJ8
profile picture

Eugene

#7
Only about 5% of your symbols can return insider transactions. Note it works for U.S. stocks only - no futures, no ETFs, no indices, no international markets, and no suffix/prefix should be there. Only a U.S. stock without a suffix is a valid symbol. Just remember, this U.S. insider data vendor "knows" nothing about BBFree's/other symbology.

Returning 0 fundamental items on valid symbols is either a server error or something has already been written to the file. Try the following:

1. Remove the InsiderTransactionsProvider folder from \Users\your username\AppData\Roaming\Fidelity Investments\WealthLabDev\1.0.0.0\Data\
2. This time, repeat update in DataSet mode on DataSets containing valid symbols only.

To do it, highlight a DataSet in "DataSets" and then click "Update DataSet" - but not "Update (Pricing)". In this case, fundamental providers selected on "Update Data" tab will be processed.
profile picture

Eugene

#8
With version 2011.11, meet our first news provider:

Google Finance RSS news feed

Update your existing installation in Extension Manager (or install the provider from Extensions if you haven't done so) and start getting news items for stocks which you can chart as any other fundamental items like "eps" or "dividend". The financial news are provided by Google Finance (up to 2 years back).

Power your analysis with non-price series like news and sentiment! More details in the Wiki:

Google News provider
profile picture

Eugene

#9
With version 2012.01, meet our second news provider:

* High "signal-to-noise" Reuters news feed!

Reuters News provider - online documentation in the Wealth-Lab Wiki

* Bonus:

Bankruptcy filings (economic items provider)
profile picture

Eugene

#10
The extension has been updated to version 2012.05. Highlights:

* New: Morningstar fundamental provider (extended dividend data) - requested by thodder
* Change: (internal) http-agent version upgrade
* Change: Wealth-Lab 6.3+ required to install this extension
* Fix: Yahoo! Analyst Ratings got broken (due to external change), now more robust
* Fix: required shared component, HTMLAgilityPack DLL, will not be uninstalled
profile picture

abegy

#11
I would like to inform that I have a problem with Reuters News provider.
When I try to download news for my stocks database (around 8000 tickers), WLD looks like to froze downloading after 1 hour (no reporting activity of new ticker updated).
If I try to close Datamanager, cancelling update is launching.

Do you have the same problem ?
profile picture

Eugene

#12
Reuters News provider is working for me. Most likely, the system froze due to some server or network error. What symbol the provider was updating? If you can reproduce it with certainty with some particular ticker(s), then we might be able to fix.
profile picture

abegy

#13
I reproduce the problem all the time. I'm using RIC code as ticker to be able to have news from reuters provider.
The last ticker updated is "CBR 25 news items updated" (ticker item number : 1195). After this ticker, normally it would be "CBRL".

The problem appears only with Reuters News provider. I have no problem with Google News on the same database.
profile picture

Eugene

#14
Updating Provider Reuters news ...
CBR 25 news items updated
CBRL 21 news items updated
Provider update complete for Reuters news
----------------

Did this several times by clicking on "Update DataSet", same result (OK). Please try excluding CBRL from your DataSet and retry.
profile picture

abegy

#15
I have retried and see that the download is stopping just after the ticker number 39 which had no problem to be downloaded before.
profile picture

Eugene

#16
Looks like a network/server problem -- there's nothing to fix because nothing appears broken. It's Sandy time, some of our vendors are also having issues today. Try again tomorrow or even later, and possibly from different IPs.

Update: still OK for me today.
profile picture

abegy

#17
I confirm you that I have still the problem. How many symbols do you try to download ? Me, it's working also for few symbols. But for more than 1500, it's not working.
profile picture

Eugene

#18
Well, I never tried to download 1500 symbols. It's a tough debugging, considering it's all multi-threaded, and at some point the website itself may even think it's a bit too much and throttle/reject your requests.

In this case I am suggesting to switch to update by DataSet (if you were updating symbols by all providers) or, if it's such large DataSet you were updating, to break it down and perform the fundamental update in parts.

If this is not appropriate, use Google news.
profile picture

abegy

#19
Unfortunately, I cannot update dataset by dataset because I'm using Metastock file format. Data Manager doesn't allow that for this fie format.
profile picture

Eugene

#20
Having made a couple of changes (enhanced an XPath query and chosen .NET 4.0 as target framework version), I've successfully updated Reuters news for the entire S&P 500 without a single hiccup:

...
ZMH 24 news items updated
Provider update complete for Reuters news
----------------

Update completed (1 hrs 9 min 7.596 sec)
----------------

You have the backup provider which is much faster (Google news doesn't have to grab dozens of HTML pages - only a single lightweight RSS document). Sorry but this is where it ends. No more time to dedicate to what I couldn't reproduce.
profile picture

Eugene

#21
It has been determined that the Morningstar dividends provider incorrectly reports "0" for the international stocks like XPAR:BNP. A patch is in the works.
profile picture

Eugene

#22
The provider has been updated to version 2012.11.

Summary of changes:

* Change: requires Wealth-Lab 6.4 and .NET 4.0 framework
* Fix: Morningstar dividends: Dividends denominated in other currencies erroneously displayed as zero
* Change: Reuters news: made XPath query more robust (internal)
profile picture

abegy

#23
I see that it is possible to "request news" by date from the Yahoo website (see this example : http://finance.yahoo.com/q/h?s=BNP.PA&t=2011-08-16). As you can see, you have more than one year online. Sources come from Bloomberg, Reuters and CNBC.

Is it possible to have an extension for Yahoo news ?
Thanks a lot
profile picture

Eugene

#24
Thank you for the suggestion. It would be possible if the content could be delivered on a single page, something like this, or at least a few items (pages, RSS feeds, CSV/XML/JSON requests etc). Just compare the speed of Google news vs. Reuters news: it's an order of magnitude. But requesting news by date from Yahoo will be much slower than Reuters. Additionally, Yahoo news have a very weak "signal-to-noise" ratio: only a few items really relate to the stock itself e.g. BNP.PA. I don't consider such newsfeeds that aren't on par with Google/Reuters.
profile picture

Eugene

#25
The provider has been updated to version 2013.01.

Summary of changes:

* New data provider: Earnings.com Fundamental Provider for extended earnings data
profile picture

abegy

#26
Great news ! Thanks a lot :-)
profile picture

abegy

#27
I would like to inform you that earnings.com website is down. Now, it's a direct link to reuters.com website.
profile picture

Eugene

#28
Sad news. Even more indicative, the domain owner has become "domain.name@thomsonreuters.com". Taking down this provider in a new update of MS123 Extra Fundamental/News. Thanks for the heads-up.
profile picture

abegy

#29
If you are interesting, there is somethings similar from the Yahoo website : http://biz.yahoo.com/research/earncal/today.html
profile picture

Eugene

#30
As advertised, the terminated Earnings.com provider has been removed starting from version 2013.07.
profile picture

Eugene

#31
ATTENTION

As fulldisclosure.com is not loading right now, it appears that Reuters has pulled the plug on this domain too. Should this change be permanent, the Earnings.com fundamental data provider will be excluded from MS123 Extra Fundamental/News package - now for good.

In addition to fulldisclosure.com, there are two more domains on 162.8.64.131 - retail.ccbn.com and companyboardroom.com - none of them is alive. :( There's a hot discussion at EliteTrader that motivated traders are welcome to monitor and repost related news:

Earnings website
profile picture

Eugene

#32
The provider has been updated to version 2013.10.

Summary of changes:

* New: ThinkNum fundamental provider (requested by dextermueller)
* New: 99wallstreet.com earnings data (fundamental provider)
* Change: finally removed Earnings.com (fulldisclosure.com) provider due to data feed demise
* Fix: FRED: minor internal fix

ATTENTION: DO NOT update if you need to access old saved earnings by the defunct Earnings.com provider, or you will lose that data.
profile picture

abegy

#33
Eugene, is it possible to add again earnings.com thanks to this website : http://staging-retail.ccbn.com/splash.asp?client=cb ?
Thanks a lot :-)
profile picture

Eugene

#34
This seems to be missing the latest releases, hopefully they'll be bringing them online eventually. Nevertheless, thanks for the heads-up Alexandre. The provider is all set to be back in business in early December.
profile picture

Eugene

#35
bobydesi123 asked:


Seems like the 99wallstreet.com site is inaccessible. Has it been taken down for good? Any idea?
Thanks.
profile picture

Eugene

#36
No, troubeshoot your network connection. Whenever you or others have such question re: 3rd party site, don't forget to check this (or similar) site in the first place:

http://www.downforeveryoneorjustme.com/99wallstreet.com

Next step is to contact the domain's admin by email in whois and/or website's tech support:

http://whois.domaintools.com/99wallstreet.com
profile picture

bobydesi123

#37
Eugene,
Thanks for quick reply.

I used the 1st link and got the message "It's just you. http://99wallstreet.com is up."
So apparently it is my network connection. ( I have verizon FIOS)

UPDATE:
I resolved the problem by having the verizon change my IP. It must be bogon..or something ...thanks for all your help




profile picture

bobydesi123

#38
Re: 99wallstreet.com not accessible.

Eugene,
As i had mentioned before I was able to resolve the access to the 99wallstreet.com by having verizon change my IP address.

It seemed to work for a week, and same problem. I can't access 99wallstreet.com. i know it is my 'IP adress' that is denied the service as a tried 'tor' browser and site is accessible.
Question:
1. Almost every day i update my data set that contains all of stocks listed on NASDAQ and NYSE with fidelity data and the 99wallstreet EPS data.
2. I am wondering that since these are too many requests to 99wallstreet.com, does the website/admin think that it is a some sort of 'denial' of service attack and blocks me out?
profile picture

Eugene

#39
Most fundamental data is not being updated every day, so it makes sense to uncheck the fundamental data sources when doing regular daily updates and only enable them on every week or even quarter - depending on your fundamental items of interest. The website admin and the antivirus development team should know the answers to your other questions.
profile picture

LenMoz

#40
Eugene,

Re:
QUOTE:
it makes sense to uncheck the fundamental data sources when doing regular daily updates and only enable them on every week or even quarter


Won't delaying the download of fundamentals wreak havoc with closing positions? Positions opened while fundamentals became out-of-date might not have been opened if the most current data were available, and so, will never be closed. Changes to fundamentals are staggered. This is unless the effective date of the new fundamental is the date it's downloaded. That would surprise me but you may know otherwise.

Can you verify this?
profile picture

Eugene

#41
Len,

Like I said, it depends on fundamental item. Some should be downloaded daily (e.g. news or insider transactions) while for others (released quarterly) this doesn't apply. I do not have anything more to add in this "blog" thread for the "MS123 Extra Fundamental/News providers" library where I only post extension updates, collect requests for new data sources and notifications about data feeds going down. If you like to continue this discussion, please consider asking your general usage question in a new thread. Thanks.
profile picture

Eugene

#42
The provider has been updated to version 2014.05

Summary of changes:

* Fix: ThinkNum provider broken (obfuscation bug)
profile picture

Eugene

#43
The provider has been updated to version 2014.09

Summary of changes:

* New: Multpl.com economic data provider. Requested by richard1000.
* Fix: 99wallstreet provider stopped working due to a website change. This fix is temporary and may break again anytime.
profile picture

Eugene

#44
The provider has been updated to version 2014.10:

Summary of changes:

* New: ThinkNum: fundamental data provider for stocks (83 items) - requested by coppola
* Fix: ThinkNum: economic provider broken due to website overhaul
* Change: Multpl: added 10 year treasury rate - requested by richard1000
profile picture

Eugene

#45
What's new in latest version 2014.11:

* Fix: 99wallstreet provider stopped working (due to legacy URL having been decommissioned)

Previous temporary fix was broken with the decommission of legacy 99WS website. This new fix is expected to keep the provider working until there's a major website change at 99WS (hopefully not).
profile picture

Eugene

#46
Happy New Year!

Version 2015.01 brings a minor optimization to Reuters News provider in hope of fixing a rare issue.
profile picture

Eugene

#47
What's new in latest version 2015.04:

* Insider Transactions provider: added "relationship" field (requested by fairone99)
profile picture

fairone99

#48
Thank you very much, Eugene for adding the "relationship" field for the InsiderCow data feed.

I have been testing it using this method:

fiIC.GetDetail("relationship")

However, I can not seem to retrieve any "relationship" information. I have made sure that I have the 2015.04 MS123 Extra Fundamental News Providers Extension installed (even uninstall and reinstall it after merely updating it won't work) and .NET 4.5.2 framework installed properly. Also, I have tried it on many single symbols still without luck.

Any advice?


Alex
profile picture

Eugene

#49
You have to reload all data by the provider from scratch. It is not available in the already downloaded data.

I double checked that it works. If your experience is different, please use this thread: Insider Transactions provider: GetDetail question
profile picture

fairone99

#50
Great, Eugene...

After I reload the data and it works. Thank you again.


Alex
profile picture

Eugene

#51
The ThinkNum feed no longer delivers neither economic nor fundamental data items due to yet another website overhaul. The remaining data availability on the website is severely delayed, partial at best, and raises serious concerns about data adequacy. Both ThinkNum's economic and fundamental data providers will be permanently removed effective v2015.07.
profile picture

Eugene

#52
Effective 2015.07, the Earnings.com fundamental data provider will be permanently removed from the library due to web data feed shutdown.
profile picture

Eugene

#53
What's new in latest version 2015.07:

* New: OpenInsider provider: free insider transactions data going back over 10 years
* New: StreetInsider provider: replacement for broken 99wallstreet feed
* Fix: InsiderCow insider transactions: internal fix
* Change: (BREAKING) ThinkNum fundamental and economic providers are removed due to decommissioned data feed. All accumulated data disappears.
* Change: (BREAKING)Earnings.com fundamental provider is removed due to decommissioned data feed. All accumulated data disappears.
* Change: (BREAKING)Bankruptcy economic provider is removed due to data feed stopping to return data past 2013. All accumulated data disappears.
profile picture

Eugene

#54
What's new in latest version 2015.11:

* Fix: StreetInsider: Believed to fix incorrect data in certain locales (Australia)
* Change: Wealth-Lab 6.8+ required to install/update
profile picture

Eugene

#55
fairone99 asked in a thread that doesn't exist anymore:

Hi!

Recently, I have encountered an error of downloading no data from OpenInsider.

For each one of the symbol, the message of "0 fundamental items updated" shown.

I go to the download directory of OpenInsider, no data is there. Each file is 1 KB in size.

I have checked the website and it is still up and running (appear to be working...)

Where and what else should I check or adjust in order to solve this?

Oh, recently, I just upgrade WLP from 6.8 to 6.9 and also update MS123 Extra Fundamental Extension to the latest (2015.11). I wonder if these have something to do with the no data downloaded problem.

Thank you for the help in advance.


Alex
profile picture

Eugene

#56
Although the website is running, they've changed something so the data isn't recognized. Happens all the time. Thanks for the heads-up. I'll look into it. There is nothing to adjust but you have an equivalent replacement: the InsiderCow provider.
profile picture

Eugene

#57
What's new in latest version 2016.01:

* Fix: OpenInsider provider: no data returned due to website change

Alex, the provider has been fixed. Update it using the Extension Manager and it'll work again.
profile picture

fairone99

#58
Wonderful, Eugene and it works perfectly now.

Thank you!


Alex
profile picture

Eugene

#59
Version 2016.02 brings a long requested feature and a bug fix:

* New: Ability to backtest deposits and withdrawals (taxes...) of funds to account (Transfer provider)
* Fix: Reuters News: data not updated due to web page layout change
profile picture

Eugene

#60
What's new in latest version 2016.06:

* New: OpenInsider provider: by customer request, added optional "remark" field that indicates amended/error filing, option exercise or multiple insider transactions summarized
* Fix: FRED static: removed artificial limit (by default) in New DataSet Wizard preventing from creating DataSets which contain more than ~6500 symbols (32767 chars)
profile picture

Eugene

#61
What's new in latest version 2016.08:

* New: Inflation economic data provider - historical data of inflation by country
profile picture

Eugene

#62
What's new in latest version 2017.04:

* Fix: FRED static: "Update all data" doesn't update any data during first-time update
* Fix: 99WallStreet provider works again following a website change [since version 2016.09]
* Change: Wealth-Lab 6.9+ required to install/update
profile picture

Eugene

#63
What's new in latest version 2017.06:

* New: MarketBeat Analyst Ratings provider
* Change: Yahoo Analyst Ratings provider is removed due to decommissioned data source
profile picture

WEALTHPRO25

#64
how do i get the following on the graph of OpenInsider - sample chart annotations as seen in wealth lab wiki where you see horizontal red and green lines on the chart, i put the code but i only get a pop up not the horizontal bars on the chart, please help, thanks
profile picture

Eugene

#65
See the Wealth-Lab User Guide > Preferences > Chart Annotations.
profile picture

WEALTHPRO25

#66
i tried but was not succesful, i went on user guide but was not able to develop the code, i would greatly appreciate if you can provide the code for chart annotations for the insider transactions for the openinsider so horizontal bar can be viewed on the chart, thanks.
profile picture

Eugene

#67
I didn't expect that this utterly simple question turns out into a support conversation in what is purely an announcement thread.

So, without starting a new discussion thread, the only advice I will give you here is that there's no need to develop any other code - simply re-read and follow the User Guide chapter. Specifically:
QUOTE:
You can choose additional fundamental items from list of Available Items (or deselect them) by clicking on an item and then the appropriate arrow button.

Given that the data has been downloaded correctly (proved by your detailed PrintDebug output which I've stripped out), it's only a matter of enabling the "[oi] insider transaction" item. There's no alternative.
profile picture

WEALTHPRO25

#68
i got it thanks
profile picture

Eugene

#69
What's new in latest version 2017.07:

* New: Yahoo Analyst Ratings provider fully restored

I discovered that Yahoo's new web page grabs up to nearly 20 years worth of anlayst ratings data (since 1998) for a stock internally but displays only the last five or so items for whatever reason.

The Yahoo! Analyst Ratings provider is rebuilt and available again.