Nasdaq Fundamental Data provider hangs during update
Author: maximgl
Creation Date: 9/2/2019 11:31 PM
profile picture

maximgl

#1
It does not crash, just hangs forever after a specific symbol. Attempts to cancel the load also hang. It seems that it always hangs at the same place, after only a few symbols being pulled. Nothing in event log, and the download log has nothing, since the load hangs within few seconds of download start. I've watched the order of symbols for other providers and assuming order is the same for all providers, I think it is hanging on .MB_ADV.A symbol.

Please let me know if there are ways to capture any additional log information.
profile picture

Eugene

#2
If I run the fundamental provider update in both modes (Update DataSet and Update All Data) on lots of invalid symbols unsupported at Nasdaq's side, the issue positively cannot be reproduced. The provider will not hang and continue without any delay as expected:

QUOTE:
Updating DataSet test ...
[3] ^VIX Error: No data
[0] .MB_ADV.A Error: No data
[2] ^VAY 4947 bars 30.08.2019 4947 bars added
[1] ^GSPC 4947 bars 30.08.2019 4947 bars added
Updating Provider Nasdaq Fundamental Data ...
.MB_ADV.A fundamental items updated: 0 dividends
^GSPC fundamental items updated: 0 dividends
^VAY fundamental items updated: 0 dividends
^VIX fundamental items updated: 0 dividends
Provider update complete for Nasdaq Fundamental Data
----------------

Update completed (10.463 sec)
----------------



Funny how it reports having found 30 dividends for an invalid stock symbol (ALT.DE...) Perhaps the feed helpfully returns the dividends for symbol ALT instead. Well, I call it a case of "garbage in garbage out" rather than a bug:
QUOTE:
Updating Provider Nasdaq Fundamental Data ...
.MB_ADV.A fundamental items updated: 0 dividends
^GSPC fundamental items updated: 0 dividends
^VAY fundamental items updated: 0 dividends
^VIX fundamental items updated: 0 dividends
1COV.DE fundamental items updated: 0 dividends
1PG.F fundamental items updated: 0 dividends
AAPL fundamental items updated: 58 dividends 25 short interest
ADS.DE fundamental items updated: 0 dividends
AFX.DE fundamental items updated: 0 dividends
AIR.DE fundamental items updated: 0 dividends
ALT.DE.20100827.A fundamental items updated: 30 dividends
...


As you can see it works for me. Apparently this hang was caused a network connectivity issue on your end. Things happen. It's a free feed that comes without any warranty.

P.S. Make sure you've updated the extension to its latest version v2018.12.
profile picture

Eugene

#3
Robert suggested that the fundamental provider should not make requests for unsupported symbols like indices, Forex, options, international stocks etc. Look for it in upcoming v2019.09 of the Nasdaq provider suite. If you have the unsupported symbols across your DataSets then you're going to notice a speedup.
profile picture

Eugene

#4
@maximgl

To get the provider version 2019.09 with changes I referred to in post #4, please update the extension.
profile picture

maximgl

#5
@Eugene, after the latest providers update, I am having issues with NasdaqDividentsProvider, ReutersNewsProvider and YChartsSecurityFundamentalProvider - they all return empty datasets for all of the symbols I have (and I have few thousands of US stock symbols). To clarify, it does not mean that produced files are 0 bytes in size, it is that all the files they download are exactly the same size for each symbol (2227 bytes for YCharts, 56 bytes for Nasdaq, 29 bytes for Reuters).

Rolling back YCharts DLL to the previous version seemed to have fixed the issue, previous version of NasdaqDivident generates empty datasets (was hanging before) and I haven't rolled back ReutersNews since it is bundled with other providers.

Could anyone replicate this issue or is it just happening for me?

Thanks,
Maxim
profile picture

Eugene

#6
1. Reuters website seems to be broken. I cannot get anything out of it for a plain vanilla stock symbol like AAPL in the browser. Also there's URL change for the page the provider requests to get stock news. No wonder it doesn't work, getting this cryptic message for every symbol:

QUOTE:
404
RIC not found


2. Nasdaq website is either on maintenance or doesn't return the data for whatever reason. Firstly, the URL has also changed and this may be negatively affecting the provider which gets this response on each request upon redirect:

QUOTE:
Apple Inc. Common Stock (AAPL) Dividend History
...
Data is currently not available


Neither I can get the historical data with the Nasdaq static provider. Let's keep an eye on it for a few days but it may turn out to be broken beyond repair. An in-house tool had been scraping the website periodically until Nasdaq's website overhaul got rolled out lately, taking down the feature altogether. :/

3. YCharts works for me as before.
profile picture

maximgl

#7
I've tried a new version of YCharts library today and it seems to be working. Strange, I wonder if they were doing maintenance over the weekend. Let's cross out YCharts from the list of issues and I'll check Reuters/Nasdaq in a couple of days to see if it returns back to the working state.

Thanks,
Maxim
profile picture

Eugene

#8
Nasdaq isn't back to life yet so no data on website.

Reuters website is working but they made a few breaking changes. Sorry but this means that the provider is to be decommissioned. Check out my link for a replacement provider.
profile picture

Eugene

#9
Nasdaq website is up and running. Under the hood there's a breaking change to the way everything is delivered: historical data, short interest and dividends. Affected are all U.S. data providers: static, streaming and fundamental. :(

None of the Nordic providers are broken yet.
profile picture

Eugene

#10
Good news: found a solution to restore all the broken providers. Quite an effort. Look for it in v2019.11 (ETA end of the month).
profile picture

Eugene

#11
@maximgl

To fix the U.S. data providers (static, fundamental, streaming) completely broken by Nasdaq website overhaul, please update the Nasdaq provider extension to version 2019.11.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).