Finam provider data download takes too long
Author: GolfPlayer
Creation Date: 2/11/2019 11:03 PM
profile picture

GolfPlayer

#1
Hi!
Please help!

I use Finam data provider. I use Data Manager then Update Dataset to download market data. It takes about five minutes to download 60-minutes history per one future contract. It is about 1000 bars. Why does it take too long?

After downloading via Data Manager, it takes practically the same time to open chart of already downloaded security. It is looks like WLD downloads all data again and I have to wait several minutes again just to open chart. Why is it so? Does not WLD store all market data after downloading on my hard disk?

If I try to download for example 50 or 60 contracts in takes several hours :(

profile picture

GolfPlayer

#2
A picture
profile picture

Eugene

#3
Hi,

As stated in post #74 dated April 2018 in this thread, "Data updates will now become much slower due to an extra delay introduced to avoid the new anti-bot protection."

At Finam, initial download of intraday data (backfill) per se has never been a piece of cake. You aren't allowed to download entire intraday history in one shot, multiple web requests are prohibited, there's anti-bot protection now etc. To avoid serious download restrictions the provider has to make multiple requests in steps and combine the data. Since April 2018 there's a 1 second delay in between those requests, adding further to the wait.

Add inability to make parallel downloads so each stock must be downloaded one by one and your intraday backfill takes a very long time. As there's nothing we can do to speed up the process, schedule your initial backfill for the night using the Data Manager. On the bright side, daily updates are are fast enough.
profile picture

GolfPlayer

#4
OK.
Thanks.
But why does it it take several minutes to open already downloaded chart?

profile picture

Eugene

#5
Because "On Demand Data Update" is enabled in the Data Manager. This causes a series of extra requests to attempt update the symbol. Disabling it should make the chart open up instantly.
profile picture

GolfPlayer

#6
Thank you! !!!!
I will use your advise this evening.
You are a Great Time Saver :)
profile picture

GolfPlayer

#7
It works!
Eugene, thanks!!
profile picture

Eugene

#8
Sergey,

Great to hear it. Glad to have helped!
profile picture

Eugene

#9
As a followup to explanation from my post #3, I think there's potential for a considerable speed improvement of intraday history backfills. In my testing, the download time was reduced twofold by employing some tweak!

There may be a couple downsides (which essentially are just borderline cases):

1. On aborted data download (e.g. network timeout or server issue) you'll be left with incomplete history which can not be fixed by "Update Data" but only by right click + "Reload Chart History".
2. For an illiquid Futures symbol, a crippled history near its inception may be delivered (but usually should be okay)

Look for v2019.03 (late February) with this improvement.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).