In the past few days, IQFeed data updata seems slow than it used to compare to couple weeks ago.
I tried to run "Update Data" and also tried to update each dataset separately and the update speed seems not to change. It used to update and complete (4K daily and 500 (5 minutes) ) take about 30 - 45 minutes. Now it takes 1.5 - 2 hours to complete.

Recently I added couple more data sets with symbol count > 4K. However the data sets have many overlap symbols. Any suggestions. Thanks

The only obvious thing that you can control that could make a big difference is the number of symbols downloaded in parallel. It's the last option in the Data Manager > IQFeed settings. Set it to 10.

Also, the more data you add, the bigger the files get. It can take a "not insignificant" amount of time to read and write those files as you update them, and over time will just take longer and longer.

Finally, if you don't need pre/post market data, using the "Regular Session only" option will help to reduce the amount of data returned. If you're not already using this option, then you should refresh your data completely after selecting it. This will reduce file size and you can stop using the Market Manager for filtering, both of which will make loading data for backtests much faster.

Thanks Cone. The number of symbols download in parallel are set to 10 and this has not change at all. I have the "Regular Session only" option ON from the beginning and this has not changed either. I also took your advice earlier from another post and disable IQFeed within Market Manager.

Actually I just noticed that WLD overall and not just data update has become slow to response. I don't think it is the hardware since there still plenty of memory and processor resources are available.

This is strange. I will remove the number of datasets to see if that change anything. Thanks.

Cone, condensing data sets with same symbols into a single data set does not improve data update speed. last night it took about 4.5 hrs to complete updating all data. If more symbols mean more files to write then it is reasonable to expect the total update time to take longer. However, here the updating speed (looking at it as it updating symbol by symbol) is slow, about 1/2 or less of the original speed. Thanks.

DataSets wouldn't matter since the symbols are grouped by scale and updated only once.

Considering we haven't changed anything since last October (the latest build only disabled streaming bars by changing 1 boolean property), the only reasons I can imagine why you're seeing such a difference are:

1. you added many more symbols to download. Just 5-minute bars or tick intervals too?
2. several of the symbols that you're downloading are no longer available.. this would result in retries, but generally even these should be quick to fail.

What Data Manager > IQFeed options do you have selected?
If you don't mind, please attach your LastUpdateLog.txt here. There may be important clues there.

The new symbols added are for daily bars only. I will scan and remove those inactive symbols. I will run a complete data update later this evening and will attach the log file then. There are about 6800 daily and 1800 5-minutes symbols. Those 5-minutes set are included within that of the daily set. Thanks.

Let me add that all my Daily ticker symbols (3000 over 30 datasets) use to take 15 minutes for static updates. After the first week of January, 2021, they now take twice as long to update. This morning's log says 32 minutes. And that was just after I got an Internet speed upgrade.

My point is that this maybe an issue at DTN's end, and not a WL client issue at all. You may want to give DTN a call. I think they are throttling updates for those with highly fast Internet speeds (which makes some sense).

Honestly, I don't care whether or not static updates for 3000 symbols take 15 or 32 minutes. Prior to market open, I just manually update the smaller datasets first on Strategy Monitor, then start trading on them while Data Manager updates everything else in the background. As a result, the time to do static updates doesn't affect my pre-market trading workflow.

---
If there were an option to limit the price data file size (8 years would be good), I would use it. But I don't think the bottleneck is at the WL client side here.

Thanks superticker. That helps. Something to try might be to reduce the number of symbols requested in parallel to 5 or so. If IQFeed is throttling, counter-intuitively, maybe it will actually speed things up a bit. It's worth a try.

Thanks Superticker and Cone. I just did a quick experiment on IQFeed data update speed based on Cone's suggestion of change the number of symbols requested in parallel between 5 and 10 (restart WL after change of setting) and at both setting of 5 and 10, most of the symbols are update at a speed of right around 1 symbol/sec. There are so often where it take about 2 to 2.5 sec to update a symbol on both settings and there are less so often when both settings have a quick burst of a symbols update at the rate around 1/2 sec. So both settings seems to have a nominal update speed of around 1 symbol/sec. These quick observations might not be a good comparisons, in the next few days I will update on both settings on different days and look at the total times it takes to update everything. In the mean time, I will contact DTN about this.

For the sake of trouble shooting, I just deleted all the big data sets and only leave a few smaller data sets (restart WL afterward) and run Data Update and the update speed seems does not change. I don't know how WL run the update and whether it requests all the IQFeed symbols on files or only just those active on WL.

QUOTE:
If IQFeed is throttling,...

QUOTE:
I will contact DTN about this.

Honestly guys, with today's ultrahigh Internet speeds, I don't think DTN has any choice but to throttle these Internet connections. And if I were working for DTN, I would be doing the same thing; otherwise, a few customers could hog all the server side bandwidth. I see throttling as a necessary evil here--throttling is required.

What would be worth trying is to throttle your Internet speed between your workstation and your router. If you hit a sweet spot, that may speed you up.

Thanks Superticker for your suggestions.

Even after I removed those inactive symbols, running Update Data will update everything, which will take more time for the update. Would it is possible to run Update Data for selective data sets? I can always select a data set and update that only but using Update Data allow it to run on a schedule.

QUOTE:
Even after I removed those inactive symbols, running Update Data will update everything

Choose "Update only symbols that are contained in DataSets".

Thanks Eugene. I missed that.

Setting Symbols download in parallel to 5 and 10 in IQFeed get the total time to complete data update to about 3hrs 40 minutes and 3hrs 20 minutes, respectively. So there is not much difference in how long it takes to update data between the two settings.

I called DTN today. In the course of our conversation, I asked them about throttling. They did confirm there was a protocol change early in January that now throttles requests to 30-per-second. But that still seems generously fast to me.

I told them that on static WL IQFeed updates, I'm getting about one stock per second in the provider log. The first several run about twice as fast, but after about 10 minutes, it slows down to one stock per second for static updates. The DTN support consultant had no explanation other than to say their change in early January may have adversely affected the response time of the IQFeed provider and to mention this to WL support.

My questions are:
1) After 10-minutes, does your 2021.1.7.0 IQFeed static provider only update one stock per second? And maybe two stocks per second before that?
2) Is the IQFeed provider for WL7 any faster than this?
3) For static updates, wouldn't it take only one or two requests to update a single stock?

1, Of course not. It hits 10 symbols on different threads simultaneously but waits for all data to return before starting a new round.
2. No.
3. It's always just one request per instrument.

QUOTE:
1, Of course not. It hits 10 symbols on different threads simultaneously but waits for all data to return before starting a new round.

So you're getting something faster that one stock update per second in the IQFeed log? So why would my logged updates be so slow? I'm assuming it's only requesting the most recent (last three trading days) Daily bars, so very little data is being exchanged per request.

As I mentioned in Post# 7, it takes over 30 minutes to update 3000 Daily-bar stocks. (Yes, the pre/post market data is included in that. The "Regular Session Only" box is unchecked.)

QUOTE:
So you're getting something faster that one stock update per second in the IQFeed log?

Yes, for short daily bar requests I'm getting at least 10 and up to 20 symbols per second returned. It slows down to maybe 1 or 2 per second when updating hundreds of intraday bars.

QUOTE:
So why would my logged updates be so slow?

I don't know.

QUOTE:
It slows down to maybe 1 or 2 per second when updating hundreds of intraday bars.

I'm thinking it slows down to 1 (maybe 2) a second for all updates whether they are daily bars or intraday bars. The throttling is a function of the number of "recent" requests made, and not the bar scale used. Since my static updates are only for daily bars, after enough requests, I'm seeing the throttling with them. (You might try it once with only daily bars.)

I'm guessing the DTN consultant's 30 requests/second is before serious throttling kicks in, which the consultant doesn't know about. And as I mentioned in Post# 11, I don't think DTN has any choice but to throttle to keep their servers functional.

The only way you could speed this up on the WL client side would be to "minimize" the number of requests on the client side (over time).

You could certainly minimize requests per second by reducing the number of threads. At nominally 100 msec round trips and ignoring processing delays, even 5 threads would exceed 30 requests per second. Try just 1 or 2 threads - that should certainly keep the number of requests per second below 30.

My download speed for daily bars is about 1 symbol per second, consistent what we all seen here. I am tried with 5 and 10 symbols download in parallel (post #15) and the speed seems did not changed. I would take on Cone's suggestion on #21 to try with 1 and 2 symbols download in parallel. BTW, DTN never return my email about throttling but from SuperTicker on #16 above, they confirmed it. Thanks.

Let me show you what it looks like for me. Maybe it's because I'm hitting different servers in Europe?

This animated gif loops after a few seconds...

I'm now getting 3000 static stock updates over 10 minutes, which is as fast as I've ever gotten. Perhaps DTN fixed something. I'm running somewhere between 5 to 10 IQFeed threads when doing this. I would employ whatever number of threads produce the most continuous download.

Strangely, just this morning I ran an update and was getting the 'ole "1-request-per-second treatment".

What I noticed in your Post# 23 gif is that all the threads (10 threads?) report each second, then halt. That results in a jerky update each second. That also suggests DTN is throttling you by port# (such that each thread has its own port#). That approach makes sense to me and is expected.

But in my case, I think the throttling was by IP address ("effectively" user ID), which makes less sense to me. Older network hardware won't typically snoop the network packet to get the IP address for throttling, and if it did, it would tax the network processor in the switching hardware. Bottom line, it's less taxing on the network switches to throttle by port# than IP address.

I'm guessing they're experimenting with the throttling approach to see what their hardware can afford to do. I think they would like to throttle by IP address (using a "modern" switch), but have limited resources to do so; therefore, they fall back to throttling by port# (thread) instead (older switch). So your throttling method will vary depending on if you get a "modern switch" or an "older switch" when you log into their cluster.

You can look at the address ranges of the IP addresses of their servers relative to throttling speeds to confirm this. Happy hacking.

QUOTE:
all the threads (10 threads?) report each second, then halt.

Like I mentioned in post #17, It hits 10 symbols on different threads simultaneously but waits for all data to return before starting a new round. So there's always a processing and round-trip delay between sets of n requests.