IB-performance-accessing-historical-data-in-backtest-live-strategies

QUOTE:
experience with IB performance has not been good, ...

Based on earlier descriptions, I don't think your data problem has anything to do with the TWS API, but instead has to do with the limitations of your "data feed quota" with IB instead. They are clearly throttling your data feed.

I would call IB and tell them you want to pay for more data quota. I think increasing that quota will solve much of your problem. It is certainly worth a try. But remember, IB is not a historical (static) data provider.

QUOTE:
you want to pay for more data quota.

It's not an option for historical data.
You can pay for more simultaneous quote streams in packs of 100, but that's another topic.

QUOTE:
You can pay for more simultaneous quote streams in packs of 100, but not historical data ...

Perhaps the blog article about data providers and brokers needs to explain--in more detail--why both a historical data provider (for backtesting) and a streaming data provider (for the placing of intraday real-time trades) is required for smooth Wealth-Lab operations.

There was a time (in my early WL years) when I mistakenly "thought" a single streaming data provider could do it all. And perhaps I'm not the only one with this misconception.

@superticker FWIW I had a revelation this week about intraday trading strategies. I thought my problem with performance and reliability was either with my coding (usually it has been) or the inability for the broker to place trades fast enough. Then, by looking carefully at the debug messages I was logging while processing, I realized that the strategy was spending most of its time waiting for fresh backtest data. This was despite the fact that @Cone has shared some clever caching patterns to reduce the load on backtest data.

What never dawned on me before is that most strategies (even live intraday ones) depend on some backtest data to establish one or more "states" (e.g., with indicators) that determine signals. From what I observed, WL makes a request for data updates on every execution bar (guess it has to). That means, with 1 min bars, every minute WL is checking to see that data is current and waits for a response or update before executing on the current bar. If this check / response / update cycle takes more than one minute, the strategy misses the bar and gets out of sync.

My revelation was that the speed of updating historical data can be even more critical to successfully processing intraday strategies than the speed of the broker placing an order, particularly as the data scale gets small. It may be more accurate to say it is likely to be the weak link. I believe some comments from @Cone on this topic above supports this supposition.

I am a novice with intraday live trading with WL, and I'm sure there is a lot I don't understand about the inner workings of WL in this hybrid backtest / live trading modality. These are just my working theories based on my recent observations and my struggle with this issue. As always, i welcome feedback and correction so I can get smarter about these topics.

QUOTE:
What never dawned on me before is that most strategies (even live intraday ones) depend on some backtest data to establish one or more "states" (e.g., with indicators) that determine signals. From what I observed, WL makes a request for data updates on every execution bar

I "think" the WL data model is even more complicated than what you realize. There are [1] static (historical), [2] on-demand, and [3] streaming data fetches. The fast data is static, but you only (probably) have one copy on disk of that handled by Data Manager. My only backtesting is done with Daily data, so I create my WL linked datasets with only Daily data; and that gets cached on disk.

On-demand data comes in if I want to run an intraday backtest, which I rarely do. Since I haven't created any linked datasets for intraday data, WL uses on-demand data from IQFeed to run that intraday backtest, not fast, disk cached, static data. But I'm only backtesting a few stocks intraday, so using on-demand data for those few cases isn't that slow.

Now if you're doing a great deal of intraday backtesting (Are you?), then it "might" make sense to create linked, static, intraday datasets especially for this purpose. But that's going to use a lot of disk space. Are you sure you want to do that for every stock? Perhaps you only need fast, static, intraday data for a few external symbols like $SPX (S&P500), so create "special" linked, intraday, static datasets to cache on disk just for those external symbols, and nothing more.

Streaming data is for live, real-time trading; and updating the Quotes window and some intraday Strategy Monitor activity. But you shouldn't be substituting streaming data (from your broker) for static or on-demand data from your primary data provider (e.g. IQFeed); otherwise, you'll have problems.

QUOTE:
WL makes a request for data updates on every execution bar

Only if you program it to do that!

Your program controls what happens. If you put a GetHistory() in Execute() without a condition, then it's going to request GetHistory() on every bar.
Do not do that.

Most strategies know the symbols they'll be trading before runtime. Consequently, if you're trading an external symbol, you request it once in Initialize() and save a reference to that data in a local variable that you use later in Execute().

Strategies that trade options based on the underlier's price (or some other condition) select contracts ad-hoc during the Execute(). Now, you could request all the option contracts in the chain for a given expiration during Initialize, but that might be 100 or more contracts.

Instead, you use the pattern that I gave everyone in the Options blog article. Request a contract ONCE, cache it, and don't repeat the same request for data on subsequent bars.

The problem is that you might be trading 5 contracts on the same day. That's 5 requests. If you don't have the history downloaded already (because you didn't update those 100 contracts from the chain before you started), you're going to be waiting a long time for data to load - especially if you're using IB.

What's the solution for live trading?
For backtesting, you have no choice but to wait for data, and, when you're backtesting there's time to wait.

But if you don't need the contract's history to trade it (usually you don't), don't request it. Use a "dummy" BarHistory to fire off a trade with no delay. If you need a option's price (for sizing the trade, for example), then I'd recommend a lightweight request using WLHost.Instance.GetHistory for an abriged History that you can use to seed your "dummy" BarHistory.

In fact, using WLHost.Instance.GetHistory (instead of GetHistory or GetPairHistory) that limits the history request to just the current session may be the only change you need to speed things up immensely. Usually you don't need to modify a trading strategy for live trading, but the "options trading scenario" is one such case.

QUOTE:
Instead, you use the pattern that I gave everyone in the Options blog article. Request a contract ONCE, cache it, and don't repeat the same request for data on subsequent bars.

I am using that pattern. GetHistory() is only invoked when the symbol is not found in the dictionary. But it doesn't appear that this is where the problem is.

The strategy is calculating moving averages (EMA) on two symbols, QQQ and SPY using 1 min data. In Initialize() the in EMA indicators are recalculated each minute using the latest bar of data. In Execute(), the indicators are tested for a buy or sell signal. Really straightforward. A buy signal would result in requesting an option symbol from a broker using [Broker]Historical. GetOption().

Using WL.Host.LogItem I set up detailed log outputs from the Initialize() and Execute() methods. From what I can tell, from watching the log, WL seems to get "stuck" in the Initialize() method. Using IB, when the strategy starts in the SM at 6:31 AM (market time) it takes about 6 minutes to output to the log from the Execute() method. By this time I have missed 5 bars of data and potentially signals.

I suspect the time is not being used to recalculate the indicator. I have no such delay on backtesting in the SW. I assume it is waiting for a data request update, as confirmed in the SM log.

The issue doesn't appear to be with option symbol history. That is not even invoked unless there is a buy signal. The delay appears to be with getting the current 1 min bars on QQQ and SPY required to complete the Initialize() method and move to Execute().

This doesn't make any sense to me. I make sure the strategy is activated and it has all the QQQ and SPY 1 min bar history (confirmed by the SM log) prior to 6:31 AM. What can be happening in the first six minutes? After this, it will process on a 1 min basis, until it randomly does not return SPY / QQQ bar for 90 seconds. And then we're out of sync again.

BTW, if it is a clue, when the SM log reports it failed to get historical data (e.g., after 1.5 min), the values of my indicators are NaN. This additionally leads me to believe the issue is getting timely updates on QQQ and SPY. The indicators are not reliant on signals, options selection, option history, etc. I suspect the NaN is coming from the last bar being null (didn't get the data).

What do you think?

ADDENDUM

During failures, such as the first 6 min, WL freezes. Can't open windows, use menus, or do anything else with WL.

It sounds to me like the connection isn't stable. I insist, each day before you start trading you must restart TWS, then WealthLab, and then connect. If TWS (or IB Gateway) restarts after connecting with WealthLab, the connection with WealthLab could be compromised.

QUOTE:
I insist, each day before you start trading you must restart TWS

Is this only true for TWS, or is it true in general? If I'm using Tradier, do I have to restart WL before market open?

I recommend restarting each day sometime before the open for any broker connection. I only assumed IB because we're usually talking about IB data delays.

If you can't figure out the source of the 6 minute delay and want to share a complete running code sample that exhibits the problem, I'll be happy to look at it. It would be best to export and send me the strategy xml file to preserve all the settings.

Something else to check in case of data slowness - Event Providers.
Do you have any selected? Which ones and are they required for live trading?

QUOTE:
Do you have any selected? Which ones and are they required for live trading?

No and no

Today I started testing live with Tradier (streaming data, intraday historical data, option selection, and bar history - no IB) with the WL Dummy Broker for a 1 min strategy. It worked fast and nearly flawlessly except for two issues.
1. I did not restart WL this morning and it did not auto connect to a streaming chart or complete a history fetch for QQQ on the first minute bar. I restarted WL and it began to work fine. (That was the source of my question about daily restarts in Post #9.)
2. Several hours into the session, it stopped streaming updates to the chart, and 1 min historical data fetch (for QQQ) was failing. I again restarted WL and it started working again.

I'm curious what causes this connection fragility. Is it the nature of the network hooks in software that IB and Tradier have in common (such as Web sockets - not sure whether that or HTTPS: streaming is used by WL for Tradier). More importantly, I'm wondering if there's any way to "harden" the connection, and if not, a way to detect and recover from data disconnects. The goal is to have an auto trade system that I don't have to watch all day, since that defeats the purpose.

You also reported numerous disconnects with Medved. You might want to check your internet. A few months ago I was experiencing disconnects with my fiber optic line and it turned out there was some faulty wiring between the box and my router. After they fixed it the connection has been rock solid.

Good point. That type of problem probably wouldn't show up except with something like streaming data. Even streaming a video won't show it because the stream is cashed. No way to cache future live market data (yet). Wonder how I can test that?

What about detection and recovery? Anything that can be done there?

I found this,

https://www.highspeedinternet.com/resources/why-does-my-internet-keep-disconnecting#:~:text=You%20can%20check%20for%20outages,of%20their%20customers'%20internet%20connections.

QUOTE:
You also reported numerous disconnects with Medved.

The MT disconnects / reconnects always happen when Tradier is processing a 1-min bar and writing to the Log Viewer