- ago
One of my strategies uses the Fidelity Select Sector funds dataset, which is one of the World Indices datasets linked to Yahoo. IQFeed is my default streaming provider.

When I run my strategy the first time, it preloads Fidelity fund data without problems. But when it begins the backtest, suddenly I get all these timeouts from IQFeed when it runs this Yahoo data. And "no", I am not subscribed to mutual fund data from IQFeed. Is that even relevant here? And Fidelity Select Sector mutual funds only update once a day; they don't stream.


What's weird is that this only happens on the first run of the strategy. On subsequent runs, no IQFeed timeouts are being generated.

So please explain what's going on here? Why is IQFeed even involved in the first place with these mutual funds when I don't have a subscription for IQFeed mutual fund data in the first place? (Remember, these are funds linked to Yahoo, not IQFeed.) And why aren't these timeouts a problem on subsequent runs of the strategy?
0
525
Solved
19 Replies

Closed

Bookmark

Sort
- ago
#1
Does IQFeed have priority over Yahoo in your Historical Providers list in the Data Manager (Streaming providers do not have anything to do with this)? If it goes before Yahoo there, it gets hit first. As the symbol cannot be found, WL7 queries the next checked provider on the list. Hence the timeout. Now that the data is cached, it's not an issue for subsequent strategy runs.
0
Cone8
 ( 28.25% )
- ago
#2
@Eugene
But the point was: the Fidelity Select Sector funds dataset, which is one of the World Indices datasets linked to Yahoo. A linked DataSet shouldn't try to go down the rabbit hole if the data are up-to-date.
2
Glitch8
 ( 12.08% )
- ago
#3
Unless Yahoo data was down temporarily, forcing it to try to get the data from other sources. It's not happening on my end when I backtest with this DataSet. Are you accessing those symbols via GetHistory calls in your Strategy? If you want to share the Strategy this is ocurring with I can try to reproduce.
1
- ago
#4
QUOTE:
Does IQFeed have priority over Yahoo in your Historical Providers list ...?
Yes it does.

QUOTE:
... is one of the World Indices datasets linked to Yahoo. A linked DataSet shouldn't try to go down the rabbit hole if the data are up-to-date.
And when Data Manager updates the Fidelity Select Sector funds dataset, a query is not made to IQFeed. Data Manager only queries Yahoo as designed during an update. It's only when it starts its backtesting (not the update step) that the IQFeed timeouts occur.

QUOTE:
Are you accessing those symbols via GetHistory calls in your Strategy?
Yes I am. So is that why the timeouts only occur during the backtest and not during the static update-dataset step? So GetHistory calls are bad?

But wait a minute. GetHistory is only querying funds that Data Manager has already cached in the static update step.... Right? Are you saying GetHistory does not use statically cached data collected by Data Manager? If so, why not?

CODE:
      public override void Initialize(BarHistory syncBars) {          if (firstPass)          {             int nullBar = Parameters[0].AsInt; //paramNullBar             int plot2highlight = Parameters[3].AsInt-1; //paramPlot2Highlight             //plot the best 9 stocks, but be sure to synchronize them with the current BarHistory before plotting them             for (int symIndx=fundSymbols.Length-1, colorIndx=0; colorIndx < plotColor.Length; symIndx--, colorIndx++)             {                BarHistory mutFundBars = GetHistory(syncBars, fundSymbols[symIndx]);                TimeSeries decorrelated = RSDif.Decorrelate(mutFundBars);                decorrelated = decorrelated - decorrelated[decorrelated.Count-nullBar];                decorrelated.Description = String.Format("[{0:0#}] {1,5:0.00}/bar {2,5:0.00}/bar", colorIndx+1,                   LRSlope.Value(decorrelated.Count-1, decorrelated, 15),                   LRSlope.Value(decorrelated.Count-1, decorrelated, 30)) +                   " decorrelated(" + mutFundBars.Symbol + ") " + mutFundBars.SecurityName;                bool highlight = colorIndx == plot2highlight; //highlight this plot?                PlotTimeSeriesLine(decorrelated, mutFundBars.Symbol, "comparison", plotColor[colorIndx], highlight? 2:1, LineStyles.Solid, true);                SetPaneDrawingOptions("comparison", 450, -2);                DrawHeaderText(decorrelated.Description, plotColor[colorIndx], 10, "comparison");                int lowestRecentBar = decorrelated.GetLowestBar(decorrelated.Count-1, 22); //lowest bar for past month                int gainBars = decorrelated.Count - lowestRecentBar; //#of bars to determine recent price gains                if (highlight)                {                   TimeSeries regressionPlot = Regress.ReciprocalFit(decorrelated,0,gainBars);                   regressionPlot.Description = null;                   PlotTimeSeriesLine(regressionPlot, mutFundBars.Symbol, "comparison", Color.Red, 4, LineStyles.Dotted, true);                }                int dipBar = decorrelated.Count - gainBars; //bar# @dip pt                DrawDot(dipBar, decorrelated[dipBar], plotColor[colorIndx], 4, "comparison");                PlotTimeSeriesLine(new WilderMA(new Momentum(decorrelated, 1), nullBar),                   "", "velocity", plotColor[colorIndx], highlight? 3:1, LineStyles.Solid);                SetPaneDrawingOptions("velocity", 300, -1);                DrawHeaderText("[" + (colorIndx+1) + "] WilderMA(Velocity(" + mutFundBars.Symbol + "))", plotColor[colorIndx], 10, "velocity");             } ...          }       }
BTW, this strategy works good. That should count for something.

Oops, the [Image] button won't let me add a graphic upon editing a Reply. Don't you just hate that?
0
- ago
#5
As you said you're using GetHistory to get the sector fund symbol's data. But the fact that a sector fund resides in the Yahoo linked DataSet does not imply that there's something wrong with GetHistory. Unless you run this very strategy on the "Fidelity select sector funds" DataSet, GetHistory has to make a data lookup under the hood. Only if this is the case, IQFeed - your first active data provider - should not be queried and the data should be collected straight from Yahoo.

If any DataSet other than "Fidelity select..." is used to backtest the strategy, IQFeed will be queried, then comes Yahoo or whatever follows IQFeed.

Makes sense?
1
Glitch8
 ( 12.08% )
- ago
#6
QUOTE:
But wait a minute. GetHistory is only querying funds that Data Manager has already cached in the static update step.... Right? Are you saying GetHistory does not use statically cached data collected by Data Manager? If so, why not?


GetHistory does not use staically cached data collected by the Data Manager. Why not? It's primary intent was to allow a Strategy to make decisions off data that does not reside in the DataSet being tested. For example, buy a stock if the corresponding index is oversold.

If you want to get at the statically cached data, use the BacktestData property, which returns a List containing the BarHistory instances being backtested.
1
- ago
#7
QUOTE:
GetHistory does not use statically cached data collected by the Data Manager.
Okay, I didn't know that. I wished that was documented better. So I need to avoid GetHistory.

QUOTE:
If you want to get at the statically cached data, use the BacktestData property,
So I need a Linq query into BacktestData to pull specific BarHistories followed by a BarHistorySynchronizer.Synchronize(...) call so each fund can be placed on the same graph.

Part of the problem is I look at the "how" of an example, but the discussion doesn't include the "why" it's done that way. Now I'm just understanding the "why" part. I need a deeper understanding of the underlying architecture.
0
Glitch8
 ( 12.08% )
- ago
#8
The “why” is no mystery. I simply hadn’t considered doing that. Where were you when i was building out that method in WL7?? 🤷🏼‍♂️
0
- ago
#9
QUOTE:
Where were you when i was building out that method in WL7?
Well, that's a totally different (and unrelated) topic. https://www.wealth-lab.com/Discussion/Building-the-final-multi-stock-plot-for-display-6596

But the problem strategy is the same as discussed in this old, original topic. I just never fully understood the "why" in your original example solution. Now I "think" I do. This also explains why you created the "bestStocks" List and sorted that independent List object. Should we be having this explanation in the old, original topic (which went off on another tangent)?
0
Glitch8
 ( 12.08% )
- ago
#10
I'm good with leaving things off here.
1
Cone8
 ( 28.25% )
- ago
#11
Doesn't GetHistory and GetHistoryUnsynched using the dataSetName overload allow you to get at cached data?
0
Glitch8
 ( 12.08% )
- ago
#12
No, using BacktestData let’s you.
1
- ago
#13
The problem is the documentation for GetHistory and GetHistoryUnsynched is incomplete. It should state these functions do not use Data Manager cached data, and then provide a hyperlink to BacktestData saying that it does.

The quick and dirty (but less than ideal) fix below removes GetHistory altogether.
CODE:
   BarHistory mutFundBars = BacktestData.Find(x => x.Symbol.Equals(fundSymbols[symIndx]));    mutFundBars = BarHistorySynchronizer.Synchronize(mutFundBars,syncBars);    //BarHistory mutFundBars = GetHistory(syncBars, fundSymbols[symIndx]);
The bad news is that IQFeed timeouts still occur, but I "think" that's because Yahoo hasn't updated the Fidelity Select Sector fund data yet (post market close), so removing GetHistory alone doesn't completely cure the IQFeed timeout problem. IQFeed is still queried for the last, missing, EoD bar from Yahoo.

The goal of this visualizer (It doesn't trade yet.) is to rank the different Fidelity Select Sector funds and plot their decorrelated lines (decorrelated with the S&P500). The two groups of plots are identical, but the top group is simply the sum of the relative changes (i.e. the decorrelated line); whereas, the bottom group is summed by an EMA instead. The "null point" where the plots intersect is artificially created (via a vertical offset) at bar 22 from the end so the plots have a common starting point.

1
- ago
#14
QUOTE:
because Yahoo hasn't updated the Fidelity Select Sector fund data yet (post market close)

Yahoo's EOD data is best updated late i.e. tomorrow a.m.
1
- ago
#15
That's the fix. After replacing all the GetHistory calls with BacktestData calls, one needs to run the strategy sometime after Yahoo updates its previous market day data in the following morning or during the following trading day before the market closes, which would release a "new" EOD bar not yet published by Yahoo.

Even if one has a mutual fund data subscription on IQFeed, the problem would still persist because Fidelity doesn't publish EOD data (on IQFeed) for the current market day until several hours after the market closes.

Bottom line, it's simplest to run a mutual fund strategy either when the market is open or over the weekends.
0
Best Answer
Cone8
 ( 28.25% )
- ago
#16
QUOTE:
documentation for GetHistory and GetHistoryUnsynched is incomplete.
Already ahead of you. I updated the QuickRef a few days ago with Glitch's comments. The QuickRef doesn't have hyperlink capability, but when the update makes it to the site's docs it will automatically be added.

Actually, I just found an extra entry for GetHistory that shouldn't be appearing.
1
- ago
#17
QUOTE:
The QuickRef doesn't have hyperlink capability,
Well that needs to be fixed. The idea is the make the QuickRef instructive by allowing users to "explore" all the hyperlinking so they realize alternative options; i.e. GetHistory vs BacktestData.
0
Glitch8
 ( 12.08% )
- ago
#18
Hey superticker, we're already working 12+ hours a day. Would appreciate a little bit less demanding of an attitude, but I agree it would be great to eventually get this done!
2
- ago
#19
With this hyperlinking request, I see this discussion going far beyond the resolved IQFeed timeout issue. Let's mark this as closed and have a new topic for hyperlinking if it's really necessary.
0

Closed

Bookmark

Sort