One of my strategies uses the Fidelity Select Sector funds dataset, which is one of the World Indices datasets linked to Yahoo. IQFeed is my default streaming provider.
When I run my strategy the first time, it preloads Fidelity fund data without problems. But when it begins the backtest, suddenly I get all these timeouts from IQFeed when it runs this Yahoo data. And "no", I am not subscribed to mutual fund data from IQFeed. Is that even relevant here? And Fidelity Select Sector mutual funds only update once a day; they don't stream.
What's weird is that this only happens on the first run of the strategy. On subsequent runs, no IQFeed timeouts are being generated.
So please explain what's going on here? Why is IQFeed even involved in the first place with these mutual funds when I don't have a subscription for IQFeed mutual fund data in the first place? (Remember, these are funds linked to Yahoo, not IQFeed.) And why aren't these timeouts a problem on subsequent runs of the strategy?
When I run my strategy the first time, it preloads Fidelity fund data without problems. But when it begins the backtest, suddenly I get all these timeouts from IQFeed when it runs this Yahoo data. And "no", I am not subscribed to mutual fund data from IQFeed. Is that even relevant here? And Fidelity Select Sector mutual funds only update once a day; they don't stream.
What's weird is that this only happens on the first run of the strategy. On subsequent runs, no IQFeed timeouts are being generated.
So please explain what's going on here? Why is IQFeed even involved in the first place with these mutual funds when I don't have a subscription for IQFeed mutual fund data in the first place? (Remember, these are funds linked to Yahoo, not IQFeed.) And why aren't these timeouts a problem on subsequent runs of the strategy?
Rename
Does IQFeed have priority over Yahoo in your Historical Providers list in the Data Manager (Streaming providers do not have anything to do with this)? If it goes before Yahoo there, it gets hit first. As the symbol cannot be found, WL7 queries the next checked provider on the list. Hence the timeout. Now that the data is cached, it's not an issue for subsequent strategy runs.
@Eugene
But the point was: the Fidelity Select Sector funds dataset, which is one of the World Indices datasets linked to Yahoo. A linked DataSet shouldn't try to go down the rabbit hole if the data are up-to-date.
But the point was: the Fidelity Select Sector funds dataset, which is one of the World Indices datasets linked to Yahoo. A linked DataSet shouldn't try to go down the rabbit hole if the data are up-to-date.
Unless Yahoo data was down temporarily, forcing it to try to get the data from other sources. It's not happening on my end when I backtest with this DataSet. Are you accessing those symbols via GetHistory calls in your Strategy? If you want to share the Strategy this is ocurring with I can try to reproduce.
QUOTE:Yes it does.
Does IQFeed have priority over Yahoo in your Historical Providers list ...?
QUOTE:And when Data Manager updates the Fidelity Select Sector funds dataset, a query is not made to IQFeed. Data Manager only queries Yahoo as designed during an update. It's only when it starts its backtesting (not the update step) that the IQFeed timeouts occur.
... is one of the World Indices datasets linked to Yahoo. A linked DataSet shouldn't try to go down the rabbit hole if the data are up-to-date.
QUOTE:Yes I am. So is that why the timeouts only occur during the backtest and not during the static update-dataset step? So GetHistory calls are bad?
Are you accessing those symbols via GetHistory calls in your Strategy?
But wait a minute. GetHistory is only querying funds that Data Manager has already cached in the static update step.... Right? Are you saying GetHistory does not use statically cached data collected by Data Manager? If so, why not?
CODE:BTW, this strategy works good. That should count for something.
public override void Initialize(BarHistory syncBars) { if (firstPass) { int nullBar = Parameters[0].AsInt; //paramNullBar int plot2highlight = Parameters[3].AsInt-1; //paramPlot2Highlight //plot the best 9 stocks, but be sure to synchronize them with the current BarHistory before plotting them for (int symIndx=fundSymbols.Length-1, colorIndx=0; colorIndx < plotColor.Length; symIndx--, colorIndx++) { BarHistory mutFundBars = GetHistory(syncBars, fundSymbols[symIndx]); TimeSeries decorrelated = RSDif.Decorrelate(mutFundBars); decorrelated = decorrelated - decorrelated[decorrelated.Count-nullBar]; decorrelated.Description = String.Format("[{0:0#}] {1,5:0.00}/bar {2,5:0.00}/bar", colorIndx+1, LRSlope.Value(decorrelated.Count-1, decorrelated, 15), LRSlope.Value(decorrelated.Count-1, decorrelated, 30)) + " decorrelated(" + mutFundBars.Symbol + ") " + mutFundBars.SecurityName; bool highlight = colorIndx == plot2highlight; //highlight this plot? PlotTimeSeriesLine(decorrelated, mutFundBars.Symbol, "comparison", plotColor[colorIndx], highlight? 2:1, LineStyles.Solid, true); SetPaneDrawingOptions("comparison", 450, -2); DrawHeaderText(decorrelated.Description, plotColor[colorIndx], 10, "comparison"); int lowestRecentBar = decorrelated.GetLowestBar(decorrelated.Count-1, 22); //lowest bar for past month int gainBars = decorrelated.Count - lowestRecentBar; //#of bars to determine recent price gains if (highlight) { TimeSeries regressionPlot = Regress.ReciprocalFit(decorrelated,0,gainBars); regressionPlot.Description = null; PlotTimeSeriesLine(regressionPlot, mutFundBars.Symbol, "comparison", Color.Red, 4, LineStyles.Dotted, true); } int dipBar = decorrelated.Count - gainBars; //bar# @dip pt DrawDot(dipBar, decorrelated[dipBar], plotColor[colorIndx], 4, "comparison"); PlotTimeSeriesLine(new WilderMA(new Momentum(decorrelated, 1), nullBar), "", "velocity", plotColor[colorIndx], highlight? 3:1, LineStyles.Solid); SetPaneDrawingOptions("velocity", 300, -1); DrawHeaderText("[" + (colorIndx+1) + "] WilderMA(Velocity(" + mutFundBars.Symbol + "))", plotColor[colorIndx], 10, "velocity"); } ... } }
Oops, the [Image] button won't let me add a graphic upon editing a Reply. Don't you just hate that?
As you said you're using GetHistory to get the sector fund symbol's data. But the fact that a sector fund resides in the Yahoo linked DataSet does not imply that there's something wrong with GetHistory. Unless you run this very strategy on the "Fidelity select sector funds" DataSet, GetHistory has to make a data lookup under the hood. Only if this is the case, IQFeed - your first active data provider - should not be queried and the data should be collected straight from Yahoo.
If any DataSet other than "Fidelity select..." is used to backtest the strategy, IQFeed will be queried, then comes Yahoo or whatever follows IQFeed.
Makes sense?
If any DataSet other than "Fidelity select..." is used to backtest the strategy, IQFeed will be queried, then comes Yahoo or whatever follows IQFeed.
Makes sense?
QUOTE:
But wait a minute. GetHistory is only querying funds that Data Manager has already cached in the static update step.... Right? Are you saying GetHistory does not use statically cached data collected by Data Manager? If so, why not?
GetHistory does not use staically cached data collected by the Data Manager. Why not? It's primary intent was to allow a Strategy to make decisions off data that does not reside in the DataSet being tested. For example, buy a stock if the corresponding index is oversold.
If you want to get at the statically cached data, use the BacktestData property, which returns a List containing the BarHistory instances being backtested.
QUOTE:Okay, I didn't know that. I wished that was documented better. So I need to avoid GetHistory.
GetHistory does not use statically cached data collected by the Data Manager.
QUOTE:So I need a Linq query into BacktestData to pull specific BarHistories followed by a BarHistorySynchronizer.Synchronize(...) call so each fund can be placed on the same graph.
If you want to get at the statically cached data, use the BacktestData property,
Part of the problem is I look at the "how" of an example, but the discussion doesn't include the "why" it's done that way. Now I'm just understanding the "why" part. I need a deeper understanding of the underlying architecture.
The “why” is no mystery. I simply hadn’t considered doing that. Where were you when i was building out that method in WL7?? 🤷🏼♂️
QUOTE:Well, that's a totally different (and unrelated) topic. https://www.wealth-lab.com/Discussion/Building-the-final-multi-stock-plot-for-display-6596
Where were you when i was building out that method in WL7?
But the problem strategy is the same as discussed in this old, original topic. I just never fully understood the "why" in your original example solution. Now I "think" I do. This also explains why you created the "bestStocks" List and sorted that independent List object. Should we be having this explanation in the old, original topic (which went off on another tangent)?
I'm good with leaving things off here.
Doesn't GetHistory and GetHistoryUnsynched using the dataSetName overload allow you to get at cached data?
No, using BacktestData let’s you.
The problem is the documentation for GetHistory and GetHistoryUnsynched is incomplete. It should state these functions do not use Data Manager cached data, and then provide a hyperlink to BacktestData saying that it does.
The quick and dirty (but less than ideal) fix below removes GetHistory altogether.
The goal of this visualizer (It doesn't trade yet.) is to rank the different Fidelity Select Sector funds and plot their decorrelated lines (decorrelated with the S&P500). The two groups of plots are identical, but the top group is simply the sum of the relative changes (i.e. the decorrelated line); whereas, the bottom group is summed by an EMA instead. The "null point" where the plots intersect is artificially created (via a vertical offset) at bar 22 from the end so the plots have a common starting point.
The quick and dirty (but less than ideal) fix below removes GetHistory altogether.
CODE:The bad news is that IQFeed timeouts still occur, but I "think" that's because Yahoo hasn't updated the Fidelity Select Sector fund data yet (post market close), so removing GetHistory alone doesn't completely cure the IQFeed timeout problem. IQFeed is still queried for the last, missing, EoD bar from Yahoo.
BarHistory mutFundBars = BacktestData.Find(x => x.Symbol.Equals(fundSymbols[symIndx])); mutFundBars = BarHistorySynchronizer.Synchronize(mutFundBars,syncBars); //BarHistory mutFundBars = GetHistory(syncBars, fundSymbols[symIndx]);
The goal of this visualizer (It doesn't trade yet.) is to rank the different Fidelity Select Sector funds and plot their decorrelated lines (decorrelated with the S&P500). The two groups of plots are identical, but the top group is simply the sum of the relative changes (i.e. the decorrelated line); whereas, the bottom group is summed by an EMA instead. The "null point" where the plots intersect is artificially created (via a vertical offset) at bar 22 from the end so the plots have a common starting point.
QUOTE:
because Yahoo hasn't updated the Fidelity Select Sector fund data yet (post market close)
Yahoo's EOD data is best updated late i.e. tomorrow a.m.
That's the fix. After replacing all the GetHistory calls with BacktestData calls, one needs to run the strategy sometime after Yahoo updates its previous market day data in the following morning or during the following trading day before the market closes, which would release a "new" EOD bar not yet published by Yahoo.
Even if one has a mutual fund data subscription on IQFeed, the problem would still persist because Fidelity doesn't publish EOD data (on IQFeed) for the current market day until several hours after the market closes.
Bottom line, it's simplest to run a mutual fund strategy either when the market is open or over the weekends.
Even if one has a mutual fund data subscription on IQFeed, the problem would still persist because Fidelity doesn't publish EOD data (on IQFeed) for the current market day until several hours after the market closes.
Bottom line, it's simplest to run a mutual fund strategy either when the market is open or over the weekends.
QUOTE:Already ahead of you. I updated the QuickRef a few days ago with Glitch's comments. The QuickRef doesn't have hyperlink capability, but when the update makes it to the site's docs it will automatically be added.
documentation for GetHistory and GetHistoryUnsynched is incomplete.
Actually, I just found an extra entry for GetHistory that shouldn't be appearing.
QUOTE:Well that needs to be fixed. The idea is the make the QuickRef instructive by allowing users to "explore" all the hyperlinking so they realize alternative options; i.e. GetHistory vs BacktestData.
The QuickRef doesn't have hyperlink capability,
Hey superticker, we're already working 12+ hours a day. Would appreciate a little bit less demanding of an attitude, but I agree it would be great to eventually get this done!
With this hyperlinking request, I see this discussion going far beyond the resolved IQFeed timeout issue. Let's mark this as closed and have a new topic for hyperlinking if it's really necessary.
Your Response
Post
Edit Post
Login is required