For one particular strategy (which includes a PreExecute{block}), the parameter the optimizer is trying to optimize (which is Profit here) is not correctly assigned in the Tabular output. See screenshot.
Some comments.
1) This problem is demonstrated with both the Shrinking Window and Exhaustive optimizer. So it's probably not an optimizer problem.
2) When you manually plug in the parameter row, the strategy always returns the correct Profit. Of course, that doesn't match up with what you're expecting from the Tabular optimization listing.
3) This is a problem whether one is optimizing one parameter or all three parameters of the strategy. Symbol-specific Preferred Values are not being used.
4) When optimizing just one parameter, only 25% of the numbers in the Profit column are incorrect. This "may" worsen if more parameters--more run permutations--are being optimized, but I'm not sure.
So how is it possible the errors only occur 25% of the time? I would think it would be all or nothing. Is there some kind of synchronization problem with handing off the parameter being optimized to the Tabular table? UPDATE: It looks to me like the Profit produced by one optimization-run thread is erroneously being assigned to another run thread in the Tabular table. I'm running WL7 Build 36.
Why is this only occurring with one particular strategy? How can this strategy confuse the Tabular output of the parameter being optimized?
I posted part of the strategy. I can email the entire thing to you if you want. I have no idea how the strategy influences the assignments of the Profit column. I can run a few tests if you tell me what I'm looking for. Is there a way to "peek" at the Profit the strategy is returning to the optimizer or Tabular table without setting a debugger break point?
This strategy does call custom Local.Components.dll code, but that code is old and common to all my strategies, which don't have this problem.
Perhaps you have a similar strategy to the above you can test against.
Some comments.
1) This problem is demonstrated with both the Shrinking Window and Exhaustive optimizer. So it's probably not an optimizer problem.
2) When you manually plug in the parameter row, the strategy always returns the correct Profit. Of course, that doesn't match up with what you're expecting from the Tabular optimization listing.
3) This is a problem whether one is optimizing one parameter or all three parameters of the strategy. Symbol-specific Preferred Values are not being used.
4) When optimizing just one parameter, only 25% of the numbers in the Profit column are incorrect. This "may" worsen if more parameters--more run permutations--are being optimized, but I'm not sure.
So how is it possible the errors only occur 25% of the time? I would think it would be all or nothing. Is there some kind of synchronization problem with handing off the parameter being optimized to the Tabular table? UPDATE: It looks to me like the Profit produced by one optimization-run thread is erroneously being assigned to another run thread in the Tabular table. I'm running WL7 Build 36.
Why is this only occurring with one particular strategy? How can this strategy confuse the Tabular output of the parameter being optimized?
I posted part of the strategy. I can email the entire thing to you if you want. I have no idea how the strategy influences the assignments of the Profit column. I can run a few tests if you tell me what I'm looking for. Is there a way to "peek" at the Profit the strategy is returning to the optimizer or Tabular table without setting a debugger break point?
This strategy does call custom Local.Components.dll code, but that code is old and common to all my strategies, which don't have this problem.
CODE:
using System; using System.Collections.Generic; using System.Drawing; using System.Text; using WealthLab.Core; using WealthLab.Backtest; using WealthLab.Indicators; using Superticker.Components; namespace WealthScript6 { public class SelSecTradeRank : UserStrategyBase { public SelSecTradeRank() { AddParameter("Buy list length", ParameterTypes.Int32, 6, 1, 14, 1) .Hint="Length of the top-buys list"; AddParameter("regEMA vel sell threshold", ParameterTypes.Double, -0.35, -1.0, 0.0, 0.05) .Hint="Minimum velocity threshold to sell"; AddParameter("RegEMA affect", ParameterTypes.Double, 0.5, 0.0, 1.25, 0.25) .Hint="EMA regularization"; SetChartDrawingOptions(WLUtil.HideVolume()); } static List<BarHistory> topBuys = new List<BarHistory>(); StringBuilder topBuysString = new StringBuilder(160); public override void Initialize(BarHistory bars) { regEmaVelSellThreshold = Parameters[1].AsDouble; double emaRegular = Parameters[2].AsDouble; //paramEmaRegular TimeSeries decorrelated = RSDif.Decorrelate(bars); PlotTimeSeriesLine(decorrelated, bars.Symbol + " decorrelated", "decorrelated", Color.Green, 2, LineStyles.Solid); SetPaneDrawingOptions("decorrelated", 150, -2); RegEmaFit regEmaFit = new RegEmaFit(this as UserStrategyBase, decorrelated, "decorrelated", 0.4, emaRegular, decorrelated.Count, false); PlotTimeSeriesLine(regEmaFit.YModelTS(), bars.Symbol + " regEMA", "decorrelated", Color.Red, 2, LineStyles.Dashed); TimeSeries regEmaVel = new Momentum(regEmaFit.YModelTS(),1); bars.Cache["RegEmaVel"] = regEmaVel; //regularized EMA velocity TimeSeries for selling PlotTimeSeriesLine(regEmaVel, "regEMA velocity", "regEmaVel", Color.Red, 2, LineStyles.Solid); SetPaneDrawingOptions("regEmaVel", 80, -3); TimeSeries emaVelTransformed = new TimeSeries(regEmaVel.DateTimes, 0.0); for (int bar=0; bar<regEmaVel.Count; bar++) { if (regEmaVel[bar] < 0.0 && regEmaVel[bar]-regEmaVel[bar-1] > -0.15) emaVelTransformed[bar] = 1.0-regEmaVel[bar]; } PlotTimeSeriesLine(emaVelTransformed, "EMA velocity transformed", "regEmaVel", Color.Orange, 2, LineStyles.Solid); TimeSeries pctRank = bars.Close.PercentRank(220); StartIndex = pctRank.FirstValidIndex; PlotTimeSeriesLine(pctRank, bars.Symbol + " fractional rank", "percentRank", Color.YellowGreen, 2, LineStyles.Solid); SetPaneDrawingOptions("percentRank", 60, 1); TimeSeries pctRankShort = bars.Close.PercentRank(200); bars.Cache["PctRankSrt"] = pctRankShort; //short-term %rank TimeSeries for selling PlotTimeSeriesLine(pctRankShort, bars.Symbol + " short rank", "percentRank", Color.SaddleBrown, 2, LineStyles.Solid); meritScore = new TimeSeries(bars.DateTimes); for (int idx = 0; idx < meritScore.Count; idx++) { //int lowestRecentBar = decorrelated.GetLowestBar(idx, 22); //lowest bar for past month //int gainBars = idx - lowestRecentBar; //#of bars to determine recent price gains (idx-1?) meritScore[idx] = 2.0 - pctRank[idx]; } bars.Cache["MeritTS"] = meritScore; //Merit TimeSeries for sorting PlotTimeSeries(meritScore, bars.Symbol + " merit scores", "meritScores", Color.DarkRed, PlotStyles.Dots); SetPaneDrawingOptions("meritScores", 60, -1); } //called once prior to the start of each bar; determine and sort by dipBuyMeritScore public override void PreExecute(DateTime dt, List<BarHistory> participants) { int topBuysLength = Parameters[0].AsInt; //for each fund in DataSet, calculate dipBuyMeritScore foreach (BarHistory bars in participants) { int idx = GetCurrentIndex(bars); //rtn index of the BarHistory currently being processed meritScore = (TimeSeries)bars.Cache["MeritTS"]; bars.UserData = meritScore[idx]; //save merit score w/BarHistory instance } //sort highest meritScore values where participants[0] participants.Sort((b,a) => a.UserDataAsDouble.CompareTo(b.UserDataAsDouble)); meritThreshold4Buy = participants[topBuysLength-1].UserDataAsDouble; meritThreshold4Hold = participants[13].UserDataAsDouble; topBuys.Clear(); int nEndTop = Math.Min(topBuysLength,participants.Count); //select top funds for (int n = 0; n < nEndTop; n++) { topBuys.Add(participants[n]); //WriteToDebugLog(string.Format("{0}=n {1:0.00}=meritScore",n,participants[n].UserDataAsDouble)); } } public override void Execute(BarHistory bars, int idx) { } //shared strategy variables static double regEmaVelSellThreshold; //PV parameter static double meritThreshold4Buy,meritThreshold4Hold; TimeSeries meritScore,meritScore4bars,regEmaVel,pctRankShort; } }
Perhaps you have a similar strategy to the above you can test against.
Rename
I have another question about the declaration below.
But what about during optimization where threads are involved? Is the threading going to mess this up? If so, how should these values be stored to be compatible with threads?
CODE:For running this strategy without the optimizer, there should be no problem with the above line of code. Once PreExecute{} assigns these values, they should remain good for all symbols during that bar. And they are.
static double meritThreshold4Buy,meritThreshold4Hold;
But what about during optimization where threads are involved? Is the threading going to mess this up? If so, how should these values be stored to be compatible with threads?
This is probably similar to the issue discussed in this post ...
https://www.wealth-lab.com/Discussion/Backtest-and-Optimization-Results-Don-39-t-Match-6718
Due to the parallel optimizers, it's likely that different runs are colliding using the same static variable. You could either use a non-parallel optimizer, or use the Backtester.Cache solution outlined in Post #6 of that topic.
https://www.wealth-lab.com/Discussion/Backtest-and-Optimization-Results-Don-39-t-Match-6718
Due to the parallel optimizers, it's likely that different runs are colliding using the same static variable. You could either use a non-parallel optimizer, or use the Backtester.Cache solution outlined in Post #6 of that topic.
QUOTE:
it's likely that different runs are colliding using the same static variable. You could either use a non-parallel optimizer,
I'm not sure if "colliding" is the right word; otherwise, using a concurrent collection datatype in the above link discussion would have solve this problem. I think the reason the concurrent collection approach isn't working (in the above link) is because it's not keeping track of "dirty" entries (or dirty page objects) in the collection. Maybe the concurrent Dictionary should be keyed by thread-ID instead of BarHistory, but that's another discussion....
At any rate, you are correct. Apparently using the non-parallel Exhaustive optimizer solves this problem. I would include the above link (which has the solution) in the documentation for PreExecute{} and eventually replace that link with a link to a blog article that sheds more details on this problem and its solution.
I don't favor using a concurrent collection approach because it ties the hands of the Windows scheduler with blocking. But I like the Backtester.Cache[...] approach. I'll try to implement that. Thanks for your help and insights. They are always appreciated.
Will update the website FAQ item "Why my backtest and optimization results don't match?" to reflect this (related) case.
QUOTE:
... update the website FAQ item "Why my backtest and optimization results don't match?" to reflect this (related) case.
That's a good idea, but I would also indicate that the prerequisite to that read is reading the PreExecute{} section of Anatomy of a Wealth-Lab 7 Strategy https://www.wealth-lab.com/blog/anatomy-of-a-wl7-strategy and the concurrency (tasking/threading) chapter of their C# language textbook.
They will not understand the Anatomy of a Wealth-Lab 7 Strategy PreExecute{} section unless they understand concurrency issues and scheduling. I would explicitly cite a textbook reference so they can read up on that first; otherwise, they are lost. They need the background first to comprehend the issues involved here.
They can learn, but only if they meet the prerequisites first.
I have a followup question. Does the Backtester.Cache[...] get purged between PreExecute{} and Execute{} ? If so, can we fix that?
I'm also wondering if I can simply declare the staticVars as "static" and not use the cache at all for them? This is basically the same question I raised in Reply# 1.
CODE:So the userData object gets populated and used in PreExecute{}. That seems to work well. However, the staticVars object gets populated in PreExecute{} and then employed (read only) in Execute{}. It appears the Backtester.Cache[...] purges it before reaching Execute{}. All staticVars are zero after reaching Execute{}. Is that by design or a bug?
public class SelSecTradeRank : UserStrategyBase { private struct StaticVars { public double regEmaVelSellThreshold; public double meritThreshold4Buy; public double meritThreshold4Hold; } ... StaticVars staticVars = new StaticVars(); Dictionary<string,double> userData = new Dictionary<string,double>(); public override void BacktestBegin() { Backtester.Cache["StaticVars"] = staticVars; Backtester.Cache["UserData"] = userData; }
I'm also wondering if I can simply declare the staticVars as "static" and not use the cache at all for them? This is basically the same question I raised in Reply# 1.
QUOTE:
I'm also wondering if I can simply declare the staticVars as "static" and not use the cache at all for them? This is basically the same question I raised in Reply# 1.
You can't; I just tried it. The declared staticVars are no longer zero when declared as "static" but the problem with the Profit column being bogus remains. So these staticVars need to be passed through the Backtester.Cache[] somehow. But they return all zeros when they are. I'm not sure how to fix that. It looks to me like the Backtester.Cache[] fails to preserve the values going from PreExecute{} to Execute{}. Does the thread-ID need to be included as part of the cache lookup key? If so, how do I grab the thread-ID in PreExecute?
No, it's not purged between PreExecute and Execute, here's a simple Strategy that illustrates this ...
Some sample Debug Log output ...
CODE:
using WealthLab.Backtest; using System; using WealthLab.Core; using WealthLab.Indicators; using System.Drawing; using System.Collections.Generic; namespace WealthScript1 { public class MyStrategy : UserStrategyBase { //Initialize public override void Initialize(BarHistory bars) { WriteToDebugLog("Initialize", false); } //Pre-Execute public override void PreExecute(DateTime dt, List<BarHistory> participants) { Backtester.Cache["Count"] = participants.Count; WriteToDebugLog("PreExecute - " + participants.Count + " written to Backtester.Cache", false); } //Execute public override void Execute(BarHistory bars, int idx) { int count = (int)Backtester.Cache["Count"]; WriteToDebugLog(count.ToString(), false); } //private members } }
Some sample Debug Log output ...
CODE:
AMZN: PreExecute - 11 written to Backtester.Cache AMZN: 11 DOW: 11 MSFT: 11 TCOM: 11 AAPL: 11 T: 11 GS: 11 GE: 11 BA: 11 XLB: 11 F: 11 AMZN: PreExecute - 11 written to Backtester.Cache AMZN: 11 DOW: 11 MSFT: 11 TCOM: 11 AAPL: 11 T: 11 GS: 11 GE: 11 BA: 11 XLB: 11 F: 11
QUOTE:
No, it's not purged between PreExecute and Execute,
Then I'm doing a stupid thing. I'll email you my code. Please look at how the cached StaticVars are handled in Execute{} and tell me what's wrong. I just don't know how to fix it. All the StaticVars in Execute{} are zero according to the debug statements there.
Email has been sent. I'm at a loss how to fix this.
I apologize. Since we already know the bug is a cache handling problem, let's simplify it to the bear bones. Please tell me why the WritetoDebugLog line below returns all zeros and how to correct that. Hopefully this is an easy question. And I'm sorry for not seeing the very obvious mistake here.
CODE:
using System; using System.Collections.Generic; using WealthLab.Core; using WealthLab.Backtest; namespace WealthScript2 { public class StaticVarsTest : UserStrategyBase { private struct StaticVars { public double regEmaVelSellThreshold; public double meritThreshold4Buy; public double meritThreshold4Hold; } StaticVars staticVars = new StaticVars(); public override void BacktestBegin() { //StaticVars staticVars = new StaticVars(); Backtester.Cache["StaticVars"] = staticVars; } public override void Initialize(BarHistory bars) { StaticVars staticVars = (StaticVars)Backtester.Cache["StaticVars"]; staticVars.regEmaVelSellThreshold = 1.0; } //called once prior to the start of each bar; determine and sort by dipBuyMeritScore public override void PreExecute(DateTime dt, List<BarHistory> participants) { StaticVars staticVars = (StaticVars)Backtester.Cache["StaticVars"]; staticVars.meritThreshold4Buy = 2.0; staticVars.meritThreshold4Hold = 3.0; //WriteToDebugLog(string.Format("PreEx: {0:MM/dd/yy} {1:0.00}=meritThreshold4Buy {2:0.00}=meritThreshold4Hold",dt,staticVars.meritThreshold4Buy,staticVars.meritThreshold4Hold)); } public override void Execute(BarHistory bars, int idx) { StaticVars staticVars = (StaticVars)Backtester.Cache["StaticVars"]; WriteToDebugLog(string.Format("Execute: {0:000}=idx {1:0.00}=staticVars.regEmaVelSellThreshold {2:0.00}=meritThreshold4Buy {3:0.00}=meritThreshold4Hold", idx,staticVars.regEmaVelSellThreshold,staticVars.meritThreshold4Buy, staticVars.meritThreshold4Hold)); } } }
It's because you're using a struct instead of a class, and structs are not allowed to have parameterless constructors. I think there's an exception happening in BacktestBegin that is not getting bubbled up to the strategy window error pane, I'll check into that.
For reference:
https://www.c-sharpcorner.com/blogs/difference-between-struct-and-class-in-c-sharp
And here is an edit of the code to make it work ...
For reference:
https://www.c-sharpcorner.com/blogs/difference-between-struct-and-class-in-c-sharp
And here is an edit of the code to make it work ...
CODE:
using System; using System.Collections.Generic; using WealthLab.Core; using WealthLab.Backtest; namespace WealthScript2 { public class StaticVarsTest : UserStrategyBase { private struct StaticVars { internal StaticVars(double v1, double v2, double v3) { regEmaVelSellThreshold = v1; meritThreshold4Buy = v2; meritThreshold4Hold = v3; } public double regEmaVelSellThreshold; public double meritThreshold4Buy; public double meritThreshold4Hold; } //StaticVars staticVars = new StaticVars(); public override void BacktestBegin() { StaticVars staticVars = new StaticVars(1,2,3); Backtester.Cache["StaticVars"] = staticVars; } public override void Initialize(BarHistory bars) { StaticVars staticVars = (StaticVars)Backtester.Cache["StaticVars"]; staticVars.regEmaVelSellThreshold = 1.0; } //called once prior to the start of each bar; determine and sort by dipBuyMeritScore public override void PreExecute(DateTime dt, List<BarHistory> participants) { StaticVars staticVars = (StaticVars)Backtester.Cache["StaticVars"]; staticVars.meritThreshold4Buy = 2.0; staticVars.meritThreshold4Hold = 3.0; //WriteToDebugLog(string.Format("PreEx: {0:MM/dd/yy} {1:0.00}=meritThreshold4Buy {2:0.00}=meritThreshold4Hold",dt,staticVars.meritThreshold4Buy,staticVars.meritThreshold4Hold)); } public override void Execute(BarHistory bars, int idx) { StaticVars staticVars = (StaticVars)Backtester.Cache["StaticVars"]; WriteToDebugLog(string.Format("Execute: {0:000}=idx {1:0.00}=staticVars.regEmaVelSellThreshold {2:0.00}=meritThreshold4Buy {3:0.00}=meritThreshold4Hold", idx, staticVars.regEmaVelSellThreshold,staticVars.meritThreshold4Buy, staticVars.meritThreshold4Hold)); } } }
I found the problem. The cached "struct" variable operations are not just adjusting a reference pointer into the cache. Rather they are copying the cache by value instead. That means changes made to the cached variables need to be written back to the cache before exiting the procedure that was doing the writing.
Also, since BacktestBegin{} isn't writing to the cache, it's not needed at all. One can just declare datatypes to be cached as field variables in the MyStrategy block. Here's the working StaticVars caching solution.
Sorry for taking up everyone's time for not recognizing what's happening (via the cache) by value rather than by reference pointer. As I said, it was a stupid mistake on my part.
Also, since BacktestBegin{} isn't writing to the cache, it's not needed at all. One can just declare datatypes to be cached as field variables in the MyStrategy block. Here's the working StaticVars caching solution.
CODE:
using System; using System.Collections.Generic; using WealthLab.Core; using WealthLab.Backtest; namespace WealthScript2 { public class StaticVarsTest : UserStrategyBase { private struct StaticVars { public double regEmaVelSellThreshold; public double meritThreshold4Buy; public double meritThreshold4Hold; } StaticVars staticVars = new StaticVars(); public override void Initialize(BarHistory bars) { staticVars.regEmaVelSellThreshold = 1.0; Backtester.Cache["StaticVars"] = staticVars; } //called once prior to the start of each bar; determine and sort by dipBuyMeritScore public override void PreExecute(DateTime dt, List<BarHistory> participants) { StaticVars staticVars = (StaticVars)Backtester.Cache["StaticVars"]; staticVars.meritThreshold4Buy = 2.0; staticVars.meritThreshold4Hold = 3.0; Backtester.Cache["StaticVars"] = staticVars; } public override void Execute(BarHistory bars, int idx) { StaticVars staticVars = (StaticVars)Backtester.Cache["StaticVars"]; WriteToDebugLog(string.Format("Execute: {0:000}=idx {1:0.00}=staticVars.regEmaVelSellThreshold {2:0.00}=meritThreshold4Buy {3:0.00}=meritThreshold4Hold", idx,staticVars.regEmaVelSellThreshold,staticVars.meritThreshold4Buy, staticVars.meritThreshold4Hold)); } } }
Sorry for taking up everyone's time for not recognizing what's happening (via the cache) by value rather than by reference pointer. As I said, it was a stupid mistake on my part.
Yes, makes sense. To be honest I've gotten to a point that I never use structs, I use classes exclusively. For me at least it is much better for my personal comfort zone.
QUOTE:So you're right. Structs are represented by value; whereas, classes are usually by reference in C#. Good point.
I've gotten to a point that I never use structs, I use classes exclusively.
I use classes if I need a constructor, and constructors are handy to have. Nearly all my datatypes have constructors. But if you can pass something by value, that's less indirection so it's slightly faster. Although today's look-ahead pipelining processor architecture makes that indirection overhead very minimal. (That's another topic.)
The Dictionary userData object is a class, so it's by reference rather than by value. It's handling is a little different. And yes, I didn't appreciate that like I should have.
When you create a blog article about PreExecute{}, I would include something about cache handling of classes (by reference) and structs (by value) just to clarify their differences. If I can get mixed up, others can too.
Yes, that will be a good point to call out, noted!
Well, the current solution is not 100%. The statements from fetching the cache to updating the cache represent a "critical section" which should be protected from time slicing. What I've done is try to minimize the statements within this critical section, but technically, all the cached elements in the critical section should be updated atomically prior to another time slice. That is, execution during that critical section should be locked to one thread; otherwise, you have problems.
Now I have a stupid question. If a "local" variable such as ...
I'm also wondering, do all the Backtester threads finish one bar (i.e. stay in synchrony) before stepping to the next PreExecute{} bar? If all the threads are operating on the same bar, then the cached contents for all those threads should be in good agreement and tolerate some mix up.
Now I have a stupid question. If a "local" variable such as ...
CODE:is strictly local to PreExecute{}, then there should be no reason to cache it right? If that's the case, then the only cached variable I have is a class with two doubles in it. And the computation of these two doubles ...
Dictionary<string,double> userData = new Dictionary<string,double>();
CODE:is fairly close together, yet the integrity of my optimizer Profit column is still not 100%, although it's over 90% perfect with 96 parameter permutations. So some kind of protection of the "critical section" code is probably necessary. I can post my final PreExecute code if it helps.
private class StaticVars { public double meritThreshold4Buy; public double meritThreshold4Hold; }
I'm also wondering, do all the Backtester threads finish one bar (i.e. stay in synchrony) before stepping to the next PreExecute{} bar? If all the threads are operating on the same bar, then the cached contents for all those threads should be in good agreement and tolerate some mix up.
Your Response
Post
Edit Post
Login is required