Optimization-parameters-when-fed-into-the-Backtest-don-t-produce-the-same-results

Please update B76 to the latest build (B84) and see if it makes difference.

Hi Eugene,
Thanks for your reply. I did update to B84. I reran some of the backtests. I noticed that the backtest itself produces different results every time with the same input parameters. I checked the trades that were made in the backtest to compare one run against the other. The trades are quite different. I would expect them all to be the same. ...not sure what is going on.

If you have any other ideas, I can try those out.

Check the position sizing. You're probably overallocating and that will cause what you described.

I would think that the position sizing settings I'm using would be consistent across runs -not sure though since I'm new to this program. Anyway if something looks strange, please let me know.

This video might help, check your Metrics Report and how many "NSF Positions" do you see?

https://youtu.be/59d3WGa0OBw

You need to provide more information. How many symbols are in your data set? For example, if you have 10 or more symbols then you could overallocate. You'll exhaust the funds available and you'll get random results.

I'm also seeing some unusual profit values, it would help if we see the Strategy completely and see your Backtest Preferences. I wonder if you're using Futures Mode and have some incorrect symbols defined?

Glitch and paul1986,
I watched the video in the link provided. The strategy I'm testing is the Bensdorp Short RSI Thrust V2 using the WealthData S&P500 data set (969 symbols in the dataset -listed and delisted I presume). From the video I see how I could get varying results. This particular setup though is ranked by the 7 day ADX. I guess it is possible that some of the 7 day ADX values could be the same and therefore ranked slightly different each time. Is there a way to rank within a rank so I can get reproduceable results?

Also there is a mention of using Futures Mode, how can I check that?

I opened that strategy in WL8 but it seems you have 3 parameters that I don't? I only have

"Entry Limit %"
"RSI Limit"
"Profit %"

where did the other parameters come from?

Also, I would advise you to use a non-parallel optimizer for this Strategy, since it uses some static data structures that would likely get overrun when multiple parallel optimizations execute against it. This reminds me, we plan to add a warning about this and detect Strategies that might experience problems with parallel optimizations so it's good this came up now!

Yeah, I parameterized other aspects of that strategy...pretty much every variable I could change I put in as an optimizable parameter. Other than that it is the same strategy.

Any tips on how to use a non-parallel optimizer?

In the optimize tab select for "Optimizer" choose either the "Exhaustive non-Parallel" or the "Shrinking Window non-Parallel."

If think you'll find much more stable results and matching backtests with the same parameter values.

Okay great! Thanks for your help

QUOTE:
for this Strategy, since it uses some static data structures that would likely get overrun when multiple parallel optimizations execute against it.

This is really a bug in the strategy implementation itself. Why would anyone include static data structures in a WL8 strategy to begin with? ... unless they were too cheap to buy more memory chips for their machine.

One of the hassles I had with converting WL6 (single threaded) code to WL8 (multithreaded) code was rewriting the strategies to eliminate static data structures.

The reason is the strategy collects ranking data in the PreExecute method. It’s not a fault in the Strategy. It’s something we need to detect and warn about in conjunction with using parallel optimizers.

QUOTE:
The reason is the strategy collects ranking data in the PreExecute method.

So you're saying all the threads need be accessing the ranking data structure via ConcurrentBag?

https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentbag-1?view=net-8.0

ConcurrentBag does not have a Sort() method. And how could you sort something that's dynamic (concurrent) in the first place? You need to replace PreExecute{} with two new procedures: ConcurrentPreExecute{} and SortingNonconcurrentPreExecute{}. Oh, we need to start a new topic before we get into WL architecture.

But I see your point. Concurrency has to be paused to do a synchronous sort.

Did I say anything about ConcurrentBag?? :)

These are methods in DrKoch's finantic.Indicators so I'm not sure exactly what data structure it's using.

But the problem is this:

The PreExecute method will populate some data in some kind of collection for all of the symbols in the backtest. By nature this needs to be a static data structure because it will be accessed in the Execute method.

The Execute method examines the values in the collection to make some decision.

It's all fine as long as two or more parallel threads don't try to utilize the data structure simultaneously. It has nothing to do with concurrency. One thread will wipe whatever information that the other thread populated in the collection.

So, this kind of Strategy will produce undefined results in parallel optimizations but will be OK in non-parallel.

QUOTE:
It's all fine as long as two or more parallel threads don't try to utilize the data structure simultaneously. It has nothing to do with concurrency. One thread will wipe whatever information that the other thread populated in the collection.

I'm confused. Only PreExecute{} has write access to this ranking collection. The Execute{block} should only be reading this ranking collection, not writing to it.

The problem is PreExecute must wait for all the threads to complete (synchronize) so the ranking collection is stable before it can sort them. After sorting, the ranking collection should become read-only and the concurrent Execute procedures shouldn't be messing with it.

Are you trying to say the Execute procedures are messing around with the ranking collection? If so, why are they doing that? The integrity of the ranking collection is solely PreExecute's job.

I'm saying that one PreExecute method from thread 2 can wipe out the collection that thread 1 is still reading in the Execute method.

It just won't work using static collections in a parallel optimizer. So I included a warning as such in Build 85.

I think the correct approach would be using ThreadLocal storage.

https://learn.microsoft.com/en-us/dotnet/api/system.threading.threadlocal-1?view=net-8.0

QUOTE:
one PreExecute method from thread 2 can wipe out the collection that thread 1 is still reading

So PreExecute is multithreaded? Then that's your problem. PreExecute is the "monitor" (arbitrator in charge) and there can only be one monitor. OSes have monitors too to keep all the processes coordinated.

If PreExecute delivers the ranking collection as a ConcurrentBag to the Execute threads, it "may" be okay for these threads to remove entries from ConcurrentBag, but they shouldn't change the ranking order established by the monitor, PreExecute; otherwise, you have chaos.

Let's clear this up.

PreExecute is not inherently multi-threaded.

Running a PARALLEL OPTIMIZER makes everything multi-threaded.

Having numerous threads hitting one static collection in the class is the problem.

Using ThreadLocal storage could be a solution.

Or, as the new warning explains, use a non-parallel Optimizer for these kinds of Strategies.

QUOTE:
PreExecute is not inherently multi-threaded.
Running a PARALLEL OPTIMIZER makes everything multi-threaded.

Okay, you're saying the entire strategy, as an atomic unit, becomes a thread. I didn't know that. I thought Initialize and Execute were asynchronous (as separate threads) and PreExecute was synchronous since it was the monitor.

Yes, I see the problem now. There is no monitor or chief-in-charge for data arbitration.

QUOTE:
Using ThreadLocal storage could be a solution.

That might be one solution. There are multiple solutions. The question remains, what is the simplest solution that non-programmers can understand?

Thank you for taking the time to explain the problem to me. This has been eye opening.

The simplest solution is to switch to a non-parallel optimizer.