Different backtest results each run
Author: sunyata
Creation Date: 4/24/2009 2:24 PM
profile picture

sunyata

#1
Hullo everyone,

Quick question haven't been able to find the answer to. I am running an intraday strategy on Wealth-Lab (not Pro) on 200+ symbols with 5-min data. If I run this backtest consecutively, I receive the same results, HOWEVER if after running this strategy on all data, I proceed to run in on a single symbol, and then run it again on all data, the all-data backtest outputs a different P&L! I notice that the trades taken on the former and latter all-data backtests are completely different. What am I missing? Is there a maximum number of symbols that you can backtest at once on intraday data?

Thanks!

Jake
profile picture

Eugene

#2
Reason: your Strategy runs out of capital to execute all the signals.

For the explanation and workaround, please see this item on our WL5 Wiki FAQ:
Every time I run a Strategy I get a different result. What am I missing?

profile picture

sunyata

#3
Eugene,

Thank you for your response. However, I would like to see the result of taking EVERY trade in one backtest. So to do this I did a little test. There should be about 4000 trades in an all-data backtest and most of the stocks are under $20. So I set my capital at $20M and a fixed position size of 100 shares. Yet, the backtest still only generates about 50 trades. At an average cost of $20/share, each position requires $2000 about. With $20M in capital, I should roughly be able to handle 2000 positions at once. But the test doesn't come near that.

Maybe I am still missing something? Do I still need to work with "priorities" even if the capital is so large?

Thanks for your help.

Best,
Jake
profile picture

sunyata

#4
Also, if I run in "Raw Profit Mode" I still only get about 50 trades. Hmm.... Maybe something's up with my code.
profile picture

Eugene

#5
This could be the reason; if you're not sure, please post the code.
profile picture

sunyata

#6
Thanks Eugene for your help. Here's the code. It's a little lengthy I believe.

CODE:
Please log in to see this code.
profile picture

Eugene

#7
First of all, your strategy peeks into the future. You can't just use *AtStop(bar) the way you did that in real life because the High/Low prices of the bar are not available until close. I mean, for example, lines 142 and 163. Please see this KB article for more:
Bars, Loops, and Bar + 1
profile picture

sunyata

#8
Ok. I will work on fixing that. Do you think this could contribute to the backtesting issue I described?
profile picture

sunyata

#9
Okay. I have adjusted my code so that it does not peek into the future. However, I am still unable to backtest every single trade at once, even in Raw Profit Mode.

CODE:
Please log in to see this code.
profile picture

Eugene

#10
Your code will not generate a single alert because you're passing bar-1 to the trading functions (instead of bar+1). Please review the KB article link above.

In addition, your code is somewhat too long to be readable, would you mind creating a more compact version for troubleshooting by creating dummy conditions instead of the actual production logic? Thank you.
profile picture

Cone

#11
What is your Raw Profit setting?

- just check to make sure that it's sufficiently high to enable the purchase of at least 1 share. In other words, if your RP size is $1,000, you won't be able to create trades on BRK/B (trading around $3000/share), for example, since even in RP mode you cannot buy 1 share.
profile picture

sunyata

#12
Hi Eugene,

I've created a simplified version, in some cases simply using written text to convey what I am trying to accomplish. This may be a start to help figure out the problem I having. And in response to Cone, I've attempted to use Raw Profit Mode by setting a Fixed Position Size to enter with on every trade, rather than setting a Fixed Dollar amount per trade. Because of this, I should not be having any problems related to lack of available funds. Thanks for your input though.

Here's the code:

CODE:
Please log in to see this code.
profile picture

sunyata

#13
Also, as of now, I am not too interested in having the strategy generate alerts. I just want to be able to generate a backtest with every single trade so that I can determine if the strategy is even profitable across a broad range of securities.
profile picture

Eugene

#14
Jacob,

Probably I wasn't clear on the code readability. Let me first quote the excellent advice given by Robert (Cone):

Best way to implement the following trading strate

QUOTE:
It turns out that this is a good technique to developing a program or application - break it down into small parts so that you can concentrate on a manageable piece. Start with one piece, define the specific requirements, and think about how you'll integrate it into the finished product.


What was on my mind is not explaining the rules in plain English, but rather stripping them out completely, leaving just a skeleton with the structure. Rules are well could be random entries and exits. Anyway, it's not important. To generate a backtest with every single trade, you need to fix the obvious peeking errors (SellAtStop(bar-1), BuyAtStop(bar) etc). This is a prerequisite to determine the number of signals in Raw Profit mode. After troubleshooting your code and verifying that it works according to your design, it becomes reasonable to approach the portfolio backtest mode.
profile picture

sunyata

#15
Hey Eugene,

Thanks for your patience with this. In regards to peeking, I am a bit confused. How is "SellAtStop(bar - 1)" considered code that peeks, given that the referenced bar has already closed. Although, admittedly, it seems unorthodox to me, given that the position is not closed until the ensuing bar, it does not seem to be an example of peeking. Am I incorrect here? Having read the KB article you sent me, I understand passing "bar + 1" to a trading signal to avoid entering a position on the open, while utilizing an indicator value that is calculated at the close. I certainly understand this as peeking. But I am not quite sure I understand "peeking" in reference to how I have utilized "bar - 1".

Also, I certainly agree that building a program in increments is more simple to manage. In fact, I did just that with this program, starting with the trade setup, working to entries, and then finishing off with exits. How would you like me to better convey the skeleton of the program?

Thanks again.

jake
profile picture

Eugene

#16
QUOTE:
But I am not quite sure I understand "peeking" in reference to how I have utilized "bar - 1".

Imagine you have created a position today (bar) on a stop (let's put aside the fact that it's impossible to enter on a stop at bar, utilizing the high(bar) price), how could you close that position yesterday with sellatstop(bar-1) ?
profile picture

sunyata

#17
Eugene,

How would you suggest entering a position on a stop? Could you do the following:

CODE:
Please log in to see this code.


What I don't like about this solution is that although the position is recorded as entered on the correct bar, the strategy does not actually record the entry until the following bar. It does not seem possible to me to utilize the bar + 1 setup when entering on stops.

profile picture

Cone

#18
Did you review Bars, Loops, and Bar + 1?

Okay, it's Thursday night, and lets say you want to buy on a stop if MSFT hits 21 on Friday. Further, lets assume that MSFT does cross 21 tomorrow and closes at 22. How will you enter this order with your broker?

Are you going wait until Monday to place your order for the previous bar (bar - 1), or, are you going to enter your stop order tonight (or any time on or before the opening bell) so that it can be live tomorrow when the price event takes place?
profile picture

sunyata

#19
Oh, I think Cone's comment sparked a realization! Prior to using WealthLab, I used eSignal, whose broker language required that I enter the prices at which positions were either opened and closed. This language had no way to decide if my stop price was hit or not; it simply executed and I had to use conditional statements to control access to this broker language. But with WealthLab, it seems as though the broker terms, such as BuyAtStop and SellAtStop, will only execute if the indicated price is hit. Am I correct about this?

Thanks again for your help.

Jake
profile picture

Eugene

#20
*AtStop (as well as other trading functions) will execute if the price is hit, and you will know when it happen by the fact a Position object is returned (or not).
profile picture

ss161

#21
sunyata, i was looking at your code and thought for a second i was looking at some of my code with all of the PrintDebug("got here") messages. fwiw, i downloaded and installed SharpDevelop (it's free) and followed Eugene's very well written links on how to debug strategy code using SharpDevelop. It's saved me hours since. Just in case your not familiar with it. At the top of this forum post are the links to the directions:
/Forum/Posts/Is-it-possible-to-debug-into-stepping-the-code-lik-29198

profile picture

sunyata

#22
Thanks so much for everyone's help. I will rewrite the code per my newfound understanding and see if I can't get WealthLab to record all the trades. And thanks ss161 for the SharpDevelop tip.

Jake