Memory consumpton issues
Author: jimcrist
Creation Date: 1/27/2010 12:51 PM
profile picture

jimcrist

#1
I like WLP5, but it's nothing but a never ending series of memory problems. Last nite I ran a backtest that used less than a Gig of memory. Today, I made a minor change and it craps out at 1.33G with a memory exception. I'm using XP with 2G RAM. I'm running 1488 symbols, 5305 bars and 14 dataseries, so it should run in 884 Meg. I compress the 15 min bars into daily data in my script, so in addition to the 6 standard dataseries, I have hATR, hAVGV20, hAVGC10, hOpen, hClose, hLow, CompBar, GetDailyBar. I do a Bars.Cache.Clear and the very end of my script after the symbol loop.

My questions are:
1) I have this at the beginning of the Execute section:
CODE:
Please log in to see this code.

I don't have to do anything to clear memory for that structure do I? Am I racking up that space for every symbol?

2) What's being done in the first step "Collecting Data"? If backtest A has 500 trades and backtest B has 1000 trades does it use more memory in "Collecting Data"?
Thanks!

3) Can you recommend a utility that clears memory, so I don't have to reboot everytime I run a backtest?
profile picture

Eugene

#2
1 - For every symbol. Yes, you can explicitly clear that memory (fList = null), although I believe that the GC (garbage collector) should take care of that.

2 - The more trades the more is memory consumption.

3 - I can't recommend an utility (try searching for "defragment memory"), but would suggest you adding at least 1 Gb of RAM more and enabling the /3GB switch. RAM is very cheap these days, and this should serve as a temporary relief until Wealth-Lab Pro x64 is released and you move to a 64-bit OS with 4+ Gb RAM.

Until x64, the best thing is to reduce the number of symbols and/or data. Here's some clear explanation by an MVP (marked as "Good answer"): OutOfMemory Exception but heaps of free memory
profile picture

Cone

#3
1. FundamentalDataItems isn't especially memory intensive. It creates a list of the 24 FundamentalItem objects, each of which has a few strings, int, and double types.

In general, there isn't anything that can do clear specific objects from memory. That's the job of the Garbage Collector, which should do it automatically.

2. Collecting data = read data from disk and/or update it (if requested). More trades requires more memory because each Position object has a bunch of data that goes along with it.

3. No suggestions from me, but if you run into limits with XP, shut down WLP. In my experience, memory management is better on Vista (and Win 7), and with WLDx64 I've had it up to the machine's 8G limit with no trouble at all.

64-bit is coming for WLP this year, so start a piggy bank :)
profile picture

Cone

#4
Nearly the same answers and we're not even in the same country! :)
profile picture

Eugene

#5
Blame it on Carl Jung's Synchronicity :)
profile picture

Cone

#6
I don't think this qualifies for synchronicity since we often answer the same question simultaneously (a likely event) and necessarily in a causal manner (after someone asks).

I think a better example of synchronicity would be me making the right trade (unlikely) at precisely the right time. :)
profile picture

jimcrist

#7
Why can't I run this little script on a 680 symbol dataset in less then 1.3G RAM? It's not even close, even on a fresh boot...
CODE:
Please log in to see this code.

Apparently, the SetContext and RestoreContext are unnecessary here, but it doesn't help with the memory hog problem...
profile picture

Eugene

#8
SetContext/RestoreContext are absolutely unnecessary here in this strategy which doesn't trade on external symbols. Try this strategy and make sure you run it in single symbol mode:
CODE:
Please log in to see this code.
profile picture

jimcrist

#9
Single Symbol Mode? I'm just trying to export the symbols from a dataset to a .csv file. How do I do that without using Multi-Symbol Backtest, in less than 1.3G of RAM?
profile picture

Eugene

#10
Yes, this code requires single symbol mode. When you try to run a DataSet-looping strategy (like this) in multi-symbol backtest mode, you're running the strategy DataSetSymbols * DataSetSymbols times for no reason. :)
profile picture

jimcrist

#11
So there's no way to export all the symbols from a dataset to a .csv file?
profile picture

Eugene

#12
The strategy code does just that. You click on 1 symbol - it exports all symbols of the dataset. Is this more clear?
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).