allow complex systems to be run in Single System Mode without generating the memory errors

Author: ss161

Creation Date: 3/18/2009 10:01 AM

Wealth Lab » WealthScript (.NET) » allow complex systems to be run in Single System Mode without generating the memory errors

ss161

3/18/2009 10:01 AM

Both Single Symbol Mode and Multi-Symbol BackTest seem to come close, but ultimately fail in their ability to support scripts designed to trade more than one symbol. Certainly at a simple level they work, but even slightly complicated scripts ultimately have issues.

Here is a simple program design:

CODE:
Please log in to see this code.

the objective is to loop through the entire watchlist (from 0 to WLCount -1) and for each name, create a pair with every name beginning 1 greater than the current symbol, all the way to the end, and then for each pair, add trading symbol logic for each Bar.

The problem is, it doesn't work. It doesn't work because when the Context is Set (to True) the two pairs are not Synched with each other, but rather with whichever symbol was clicked to launch the script.

so, i thought about another method: just do one pass through the list, starting from the clicked symbol and going all the way to the end, and then run the script in MultiSymbolMode -- thereby letting Wealth-Lab do the outer loop. This way, each pair will always be synched to the first clicked symbol from each run. Here is a code fragment for this:

CODE:
Please log in to see this code.

The problem with the above is that it is almost guaranteed to generate System.OutOfMemory errors if the dataset is too large.

Here is my enhancement request:

Write an Override for the SetContext function to make the SetContext "permanent", i.e.

CODE:
Please log in to see this code.

i believe this would allow complex systems to be run in Single System Mode without generating the memory errors and i think it would provide a lot of flexibility in other non-pair types of systems as well.

Size:

Color:

Both Single Symbol Mode and Multi-Symbol BackTest seem to come close, but ultimately fail in their ability to support scripts designed to trade more than one symbol.  Certainly at a simple level they work, but even slightly complicated scripts ultimately have issues.

Here is a simple program design:

[CODE]
	public class TEST1 : WealthScript {
 
		protected override void Execute() {
		
			string ClickedSymbol = Bars.Symbol; 
			PrintStatusBar(ClickedSymbol);

for (w = w; w < DataSetSymbols.Count -1; w++) {
				
				Bars.Cache.Clear();
				PrintStatusBar(DataSetSymbols[w]);
				SetContext(DataSetSymbols[w], true);
						
				for (int x = w +1; x < DataSetSymbols.Count; x++) {
							
					PrintStatusBar(DataSetSymbols[w] + ":" + DataSetSymbols[x] ); 
					SetContext(DataSetSymbols[x], true);
						
					for (int Bar = 0; Bar < Bars.Count; Bar++) {

//------------------------------
						// PROCESSING LOGIC GOES HERE
						//------------------------------
					}
				}
			}
		}
	}
[/CODE]

The problem is, it doesn't work.  It doesn't work because when the Context is Set (to True) the two pairs are not Synched with each other, but rather with whichever symbol was clicked to launch the script.

so, i thought about another method:  just do one pass through the list, starting from the clicked symbol and going all the way to the end, and then run the script in MultiSymbolMode -- thereby letting Wealth-Lab do the outer loop.  This way, each pair will always be synched to the first clicked symbol from each run.  Here is a code fragment for this:

[CODE]
	public class TEST2 : WealthScript {
 
		protected override void Execute() {
		
			string ClickedSymbol = Bars.Symbol; 
			PrintStatusBar(ClickedSymbol);

int w = 0;
			while((ClickedSymbol != DataSetSymbols[w])) w++;

Bars.Cache.Clear();
			if (w < DataSetSymbols.Count -1) {

PrintStatusBar(DataSetSymbols[w]);
				SetContext(DataSetSymbols[w], true); 
						
				for (int x = w +1; x < DataSetSymbols.Count; x++) {
							
					PrintStatusBar(DataSetSymbols[w] + ":" + DataSetSymbols[x] ); 
					SetContext(DataSetSymbols[x], true);
						
					for (int Bar = 0; Bar < Bars.Count; Bar++) {
						//------------------------------
						// PROCESSING LOGIC GOES HERE
						//------------------------------
					}
				}
			}
		}
	}
[/CODE]

The problem with the above is that it is almost guaranteed to generate System.OutOfMemory errors if the dataset is too large.

Here is my enhancement request:

Write an Override for the SetContext function to make the SetContext "permanent", i.e.  
[CODE]SetContext(DataSetSymbols[w], true, true);  //<- the second "true" would reset the context permanently to this symbol,  then the next call ...
SetContext(DataSetSymbols[x], true); //<-   would synch DSS[x} with the prior DSS[w]
[/CODE]

Cone

3/18/2009 2:30 PM

QUOTE:
The problem is, it doesn't work. It doesn't work because when the Context is Set (to True) the two pairs are not Synched with each other, but rather with whichever symbol was clicked to launch the script.

You have to work within the framework of the design. If you click a symbol, then it's the "context symbol".

The solution is to not create an explicit DataSet loop, requiring your to click a symbol. Instead, click on the folder and let the MSB cycle through the symbols, each in its own context.

ss161

3/18/2009 2:55 PM

Sorry, i must not have been clear. Design method 2 above does not click on the symbol, but instead clicks on the folder and runs through each name in MSB as if each name has been clicked (atleast that's how it appears to me). You're right, that should work, and it does for small datasets. It generates the System.OutOfMemory issues that i referenced above for medium sized datasets though.

I am able to use Design 2 to run my scripts through a portfolio of 137 names (it takes about 6 hours), but i can't run it through a portfolio of 1500 names.

-- which is why i want to Create Data Sources based on Industry SubGroups programatically (/Forum/Posts/Generate-DataSources-programatically-based-on-Indu-29055), but since there is no API i can't do that. Atleast, i haven't invested the hours of time yet to do what documentation could have me complete in about 10 minutes (I did call Fidelity). I do think the optional ability to "reset" permanently the context would be very helpful.

Cone

3/18/2009 5:44 PM

The documentation for the Static Provider won't help you create subgroups - it's easy enough to do that without creating a completely new data provider as I suggested in that thread, but I like the idea of using Dave Aronow's component for the job if you can figure it out.

QUOTE:
reset the context permanently to this symbol

What do you mean by that?

Based on that statement and looking at the logic in your snippets, I'm not entirely sure that you're using SetContext for the right reason(s). The ClickedSymbol is the "permanent" context symbol (primary symbol) to which secondary symbols will be synchronized when their data are referenced by calling SetContext or GetExternalSymbol - provided that you RestoreContext! In that sense, the context is "permanent" for the ClickedSymbol for the duration of script execution.

SetContext is used for secondary symbols in order to create indicators (with pre or post synch'd data) using the Bars, Open, Close, Volume, etc. syntax as well as to open Positions for those symbols. Once you've SetContext for a secondary symbol and you're done operating in that context, you should return to the primary symbol context by calling RestoreContext().

Using RestoreContext may help somewhat with the memory issue (probably not enough), but maybe the explanation makes your request (which you should call in to Fidelity) not applicable.

Related: WealthScript Programming Guide: DataSeries > Accessing Secondary Symbols > Secondary Series Synchronization

ss161

3/24/2009 8:25 PM

a much delayed response:

QUOTE:
What do you mean by that?

here is a hypothetical example of how i'd like it to work, and why i think it would be useful:

CODE:
Please log in to see this code.

SetContext( sym, true/false ); would work exactly like it does now.
SetContext( sym, t/f, true ); would act to reset the primary symbol to sym and and have subsequent calls to RestoreContext) restore the context to it, not the previously selected symbol on which the script was clicked to initiate the run.

Why would this be helpful?
MSB is extremely efficient and fast. It seems to load all the data in the dataset into memory first, and then execute repeatedly execute the script over and over again on the cached data. As I indicated, on backtests that I run, I'm constantly running into memory issues. I'm not sure if the picture will load, but even on a 35 symbol dataset (granted many of the symbols have between 100k and 500k 1 minute bars) backtest ran out of memory on the first symbol (see below if the picture loaded, but i'm sure you get what i mean even if it didn't).

If that (seemingly minor) enhancement to SetContext was made, then a SSB script could be designed that would have the script loop through the entire watchlist. I assume since the watchlist looping would be done in SSB mode, the entire watchlist would not be loaded into memory at the start, instead, as each symbol was accessed it would be loaded into memory. That way, by using Bars.Cache.Clear memory would be freed up and my guess is that the script would not generate the same System.OutOfMemory issues (note, my computer is a few years old, but does have 2 gig of memory).

Thanks,

Steve

Size:

Color:

a much delayed response:

[QUOTE]What do you mean by that?[/QUOTE]

here is a hypothetical example of how i'd like it to work, and why i think it would be useful:
[CODE]// assume a script clicking on the symbol "A"
SetContext("SPY", true);   		// Symbol SPY is synched with "A"
DataSeries spy = Close;
RestoreContext();				// Context back to symbol "A"

SetContext("C", false, true);	// in the example, the context would be set to "C"
								// C would NOT be synched back to A, and the second boolean
								// true, would reset the primary symbol to C going forward
								//
SetContext("SPY", true);   		// Now, Symbol SPY would be synched with "C"
DataSeries spy = Close;			
RestoreContext();				// Context would now be restored to "C"

SetContext("D", false, true);	// in the example, the context would be set to "D"
								// D would NOT be synched back to C, and the second boolean
								// true, would reset the primary symbol to D going forward
								//
SetContext("SPY", true);   		// Now, Symbol SPY would be synched with "D"
DataSeries spy = Close;			
RestoreContext();				// Context would now be restored to "D"
[/CODE]

Why would this be helpful?
MSB is extremely efficient and fast.  It seems to load all the data in the dataset into memory first, and then execute repeatedly execute the script over and over again on the cached data.  As I indicated, on backtests that I run, I'm constantly running into memory issues.  I'm not sure if the picture will load, but even on a 35 symbol dataset (granted many of the symbols have between 100k and 500k 1 minute bars) backtest ran out of memory on the first symbol (see below if the picture loaded, but i'm sure you get what i mean even if it didn't).

[IMG]http://files.iclippy.com/29405/1237817056.jpg[/IMG]

If that (seemingly minor) enhancement to SetContext was made, then a SSB script could be designed that would have the script loop through the entire watchlist.  I assume since the watchlist looping would be done in SSB mode, the entire watchlist would not be loaded into memory at the start, instead, as each symbol was accessed it would be loaded into memory.  That way, by using Bars.Cache.Clear memory would be freed up and my guess is that the script would not generate the same System.OutOfMemory issues (note, my computer is a few years old, but does have 2 gig of memory).

Thanks,

Steve

Cone

3/25/2009 6:41 AM

Whenever you start messing around with something like synchronization, it's not a minor enhancement because it can easily have negative side effects. We've been there before.

QUOTE:
granted many of the symbols have between 100k and 500k 1 minute bars

There's a lot of overhead for backtests, and largely due to the use of double precision numbers that our community repeatedly asked for, memory requirements have gone way up - especially for intraday MSBs. Part of the solution for this is going to be full 64-bit support (we'll soon have that for WL Developer 5, but it will take a while longer for WL Pro 5 since Fidelity needs to rewrite an important legacy component) and a computer with tons of memory.

The fact is that this change that is very very specific to a certain type of backtest that only perhaps 1 or 2% of users may ever do. In fact, we didn't even allow that type of backtest in Version 4 due to the implication on memory - even with single precision numbers.

All that said, I'm absolutely not the one who decides what goes in to the main WL app, so I encourage you to call Fidelity and ask them for this enhancement. If you have a large commission-generating account, it may help you to front run other planned enhancements that generally 90% of the users will actually use. (The "bang for the buck" criteria.)