Strategy Monitor lingers during execution or data updates in 6.9.22+

ronc

#1

4/27/2020 5:08 PM

My strategy (code below) writes the latest bar data for all symbols in a dataset to a text file. The dataset contains 6 symbols. Before running the strategy in Strategy Monitor I manually update the dataset via Data Manager. The objective: on 1-minute intervals, overwite the text file with the latest bar data. The Strategy Activation Settings are:
Account: PaperAccount1
Data Range: 1 Bars
Position Size: $5000 (RP)
Scale: 1 Minute
No boxes at bottom of Activation Settings are checked.
No alerts or autostaging are used (the strategy does not trade; it just downloads data each minute.).

My expectation was that the data would download within a few seconds. However, it takes approximately 1 minute to download the latest bar for 6 symbols, by which time of course the data is out of date.

The Strategy Monitor Action column typically displays, at each cycle, messages such as:
Entering mointor thread
Execute <symbol name> <strategy name> (lingers for ~5 to 20 seconds)
Update Completed(2) <symbol name> (5 symbols left) (lingers for ~5 to 20 seconds)
Leaving streaming filter monitor thread (lingers for several seconds)

This is not due to network connection speed. My speed is about 200 Mbps down, 11 Mbps up.

I have tried varying the Strategy Monitor settings but have observed no change.

In Data Manager, the data provider is Fidelity.

What, if anything, am I doing wrong?

Thanks in advance.

CODE:
Please log in to see this code.

Size:

Color:

My strategy (code below) writes the latest bar data for all symbols in a dataset to a text file. The dataset contains 6 symbols. Before running the strategy in Strategy Monitor I manually update the dataset via Data Manager. The objective: on 1-minute intervals, overwite the text file with the latest bar data. The Strategy Activation Settings are:
Account: PaperAccount1
Data Range: 1 Bars
Position Size: $5000 (RP)
Scale: 1 Minute
No boxes at bottom of Activation Settings are checked.
No alerts or autostaging are used (the strategy does not trade; it just downloads data each minute.).

My expectation was that the data would download within a few seconds. However, it takes approximately 1 minute to download the latest bar for 6 symbols, by which time of course the data is out of date.

The Strategy Monitor Action column typically displays, at each cycle, messages such as:
  Entering mointor thread
  Execute <symbol name> <strategy name>  (lingers for ~5 to 20 seconds)
  Update Completed(2) <symbol name> (5 symbols left)  (lingers for ~5 to 20 seconds)
  Leaving streaming filter monitor thread (lingers for several seconds)

This is not due to network connection speed. My speed is about 200 Mbps down, 11 Mbps up.

I have tried varying the Strategy Monitor settings but have observed no change.

In Data Manager, the data provider is Fidelity.

What, if anything, am I doing wrong?

Thanks in advance.

[code]
using System;
using System.Collections.Generic;
using System.Text;
using WealthLab;
using System.IO;
using WealthLab.Indicators;

namespace WealthLab.Strategies
{
	public class RT : WealthScript
	{
		protected override void Execute()
		{
			// Write current bars for symbols in dataset
			const string sep = ",";
			const string fmt = "0.00########";
			string datefmt = "s";
			List<string> datalist = new List<string>();
			foreach (string symbol in DataSetSymbols)
			{
				SetContext(symbol, true);
				Bars currentSymbolBars = Bars;
				int bar = Bars.Count - 1;
				{
					string csv = symbol + sep
						+ Date[bar].ToString(datefmt) + sep
						+ Open[bar].ToString(fmt) + sep
						+ High[bar].ToString(fmt) + sep
						+ Low[bar].ToString(fmt) + sep
						+ Close[bar].ToString(fmt) + sep
						+ Volume[bar].ToString("0");
					datalist.Add(csv);
				}
			}
			string barFile = "Z:barFile.txt";
			File.WriteAllLines(barFile, datalist);
			RestoreContext();
		}
	}
}[/code]

Eugene

#2

4/27/2020 5:24 PM

Please attach the complete SM log file.

superticker

#3

4/27/2020 5:49 PM

CODE:
Please log in to see this code.

There's a typo in your Windows file name. Try:

CODE:
Please log in to see this code.

Eugene

#4

4/27/2020 6:14 PM

It should work either way.

ronc

#5

4/27/2020 8:00 PM

SM log is attached.
Since first observing the issue I:
- Updated WLP to v 6.9.22.7. Issue persist.
- Changed indices to include period at start of symbol name, e.g. .SPX

Note that in the log, at 12:54:06, I changed strategy settings. I was trying most recent 5 bars and then changed to 1 bar.

SM log.txt

Eugene

#6

4/27/2020 8:40 PM

What is disk Z exactly?

Is your PC clock in sync?

ronc

#7

4/27/2020 8:59 PM

PC clock is in sync. Date/time settings say the last successful sync was 4/27/2020 9:57:25 AM, time server = time.windows.com

Disk Z: is a shared drive shared with my Mac. I run WLP on Windows in a virtual PC in VirtualBox. Physically, all storage is SSD. Z: is not a separate physical drive; the virtual PC sees it as a drive and the Mac sees it as a directory.

The Windows time and Mac time are in sync.

The reason for the shared drive is I have Python code that runs on Linux/Unix (including Mac) that reads the file containing the bars, computes, and returns another file which will be read by other code in WLP.

I thought this could be a performance problem due to using a virtual PC vs. native but I've never observed any performance problems before, everything else in WLP runs fine, e.g. streaming charts, etc.

I could do a test that does not involve writing the files to Z: to see if that speeds things up. But of course without the output file the whole scheme won't work.

If the Z: drive is a potential root cause, I think I can share the entire C: drive with my Mac/Python code. But that might just cause the same problem in C;.

Cone

#8

4/28/2020 7:29 PM

We're doing some testing with the new Fidelity provider. I'm seeing data updated very quickly, but it's followed by "lingering" and suspicious gaps in which it appears not to be processing anything. We'll look into it.

In the time being, if you're trading fewer than 15 or even 20 symbols, use Streaming Windows for minimal delay.

ronc

#9

4/28/2020 9:03 PM

Thanks Cone. Yes, I am using only 6 symbols at this time.

Regarding Streaming Windows, I am looking at User Guide p94-95 but am not clear how this should work.

My objective is to write the bar data for each symbol to a text file (at each 1-minute bar).

User Guide p94 says that each new bar in the Streaming Chart triggers a strategy in the Strategy Window. Does this mean that I should no longer use the Strategy Monitor to trigger the strategy?

Thanks again.

Cone

#10

4/29/2020 10:30 AM

For sure, with only 6 symbols, open a strategy once for each symbol. Save the Workspace to open it again in a new session.

QUOTE:
Does this mean that I should no longer use the Strategy Monitor to trigger the strategy?

Right, the S. Monitor is not required for that. You shouldn't have a problem, though, using either the S. Monitor or Streaming Windows. If delay is not an issue, the S. Monitor is more convenient - but only because you don't have to set up a Workspace for it.

Now, that I'm looking at what you're doing - accessing static data from external symbols - this is actually not straightforward to do from either tool because it assumes the data are available for the other symbols when the script's symbol is triggered. Bad assumption.

The only way that I know to be successful is to use the 2-script technique outlined here:
http://www2.wealth-lab.com/wl5wiki/kbOptimalTradingTool.ashx

Otherwise, the very simple way is for each symbol to write it's own data to a separate file:

Note! This script will create and save data in a directory called: C:\Data\ASCII\ronc

CODE:
Please log in to see this code.

But if you need a solution for all the symbols in the same file, let me know. We can modify the script from the wiki to do that.

Size:

Color:

For sure, with only 6 symbols, open a strategy once for each symbol.  Save the Workspace to open it again in a new session.

[quote]Does this mean that I should no longer use the Strategy Monitor to trigger the strategy?[/quote]Right, the S. Monitor is not required for that.  You shouldn't have a problem, though, using either the S. Monitor or Streaming Windows.  If delay is not an issue, the S. Monitor is more convenient - but only because you don't have to set up a Workspace for it.

Now, that I'm looking at what you're doing - accessing static data from external symbols - this is actually not straightforward to do from either tool because it assumes the data are available for the other symbols when the script's symbol is triggered.  Bad assumption.

The only way that I know to be successful is to use the 2-script technique outlined here: 
[link]http://www2.wealth-lab.com/wl5wiki/kbOptimalTradingTool.ashx[/link]

Otherwise, the very simple way is for each symbol to write it's own data to a separate file:

Note! This script will create and save data in a directory called: C:\Data\ASCII\ronc
[code]using System;
using System.Collections.Generic;
using WealthLab;
using System.IO;

namespace WealthLab.Strategies
{
	public class RT : WealthScript
	{
		protected override void Execute()
		{
			// Write current bar for symbol
			const string dir = @"C:\Data\ASCII\ronc";
			List<string> datalist = new List<string>();
			
			int bar = Bars.Count - 1;
			string csv = string.Format("{0},{1:s},{2:0.00##},{3:0.00##},{4:0.00##},{5:0.00##},{6:0}", 
				Bars.Symbol, Date[bar], Open[bar], High[bar], Low[bar], Close[bar], Volume[bar]);
			datalist.Add(csv);
			
			if (!Directory.Exists(dir))
				Directory.CreateDirectory(dir);
			
			string filename = string.Format("{0}.txt", Bars.Symbol);
			string fullpath = Path.Combine(dir, filename);
			File.WriteAllLines(fullpath, datalist);
		}
	}
}[/code]

But if you need a solution for all the symbols in the same file, let me know. We can modify the script from the wiki to do that.