Strategy Monitor not getting new Fidelity 1-minute data
Author: kazuna
Creation Date: 3/2/2020 3:27 PM
profile picture

kazuna

#1
It's been working fine until this morning but all 1-minute scale strategy in Strategy Monitor not getting new bar data.

At 0th second, the strategy turns into red with Action tab saying "Entering Momitor Thread", then at 5th second, it turns into blue with Action tab saying "Leaving Streaming Filter Monitor Thread". When it's been working, it goes to leaving state at around 50th second, I think.

SCLog.txt looks lie this.
CODE:
Please log in to see this code.
The thread is leaving almost immediately without getting the bar update.

Any idea what's going on?
I'm seeing this problem on all machines this morning.

What changed this morning is, instead of having 50 threads updating a single symbol, I have one thread updating 50 symbols.
Once I restart WLP, this problem goes away and Strategy Monitor starts getting bar data as usual by waiting for monitoring thread to leave until 50th second.
It sounds like something bad happens to "Monitor Thread" while updating many symbols at the very first bar of the day and "Monitor Thread" stays in the bad state until WLP is restarted?
profile picture

Cone

#2
I'll take a look now, but I stress over and over, make sure you restart WLPro within and hour or two before the market open. That streaming connection "keep alive" and "reconnect" is not working as it should.

There is a new streaming provider in the works.. and its signature was inadvertently included in the Preferences > Streaming. For now, make sure that the ".. Websocket Data" option is NOT selected.
profile picture

kazuna

#3
QUOTE:
you restart WLPro within and hour or two before the market open
I start WLP 10 minutes before the market open on all machines every morning. So the streaming connection should have been working fine unless there was something going wrong on Fidelity side.

QUOTE:
its signature was inadvertently included in the Preferences > Streaming. For now, make sure that the ".. Websocket Data" option is NOT selected.
I cannot find it on 6.9.22.7. Are you talking about upcoming version?

One thing I noticed when I restarted WLP while Strategy Monitor was in the bad state is the strategy seems downloading symbols at the first run and it's taking long time for more than a minute, resulting the next update being skipped.

I'm just guessing but if the very first run misses the update, the "Monitor Thread" may get in the bad state?
profile picture

Cone

#4
5 minute updates worked normally for me. Maybe you were hitting a bad server?
profile picture

kazuna

#5
QUOTE:
5 minute updates worked normally for me.
I guess it has to be 1-minute updates and it needs to run from the first bar (9:31) of the day.

QUOTE:
Maybe you were hitting a bad server?
Very unlikely. Because I have 7 machines (4 physical and 3 virtual) running with exactly the same WLP configuration and they all exhibit exactly the same problem.

Could you try this setup to see if you can duplicate the problem?
- Create a 1-minute scale dataset consisting 50 symbols.
- Add a strategy (simple one is fine, mine is just to call SetGlobal with the last close price) to Strategy Monitor.
- The strategy is activated with the dataset, 1-day range and 1-minute scale.
- Account can be either paper account or fidelity account.
- Observe from the first bar of the day at 9:31.

If you have more than one system to test, you may also try larger dataset like 100 symbols and 200 symbols to accelerate the problem.
profile picture

Eugene

#6
QUOTE:
Very unlikely. Because I have 7 machines

I'd say it's rather the number of your network connections that counts than the number of machines. If they all share the same IP address they may be hitting the same "naughty" server.
profile picture

kazuna

#7
There apparently something going wrong with Fidelity since Monday morning?
Regardless of the dataset sizes (I tested 8, 10, 15, 25, 50 symbols), all 1-minute scale strategies in Strategy Monitor on all 7 machines are running into the problem this morning.

At 9:20am, WLP started.
At 9:30am, Strategy Monitor ran daily scale strategy with the data updated, no problem.
At 9:31am, Strategy Monitor didn't run 1-minute scale strategy because the data update didn't complete (monitoring thread leaving in 5 seconds), this is the problem.
(the problem continued every minute)
At 10:00am, WLP restarted
At 10:01am, Strategy Monitor ran 1-minute scale strategy with the data updated, no problem.
(the problem went away once WLP restarted)

Did you have a chance to test it this morning?
profile picture

kazuna

#8
It's been consecutive three days since Monday, I'm running in this problem.
Is anyone else having the same problem?
profile picture

Cone

#9
I didn't run it today, but didn't notice a problem yesterday.

Please make sure that you have not selected the "Websocket Data " option in Preferences > Streaming Data. The websocket selection was inadvertently displayed in 6.9.22 and is currently in development test.
profile picture

kazuna

#10
No, I have never selected the "Websocket Data" option.
I'm having this problem on both 6.9.20 and 6.9.22, so the option doesn't even exist on 6.9.20.

CODE:
Please log in to see this code.
What makes the monitoring thread leaving in 5 seconds without actually updaing the data?
profile picture

Cone

#11
The only thing that comes to mind is to make sure to synchronize your local clock. I'll take a look today too.
profile picture

Cone

#12
I'm not seeing a problem here. 1-minute updates, 5-minute updates. Appears to be normal operation to me.
profile picture

kazuna

#13
Thank you for looking at it.

I think I found what's causing the problem.

It is due to the data range set to 1 day.
If the data range is set to 2 days or 1 week, the problem doesn't happen.

Apparently when the strategy runs in 1-minute scale with 1-day data range, the very first execution of the day screws up the monitoring thread.
Parhaps, it is because the very first execution not having the previous bar?

That may explain why restating WLP fixes the problem because the very first execution will have previous bars at that time.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).