Last-date-value-repeating-on-external-symbol

Looks like synchronization at work. Check if the symbol has a missing date (reload if necessary).

Your index symbol is likely coming from Yahoo or another provider (open a chart for that symbol to check.)

Certain providers update their data much later, so the only solution is to wait until the data is updated.

@Eugene I presume you mean the index symbol (^GSPC)? VIX is external (not stored locally), so I don't think I can update it. I used Data Manager to update folder containing ^GSPC. Is that the reload you wanted me to try?

No change after reload. Not surprised because this issue is not limited to this symbol.

QUOTE:
Often times Yahoo won't update to Friday's prices until Monday.

@superticker You may be right. I just tried again, and it appears to be working OK now. I will monitor on other days of the week to see if Monday is unique. Thanks.

^GSPC 1/18/2022 22.790001
^GSPC 1/19/2022 23.85
^GSPC 1/20/2022 25.59
^GSPC 1/21/2022 28.85

I've been tracking the @superticker theory about Yahoo's late update of certain daily symbols. This morning, my strategy again did not get yesterday's values for VIX or VXN. I figured I'd wait it out and see if Yahoo updated later in the day.

An hour before market close it was still not updating. I started getting suspicious about the theory. I know I did something to get this working in the past but didn't know what.

I tried a V7 restart. No change. I went to the symbol manager. However, these volatility indices are external symbol in my strategy, so there's nothing the DataManager can do with them (I don't think.)

Then I opened a chart window and charted VIX. The chart had yesterday's data. So, I ran my strategy and sure enough it resolved the VIX data properly in my strategy. But not VXN. So, I repeated this process for VXN and the strategy ran correctly for both symbols.

It appears that DataSet symbols and external symbols are treated differently when it comes to updating. (Which makes sense, since external symbols don't need to be updated because they are not stored locally. Right?) I would expect external symbols to fetch whatever data is available from the provider. Or are they cached locally, even after a V7 restart?

Would appreciate any clarity on this behavior and how to avoid it. Should I be saving my volatility symbols in a DataSet instead of making them external?

They're not treated differently at all actually.

Interesting. So where are external symbols "stored"? This implies that the price data for external symbols is persistent and needs regular updating by the program? Any idea why that's not happening for my strategy?

Each data provider has their own folder under your WL7 user data folder (file menu.) All of the symbols are stored there. The data update and persistence doesn't know or care if the symbol is part of any DataSet or not, external or not, or any of those kinds of factors.

I see ^VIX.QX and ^VNX.QX in my Yahoo Finance > Daily folder. I think that's what you are referring to.

When a strategy executes, each symbol in the DataSet is updated automatically, correct? When are external symbols that are populated in code updated? Should using

CODE:
            vix = SymbolData.Series(bars, "^VIX", PriceComponents.Close);

force an update or is there some other function I should execute?

QUOTE:
This morning, my strategy again did not get yesterday's values for VIX or VXN. I figured I'd wait it out and see if Yahoo updated

Yahoo is slow to update. In the middle of the week, I usually don't trying updating Yahoo data until after midnight. (I'm on CST Central Time).

For your indexes like VIX and VXN, you might consider using Q-Data's $VIX and $VXN instead of Yahoo. It's free as well for EOD prices.

Even the CBOE website itself may delay to post a daily update for up to a day on their Put/Call Ratios (including the VIX P/C).

QUOTE:
Yahoo is slow to update. In the middle of the week, I usually don't trying updating Yahoo data until after midnight.

@superticker. This was well after midnight, more than 15 hours after midnight.

Thanks for the tip on the alternate data provider. I changed symbol in code to Q-Data compatible ($) and enabled Q-Data provider, ran, and it worked (for now). I will have to wait until tomorrow to see if Q-Data has the same update issue.

CODE:
Even the CBOE website itself may delay to post a daily update for up to a day on their Put/Call Ratios (including the VIX P/C). 

@Eugene But as I wrote above, I am able to see up-to-date data if I open a chart of the symbol. And after I do that the data is available to my program. Doesn't that indicate that it is available from the provider?

It seems like V7 is using the locally cached out-of-date data instead of refreshing as it does with symbols in the DataSet. Charting the symbol appears to refresh the locally cached symbol data.

When I use Data Manager to update all Yahoo daily symbol data, the up-to-date price data become available immediately. Clearly the data is available from the Yahoo provider but V7 is not triggering an update for an "external" (not in active DataSet) symbol.

This is easy to disprove. Here's a screen shot of my Yahoo ^VIX file before running the Strategy below, and a screen shot of it after. You can see it updated the external file.

Are you leaving the same strategy window open and running it hours later perhaps? In this case, WL7 won't reload data because it's already loaded. You can force a reload by either re-opening the strategy or clicking the little message in the status bar that says "N Symbols Loaded."

CODE:

using WealthLab.Backtest;
using WealthLab.Core;
using WealthLab.Indicators;

namespace WealthScript1 
{
    public class MyStrategy : UserStrategyBase
    {
        //Initialize
        public override void Initialize(BarHistory bars)
        {
         IndicatorBase vix = SymbolData.Series(bars, "^VIX", PriceComponents.Close);
         PlotIndicator(vix);
        }

        //Execute
        public override void Execute(BarHistory bars, int idx)
        {
        }
    }
}

QUOTE:
It seems like V7 is using the locally cached out-of-date data instead of refreshing as it does with symbols in the DataSet. Charting the symbol appears to refresh the locally cached symbol data.

I agree this is a "problem" with Yahoo data. But it's also a "feature" built into WL7. WL7 is specifically designed to not re-load data for a given day. But if you wait until after midnight (local time?), it resets itself and Yahoo data is refreshed.

QUOTE:
Are you leaving the same strategy window open and running it hours later perhaps?

Yes I am. Is that a problem? (rhetorical)

QUOTE:
In this case, WL7 won't reload data because it's already loaded. You can force a reload by either re-opening the strategy or clicking the little message in the status bar that says "N Symbols Loaded."

I didn't know about know about clicking the message in the status bar. I'll try that to see if I can get Yahoo to reload mutual fund data prior to midnight. Thanks for the tip!

@Glitch

QUOTE:
This is easy to disprove.

That's a good test. I will try it tomorrow. If mine works and yours fails, I will have a question about what I am doing different in my code than you are.

QUOTE:
Are you leaving the same strategy window open and running it hours later perhaps?

No. Strategy windows are opened fresh each morning and this problem happens even after V7 restart. When are external symbols updated? When the code calls for them as in your example? Strategies appear to update all symbols in the target DataSet before running strategy. I'm wondering when symbols not in DataSet (aka External, like "^VIX" in your example, are updated.

@superticker

QUOTE:
I agree this is a "problem" with Yahoo data. But it's also a "feature" built into WL7. WL7 is specifically designed to not re-load data for a given day. But if you wait until after midnight (local time?), it resets itself and Yahoo data is refreshed.

This strategy is run in the morning, well after midnight. But the question is when are External symbols updated and by what triggers an update?

QUOTE:
when are External symbols updated and by what triggers an update?

I think the update happens after midnight (local time?) and after the strategy is restarted. Both conditions must be in play.

At any rate, when I fresh start WL7 in the morning, I don't have this problem unless it's a Saturday morning. In that case, I need to wait until Monday morning for the Yahoo data to update. I can't speak for Q-Data updates.

QUOTE:
I think the update happens after midnight (local time?) and after the strategy is restarted.

When a strategy is started first time in the morning, V7 appears to update all symbols in the target DataSet for that strategy. I don't think it updates External symbols at that time, but I've yet to get an official confirm on that.

>>When are external symbols updated?<<

They are updated on demand as needed, unless you turned Offline Mode on under the File menu.

I just proved in my example that it does update the external symbol, not sure how to be any clearer?!?

No, it's clear. I was just wondering if there was another time these symbols were automatically updated. I plan to run your program tomorrow if I get a failure.

I don't understand why my program does not yield the same immediate update that yours does. I'm trying to figure out what else I might be doing that's different.

I checked offline mode and it is not selected.

When I ran my strategy this morning using $VIX and $VXN I see the same problem.



I ran @Glitch program above but does not update file

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Indicators;
using System.Drawing;
using System.Collections.Generic;

namespace WealthScript1
{
   public class MyStrategy : UserStrategyBase
   {
      //Initialize
      public override void Initialize(BarHistory bars)
      {
         IndicatorBase vix = SymbolData.Series(bars, "$VIX", PriceComponents.Close);
         PlotIndicator(vix);
      }

      //Execute
      public override void Execute(BarHistory bars, int idx)
      {
      }
   }
}

Must be something in my system configuration then?

You're using QData. On Glitch's screenshot the data is by Yahoo. They get updated at different time of day.

@Eugene here's the results for Yahoo data. Regardless of whether the data has been updated by the provider, shouldn't the file timestamp change?

Before running @Glitch program



After running @Glitch program

CODE:
using WealthLab.Backtest;
using System;
using WealthLab.Core;
using WealthLab.Indicators;
using System.Drawing;
using System.Collections.Generic;

namespace WealthScript1
{
   public class MyStrategy : UserStrategyBase
   {
      //Initialize
      public override void Initialize(BarHistory bars)
      {
         IndicatorBase vix = SymbolData.Series(bars, "^VIX", PriceComponents.Close);
         PlotIndicator(vix);
      }

      //Execute
      public override void Execute(BarHistory bars, int idx)
      {
      }
   }
}

These providers do not update until well into after the market close. Did you expect to see a daily bar for 1/27? You won't see a daily bar for 1/27 until sometime later tonight.

No, I'm expecting data from yesterday 1/26/21 but it's not updating. Here's what I get when I run the program (yours or mine).

Note data for 1/26 is duplicated from 1/25




And the data file (like you showed in your test) timestamp has not changed since this morning, although I've run the program several times.



I'll ask the same question I asked of Eugene: Wouldn't you expect the data folder file timestamp to change, independent of what's going on at the provider?

what do you see if you open a chart of that symbol?

Today's price will be in the chart for ^VIX, and once I display the symbol in a chart, the problem goes away. (Yahoo data file is updated).

I noted the ways I can manually correct the problem each day in Post #14 above.

I truncated ^VIX by Yahoo by one bar and then ran a test using code by Glitch on another symbol recognized by Yahoo. The truncated ^VIX.QX file was NOT updated to include the 1/27/2022 bar. Then I did a 'Clear internal tracking info' which wasn't helpful either.

@Eugene, I'm not sure if this is a response to @superticker or me, but I was looking for 1/26 data yesterday, not 1/27

With this morning's (1/28) run, there is still no update to data files

BEFORE RUN



AFTER RUN



Now I am two days behind in VIX data



And chart is not showing price or indicator data for yesterday

@innertrader, my findings are the same.

@Eugene, and what is your current thinking? Is this a problem with my system setup or a potential issue with WL7?

This is a very strange day. I am unable to get yesterday's (1/27) daily data on *any* symbol. After trying PC restart with no improvement, I gave in and used Data Manager to update Yahoo symbols. As expected, it corrected the "duplication" problem, but still cannot get yesterday's close data.



Inexplicably, I enabled Q-Data and I still can't get daily data from yesterday, even on QQQ. Are you seeing anything like this?

I see that Yahoo historical data at their site for 1/27/22 is "null." I'm guessing Q-Data relies on the same data source.

What an untimely coincidence in the midst of the other data update issues we're trying to resolve.

Hmm, I am getting the 1/27 bars for both ^VIX and ^GSPC from Yahoo in my charts.

QUOTE:
I see that Yahoo historical data at their site for 1/27/22 is "null." I'm guessing Q-Data relies on the same data source.

1 - On Yahoo's website only the downloaded CSV file matters (and not the web page itself), did you check it?
2 - No.

P.S. I wonder whether DrKoch suggested you give finantic.BestData a try or he was talking to somebody else.

Re: QData
We had an undetected error in the update process for 1/27.

My External symbols are still not updating on-demand from C#. I ran my strategy and @Glitch test in Post #15 today (2/1/22), and ^VIX is not updating.



Have you been able to duplicate this problem? What can I try to assist troubleshooting?

I would not expect it to update because the market is still open. I would not expect that symbol to update until later this evening.

@Glitch I'm looking for yesterday's prices, not today's. The data file is not updating (see time stamp in image above) like it did in your test.

Please confirm that the timestamp on data file should be updated when External symbol is accessed in code, regardless of what's happening at provider. Or do I have that wrong?

I have a "duplicate" price that appears to be the result of not having data for yesterday. Here's the chart and debug values. Note duplicated value for yesterday in VIX.





I am repeating what I have already documented several times above.

Here's the ^VIX data downloaded from Yahoo site just now. Clearly yesterday's data is there, but WL7 is not updating file for some reason.



What else can I do to help?

And what if you look at these symbols in a chart? Is yesterday’s date still not updated?

Just open a new chart and enter ^VIX without clicking any DataSet or symbol.

Dion, I just made the following test:

0. Cleared any Yahoo data
1. Created a Yahoo DataSet with ^VIX being the only symbol, and updated it to 1/31/2022
2. Truncated the last bar so the data is up to 1/28
3. Exited WL7
4. Restarted it and ran your code from Post #15 on SPY

Expected: ^VIX is updated to 1/31/22 again
Outcome: "Last date value repeating on external symbol", ^VIX last date: 1/28/2022

can you open a blank chart and type in ^vix?

Now If I open a blank chart the ^VIX does get updated to 1/31/22.

Ok, at least we have a workaround while we investigate this.

QUOTE:
Ok, at least we have a workaround while we investigate this

We've had multiple workarounds for a while (see Post #14). The issue is it will not update on-demand as an external symbol in C# code.

@Glitch test (Post #15) does not work on my PC the way it worked on his. I am gleaning from Post #44 that you have confirmed this behavior on your own system, and you don't require anything further information from me at this time.

Has there been progress identifying the cause of this issue?

Yes, I believe I isolated the cause, and it should be resolved in B50.

Excellent. Thank you. Once again, I'll offer to test a beta if that would be helpful.

QUOTE:
Yes, I believe I isolated the cause, and it should be resolved in B50.

I updated to v.50 yesterday. This morning I ran my strategy that uses External Symbols for volatility indices and, unfortunately, it is showing the same problem. It manifests as yesterday's symbol not updating but repeating the previous day's value. Below are examples of ^GSPC vol index (^VIX) pulled from Yahoo data and EEM vol index ($VXEEM) pulled from Q-Data.

After this failure, I ran the Historical Data Loading Testbed for $VXEEM and it showed data up to date to yesterday (a success result since this is daily data.). I then ran my strategy again and got correct data for $VXEEM (but not for other symbols).








After Running Historical Data Loading Testbed

Thanks for checking, I'll continue to keep this issue under development going forward!

Quick question: you are seeing current values of ^GSPC in the thirties range? That's what I see in your Post #52.

I'm seeing ^GSPC at 4363.49, and my GetHistory call did update it to current date. Perhaps we can schedule a troubleshooting Zoom call to help get the bottom of this? If you're interested write us at support@wealth-lab.com.

When I tried with ^VIX it also updated correctly to yesterday's value:

QUOTE:
Quick question: you are seeing current values of ^GSPC in the thirties range? That's what I see in your Post #52.

I'm seeing ^GSPC at 4363.49, and my GetHistory call did update it to current date.

Yes, please note the value is for ^VIX, which is the volatility index for ^GSPC. But it looks like you already figured that out.

QUOTE:
When I tried with ^VIX it also updated correctly to yesterday's value:

I assume you did this by entering ^VIX into a new chart window? Yes, that is one of the workarounds. There are several other workarounds listed in the above that will cause a refresh of the last daily value and let the strategy will run properly.

The problem, as always, seems to be limited to calling ^VIX as an External Symbol, where it doesn't get the most recent daily value.

CODE:
            vol = SymbolData.Series(bars, "^VIX", PriceComponents.Close);

Post #15 is a test you created for me that does not work the same on my system (duplicates last value) as it did on yours (shows latest value correctly).

I'm happy to have a Zoom call if I can be helpful. Based on Post #44, I thought @Eugene had already confirmed the issue and provided a step-by-step on how to recreate it.

Let me know.

No, I didn’t. I made sure that I had a stale copy saved and then I ran your Strategy which uses SymbolData.

Was WL7 perhaps left running more than 24 hours? Just trying to think of other potential areas to focus on.

I’ll follow that procedure next and report back.

QUOTE:
Was WL7 perhaps left running more than 24 hours?

Not sure. I updated the version yesterday and I believe it was less than 24 hours. At any rate, after I saw the problem, I restarted the application and tried again but that didn't fix it.

Since you seem open to brainstorming on the cause of this issue, here are my thoughts.

This is what I think we know:
1. At the time a fetch of yesterday's daily data fails when called as an external symbol, the data *is* available at the provider(s). We know this because simply charting the symbol, or updating data, updates correctly and the data persists.
2. The problem seems to correct itself (go away) sometime later in the day, possibly 12 hours after you expected data to be available. When you notice the problem has disappeared, it seems to be true for all symbols (even from multiple providers) at that same time.
3. A simple C# program you developed to fetch external symbols works for you but fails for me (one day late).

So, with absolutely no knowledge of the internal workings of the External Symbol code, is it possible we are dealing with an issue related to time or time zones being mis-set? For example, if there is a generic function that checks the date (relying on time zone) and it thinks it's still yesterday *and* this information is used to determine if the external symbol requires updating, possibly it thinks local data is up to date, when it's not? Eventually, as time passes the code detects it's a new day and the local data is stale, so it updates.

I'm wondering if you set your system to Pacific time when you did the testing of my code (or your test).

If that was the case, then the symbol would have not updated in the Data Manager either.

Hmmm.... Have you been able to replicate the issue?

Unfortunately not. With the workaround in place i’m assuming it’s not a critical matter at this point? I’m working nonstop now getting WL8 ready and unless this issue is really holding you back i’d like to keep the momentum going.

I know we will nail this one eventually but it will probably have to wait until after WL8 is released.

@Eugene Post #44 didn't help you to reproduce it? Did @Eugene reproduce it?

QUOTE:
With the workaround in place i’m assuming it’s not a critical matter at this point?

Well, it means I will have to refresh my symbol database every day before running strategies. Otherwise, strategies will not give correct signals. I don't feel like that should be required.

QUOTE:
I’m working nonstop now getting WL8 ready and unless this issue is really holding you back i’d like to keep the momentum going.

I'm assuming the WL team had good reason to shift development to WL8. Is there a discussion anywhere that outlines the urgency?

While WL7 was a tremendous step forward in many ways, I'm feeling like it still has usability issues that have not been resolved and I'm concerned about having to start all over again with WL8.

I recently recorded a video on our Youtube channel explaining the reasons:

https://www.youtube.com/watch?v=9JxJjqevTeU&t=8s