I have datasets that are not updating on multiple intraday scales during premarket (at least) where my only historical provider is TD Ameritrade. I'm using WL 8 Build 49 and the latest TDA extension. I updated my token, successfully. No event providers are enabled.
Here is an example dataset where I updated the 1-minute scale at 9:00 AM EST 2023/09/26...
As shown, many symbols are not up-to-date. I checked a few symbols in ThinkOrSwim and they do have data later than that shown above. For a couple of the symbols that have a timestamp later than 08:37 AM 2023/09/26 - for one of them I deleted the data from 2023/08/26 forward (and updated the dataset) and the other I streamed it on a chart. For those two symbols the data became up-to-date. But subsequent updates requested from the data manager do nothing.
The problem is also occurring on the 5 scale. Also, for TDA I cleared the internal request tracking info and that had no effect.
Here is an example dataset where I updated the 1-minute scale at 9:00 AM EST 2023/09/26...
As shown, many symbols are not up-to-date. I checked a few symbols in ThinkOrSwim and they do have data later than that shown above. For a couple of the symbols that have a timestamp later than 08:37 AM 2023/09/26 - for one of them I deleted the data from 2023/08/26 forward (and updated the dataset) and the other I streamed it on a chart. For those two symbols the data became up-to-date. But subsequent updates requested from the data manager do nothing.
The problem is also occurring on the 5 scale. Also, for TDA I cleared the internal request tracking info and that had no effect.
Rename
- ago
#1
Which symbols are not up to date here? They all have an end date of 9/26 which is today. This is pre-market. Maybe those symbols just don't have as much pre-market data? Wait until the market opens and then check again. In any event there doesn't appear to be a bug here because we are getting whatever data TD is returning.
All of them are not up-to-date. The scale is one minute, not one day.
- ago
#3
Read edited post above.
In premarket, I checked for data in ThinkOrSwim, and there was data later than that shown in the image I posted - no doubt.
After the market opened (9:30) I did an update and the symbols updated. I waited a few minutes and did another update and that worked.
So, it seems like it may be a problem with premarket.
After the market opened (9:30) I did an update and the symbols updated. I waited a few minutes and did another update and that worked.
So, it seems like it may be a problem with premarket.
- ago
#5
ThankOrSwim uses a different back-end service than the TD API uses, so we expect differences.
For one symbol, using the data manager, I deleted the data from 2023/08/26 forward. I did an update in the data manager, and that brought the symbol up-to-date.
For another symbol, I created a chart and enabled streaming, and that brought the symbol up-to-date. (I believe it was behind by about 9 or 10 bars).
For another symbol, I created a chart and enabled streaming, and that brought the symbol up-to-date. (I believe it was behind by about 9 or 10 bars).
Data updates are 1 symbol at a time. If you're updating hundreds of intraday symbols, some will update minutes before the others.
What's the problem with that exactly?
What's the problem with that exactly?
@Cone - the dataset is 16 symbols. For the one minute scale, that dataset typically updates in less than 10 seconds.
Please refer to my original post for details.
As noted, it seems to be a problem in premarket data/updates. Assuming I have time Wednesday, 2023-09-27 during premarket, which is likely, I'll perform additional tests.
Please refer to my original post for details.
As noted, it seems to be a problem in premarket data/updates. Assuming I have time Wednesday, 2023-09-27 during premarket, which is likely, I'll perform additional tests.
- ago
#9
If you find an issue with pre market TD historical data you might also want to contact TD about the issue. We’re simply consuming what data they publish via their API.
@Glitch - thanks, I understand. This morning (2023-09-27), in premarket, I did find some differences between what TDA is publishing for data via their API and what appears in ThinkOrSwim. Who knows other than TDA why they do that, but it is what it is.
In any case, I did perform additional testing this morning (2023-09-27) as I stated I would. What I found is behavior of WL8 that may indicate an issue. For the following tests I used only a small dataset consisting of symbols RBLX and U. All tests were performed using the one minute scale. All data was retrieved via the Data Manager. I have only TD Ameritrade checked as a provider.
On 2023-09-26 I updated the dataset and took a snapshot:
On 2023-09-27 07:43 AM EDT I updated the dataset. The symbols updated fine...
I waited until 2023-09-27 08:09 AM EDT (26 minutes) to perform another update. ThinkOrSwim showed that additional bars had been generated for both symbols. However, when I updated the dataset, via the Data Manager, no changes occurred as shown in the following snapshot. (I understand there are differences between what TDA is pumping out via the API and ThinkOrSwim.)
I waited until 2023-09-27 08:21 AM EDT and performed another dataset update. Again, nothing changed...
So, for both symbols in the dataset, I truncated all of the data for 2023-09-27, hence resetting it back to 2023-09-26 as shown here...
After capturing the above image, I performed another dataset update. Both symbols came up to date to 2023-09-27 08:28 AM EDT...
I took a snapshot of the RBLX one minute chart and it shows bars after 2023-09-27 07:43 AM EDT...
So, it seems like a certain amount of time must pass since the most recent update before an update of the dataset will occur. If you're using the TDA Price History method then maybe the issue is with the startDate parameter (just a guess). If WL8 is ok then I guess TDA is not too friendly with publishing data via their API in premarket :(.
In any case, I did perform additional testing this morning (2023-09-27) as I stated I would. What I found is behavior of WL8 that may indicate an issue. For the following tests I used only a small dataset consisting of symbols RBLX and U. All tests were performed using the one minute scale. All data was retrieved via the Data Manager. I have only TD Ameritrade checked as a provider.
On 2023-09-26 I updated the dataset and took a snapshot:
On 2023-09-27 07:43 AM EDT I updated the dataset. The symbols updated fine...
I waited until 2023-09-27 08:09 AM EDT (26 minutes) to perform another update. ThinkOrSwim showed that additional bars had been generated for both symbols. However, when I updated the dataset, via the Data Manager, no changes occurred as shown in the following snapshot. (I understand there are differences between what TDA is pumping out via the API and ThinkOrSwim.)
I waited until 2023-09-27 08:21 AM EDT and performed another dataset update. Again, nothing changed...
So, for both symbols in the dataset, I truncated all of the data for 2023-09-27, hence resetting it back to 2023-09-26 as shown here...
After capturing the above image, I performed another dataset update. Both symbols came up to date to 2023-09-27 08:28 AM EDT...
I took a snapshot of the RBLX one minute chart and it shows bars after 2023-09-27 07:43 AM EDT...
So, it seems like a certain amount of time must pass since the most recent update before an update of the dataset will occur. If you're using the TDA Price History method then maybe the issue is with the startDate parameter (just a guess). If WL8 is ok then I guess TDA is not too friendly with publishing data via their API in premarket :(.
QUOTE:Did you click off the DataSet or change scales?
However, when I updated the dataset, via the Data Manager, no changes occurred as shown in the following snapshot.
I'm pretty sure that time won't refresh even after an update if you didn't change something first. In any case, I'd be looking at charts if you really want to analyze something.
I don't know if I clicked off the dataset. I did not change scale.
Tomorrow (2023-09-28) I'll try to repeat the test in premarket. But, I'll click off the dataset, wait until numerous bars appear in TOS, go back to the dataset, and do an update. If the times do not change then I'll display a chart. I'll let you know what happens.
It seems like I did that the other day, and no new bars appeared in the chart (there were additional bars in TOS). But, I'm not sure, so I'll try again.
Tomorrow (2023-09-28) I'll try to repeat the test in premarket. But, I'll click off the dataset, wait until numerous bars appear in TOS, go back to the dataset, and do an update. If the times do not change then I'll display a chart. I'll let you know what happens.
It seems like I did that the other day, and no new bars appeared in the chart (there were additional bars in TOS). But, I'm not sure, so I'll try again.
Additional tests show the same issue. Per Cone, I clicked off the dataset and that made no difference.
Here's the info on what I captured. The dataset contains two symbols, RBLX and U. Tests are performed premarket using one minute scale in the data manager. TD Ameritrade is the only data source.
At 2023-09-28 08:08 AM I did an update. The dataset updated from data that was most recent as of one day before. The dataset updated a-ok...
I took a snapshot of the RBLX.QX file properties. Modified date is as expected. Same for U.QX (not shown here)...
Approximately 30 minutes later, in the data manager, I clicked on another dataset, came back to the test dataset and performed an update. There was no change. The .QX files' modified times did not change. ThinkOrSwim showed bars subsequent to 08:08 AM for both symbols. (I understand what Glitch noted about API versus TOS, but there were numerous new bars in TOS, and as you'll see later in this test sequence they do get captured, but forcibly for RBLX).
Next step - I truncated the data for symbol RBLX. The "Last Date" went to the day before as expected...
And RBLX.QX was modified as expected...
I clicked on another dataset, came back to the test dataset and performed an update. RBLX updated, but U did not...
RBLX.QX properties showed the file was updated...
Clicking on another dataset and going back to test dataset showed the same as above. U was not up-to-date.
So, I double-clicked on U (in the data manager) to bring up a chart. I switched the scale to one minute and no new bars appeared.
So, it seems it as something to do with (TD Ameritrade) premarket and the amount of time that has elapsed since the last update. I'll take a guess here... the API call is including a startDate. Either the startDate is wrong in the request OR TDA is misreporting the data if the startDate is too close to the current time. If its the latter, that would stink because then you have to back-off to some "safe" startDate, and then do a data merge. Again, just a guess, not having the code.
Here's the info on what I captured. The dataset contains two symbols, RBLX and U. Tests are performed premarket using one minute scale in the data manager. TD Ameritrade is the only data source.
At 2023-09-28 08:08 AM I did an update. The dataset updated from data that was most recent as of one day before. The dataset updated a-ok...
I took a snapshot of the RBLX.QX file properties. Modified date is as expected. Same for U.QX (not shown here)...
Approximately 30 minutes later, in the data manager, I clicked on another dataset, came back to the test dataset and performed an update. There was no change. The .QX files' modified times did not change. ThinkOrSwim showed bars subsequent to 08:08 AM for both symbols. (I understand what Glitch noted about API versus TOS, but there were numerous new bars in TOS, and as you'll see later in this test sequence they do get captured, but forcibly for RBLX).
Next step - I truncated the data for symbol RBLX. The "Last Date" went to the day before as expected...
And RBLX.QX was modified as expected...
I clicked on another dataset, came back to the test dataset and performed an update. RBLX updated, but U did not...
RBLX.QX properties showed the file was updated...
Clicking on another dataset and going back to test dataset showed the same as above. U was not up-to-date.
So, I double-clicked on U (in the data manager) to bring up a chart. I switched the scale to one minute and no new bars appeared.
So, it seems it as something to do with (TD Ameritrade) premarket and the amount of time that has elapsed since the last update. I'll take a guess here... the API call is including a startDate. Either the startDate is wrong in the request OR TDA is misreporting the data if the startDate is too close to the current time. If its the latter, that would stink because then you have to back-off to some "safe" startDate, and then do a data merge. Again, just a guess, not having the code.
I'll try to remember to take a closer look tomorrow morning to figure out what's happening.
Has the TDA data update issue been resolved? I have a similar issue with all TDA scales (D, 5m, 1m). As suggested above - have truncated some bars of symbol FAZ, then updated without success. see pic below for daily data
As the Source column is cut off we can't really be sure that this data belongs to TDA.
$VIX.X is TDA format
More interesting would be to know if any folks have been able to update TDA data recently. I have not found any info by TDA suggesting a stop to their API data
More interesting would be to know if any folks have been able to update TDA data recently. I have not found any info by TDA suggesting a stop to their API data
As of Saturday 2023-11-04 08:51 AM U.S. EDT, it seems that TD Ameritrade may be having issues and/or is in the middle of an upgrade.
I too experienced the problem that mb5413 described at the time noted above. Datasets, specifically tied to TD Ameritrade, do not update at the one minute scale (at least). Some of the last dates on the stocks show dates from many days ago. The stocks are very commonly traded (e.g. CVNA, PTON, RBLX to name a few). The update of the dataset cruises along as if it is updating, but the data is not updated. No errors appear in the log. Also, I brought up a chart and hit the refresh button and the chart did not update (the last bar was from yesterday, early.) Turning on streaming didn't update the chart.
However, I attempted to login to TD Ameritrade via ThinkOrSwim and it just sits stuck at the "Installing Updates" message for 10 minutes. So, looks like a TD Ameritrade issue at this time (2023-11-04 08:51 AM U.S. EDT).
I too experienced the problem that mb5413 described at the time noted above. Datasets, specifically tied to TD Ameritrade, do not update at the one minute scale (at least). Some of the last dates on the stocks show dates from many days ago. The stocks are very commonly traded (e.g. CVNA, PTON, RBLX to name a few). The update of the dataset cruises along as if it is updating, but the data is not updated. No errors appear in the log. Also, I brought up a chart and hit the refresh button and the chart did not update (the last bar was from yesterday, early.) Turning on streaming didn't update the chart.
However, I attempted to login to TD Ameritrade via ThinkOrSwim and it just sits stuck at the "Installing Updates" message for 10 minutes. So, looks like a TD Ameritrade issue at this time (2023-11-04 08:51 AM U.S. EDT).
thank you Paul1986 - TDA updates are working now
Your Response
Post
Edit Post
Login is required