Datasource Truncation utility (Scrub Fidelity) for WLP5?
Author: Sammy_G
Creation Date: 5/31/2010 12:58 PM
profile picture

Sammy_G

#1
Hi Cone,

I am a long time WLPro 4.5 user, now just installed v5.6. I notice that there doesn't appear to any equivalent of your very useful ScrubFidelity utility for v5. Are you planning to update it soon for v5? For the record, it saved my goose on more than one occasion in v4.5. You don't need that utility every day, but when you need it, boy you *really* need it!
profile picture

Cone

#2
If I recall, the problem for which that app was built was to correct a temporary server problem that caused weekend bars to be inserted in the data. Since that problem no longer exists, I don't think it anyone needs to develop a solution to work around it.
profile picture

Sammy_G

#3
Cone, I remember the original reason, but later on you expanded the utility to be able to truncate a Fidelity datasource starting on a specified date, and its *that* functionality I was thinking about. You may recall the Flash Crash of 5/6/10. A lot of those prices, dutifully reported by data providers, were completely bogus. Later on, some of the trades were busted (it took some time for that to happen) but data had already been downloaded by users everywhere. Even though sometimes data corrections are made by the provider during data download (e.g. Fidelity may correct upto 1 previous bar of daily data), its an open question when they actually implemented all of the corrections into their own database. So if one is past the data correction window of the provider, you are stuck with the bad data. What do you do in this situation? Its not practical to manually delete one or more bars of data when you have > 10,000 securities in a dataset. Clearly, you have to use a truncation utility like ScrubFidelity.
profile picture

Cone

#4
Here's something quick and dirty for you.

NOTES!
o To avoid inadvertent operation, or multiple runs in an MSB, the code works only by clicking on the first symbol of any DataSet.

1. Works only on Fidelity DataSets that are in the default UserApp location.
2. Backup your data first. Who knows if this will work for all cases?!
3. Edit the truncate data in the Execute procedure.
4. The code will ask to confirm each DataSet before it truncates it. You can Cancel the entire process if required
5. Because of read and write times for large symbol files, it's expected to take a significant amount of time to complete the process for intraday DataSets - it could take hours for those. The code below has a switch to only process Daily DataSets; so you have to edit that to process intraday DataSets.
6. Cross your fingers! (Did you back up your data?)
7. When finished, check the Debug window for messages.

Suggestion: Turn off On-demand data so that charts don't update immediately when you go back to check their last date.

CODE:
Please log in to see this code.
profile picture

Sammy_G

#5
Thanks, Cone, I will test drive it soon.

My ignorance of v5 script language forces me to ask: Does it have a built-in safeguard, like the
CODE:
Please log in to see this code.
argument in v4.5?
profile picture

Eugene

#6
Nice code Robert.

QUOTE:
{$NO_AUTO_EXECUTE}

You will find the answer in the Knowledge Base section of Wealth-Lab Wiki (click Support > Knowledge Base to get there.) There's an article.
profile picture

Cone

#7
Hi Eugene. That code has a little of you and Aleksey in it too. Don't write what you can't borrow or steal :)

As for the safeguard, yes, see Note #4. It asks you to confirm for each DataSet and you can cancel the entire process immediately.
profile picture

Sammy_G

#8
Based on the KB article, I made the following change to the code, as a double safeguard:
CODE:
Please log in to see this code.
If its wrong let me know, but it appears to be OK.

Another Q on the side: How do you get the code to align nicely after editing?
profile picture

Eugene

#9
QUOTE:
If its wrong let me know, but it appears to be OK.

Looks OK.
QUOTE:
Another Q on the side: How do you get the code to align nicely after editing?

For example, you can force the editor to re-align a code block by selecting and typing in one of the curly braces (replace triggers realignment).
profile picture

Sammy_G

#10
Any curly brace would do or one just above/below the misaligned code? Any keyboard shortcut to force realign?
BTW, neat tip.
profile picture

Eugene

#11
QUOTE:
Any curly brace would do or one just above/below the misaligned code?

It's the curly brace below. And the lower the brace, the more misaligned blocks above it fixes.
QUOTE:
Any keyboard shortcut to force realign?

Please let me know if you find one - would help me too.
profile picture

Sammy_G

#12
Thanks for the tip, Eugene.
profile picture

Eugene

#13