Data Truncation
Author: Sammy_G
Creation Date: 10/8/2010 9:46 AM
profile picture

Sammy_G

#1
Fidelity WLPro 6.x
===================

I downloaded the data for symbol KKR from Fidelity. It was a recent IPO on 7/15/10. However, the d/l data extends back to 1/11/07, meaning its a recycled symbol and the data provider hasn't bothered to fix the data.

Rather than delete the data bar-by-bar prior to 7/15/10 (and develop a carpal tunnel in the process), I was wondering if you guys have a handy (or can create one) data truncation utility that will do the job. A safe version would be preferred that will *only* do the job on the clicked symbol, or a version that can let the user decide if the truncation job is to done on the clicked symbol or an entire dataset (I can move the bad symbols to a safe dataset where the utility can merrily chop away at the data w/o affecting the other datasets).
profile picture

Eugene

#2
This should be sufficient for a single symbol :

Data | Removing data using a WL6 Strategy > Request #1

You just have to modify this condition accordingly:
CODE:
Please log in to see this code.

Like this:
CODE:
Please log in to see this code.

P.S. And this line:
CODE:
Please log in to see this code.

Is better changed to:
CODE:
Please log in to see this code.
profile picture

Sammy_G

#3
Many thanks, Eugene. With the right mods, it worked great; I post the code here for benefit of newbies (suggestion: the code could benefit from having the date input from an input box, and 1-2 warnings thrown in):
CODE:
Please log in to see this code.

============================================

Another data-related question: Is there a script or utility that will identify, and optionally delete (actually, much prefer to have it copy the files to another location - either a dataset or a temp directory - just in case), data files with no data?
I have chunk of those; what had happened was that when I started using WL.NET a few months ago, I copied entire datasets over from WL4; quite a few of those symbols don't trade any more so I have a bunch of these symbols scattered over various datasets.
Note: I already tried the Bad History Data Check utility; it doesn't work with zero-data symbols.
profile picture

Eugene

#4
It's easy:
CODE:
Please log in to see this code.

Now just copy this subset of symbols from the Debug window and paste it in the Modify DataSet window.
profile picture

Sammy_G

#5
I guess one would have to delete symbols from the Quarantine list, and the data files, manually?
profile picture

hankt

#6
Sammy,

I want to just delete bars before my check date rather than write out to a new file. I'm working with Yahoo historical data and they concatenate previously used tickers within 2 companies on 1 ticker.

I can't find any documentation for Bars.Delete
profile picture

Eugene

#7
Bars.Delete is an intentionally undocumented method, so you're using it at your own risk. Furthermore, you won't be able to "just delete" bars because modifying the locked Bars collection is not permitted.

For simplicity's sake, I'd recommend you not process the trading system's logic before the cutoff date for that symbol:
CODE:
Please log in to see this code.
Voila.
profile picture

Sammy_G

#8
Hank,

The following is the code I use to truncate data BEFORE a date. This script pops up with an input box for the date; I find that much easier to use than hard-coding the date into the script and also allows me to use it on multiple symbols. Just make sure you have a C:\temp directory.
CODE:
Please log in to see this code.
profile picture

hankt

#9
Eugene,
I have a portfolio of securities, not a single security, so that idea won't help me.
Check date on previous bar once moved, ignoring the first bar in the dataset possibly?
{EDIT}
Ah, the collection, not current bar object...hmmm

Sammy,
If I use your method, then copy the temp file back into the Yahoo provider directory?
profile picture

Eugene

#10
QUOTE:
I have a portfolio of securities, not a single security, so that idea won't help me.

Incorrect. Try setting up a Dictionary<string,DateTime> where the Key is a security and the Value is its cutoff date:
CODE:
Please log in to see this code.
profile picture

Sammy_G

#11
QUOTE:
If I use your method, then copy the temp file back into the Yahoo provider directory?

Yes. Basically, the script truncates the data and plots the corrected data for the symbol in the temp directory; if you are satisfied with the result, then copy the new .WL file over, replacing the old one.