How to create WL5 Data Files from Ascii using WL5
Author: ss161
Creation Date: 4/1/2009 2:01 PM
profile picture

ss161

#1
i've followed the post below and create WL4 binary datasources using the BinaryWriter class.

i'm wondering about the following:

1. Can wl5 binary datasources be similarly created?
2. Can custom fields be added to either a wl4 or wl5 binary datasource.

Background, the "cache data" option is certainly helpful, but for large intraday datasources, even the first pass can be painfully slow taking multiple hours to make one pass through the data and create the cache vs about 30 seconds to access a binary data source with the same number of rows of data.


/Forum/Posts/How-to-Create-WL4-binary-files-from-ASCII-using-WL5-27018
profile picture

Eugene

#2
If the V4 solution for binary writing already works, why the need to complicate things?

#1
Anyway, the V5 file format is not published but you can create individual WL5 binary data files using a pair of helpful methods Bars.SaveToFile and Bars.LoadFromFile (QuickRef or forum search for more details). (Creating an associated dataset would be a matter of creating an XML file.)

#2
No/no. The file format spec for V4 doesn't contain a field for custom series, and in V5 it's a data provider's job (you won't be able to register a custom data series with a Bars object).
profile picture

ss161

#3
QUOTE:
If the V4 solution for binary writing already works, why the need to complicate things?


only because i was hoping that i could add a custom field to a v5 file, while i was fairly certain that i could not add a field to a v4 file.


profile picture

ss161

#4
don't cringe when i write this, but if i don't care about volume in this custom dataset, maybe i can replace the volume field with the custom field that i do care about. do any immediate adverse consequences come to mind? i believe, at least in wl4, that volume is stored as a float, so it should work to hold the value.

thanks.
profile picture

Cone

#5
Yes, you could probably use Volume like that, although the price data adapter may skip loading the entire bar if "Volume" is zero. There also could be other side effects if the "Volume" is negative.

QUOTE:
hoping that i could add a custom field to a v5 file
The WL price data files don't allow for custom fields, not even in V5. Custom fields, or fundamental data, are generally added to the Bars cache by the fundamental provider, not the provider for price data.

The exception to existing providers is the ASCII Provider, which does register custom fields. Furthermore, the latest release of the ASCII Provider can automatically convert ASCII data to binary data if you enable the "ASCII Cache". See User Guide: Data > Data Manager > Create New DataSet > Data Providers > ASCII
profile picture

Eugene

#6
QUOTE:
Background, the "cache data" option is certainly helpful, but for large intraday datasources, even the first pass can be painfully slow taking multiple hours to make one pass through the data and create the cache vs about 30 seconds to access a binary data source with the same number of rows of data.

What about creating the Cache before running the backtest for all these large intraday files?
profile picture

Panache

#7
When are you planning to release the V5 file format?

Fidelity's data is horrible. I have over 2,000 symbols with bad data as shown by the Bad History Data Check utility, including the NASDAQ Composite Average .IXIC whose Open > High on 12/22/2008. I have tried reloading some of these symbols, but many of them still have the same problems.

I'd like to be able to manually go in and fix the data. Has anyone created a utility to do that?

Obviously, back testing is only as good as the data you have, so I'd really like to have data I can rely on.
profile picture

Eugene

#8
QUOTE:
When are you planning to release the V5 file format?

We are not planning that, because we are not Fidelity - we're a third party company who is not in charge of this aspect.
QUOTE:
I'd like to be able to manually go in and fix the data. Has anyone created a utility to do that?

Having an utility like that would be a good idea, but unless someone creates it, you can do it like this KB article illustrates:
Data | Removing data using a WL5 Strategy
profile picture

Panache

#9
Thank you. I understand your limitations, and I didn't mean to be critical. Wealth-Lab is very useful, but I am continually hindered by the limitations in the quality of the data.

I'll try playing with a script to allow the user to input corrected data, and if I get something to work, I'll post it. For my example where the Open > High, it seems all I need to do is change Bars.Open(BadBar) and save the file using bars.SaveToFile(Name).

It looks like bars.Add is what allows me to change Bars.Open(BadBar). Is that correct? I can't find any mention of how Bars.Add works in the QuickRef. In the example in Bars.SaveToFile, it appears to be used to write new data to a bar, which also appears to be what is happening in the KB article. Is there an explanation of what bars.Add does?

Second question -- Is there a way to allow user input? If I display the bad bar information in the Debug Window, can the user input the corrected information, or do I have to code the corrected information into the script (which is obviously a lot less useful as a general application).
profile picture

Eugene

#10
QUOTE:
For my example where the Open > High, it seems all I need to do is change Bars.Open(BadBar) and save the file using bars.SaveToFile(Name).

Yes.
QUOTE:
It looks like bars.Add is what allows me to change Bars.Open(BadBar). Is that correct?

No, not exactly right. Read below.
QUOTE:
I can't find any mention of how Bars.Add works in the QuickRef.

That's right, this internal method was purposefully left undocumented and is not intended to be used in Strategies (read: unsupported).
QUOTE:
Is there an explanation of what bars.Add does?

The code essentially creates a new Bars object, that's why it uses Bars.Add for its purposes. It's only an illustration what can be done.

The article is about removing a whole day of intraday data. However, if your task is just to change an occasional bad bar, forget Bars.Add. See The Bars Object > OHLC/V Series > How to: Change OHLC/V Series Values in the WealthScript Programming Guide on how to change a bar, and just call Bars.SaveToFile when your data is finally in good shape.
profile picture

Eugene

#11
QUOTE:
Second question -- Is there a way to allow user input?

Sure, it's all about .NET:
How to have String or Boolean input parameters
profile picture

Eugene

#12
QUOTE:
I'd like to be able to manually go in and fix the data. Has anyone created a utility to do that?

Although I know beforehand it would become a big pain you know where with 2000 symbols (and probably that's why you're looking for an utility), still I have to ask: have you used the Edit Bar Data feature?
profile picture

Panache

#13
I'm getting close, but I have two things that aren't working:

1. At the beginning of the script, I inserted

Bars bars = new Bars( Bars.Symbol.ToString() + " (NewY.WL) ", BarScale.Daily, 0 );

and at the end, I inserted

string file = @"C:\Documents and Settings\User\Application Data\Fidelity Investments\WealthLabPro\1.0.0.0\Data\FidelityStaticProvider\Daily\Y\NewY.WL";

bars.SaveToFile( file );

When I run the script, it creates the NewY.WL file. I manually rename it Y.WL, but when I attempt to chart the new Y.WL, it has no data. What am I missing?

2. In order to get user input to change the price, I inserted

Open[bar] = Microsoft.VisualBasic.Interaction.InputBox("Open = " + Open[bar] + "Low = " + Low[bar], "Open < Low",Low[bar] + "", 50, 50);

I get a compile error that I cannot implicitly converting a string to a double. Is there a way to define the user input as a double rather than a string? Alternatively, is there a simple way to convert the string to a double?

Thanks for all the help.

I realized that I probably am going to have to learn how to use bars.Add to pad some of the symbols with gaps, but before I tackle that, I wanted to make sure I got the data replacement working correctly.
profile picture

Panache

#14
I just figured out how to use Convert.ToDouble correctly, so #2 is no longer a problem, but I still don't understand how to use bars.SaveToFile correctly.
profile picture

Eugene

#15
QUOTE:
but when I attempt to chart the new Y.WL, it has no data.

It means that something is wrong about your code.
QUOTE:
Alternatively, is there a simple way to convert the string to a double?

Yes, just think .NET: Convert.ToDouble( your_string ) or double.Parse. Example:
C# / CSharp Tutorial » Data Type » double parse + 2. 28. 1. Parsing strings to create data types: double
profile picture

Panache

#16
I think I have everything coded to do a very good data check. However, there is one problem I can't seem to get around. If a symbol is missing a bar of data, I can write the Open, High, Low and Close for that bar (and save it using Bars.Add), but when I try to write the correct date for that bar, I get a runtime error: Collection is read-only.

Is there any way to write a new date to a bar?

This is what I'm trying to use:

Date[correctedBar] = dji.Date[bar];
.
.
.
correctedBars.Add(Date[correctedBar], Open[correctedBar], High[correctedBar], Low[correctedBar], Close[correctedBar], Volume[correctedBar]);

I am using .DJI to make sure I have bars for all the correct dates. Instead of inserting each new bar, is it possible to SetContext to .DJI, use Synchronize with the primary symbol to create bars for the missing dates, and then save it using Bars.Add?
profile picture

Eugene

#17
1 - As stated in this WL5 Wiki FAQ, Is there ChangeBar method in WL5?, the Date collection is read-only, making it impossible to change the bar's date. This explains the issue you're facing.

FYI: the Bars.Add method was intended to be used inside a data provider, not in Strategies.

2 - I guess it could be the workaround.
profile picture

Panache

#18
Using .DJI as the primary symbol and synchronizing the stock symbol to .DJI creates the missing bars. However, before I do that, I wanted to make sure the .DJI data was good, so I modified the Bad History Data Check a bit to do that. I use Yahoo data for ^DJI to check the Fidelity data for .DJI. Since I was using ^DJI as the primary symbol anyway, I decided to add the option of adding additional bars of Yahoo history not available for download into Wealth Lab Pro from Fidelity.

The only problem I can't get around is that when I chart the saved .DJI data, it doesn't display the name "DOW JONES 30 INDUSTRIAL AVERAGE", which I assume is the symbol name and should be transferred by the following code:

CODE:
Please log in to see this code.


It's not a big deal, but I thought that if others want to use this, they wouldn't want to lose anything in the process of checking their data.

Here is the full script if you're interested:

CODE:
Please log in to see this code.
profile picture

Eugene

#19
Thanks for sharing.
profile picture

Panache

#20
I figured out the problem with not being able to carry over the symbol name. I was assigning it to the bars object instead of the new bars object. The correct code should be:

CODE:
Please log in to see this code.
profile picture

Panache

#21
There are two more scripts that go with the one to check .DJI that I think I have working. The first automatically fixes most errors in the symbol data, and the second allows for manual correction of the ones that can't be fixed automatically. I'd be happy to share them also. Is there a place?
profile picture

Eugene

#22
The web site is going to have a place to share soon later this year, just like it used to be on wl4.wealth-lab.com. WLP/WLD 5 will also support downloading strategies like it was before. This functionality is being implemented.

For now, please share them on the forum.
profile picture

Panache

#23
If there is no data for the symbol A, is there any way to capture the error created by the following statement to allow the script to run anyway.

Bars symbol = GetExternalSymbol("A", true);
profile picture

Eugene

#24
Yes, by enclosing the statement in a try/catch block. Please see the QuickRef for GetExternalSymbol.
profile picture

Panache

#25
Is there any way to keep Wealth-Lab from trying to update MSN data? If I uncheck On Demand Data Updates in the Data Manager, when I run the following script, Wealth-Lab does not update the Fidelity or Yahoo data, but DOES update the MSN data. I am running this script against the data for the Fidelity symbol .DJI.
CODE:
Please log in to see this code.
profile picture

Eugene

#26
Good catch, thanks for the heads-up. You can create a support ticket for tracking purposes.
profile picture

Panache

#27
Glad to know for once I wasn't asking a stupid question.

However, here is my next dumb question. The Y! adapter gives me the option of either using the pre-split Yahoo prices and volume or adjusting both the prices and volume for the split. However, both Fidelity and MSN adjust pre-split prices, but NOT the pre-split volume. In order to be able to compare apples to apples, I want to use the split adjusted Yahoo prices, but reverse the Y! adapter's pre-split volume adjustment.

This code does that, as long as there is Yahoo data. However, I can't find a way to deal with the situation where there is no Yahoo Data without generating either a compile or runtime error. I use if(yahooDataExists == true) before each time I use the yahooVolume series, so I don't care what the data series contains if there is no Yahoo data.

CODE:
Please log in to see this code.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).