How to create new bars between bars made of historical data?
Author: aokorokov
Creation Date: 10/22/2019 9:06 PM
profile picture

aokorokov

#1
Hello.
I have historical data which imported in wealth lab. Wealth lab process it and show some bars.
I have to add bars between existing bars with same atributes as last existing bar except time if there is no bar. Time and date should be adjusted. When next bar there is no creation. Time scale is minute.
How to do it?
profile picture

Eugene

#2
Hi,

Why do you think that you need to insert the artificial bars between existing bars? Please clarify your objective.

P.S. By chance have you seen the WealthScript Programming Guide > DataSeries > Accessing Secondary Symbols > Secondary Series Synchronization?
profile picture

aokorokov

#3
Hi Eugene,it is good to read you here. I had read 4.4.1 Secondary Series Synchronization one time and will read next time. Have you reference on or part of script with example writen to do external data series by number of dates with standard bar information and more dates than primary data series?
profile picture

Eugene

#4
Hi Anton,

Nothing comes to mind but if you explain your issue in greater detail we might be able to help you. I mean what problem are you trying to solve with all this?
profile picture

aokorokov

#5
I have to make data series with 413 280 minute bars. It is nessary to sucsees. Dated now and back to past. Each bar should have open and close price, high and minimum price, volume of trade. I thought I can make it from data series which I got but, I do not know how. This is the problem which could be solved if there would be code with detailed task as that:

Create Data Series with 413 280 minute bars. Dated now to back in past. Each bar should have open and close price, high and minimum price, volume of trade, datetime.

Thank you.
profile picture

Eugene

#6
Sorry for not being clear. I'm trying to find why do you think you need all that i.e. "create new bars between bars made of historical data" or "make data series with 413 280 minute bars". Why do you think it's necessary to create that DataSeries?

To put it in the right perspective: can you tell us technicians what your goal is and let us decide what would be the optimal way?
profile picture

aokorokov

#7
The goal is success. Answer on "Why" is to complete the task. It would be nice if someone has programmed routine for task and offer it for viewing. I will appreciate any answer here.
Thank you mr. Eugene.

p.s.
I like to get tasks and make it done. :)
profile picture

aokorokov

#8
Well, I would make one data series with data and then synchronize it with historical data to get new data series to use it later in derivative works.
profile picture

superticker

#9
QUOTE:
... use it later in derivative works.
If I'm reading this right (Am I?), he needs the derivative to exist at all points in the time series (i.e. DataSeries) to perform some numerical analysis (like for a system of differential equations or state-space model). The problem is stock prices are too noisy to take the derivative directly. What we typically do instead in signal processing is resample (interpolate) the data into a new time series under a different time base. I think that "may be" what he wants to do.

To resample:

1) Run the data through a smoothing filter that's not a multiple of the old or new time base (to prevent sampling harmonics or aliasing).
2) Take that smoothed (interpolated) result and sample it at the new time base to create a new time series.
3) Now you can take the derivative successfully of that new time series.

I think if he Googles "resampling", he'll find some code that does this. Math.Net does have a signal processing package (separate download) that has the pieces for putting this together, but it's not turn key like PhotoShop. PhotoShop can change the resolution of an image (resample) without requiring the user to compute filter coefficients (to prevent harmonics; called "herringbone aliasing" in image processing).

If he has signal processing background, we can just refer him to the Math.Net signal processing package. https://filtering.mathdotnet.com/ He can take it from there.

If he doesn't have signal processing background, then he can just perform step# 1 with a slow (low frequency) filter (WL has two Butterworth filters in Community.Indicators and a Gaussian filter in TASCIndicators), then perform steps 2 and 3. He'll loose the high frequency components, but he may not care about that--or even want the high frequencies--for step 3. Part of the goal of this resampling process is to filter out the noise before we take the derivative in step 3; otherwise, the computed derivatives will be useless (way too noisy).

---
There are some interpolation routines https://numerics.mathdotnet.com/api/MathNet.Numerics.Interpolation/index.htm in the main MathNet.Numerics package that might "kind of" do what he wants as well. But I haven't used them, and I don't know which methods are "noise tolerant". This would be a much slower approach than general resampling.
profile picture

aokorokov

#10
I got that nobody knows how to do the simple task to make data series with n bars. Thanks for help here.
profile picture

aokorokov

#11
I do not need any samples and filters and smoothing, I got tired from all that military radio instruments. I can say that you can go eat dna free food to start think different way. Buy it or leave.
profile picture

superticker

#12
QUOTE:
I do not need any samples and filters and smoothing,
So if you don't want to resample or interpolate the missing bars (which is the normal way), how do you mathematically want to do it? What's your plan? If you tell us how you want to compute the missing bars without interpolating or resampling, someone can probably help you. What approach do you want to use to fill in the gaps? WL won't do this on its own.

The reason there are missing 1-minute bars is because a stock doesn't necessarily trade every minute, so some minute price data will be missing--that's expected. You could simulate (backtest) on 10-minute bars instead without many gaps because chances are a stock would trade at least once every ten minutes. But you can't create bars that aren't there (no trades that period) without some kind of interpolation, which WL won't do on its own.

Even if you did interpolate to fill in gaps in the price data, doing so isn't giving you "real data". This is a bit like taking a 480pixel image (std def) and interpolating it to 1080pixel (high def) to view on your HDTV. That interpolation/resampling process isn't going to give you high definition from a standard definition image--that just isn't possible.
profile picture

aokorokov

#13
mr Superticker, I read your post that resampling or interpolating on data series could fill missing gaps in current data series? Is there so, could write example which would good with library of wealth lab?
profile picture

aokorokov

#14
Moderators, close this thread please.
thank you in advance.
profile picture

Eugene

#15
QUOTE:
Create Data Series with 413 280 minute bars. Dated now to back in past. Each bar should have open and close price, high and minimum price, volume of trade, datetime.


That task is explained in the WealthScript Programming Guide > DataSeries > Filling a Custom DataSeries (How to: Create a DataSeries filled with zeroes).

Care to explain why you aren't satisfied with Wealth-Lab's synchronization of DataSeries?
profile picture

aokorokov

#16
Thanks to mr. Eugene.
I do not think about satisfaction but more about task complete focus. I planing to spend days for make work done starting with that Wealth-Lab's Data Series synchronization perk.
profile picture

Harapa

#17
This will do the trick.
You need a reference series* to use to synchronize your target data series.
To begin, open the reference series as a new chart and apply this code after adjusting some inputs.


CODE:
Please log in to see this code.


*Your reference data series should have all the dates and times as desired (and should be in sync with the data series to be purged/synchronized). Since you are in 1 minute mode your best bet is to use data for a some very active ETF or index, such as SPY or SP500 index as a reference ( if your data series is related to US stock prices). Even if you do this, I bet that you will have some (but not so many) missing minutes. Only way you can be sure that not a single minute is missing is to do this either programmatically ( probably outside the scope of WL coding) or manually. Above code will use the date and time of reference series and price information of the series being synchronized to create a new series as a new file. Missing bars will be added as per "synchronization" routine and those not in time frame will be removed. Once created this file is ready to read in WL (if placed in a correct location).
** There are many ways to call in an external data source in WL. Ask if you need help with this step.

Good luck.
This website uses cookies to improve your experience. We'll assume you're ok with that, but you can opt-out if you wish (Read more).