We'll use this thread to show some sneak previews as we develop WL7. One interesting change is that we are focusing on Indicators as a flexible way to present data series. If something can be expressed as an indicator, it will be. This will make it easily usable in the flexible Building Block Strategy structure. Eugene has been working hard on this new paradigm. Look at the FRED and CBOE data series, easily drag and dropped on the chart in the form of an indicator!

One question Glitsh : is it a flotting windows the Indicators ? This is better when you have multiscreen.

No it’s not, but you can open multiple main windows and each one has its own indicator panel so that should alleviate that concern.

WL7 will have dynamic DataSets. Sort of what the Wealth-Data provider currently does in WL6 when a symbol leaves a supported DataSet like Dow 30 and is replaced with another. But unlike W-D, these new DataSets can be providerless and use whatever data provider is preferred by the user.

Take these "IPO DataSets" on the screenshot below. For example, the expanded DataSet gets periodically and transparently refreshed with the latest IPOs:

Can the dynamic datasets be used as a historical scan? (i.e. top down trading approach)

For example:

For a given strategy the process entails:

a) scanning weekly for a combination of fundamentals and indicators for trade candidates
b) running a portfolio of strategies on those trade candidates

In the past I would have to run the strategies on the entire universe of securities (say NYSE) to allow for any candidate to be available through the backtest. Extremely slow and data intensive. If I could dynamically create a DataSet which is a scan of fundamental and technical indicators through time then it makes backtesting this top down approach very doable.

Technically, a custom DataSet provider could be developed to import candidate symbols from say a file (database, clipboard, web service etc.) if you mean that.

Also, these DataSets can have dynamic executable ranges. That's how a backtest runs on Wealth-Data on say all Nasdaq 100 constituents through its history but the trades are applied only to the data ranges when an index component was actually part of that index.

One question Eugene : you say that there will be a dynamic executable ranges. How positions will be managed if a symbol is removed ? They will closed automatically the position for this symbol in this case ?

>>If I could dynamically create a DataSet which is a scan of fundamental and technical indicators through time then it makes backtesting this top down approach very doable.<<

Hmmm, that would be a killer feature! I don't think we will have that out of the box, but it's theoretically possible. We have something called DataSetProviderBase which can, as the name suggests, provide DataSets. This is how I implemented the dynamic WealthData DataSets that account for survivorship bias. There's no reason why we (or someone else) couldn't build a DataSetProvider that creates dynamic DataSets based off some Condition Building Blocks, for example. We could probably make the screen run off a remote service too, like Quantacula's Screener does now. Lots of possibilities!

>>They will closed automatically the position for this symbol in this case ?<<

Yes, the position is closed at market close of the last date the symbol was in the index.

Happy to take a crack at it myself once I get my hands on WL7 ... or is it the same as UniverseProviderBase in Quantacula ?

Quantacula's Universes are analogous to WL7's DataSets, yes.

For your information, I have a trading approach similar from mjj3. I'm scanning every month a global exchange to find "best candidates stocks" according Fundamentals (base on estimated earning for example) and technicals (like volume for liquidity) criterias. After that, I take the "candidates stocks" during the month to apply a dip trading strategy. By this way, I have every month a new watchlist.

one question Glitch : is there will be a possibility to create dynamic watchlist programmatically ?

abegy - for your use case I would personally just update a plain ole' DataSet manually once a month. But ... of course you could create a custom DataSetProvider that reads the symbols, for example, from a file that is saved from your other process. The possibilities are there :)

I put abegy's idea on my todo list for later since there's some demand.

QUOTE:
a custom DataSet provider could be developed to import the trade candidates from say a file if you mean that.

Well, I guess I could make that work.

What I really want to do is group "some" datasets by earnings dates so I can precisely time updates to their earnings data and trade accordingly. What you're suggesting is the write the stocks with their earnings dates to a file (The EarningsDateCache.txt file is effectively already doing that.), then import that as a dataset. That works. If you could add a Data Manager plug-in to do that automatically or thoroughly document the API for the Data Manager so I could write the plug-in (including calls to update the dataset objects without restarting WL), that would work better.

It's possible to create a DataSetProvider that would dynamically include symbols based on some condition (for example, upcoming earnings dates) and update them each time WL starts. Even if the process is slow to load (say it makes web requests or database queries) that won't bog down loading WL7. That's how the IPODataSets will work. Hmm maybe there's sense to build an UpcomingEarnings DataSetProvider?

>>Hmm maybe there's sense to build an UpcomingEarnings DataSetProvider?<<

Yes, I'm looking for stocks to short!

QUOTE:
It's possible to create a DataSetProvider that would dynamically include symbols based on some condition (for example, upcoming earnings dates)

That's great news.

QUOTE:
maybe there's sense to build an UpcomingEarnings DataSetProvider?

Well, that's kind of what the EarningsDate class does now when it creates the EarningsDateCache.txt file. What you could do is integrate them so the DataSetProvider and EarningsDate class can use the same cache file.

I also have a standalone command line program that reads the Excel file created by the Fidelity stock screener and writes a WL dataset. I might consider converting that program to a DataSetProvider. The problem is there are command line parameters passed telling it what to do. For example, ETFs take different command line parameters than stocks (and they write different WL datasets).

I wouldn't try to get every suggestion implemented on the initial WL 7.0 release. Only the breaking changes, like converting to .NET Core and WPF, need to be included on the initial release. As long as WL is extensible, other features can be added later.

I was thinking more along the lines of this:

A DynamicDataSetProvider that accepts:
Inputs:
a) DataSet (or Universe) superset of symbols to screen against
b) Criteria in the form of a model or (Quantacula Block) where you can specify fundamental/technical criteria (scan criteria)
c) Frequency (Daily, Weekly, Monthly, etc) Frequency that the screen is updated

Iterates through the DataSet and populates a new DataSet using the Quantacula Universe syntax with {symbol}.{StartYYYYMMDD}.{ExitYYYYMMDD} matching when the symbol meets the criteria specified in the inputs.

The only question is can the DataSet/Universe handle a symbol being in the dataset multiple times for different time periods? Such as:
AAPL,19950101,20001231 (Apple from 1995 to 2000')
AAPL,20020101 (Apple from 2002 to current)

The other caveat is what symbols do you want in the execution loop when you run a model against this dynamic dataset? Should it be the distinct symbol list of filtered symbols (any symbol that at one point met the scan criteria)? This would work if you had a Dictionary of <barNumber, bool> that flags that indicate that the symbol meet the scan criteria. You could then incorporate some trade logic that allows you to either exit the trade, trail a stop, etc. if it no longer met the criteria.

Does this make sense?

Just like WL6 has extendible Performance Visualizers for backtesting, WL7 has extendible Optimization Visualizers for optimizations. I'm neck deep in writing the optimization support in WL7!

Hi Glitch, I take this opportunity to request you some little changes if it is possible.

In the others backtesting platforms, you have the 2D and 3D graphs available for any type of optimizer.
In WLD, this is limited to the exhaustive optimizer. Can you display theses sheets for all optimizers (custom and buld in) ?

Regarding the Results table, can you allow multisort columns and the possibility to add a filter criteria (greater/less and a value) for the selected metric ?

And the best would also be to extend the optimizer features to allow to optimize parameters of the PosSizer selected. If you can add the possibility to push as parameters all available criterias of the posSizer selected, that would facilitate to define best values in a simple way. I saw only few backtesting platform with this build in feature

Hi abegy - based on your request I made the Surface Graph Optimization Visualizer work with partial optimizations, so consider this done. Multicolumn sort and filtering in the Tabular Optimization Visualizer would, I agree, be valuable. We'll add them to the enhancement list!

Glitch,

A suggestion - Currently Optimization metrics are limited to 2 decimal places during the Optimization process. While that is fine for some metrics (e.g. Profit) it is unsatisfactory for other metrics (e.g. Lake Ratio).

Can you please allow the optimizer to access the metrics with many more significant digits? Thanks!

Vince

Yes, this is a great point we will address when we develop the "Scorecard" analog!

Hi Glitch,
I've been messing around with the multi-objective optimizer in Quantacula. It would be good if you could expand that optimizer's fitness conditions to be more like AdaptTrade Builder's where you can specify for each fitness element whether it should be Maximized/Minimized, Weight (which you already have) and if there are conditions for a given element to be removed from the training segment.

Also, they have a nice feature where it can automatically filter for top strategies given a flexible series of conditions.

See attached pics ...

Additionally, to add some more customization on the Genetic algorithms ... again AdaptTrade does this well. see pic

Glitch,

As long as we are passing along suggestions,,,

Optimizations take time, a lot of time. Consider having a "Pause" feature to allow the user to assess the progress by running some of the solutions found to date, and then a "Resume" feature to continue the process if desired.

Vince

Thank you mjj3 and Carova for the optimizer suggestions. It's good to be back in the WL forum where there is never a shortage of great ideas. I have a feeling WL7 will break new ground in trading system dev/testing software :)

Re: post #25. Sorry, couldn't resist reposting this comic having seen that customization on genetic algorithms:




Compare that AdaptTrade screenshot to Aleksey's design in the WL6 version:

Ha ha ... yes probably a bit overkill ... just food for thought. I do like the flexibility that it provides for the fitness criteria and to auto filter for ideal candidates though.

QUOTE:
Compare that AdaptTrade screenshot to Aleksey's design in the WL6 version:

Should there be a highest/lowest parameter? Highest "Net Profit", but lowest "Seykota Lake", for instance.

QUOTE:
Should there be a highest/lowest parameter? Highest "Net Profit", but lowest "Seykota Lake", for instance.

That is a good suggestion Len!

That's what I was referring to above:

Have you considered using dockable windows for the Gui to WL7? And potentially using tabbed workspaces?

Glitch,

Will WL7 have the ability to select which variables to optimize when using Optimizers other than Exhaustive? Thanks!

Vince

@kazuna -> The future of WLD

Vince, regarding other optimizers, the sky is the limit as each optimizer can specify its own unique configuration UI. To kick things off we will include Exhaustive and Shrinking Window (which lets you specify the metric) and will add more post release and document the API so third parties can build and potentially sell or share their own.

And yes they run in parallel to take advantage of multi core!

Glitch,

You beat me to it with your last reply! :)

Shrinking Window - I am not familiar with that algorithm (did not find much in a VERY QUICK search). Is it similar to a Bayesian Optimizer?

Vince

EDIT: It seems that "Shrinking Window" might be some adaptation of the Prony algorithm. Need to dig a bunch more.

@LenMoz -> The future of WLD

Guys, as per post #1: "We'll use this thread to show some sneak previews as we develop WL7." Please use the other thread for general Q&A. Otherwise it becomes impossible to differentiate topics. Thank you.

Glitch,

QUOTE:
It seems that "Shrinking Window" might be some adaptation of the Prony algorithm. Need to dig a bunch more.

Well, I did dig a bunch, and it appears that there are a number of versions and variations of this algorithm. When you get a chance could you post a reference to the version that you are planning to use in WL7? Thanks!

Vince

"Shrinking Window" is just the name I came up with for a quick and dirty intuitive algorithm I created. It performs a number of passes through the parameter ranges. Each pass has a number of "runs". Each run randomizes the parameters. The first run goes across the entire parameter range. After each pass, the optimizer sees which parameter combination performed best, and then shrinks the range window, centered around that parameter value. After each pass, the window shrinks more, so the optimizer can gradually narrow down on a set of fitter values.

Glitch,

Thanks for the explanation! Is this what you have been using on Quantacula? Are you planning a "standard" Genetic Algorithm or other Evolutionary Optimization, or is "Shrinking Window" just an enhanced version of them?

Vince

Hey Vince, I’d say my Shrinking Window is probably a less sophisticated optimizer, but the groundwork will be there for others in the future. Right now I’m working 10+ hours a day getting this version up to speed, just like the old days 20 years ago 🙂

Glitch,

Understand!

QUOTE:
Right now I’m working 10+ hours

Does that mean you still have 14 hrs/day to expand into? ;)

Vince

That’s right, just warming up!

We just added Help buttons to the top window frames of the child windows. A nice, integrated, way to get context sensitive help from any window. Of course, F1 still works :)

I assume WL7 is using .NET core. if yes, can it run on a Mac/Linux?

It’s using .NET Core 3.1, but the user interface library, WPF, is Windows only.

On this Sunday I got Advanced Position Sizers integrated into WL7! They have basically the same API as WL6. This feature did not make it into Quantacula Studio, so it's good to see it surface again in WL7.

And here is the new and improved "Meritocratus" PositionSizer ...

In terms of accuracy, what would it be the difference from a "building block" user and a "coder"?

The Building Blocks are a way to build a strategy visually by drag and drop. They can also be used to get a jump start to coding, because you can convert a Building Block strategy to a C# Coded strategy . That can be helpful for those who want to learn how to code strategies.

Hi Glitch,

I see with interest that now, it will be possible to specify index benchmark by strategy. This is a very good news.
But, what's about Commissions ? It is possible for you to add also this possibility ? As you know, brokers applies several type of commissions logics depending of the instrument type (or exchange/market).

As a strategy is linked to a dataset (or watchlist), this would be more simple to allow us to specify the commission logic in thestrategy instead of a global parameter. In the case of a global parameter, you need to complexify your commissions logic script to take into account the instrument that you will trade.

Also, I don't see in your screen shot the possibility to attach an account like what we can do in WL 6.9. Why ?

Currently commission is still a global backtest setting. We will consider making it a strategy-specific setting!

You now assign an account once the orders are sent to the Order Manager. The Order Manager has individual "Order Blocks" that let you easily manage orders from different sources. When you assign an account to an Order Block that comes from a Strategy, WL7 remembers that automatically and will default to that account the next time it sees orders come in from that Strategy.

I have multiple Fidelity accounts that I trade. As a result, I group stocks by trading success and slot them into different datasets associated with different accounts. For example, one account is for lower risk stocks (primarily large cap), another for higher risk (primarily small/micro cap), and another for ETFs. I use the same strategy for everything, but I vary the position size by account/risk.

QUOTE:
WL7 remembers that automatically and will default to that account the next time it sees orders come in from that Strategy.

I'm just wondering how that's going to work with my way of doing things above? Can I still associate specific datasets to specific accounts (and position sizes) like I'm doing with WL6?

Each Strategy/DataSet combination gets its own Order Block, and you can assign a different account, or even a different broker, to each Order Block. You could also create a copy of the Strategy and then you’d get the benefit of WL7 remembering your assigned accounts automatically.

Mining for patterns in the Candlestick Genetic Evolver for WL7!

Wow: very impressive!

An update would already be worth it by just adding multi-core optimization to the current product.
And an highly appreciated plus would be the addition of a bit more flexibility to the current WFO module; to make it easier to apply the recipe given in the bible “The Evaluation and Optimization of Trading Strategies”, by Robert E. Pardo. And also extra customization options for the Genetic Optimizer, as suggested by mjj3 (or something in between).

But, adding this type of functionality brings the product to a whole new level. There is almost as much innovation showcased during the last month in these sneak previews, than during all the years the product was with Fidelity.

I just have one question: one of the biggest assets of WLD6 is the great corpora of existing public strategies - it’s very admirable the effort put into developing all those monthly TASC strategies. Will these be ported to the new code?


Thank you and keep up the good work, please!

QUOTE:
I just have one question: one of the biggest assets of WLD6 is the great corpora of existing public strategies - it’s very admirable the effort put into developing all those monthly TASC strategies. Will these be ported to the new code?

Thanks for all the kind words. So far *everything* WL6 is incompatible and has to be ported. We'll approach the public downloadable strategies on demand, by request etc. Those with sound logic, good performance or exclusive algo behind them will get ported in the first place - like 17-Liner or FOMC Meetings, for example. The ones illustrating yet another indicator crossing a threshold because the S&C author did not care to include some trading rules in his article, will not - we believe they just confuse the users who download hundreds of strategies and get lost. That's the idea.

QUOTE:
... *everything* WL6 is incompatible and has to be ported.

By "everything", are we including formal WL6 indicators too? Are the WL7 primitives truly different from their WL6 counterparts? If so, is this to support multi-core hardware or are these differences to capture a greater variety of data for the Performance Visualizers?

By everything I mean everything. All the APIs have changed, and even the backtester works differently now. Indicators are among the simplest to port to V7 with one minor precaution to consider: TimeSeries (the new word for DataSeries) must be properly initialized.

Thank you very much Eugene for the clarification and great work!

QUOTE:
Indicators are among the simplest to port to V7 with one minor precaution to consider: TimeSeries (the new word for DataSeries) must be properly initialized.

Thank you for not grouping both the indicator constructor and Indicator.DefaultFunction into a "single call" like WL6 does. I really don't like that anti-oops architecture! (oops: Object-oriented programming)

That grouped constructor-function architecture has complicated my formal WL6 indicators, which have properties and by-reference followup function calls that employ the precalculated state (i.e. "field") variables of the indicator object. The more conventional WL7 oops architecture should now let WL7 indicators work more like the WL6 ArmsIndex class, which is more powerful and efficient. If you want to try to "optionally hide" some of the steps of calling the ArmsIndex operations as a single call for WL7 indicator design, go ahead. But the WL7 indicator API should still treat the constructor separately from the member function (i.e. method) implementations as the WL6 ArmsIndex class does.

Also, the term TimeSeries fits better with signal processing nomenclature, although a WL6 DataSeries is much more than a single time series.

SupertTicker - wait until you see how IndexLab is implemented!!

With the new "Basis Price" position sizing option we will greatly reduce one of the most common support questions: "I set my equity to 100% but I'm missing so many trades! Why???"

Glitch,

Will WL7 sandbox each instance of an optimization running under WL7? It is frustrating when some error takes down the whole program.

Vince

Carova - it will catch exceptions during optimizations, yes.

Here's a sneak peak at the turbo-charged Indicator Profiler Extension for WL7:

Glitch,

I assume the Indicator Profiler is similar to the current Analyze Series in WL6. What are the major differences?

Vince

The idea behind the classic Evaluator (a WL4 tool) is that it looped through data to calculate the chosen indicator value at each bar and determine the percentage return after a certain number of bars, displaying a plot of the indicator value by its future N-bar return.

The Analysis Series visualizer acts as a tool which helps you see how an indicator would impact the trading system's performance if applied as a filter.

Eugene,

I do remember the Evaluator (which somehow got lost on the way to WL5). Good to see its return. Thanks for the explanation!

Vince

Glitch,

Do you plan to bring back "Lead Bars" in WL7? They were a great feature in WL4 that I know I miss.

Vince

It's been implemented 8 years ago as GetAllDataForSymbol in WL6... 🤦

We have something called StartIndex which does functionally the same thing.

Hi Glitch,

I would be a powerfull feature if there was an option to "link" the Analysis Series and the Indicator Profiler.

In the Analysis Series, you access only your dataseries in your script. This is painfull to add several dataseries to hope to find the right filter after the backtest with the Analysis Series. Why not to offer a way to run indicator profiler on a strategy script and offer the possibility to see the result like the Analysis Series to find the best filter indicator ?

I have also a question about how works the indicator profiler with indicator like rsi. Is there a way to specify the bellow value and above value as a parameter like the period ? Is there a easy way to select only a specific category of indicator to run the profiler ? Indeed, if you link the strategy run with the indicator profiler, the user know if he is looking for the best trend indicator (or another category).

QUOTE:
Community Indicator 'Beta': While it can be correctly implemented inside strategy code, as a drag-and-drop it's essentially useless as it only plots 1 series against itself.

In WL7 Alpha and Beta will be perfectly usable in drag & drop mode:

QUOTE:
We have something called StartIndex which does functionally the same thing.

Will the statistics of the Strategy and B&H be calculated from that point forward?

Vince

Honestly, I would prefer if the StartIndex is zero like it is in WL6. If there are bars before that, they can be negative bar indexes.

>>Will the statistics of the Strategy and B&H be calculated from that point forward?<<

Yes, this way it will be truly a fair comparison.

QUOTE:
>>Will the statistics of the Strategy and B&H be calculated from that point forward?<<

Yes, this way it will be truly a fair comparison.

Fantastic! :)

Vince

WL7 questions => The future of WLD

New dynamically updating DataSet in a WL7 extension: stocks with high short interest! Of course GME is there ;-)

Eugene,

How do I install this on the Beta? I could not see a process.

Vince

It's in one of our extensions called DataExtensions. We are for now focusing the beta testing on WL7 proper, though.

Understand.

Vince

Wealth-Lab 7 will be releasing the Kraken!

A dynamically updated DataSet for events like premarket gap up/down, splits/diviends/earnings for today:



Part of DataExtensions package.

Will the extension allow for custom dynamic datasets? As an example stocks with a positive ROC and some value from a custom indicator. Essentially scanning the market and returning those stocks that meet the desired criteria.

Edit: This was answered earlier in the thread....

QUOTE:
Will the extension allow for custom dynamic datasets?

Yes, that was discussed in Posts# 8-17 above. And I see from Post# 87, WL7 has a DataSetProvider to create datasets [1] by earnings announcement date and [2] from premarket gap up, which interests me.

One thing I'm interested in (eventually) is a DataSetProvider for scraping "stock screener" websites. I'll need to find a good website scraper with jQuery-style selectors for this that can place website listings into a System.Data.DataTable type. And if said scraper can handle login credentials as well (I'm wishing), then this will be easy to do. Suggestions are welcomed after WL7 is released.