Do you want to get informed about new posts via Email?

Concept to improve Generalization during Optimization

Author: Carova

Creation Date: 6/19/2016 1:15 PM

Hi All!

I am suggesting that we should consider taking a concept that is common in many branches of "automated learning", i.e. using a combination of a Training Set and a Test Set to provide a preliminary assessment the generalization of an optimizing system. This might include either a different time period of a current WL, a different WL during the same time frame, of a different WL during a different time frame. This would in no way guarantee that generalization would occur, but it would definitely remove instances when generalization did not occur.

Any thoughts?

Vince

I am suggesting that we should consider taking a concept that is common in many branches of "automated learning", i.e. using a combination of a Training Set and a Test Set to provide a preliminary assessment the generalization of an optimizing system. This might include either a different time period of a current WL, a different WL during the same time frame, of a different WL during a different time frame. This would in no way guarantee that generalization would occur, but it would definitely remove instances when generalization did not occur.

Any thoughts?

Vince

Hi Vince,

I don't pretend to have understood what you saidbut maybe you'd have better success through an example? If you could show us what you mean:

Creating Optimizers in Wealth-Lab Pro

I don't pretend to have understood what you saidbut maybe you'd have better success through an example? If you could show us what you mean:

Creating Optimizers in Wealth-Lab Pro

Thanks Vince. While I don't think there's appetite for such major changes, where this enhancement would be destined for - perhaps WFO?

Hi Eugene!

This would be for the standard Optimization primarily, since WFO does provide some info already about generalization. Understand your thoughts, but would adding another WL evaluation be that difficult?

Vince

This would be for the standard Optimization primarily, since WFO does provide some info already about generalization. Understand your thoughts, but would adding another WL evaluation be that difficult?

Vince

Thanks Eugene! I would code it myself if I had the appropriate expertise, but that is not the case. Additionally there are already PSO and GA Optimizers available that I might tackle if it were in C, but could not even dream of doing in C#. As I said, I still do not have that level of skill.

Vince

Vince

QUOTE:

Carova: This would be for the standard Optimization primarily, since WFO does provide some info already about generalization. Understand your thoughts, but would adding another WL evaluation ...

I've considered trying to provide another method of optimization as well, but have run into numerous problems.

Truth being said, trading optimization is a non-linear, fuzzy-system programming problem. I say fuzzy system because you're trying to fit an

So if we try to fit our fuzzy plane model with two points (which defines a line), we will get a solution vector space (infinite possible planes revolving around this line) rather than a unique solution.

During Wealth-Lab optimization, we are also performing a fuzzy-model fit. We employ a strategy with say 5 parameters to fit. That means

So how can you make your fuzzy WL strategy less fuzzy? Well, you can start by minimizing the number of parameters it has. That can be done by employing self-adjusting adaptive indicators to some degree. You can also include more external information (i.e. market sentiment) in your strategy.

I've also looked at trying to optimize a metric other than Profit-per-Bar that would provide

Regarding training and test sets- You can do that now albeit with some effort. Divide your target symbols into two portfolios, say evens and odds. Optimize one, then test the other with the best optimization parameters. You're obviously looking for similar results, or you may be benefitting from having overfit or outliers.

Len

Len

QUOTE:

superticker

As you point out, we rarely have enough trades to justify the number of adjustable parameters that we need to get reasonable generalization. In other Machine Learning contexts I have use Principal Component Analysis (a purely linear process) to extract the EigenVectors having the greatest contribution to precondition inputs into a Neural Net. While that type of approach helps control the vector space somewhat, it comes at the expense of eliminating all the non-linear information.

However, we do not even have this luxury in a rule-based system. As you point out we need to use external information to help improve the situation, but I would also argue that your concept of adaptive "indicators" is even more important. It is only in these can we capture some of the dynamic non-linearity of the Market.

QUOTE:

LenMoz

Yes, Len, that is a possible work-around (one that I have been using for the last couple of years), but it is highly labor-intensive (which is the reason we have computers! ;) ), kludgy, and SLOW! This was the reason that I initially brought up the suggestion.

Vince

QUOTE:This is what makes this a fuzzy system and is the

we rarely have enough trades to justify the number of adjustable parameters that we need to get reasonable generalization.

QUOTE:Yes, Principal Component Analysis, or even Factor Analysis, are excellent ways to extract

In other Machine Learning contexts I have use Principal Component Analysis (a purely linear process) to extract the eigenvectors having the greatest contribution to precondition inputs into a Neural Net.

QUOTE:Well slow down a moment. The power of a neural network solution is to solve a

While that type of approach helps control the vector space somewhat, it comes at the expense of eliminating all the non-linear information.

Since the neural net can solve nonlinear problems directly, one would skip the eigenvector solution altogether and feed the raw nonlinear data into the neural network directly. This way you'll realize a nonlinear solution rather than a linear one. However, the eigenvectors may reveal which parts of the raw data are orthogonal and which are redundant. That's essential in setting up your neural network problem because you must avoid providing any redundant raw data into the neural network; otherwise, you'll have a grossly unstable neural net solution.

Disclaimer: I know nothing about WL's neural network solving facility. I'm just talking theory here, not WL implementation. Also, I have never attempted solving for eigenvectors for steering a stock trading strategy--no experience here either. And solving for eigenvectors on most fuzzy systems may be extremely difficult if not impossible. If you've actually done this, I would like to talk to you more about your experience. For example, what was the final condition number of your system matrix in your solution? What external inputs were most relevant for solving this problem?

QUOTE:I would say we need all the help we can get--use

... we need to use external information to help improve the situation, but I would also argue that your concept of adaptive "indicators" is even more important.