Bayesian optimization works well, but have you considered using a rational expectations approach

My masters thesis was in part based on this field. Essentially, you are aiming a missile at a moving target, and there are a large number of extraneous disturbances, and some disturbances are self generated by the response of your missile. For trading, profits are one of the goals, but also the you can have others , like risk management, or building legs in a multi instrument financial strategy. Also, Kalman filtering, Bayesian models and other conditional path modelling strategies allow you to get away from time series models. Real nice for event driven forecasting. You can ask many questions, such as: if the market is up by 11%, in the time partition 10 to 15 minutes after the bell, what is the 100% probability envelope for put/call premiums in each of the implied vol smile partitions ? Also you can embed transition probability matrices within another model, and get multiple vector inferences as 'plug and play'. Real fast, easy, and target aiming precision.

I think you are both looking at it from the wrong ankle. The goal should not be to find the "optimum parameter" but the most stable parameter area.

I don't even like the word "optimization" I call it:

"Parameter Stability Testing" (I should copy write it)

This is what you really should be looking at. You actually want all parameter combinations to be profitable but look for the ones that work best in all market conditions.

What good is it to find the parameter that made 500% in 3 years but had a drawdown of 70% during "bad times".

Completely agree Volker. This was but 1 simple example. The objective function over which the optimization is run over could easily be made into a stability metric. I've done this in the past using a more grid search approach--I called it Clustering and Dynamic Range optimization, but same idea as you describe--and it's on my list of things to try with Baysian Optimization techniques.

Bayesian models just means conditioning for rational expectations.

You are conditioning your probabilities based on available information.

Rational expectations, in economics and game theory, implies a second stage response. As in , my opponent knows what I will play, hence this changes my strategy.

First level: I know what she is going to do, so I play A.

Second level: I know that she knows that I know what she is going to do, so I play B. Third level: I know that she knows that I know that she knows that ..

Thankfully, combinatorial math, shows diminishing returns to recursive predictions, and can destabilize the system, giving very small determinant values. Indeed rational causality, concludes that there is no value in observable information, and all models are discounted. Hence, you cannot beat the spread, and trading is not a good business.

My profits stem from irrational markets in the very short term ( 30 minutes) and then given time, they become rational.

Jenny: My previous comment is in response to you . Thanks.

Volker - Parameter Stability Testing - how true - how true.

I think what is wrong with the approach you have already recognised when you said "A discussion of strategy model overfitting, and evaluating how overfit a trading backtest may be, will not be addressed here, and will be the topic of a future blog post."

It's not really important what model is used. What is really important are the considerations used to reduce overfitting.

Graeme - you're able to verbalize something that has been very hard to get across to many aspiring system builders and I thank you. There are numerous examples of over optimized systems touted even right here on this forum all the time.

Greed and lust for the unearned blinds many into believing someone is going to deliver them the winning lotto ticket. Developing trading systems is hard enough work, but add to the that the self discipline to be honest with the results is something very rare.

System builders are delusional and lie to themselves all the time. But independently audited track records separate the men from the boys.

I liked the post, Justin is merely offering discussion of a possible technique to short cut the time taken to investigate this sort of problem, where you may have a large number of parameter combinations. I am not going to cover every consideration here but In researching trading systems parameter investigation is only a small part of the work required. In a way by fitting the parameters of your model by this or some other technique, it starts to give you a sense of how sensitive the trading strategy is to these parameters. It is then that a researcher's thinking must transcend the parameterisation of the model to a place where the focus is more on what this kind of thing is telling the researcher about the underlying market behaviour. What behaviour this model is capturing (if any ) and whether that behaviour is persistent or some how dependent on a specific environment. This should lead to different testing and experimentation of the phenomena you are trying to capture.

following on from above:

Ultimately it is only after you have actually learned something about market behaviour from your model that you can be in a position to understand how to parameterise it, so it is not curve fitted and will offer persistent performance into the future.

Mark,

If you try to work with events , as an alternative to time series least squares mirrors, or the tools of technical analysts, you may get different forecasting results.

For instance : a fighter jet spots a target. The computer model aims it's rockets. Once the enemy detects the threat, its historical path is secondary. The escape choices is limited by the dogfight's dual fighter planes dynamic constraints. Contraints of structural mechanics, behavior strategies, and immediate information.

For financial market prices, modelling continuously implies you have a forecast at all times. A forecast obtained perhaps from a data feed, of many markets simultaneously, and optimizing cross correlations. This may be profitable for some, but spurious for others.

Event simulations states that the next price is the current price. Unless your observations and data manipulations, identify and trigger an event.

Bayesian optimization, BO, was discussed in some of my messages on the LinkedIn discussions. Here the author presents results. I have also read a few messages from which it follows that participants are not aware about BO. BO is one way to overcome over fitting and economically use data, which are often divided in parts, where one is used for training and another for testing. This is exactly, where BO can be involved. It is also well known that with increasing samples BO converges to other optimization techniques such as least square method, etc. It is crucial that in all such applications the underlying random system from where the data samples are drawn has "the same" random properties. If this is not so, and markets gives us many ways to think that this is not so, then BO can also yield misleading conclusions. Variations of random market properties (time and price distributions) is one of the reasons why, even, advanced statistical methods can fail. Best Regards, Valerii

I have used this technique a number of times, in the field of econometrics and forecasting with excellent results.