Great post! These issues are definitely non-trivial. Just like how traders learn over different regimes through errors, machines will likely have to do same ;-)

You are so right, how to avoid that over fitting is the key issue, you just have to squeeze the neural network smaller and run less epochs, I guess. I have done several machine learning models and most of those have that over fitting problem: http://pvoodoo.blogspot.com/search/label/ML?view=flipcard

I think that even to detect that you have an over fitting is hard. Do you measure the performance decrease ? But maybe the market has really changed in a way that the model could not possibly learn (i.e. new laws, new technology, new fears, more players doing the same thing) and the statistics are going to be worse

So what exactly is the paradox?

"One of the most difficult problems with creating predictive models in financial markets is to find a system that will have a high accuracy in different market conditions." This statement is a paradox as there is no single model fitting all market regimes.

The answer is very simple

The Markets Never REALLY Changes - Only The People Do. If everybody keeps on looking inside the 'same box', two things are most likely to happen:

1. The majority (despite the level of their sophisticated codes) continue to try with little or no success.

2. The minority (who indeed know the simple secret of the markets) continue to get the benefits simply because they disagree with the popular notion that the markets is fast changing - and that's because they understand that 'there is nothing new under the sun' - whatever trend that 'IS' now 'has BEEN' before and will 'continue to BE' - as long as that secret trend remains uncovered (only to the discoverers).

Thank you for your post. I am very interested in the various imperfections of backtesting and the problems you raise from your practical experience. I intend to start a research work using stochastic modeling and a recent game theory framework to study a system of rational Algorithmic traders interacting in a market. Assuming that they only make their predictive decisions by testing on the available data at the moment and updating. This is quite close to a model I had developed in a previous work. This could reveal some information on the quantification of the risk (related to the lack of information on future), on regime changes in the market (volatility and filtration evolution), on the stability of the market, and could provide some answers to the issues you raised.

Behind a good strategy, there is an idea, a conviction of how one thinks markets work. This idea, this conviction comes from real live trading experience. You code that idea and then backtest that idea on data. I have the impression that more and more people nowadays venture into mere "data crunching" without proper trading experience. Their program comes up with fantastic backtested results but often they do not know why or how. Then, once in the real (unoptimizable) world their programs fail. First learn. I know it is not sexy advice, but it is the best I can give.

Well, for me the only good indication of valid back test method is when risk metrics of back test matches risk metrics of real trades.

"A priori" analysis always has a certain degree of predictiveness in it, so you can only "almost" predict the future if and only if the future is not so different from the past you consider. In other word there is no mean to predict the future whatever it is. And stationarity is the other big assumption you make when using technical analysis.

Predict is almost impossible. What i see is reaction algorithm, trading both sides (long and short), on pre defined value interval and a large amout of money for value average.

Marcos Lopez de Prado issued a very good paper about why the most machine Learning funds fail : https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3031282

“..to find a system that will have high accuracy in different market conditions”. How can this be a paradox in a sense of inate contradictory projection or trajectory that prevents accuracy in different market conditions due to the paradoxical nature of market conditions. To me the very nature of the predictive model is it’s power to dissect market conditions and predict outcomes to a degree of probabilistic accuracy. Therefore, market conditions are part of the predictive model as reflected by the underlying considered data be it cross sectional, time series or a cross-breed of the two or the established theoretical relationship between the different variables and the mathematical/statistical model used to summarise or reduce the time space dimension of the data. Hence the market conditions are explicit inputs to the predictive model. Of course markets evolve and market conditions change but this is not a paradox this is simply a matter of addressing corner conditions ....