The best story I read about backtesting limitations!

Only the point 2 isn't true for asset classes like commodities and currencies.

Thanks for comment Vladimir- i thought article was interesting.

These are not new. The real life problems are 1) some quants (and their supervisors) do not understand these; and some others intentionally make backtesting in these ways (probably to make a living).

What Zheng said!

I think the most common one is authors using in-sample data as proof of their models. I've lost count of how many market timing articles I've read on seeking alpha which are fitted to use a simple parameter that would have minimized damage or made a profit in the only two US bear markets of the past 20 years. However even simple attempts to increase sample size by trying to use the same models on non-US markets show that they are not robust.

All very true

Excellent points! Focusing on one sub-topic:

The following are illustrations of the magnitude of survivor bias within hedge fund and large hedge fund universes:

http://abwinsights.com/2015/03/26/hedge-fund-survivor-bias/

http://abwinsights.com/2015/03/31/large-hedge-fund-survivor-bias/

Most guru/whale following strategies have fallen and will continue to fall victim to these.

nothing mentioned about the pollution of time that plagues most used data. time is an element that alone will destroy any method. to obtain a valid backtest you have to eliminate time from the data.

What means to eliminate time from the data.

Using the chart tick by tick. You mean this?

We can eliminate time with Renko charts, yes? how about if we backtest multiple time frames...if the backtest is successful across multiple time frames is it more valid? Newbie...thanks for the insights.

Portfolio backtest in my profile. It's more interesting. There are only 2 entry condition and 2 filters. In this case I avoided the 8 errors. For more information contact me. I Hope :)

Eric P, Multiple time frames is similar to using longer periods.

Thank you Tom. I appreciate your insights.

There is a confidence inspiring statement. Head of MSCI research; an allegedly competent, experienced and aware actor. Rarely has a more ignorant statement been openly and unabashedly uttered. If seriously spoken, heaven help anyone with money any where near an MSCI related or based product.

Mark B., can you detail your thoughts on "eliminate time from the data"?

there are few ways to eliminate time from a trading strategy. you could use renko or range bar or point&figure charts as input. essentially, you concentrate on price action only...

Through time immemorial two things matter in the trading and investment business; time and price. Why not eliminate both and haul out the darts.

@Larry, thanks for a good laugh, I needed one today.

I can't speak for Mark but using constant volume is one of the easiest ways to eliminate time from charts.

Happy to provide the entertainment. Everyone is undergoing a wee bit of stress this week; long, short or flat.

I track order flow type metrics pretty carefully and over the last couple of months I have seen a new kind of algo show up very regularly. It is creating issues that have to be dealt with across the spectrum of traders from hft to manual traders.

To a large extent I have figured out how to at least get out of the way in time to keep from getting run over and am pretty close to actually being able to take advantage of the situation - which brings me to my point for this thread.

I believe a common mistake among quant traders is the reliance on back testing when they don't have the fine grained data and/or back test engine to test very short term trade models. My advice is to either test live with real money (but small size) or don't trade those algos. If you go with the live trade approach, have a budget for losses and expect them. Rarely does an algo of that nature perform at peak capacity (or even breakeven) on the first couple of iterations.

Scott, hopefully people will listen to your guidance which is based upon pragmatic experience. Back testing methods, including the most robust versions, merely assess the hypothesis. The results direct further pursuit including indicated adjustments or compel discarding the application. Real time with real money is the only meaningful test which includes testing the emotional component and discipline required for adherence to the method and system.

@Larry - Every trader should have your last sentence on their wall if not tattooed on their arm. Upon reflection maybe the tattoo idea is a little extreme. One thing that it seems is often left out of the equation is how difficult it is to sit an watch an algo trade 1 lots and often lose money at that during the live testing phase of algo development.

And thank you for the kind words; they are much appreciated.