My List: The Seven Sins in System Testing

QUESTION: Anyone have some more sins to add? § Calculation errors § 1.Data quality If your data contains “fake prints” or has not been adjusted for splits, this would certainly negatively affect your trading decisions from historical Peaks and Troughs based on the wrong values. Especially if the traders who moves the markets (e.i institutional investors and large hedge funds) with big money make their trading decisions with correct values from professional data feeds. Solutions: Use professional data feed 2.No broker commissions If broker commissions were not included, that would mean that 50 percent of all strategies that is based on “ping-pong” trades between Long and Short positions would be successful. Meaning that if your strategy is showing a negative result, just reverse the strategy and you will end up with the same result but with positive numbers. Solution: Using 0.2 + 0.2 percent commission on each trade § Statistical errors § 3.Illiquid stocks The reason why technical trading system is working comes from how you interpret the statistical probabilities from mass psychology, crowd behavior and numeric cognition. If a handful of people are trading a penny stock back and forth, that will not make the best environment to successful use theories of mass psychology. Solutions: Use instruments with more than 20 millions USD in average daily turnover 4.Short periods of data If you are only testing your strategy against data from only a couple of years during a consistent bull market like we had from 94-88, 03-07 or the present since 2011 it is obvious that the market conditions where a lot different than during the bear markets after dot-com bubble and the last subprime crisis. Solution: Using only stocks with consistent data from 1996 that includes 2 major bear markets and 3 major bull markets and check that the gains are evenly spread during each year during that time 5.Small sample of stocks It makes sense that having data from more symbols gives less variation (and more precision) in your results. Solution: Using at least 200 global stocks from various exchanges with highest daily turnover 6."Buy and Hold" bias The stock market is in some way a reflection of how the economy in general is developing. As long as the major economies in the world have a general positive development in GDP together with an increasing volume in the monetary system the numeric value of the stocks tend to go up. Especially if you also take in to account that all the exchanges around the world is no longer are holding stocks of companies that gone broke and been delisted. That is why almost any buy indicator has been successful if you also have a sell strategy from your long holdings to be more than 10 – 15 years. Solution: Only use list of stock with negative performance under used time period with Long Strategies and vice versa § Programing Errors § 7.Look-ahead bias Some back testing software let you use values from indicators that in reality has been established after the time where the trade took place. One of the most notorious indicators is the ZigZag function and its sub functions. On the other hand it can be used to an advantage if you know how to code it and you want to make experiments how great the maximum gains would be if you could manage to consistent make perfect entries and exits in the short term. Solution: Double check with graphical simulation and lists of all trades

8 comments on article "My List: The Seven Sins in System Testing"

 Marko Rantala, Indicator & Strategy Developer. Futures trading. CEO & Founder seeking new partnerships ► http://tradingmaestro.com

 Tuesday, December 30, 2014

Hi, I think that the most common is missing, curve fitting. Solution: use enough out of box data (step forward test) after initial tests.

BTW: I think that that is the reason why machine learning/neural nets fails so often, blind curve fitting.

 Valerii Salov, Director, Quant Risk Management at CME Group

 Tuesday, December 30, 2014

There are different books reflecting different aspects of trading systems development. The first edition of the book cited below was intensively used by a group of C++ developers, analysts, and traders during a "Robot Trader" project in the beginning of 1990th. The second edition of the book is

Pardo, Robert. The Evaluation and Optimization of Trading Strategies, 2nd Edition, New Jersey: Hoboken, 2008.

It has a consideration of the over fitting. There are at least two discussions going on around on LinkedIn, where over fitting and the central topic. Particularly, I have posted the following message:

Ken Duke: "How do you prevent curve fitting?"

The following techniques are applied to prevent over fitting: Bayesian optimization, regularization, cross-validation, early stopping, pruning.

Their penetration to trading and developing trading systems and investigation of their applicability is going on with an acceleration. At the same time markets is a rich source of information for developing these and new methods.

Best Regards,

Valerii

 Deo Jaiswal, Quantitative Research at Liquidnet

 Tuesday, December 30, 2014

what about slippage?

 Henri P., Founder, CTO at Streamr

 Tuesday, December 30, 2014

Common issues not yet mentioned:

The bid-ask spread is amazingly often ignored.

Simulating the execution of orders that enter the book is difficult.

Your own actions have an effect on the market.

 Federico M. Dominguez, Founder & Boardmember at GAANNA

 Saturday, January 3, 2015

Lets ad some that have left its mark on the skin:

- System is properly tested and paper traded, then not following signals

- Unable to balance positions accordingly, jumping from strategy to strategy

- The notion that system is "for ever" in a rapidly evolving market

- Ignoring commissions

- Commissions taken in account, ignoring slippage

- Ignoring Macro fundamental events, for example,

"LONG RUSSIA...

hey but someone just shot down a passenger airplane there, ...

doesn't matter, system says LONG..." (real conversation)

 Alex Krishtop, trader, researcher, consultant in forex and futures

 Sunday, January 4, 2015

8. Basing a model on visual patterns, price-volume time series analysis and similar abstract things rather than on the analysis of real market processes that drive prices.

9. Believing in high frequency trading without enormously expensive infrastructure.

10. Temptation to manually override the rules in cases that seem "important".

 Jeremy Roseberry, President at Granite Capital, LLC

 Monday, January 5, 2015

8) Too many input parameters used in the optimization - simpler is better

9) Listening to or taking system advice from academics who aren't actually trading real money

10) No position sizing methodology in place or a position sizing methodology that is a mismatch with the strategy risk/reward

11) Not being honest with yourself - don't look for only favorable outcomes and base decisions to trade live money on those. Analyze your system as if you were a third party being paid to do so

12) Per #11 above, ALWAYS question a system that performs in an outstanding manner. Chances are there is something wrong with it.

 private private,

 Wednesday, January 21, 2015

Perhaps help here on what we define as "systems trading"?

Please login or register to post comments.