A simple measure I find useful is the ratio of upside volatility to downside volatility. It gives an indication whether the strategy in letting the 'winners' run while cutting the 'losers'. Not very scientific but probably the best measure for 'normal' times.

There may be a case with a very high Total Abs Return/Max Abs Drawdown but with very few instances in the Back Testing period. This would have a very high average per trade, but your confidence level bases on fewer trades would be low as well.

Is there any specific no of trades one must have to be reasonably assured of the strategies future performance? Difficult to give a number to it perhaps, but intuitively a strategy based on 20 trades would be unacceptable and 200 would look convincing (irrespective of the period of back test and time scale used). Isn’t it so?

The factor to evaluate the strategies should also have no of trades as an input, where a higher number would reduce the Total Abs Return (after incorporating cost of execution); while a very low no won’t give you enough confidence to move ahead. How to handle this trade off?

What I have found is that the more I have tried to avoid the “Y’s, the more the overall profitability is reduced. There is an old saying that goes something like this “if a farmer was worried about losing some seed, he would now sew his field”.

So, it comes back to risk management, if you do enter into a Y trade, can you identify it and minimise the exposure as soon as possible, in some kind of systematic way.

On the point of which ratio to use, once you get results for individual trades that are normally distributed, the selection of which ration to use to compare different strategies probably them comes down to personal preference.

It's true. We can not rely any simple ratio.

At our firm, apart from the Sharpe (rather its variant Information Ratio) we also look at:

1. Annualized Return (Final Return)

2. Max Drawdown

3. Ratio of Annualized Return / Max Drawdown

4. Volatility of returns

5. t-Stat

Whether it’s the Sharpe Ration, CALMAR, CAPM, etc, the selection and implementation of algorithmic strategies is a complex task, and these ratios only show a small part of the picture.

Lets take an example:

Strategy 1 has a strong return (for ease of calculation 100%), returns have a normal distribution, the standard deviation of returns is say 10%, but requires $2m to develop, will cost $30k per month to host, needs 50% of a Systems Administrator’s time to make sure the infrastructure doesn’t fall down and needs 20% of a traders time to review progress and make sure that the macro environment doesn’t significantly change.

Strategy 2 has a return of 50% with a standard deviation of 5%, can be developed or $1m, costs $2k per month to host (does not need colocation), needs 10% of a System Administrators time and 20% of a traders time.

Both of these strategies have fixed costs, variable costs, upfront investment in IT software and allocation of capital into the strategy itself. Each one is profitable, but will need different amounts invested in order to gain the best return. On the flip side, there is also a cap to how much could be traded, no use putting $1b in if individual transactions average out at $100k.

The Sharpe Ratio only takes into account the strategy itself. It does not take into account development of it, fixed and variable costs associated with running it etc. As such, these can only be properly taken into account when the time value of money is properly incorporated via something like a Cost Benefit Analysis that includes an NPV of all costs and returns. See http://en.wikipedia.org/wiki/Cost%E2%80%93benefit_analysis

Some people are lucky enough to have very strong IT skills, can roll their sleeves up, do the statistical analysis and develop the strategies in C# (or whatever the preferred language id), but they should be seen as Guerrilla Traders as achieving the same result via a multidisciplinary team is always more expensive.

The other point is that are we being constrained in our thinking by the capabilities and tools that are available to us? Is there a different way to look at the markets, are we only looking at them through our (or our organisation’s) own frameset? Do we have an appetite for continuous or discontinuous innovation? These are all really important criteria and should not be discounted (excuse the pun). If they are, maybe we need to challenge our own thinking on what can be achieved?

You cannot evaluate a trading system if you do not know how it is done. I say the method counts. On statistics you cannot say it works or not. I have seen too much here.

Exactly, statistics and worrying about them deviates from the main game. Does the strategy work, what return, what risk and then comparing the different options.

As Dimitrov said, the most important criterion is some measure based on the ratio between profit and loss. In the case of "Calmar ratio" considering the maximum loss. If the "Sharpe ratio" is considered the dispersion of gains and losses. Both are measured reasonably well founded.

The measure of the profit no have upper limit, but the maximum drawdown has a limit of 100%. Then to maintain the proportion correctly, a small adjustment is required: instead of P / MDD simply using P / [MDD / (1 + MDD)].

The maximum period of stagnation or the average of stagnation is another important factor. No one supports 10 negative years with only 2% negative. It may be better to suffer a 20% drawdown and recover it in 1 to 2 years, instead 10 years with 2% drawdown.

There are several other criteria to consider. The ideal is to make a list of relevant criteria, assign weights to each and generate a score based on all combined criteria. This score will be the appropriate way to determine the quality of a strategy.

Alex,

My comment complements other suggestions. I was implementing software and reviewing literature on properties collected for the evaluation of trading strategies. In my book "Modeling Maximum Trading Profits with C++: New Trading and Money Managements Concepts", Hoboken, New Jersey: John Wiley & Sons, Inc, 2007 I have summarized on pp. 172 - 174 the 31 properties including those found or suggested by me (empirical distributions of different quantities) and present the list below. I see that some properties proposed in the previous discussion are either in the list or can be computed, especially, if the empirical distributions are already generated. Chapter 4 of the book also continues evaluation of Kelly approach undertaken by many authors with respect to futures trading in mentioned in the message above.

Other applications of this framework can be found in the evaluation of the maximum profit strategies in "Optimal Trading Strategies as Measures of Market Disequilibrium" (it can be downloaded free and without registration as PDF http://arxiv.org/pdf/1312.2004v1.pdf), particularly on pp. 88 - 89. I have omitted output of empirical cumulative distribution and density functions because they get extensive lines.

I'd be glad, if this will be found useful.

Best Regards,

Valerii

P.S. The list of properties

1. Total P&L

2. Total P&L per unit

3. Gross profit

4. Gross profit per unit

5. Gross loss

6. Gross loss per unit

7. Total number of trades

8. Number of winning trades

9. Number of losing trades

10. Average profit per trade

11. Average profit per unit per trade

12. Average loss per trade

13. Average loss per unit per trade

14. Largest winning trade (largest profit)

15. Largest winning trade per unit (largest profit per unit)

16. Largest losing trade (largest loss)

17. Largest losing trade per unit (largest loss per unit)

18. Maximum number of consecutive winning trades

19. Maximum number of consecutive losing trades

20. Maximum consecutive profit

21. Maximum consecutive profit per unit

22. Maximum consecutive loss

23. Maximum consecutive loss per unit

24. Distribution of trades versus P&L (typically not reported)

25. Distribution of trades versus P&L per unit scale (typically not reported)

26. Return on account

27. Maximum account value (typically not reported)

28. Minimum account value (typically not reported)

29. Largest drawdown

30. Average drawdown

31. Distribution of drawdowns (typically not reported)

For some reasons after pressing "Add Comments" the posting was changed and the numbers from 1 to 31 (except these two) were replaced with 1. They are still readable.

Best Regards,

Valerii

Alex wrote:

"

6.So, the key calculation (once we have a normal distribution) is;

E = Expectancy of Strategy being profitable

P = Average profit made for profitable trades

L = Average loss made for trades that lose

RP = The ratio of profitable trades observed in the sample

RP = The ratio of losing trades observed in the sample

7.Assuming the following figures for the values (these values are examples only):

P = 0.015

L = -0.01

RP = 0.55

RP = 0.45

Which would look like this:

E= 0.015*(0.55)+ -0.01 *(0.45) = 0.00375

"

Alex,

If you have a sample of profit and losses PL1, PL2, ..., PLn, then whether you compute first P, L, RP, RL (I believe you have made a typo repeating two times RP) and then E from them as P*RP - |L|*RL or simply add all values algebraically (with sign) and divide the sum by the number of values, then the value E will be the same.

This value is referred to as the "sample mean". If a sample is drawn from one random distribution, then it is a random variable itself. Toss a coin 10 times in a series and made several series and you will get different samples of tails and heads. If the underlying distribution does not change, then the "sample mean" is unbiased, efficient, and consistent estimate of the so-called general population mean or the true mean of the distribution in question. The probability distribution function or characteristic function of a distribution are full descriptors of a random variable. The mean, variance or other beginning and central moments, in general, are incomplete characteristics of a distribution. Although, they give some ideas about it.

If you have the sample PL1, PL2, ..., PLn and believe that it is drawn from one distribution, then not only you can evaluate its sample mean and sample variance and sample central third and fourth moments, and sample standardized central third and fourth moments (skewness and kurtosis), but you can build the so-called empirical cumulative distribution function, ECDF, and check, if the sample is drawn from normal or some other distributions. An ordinary test is the Pearson goodness of fit test (there exist other and some are specialized for normal distribution verification). Then, you do not need to guess about normality of your sample. One misses this information when computing only P, L, RP and RL.

When one divides a sample mean (estimate of a mathematical distribution) buy a sample standard deviation (square root of sample variance) what typically relates to Sharpe ratio, then he or she should also estimate an error of the fraction E/STDdev. This ratio and its error are random variables themselves. While E is estimated with more or less reasonable accuracy (for larger samples) the error of STDdev estimate is typically larger. As a first approximation a relative error of a ratio is the sum of relative errors of enumerator and denominator. For the sum the absolute errors are added as a simple estimate. If E is estimated with 10% and STDdev 50% (not uncommon but many do not compute it), the E/STDdev gets ~60% of relative error. For the ratio 1.8 this would be plus minus 1.8 * 0.6 = 1.08 and the ratio is within the interval with the lower bound 0.72.

The problem with ECDF and statistical estimates is that, if the underlying distribution is not constant (both parameters and, even, type change), then all such computations objectively become a pseudo science, often, still done because of a lack of better approaches. Understanding this emphasizes the means of determination dependence between random variable and the facts itself - are they random and, even. what does it mean "random". The question is formulated in a very simple manner:

Given samples PL1, PL2, ..., PLn or Price1, Price2, ..., Pricen, or dPrice1, dPrice2, ..., dPricen. Are they random?

Best Regards,

Valerii

Sharpe ratio measures what it measures, and is useful for that. If a strategy has a normal distribution of returns, stock only strategies are relatively (log) normal, it is a fairly good way of keeping score. Obviously, like any measure, it can be gamed and should not be used for any strategy that uses derivatives or the like.

What I find more difficult to understand is why people use max drawdown. This depends completely on the length of time for the record. Realistically, I'd say, unless that period is 20 years or so, the max drawdown is not particularly useful. If you look at the S&P500, 1987-2007 or 1989-2009 were long enough periods to have a fair enough idea of what the maximum drawdown would be for holding stocks. But even then it may not be representative, other stock markets across the world have had 80% and higher drawdowns.

At its simplest standard deviation of returns is a less bad predictor of max drawdown, than max drawdown itself. It is flawed, especially if there is auto-correlation, but it is useful.

Graeme,

"... why people use max drawdown. This depends completely on the length of time for the record. ... unless that period is 20 years ..., the max drawdown is not particularly useful. If you look at the S&P500, 1987-2007 or 1989-2009 were long enough periods to have a fair enough idea of what the maximum drawdown would be for holding stocks"

The paragraph above implies a strategy - "holding stocks". If the strategy is buy and hold, then looking on the price or index time series/charts does give an idea about maximum drawdown and the more data available the more chances to capture what was the maximum one.

However, strategies are different and "holding stocks" is only one of them. A strategy dependently on the product can go short and what looks a drawdown on the price chart will be a profit outcome from the strategy. This means that maximum drawdown does not necessarily depend "completely on the time period" but what the strategy did during that period. The result typically depends on both: the market and the strategy applied. From this point of view the list of profits and losses characterizes the market and the strategy. Then, it becomes useful to know all equity drawdowns on the way from the beginning to the end of trading - applying the set of rules and making transactions accordingly.

What is usually underestimated is a distribution of drawdowns. Similar to other random quantities collecting only means and/or extreme values is less informative than working with the empirical estimate of a distribution. Ryan Jones advocates for using not only the maximum drawdown "The Trading Game: Playing by the Numbers to Make Millions", New York: John Wiley and Sons, 1999 but at least the average drawdown too. Ironically, during estimation of averages and extreme values a full sample sufficient for the work with an empirical distribution functions is available. It is matter of extending the focus of interests.

Even, when we speak only about the maximum value of a random variable, we should be interested in a probability to get it. The extreme values do not necessarily obey the same distribution as the variable itself. This is how we come to the Extreme Value Theory. Sections "17 Extreme b-Increments" and "18 A Second Comment on Discrete Distributions" (pp. 50 - 58) of the mentioned "Optimal Trading Strategies as Measures of Market Disequilibrium" (it can be downloaded free and without registration as PDF http://arxiv.org/pdf/1312.2004v1.pdf) describe its applications to price increments and contain a few classic literature references to works of Frechet, Fisher, Tippett, von Mises, Gnedenko, Gumbel, Haan. I also investigate using discrete distributions (for prices and increments this looks important for me) with respect to extreme values.

Best Regards,

Valerii