@Hock, I think the best answer was given by @Vasily when he said: <...The permutation of returns does not matter for the terminal wealth but obviously does matter for the maximum DD > And that is what you are experiencing when doing a Monte Carlo simulation.

I like @Alex's question: <...Has anyone using MC thought why this method can at all be applied to testing strategies and what level of relevance such a result would have? > I tend to the totally non-relevant side.

When @Chang Min says: <...However, I am always doubtful to use artificial data series for simulation. > I would suggest a second look. I've studied randomly generated prices series made to mimic stock prices. And when you look at these chart's right edge, no Monte Carlo simulation of what ever duration could help you improve on what ever trading decisions you might want to make. The more you want to trade short term, the higher the degree of randomness in price variations and therefore the less useful a Monte Carlo simulation will be. Try a Monte Carlo simulation on a 100,000 tosses of a fair coin, and then based on “your” observations, make the next bet.

@Alex added: <...This test as many other statistical tests bases on the premises that the price time series are random (or pseudo-random). > Right. I too don't go the pseudo-random, but I do accept quasi-random price series in the sense that short term predictability is quite low, like near zero. The reason is simple. Imagine that there is a short term edge to be had. Then why over 75% of professionals don't beat a long term benchmark like the DJIA? In a way, it's funny, the game needs to stay quasi-random on the short term in order to give everyone the ability to trade.

One has to look at trading methodologies, and Monte Carlo simulations have little practical use in trading. They are at best unlikely visions of a probable past, but most importantly, that is not where people trade. It's at the right edge of any chart that a trader has to make his decision and take or not his next trade.

I am not a trader. I am just offering an alternative viewpoint. There is a lot of empirical research on the randomness of the markets, so in my opinion, a time series only gives you 1 possible future path.

Guy, you raised an interesting topic. You somehow confused predictability, volatility, data resolution and randomness. And at first glance your conclusions look very logical and unambiguous. However let me disagree with them.

Predictability (I personally don't like this term, but let's use it anyway as it has been already introduced here) basically means our ability to generate rules that have positive mathematical expectancy.

Randomness can be understood in a number of ways, but mostly is used in our world (trading) as a measure of directional bias in the price time series.

Volatility normally means just the distance price can go per sample (not necessarily time-based).

And by resolution I assume the scale at which we consider price time series. I intentionally avoid using the term "time frame" as sampling can be done in various ways, not only time-based.

Now, I do agree that relative volatility decreases as resolution increases. Moreover, this dependency is proportional to square root of 2.

I also do agree that the larger the scale (the higher the timeframe for example) the more affordable is directional trading — that is most frequently attributed to "predictability".

However if you don't do directional trading then predictability as per definition above may be either independent of the resolution, or — even more interesting — increase with the decrement of resolution.

To illustrate it with a simple example — consider market making using aggregated prices from several ECNs and STPs.

Therefore although I maintain that it's far more complex to do directional trading on small resolution, I wouldn't say that predictability per se is a function of resolution.

While Monte Carlo simulations are very effective at determining the general reliability given that the trades which occurred in the backtest are a constant, some systems do not inherently benefit from this type of analysis.

The first is to look at how many datapoints are available, for a system that has a relatively low number of trades to cycle, the results may become a bit askew from the initial expectations. If the initial results are significantly better, determine the process of optimization and determine if it is prone to being overfit.

Second, look at how many simulations were done with the MC analysis. One run may yield drastically different results as it is probable that a string of losing (or winning) trades end up batching together in the sample. Running a single simulation will give you different results every time. Ideally, you will want to run the simulation several thousand times and look at the aggregate results. It is also not a bad idea to remove the top couple percent of negative and positive outliers.

I think for your system, the best route to take for confirming its relative effectiveness (without forward testing) is to simulate a walk forward optimization. The walk forward optimization will reoptimize the parameters upon a certain performance metric on past sample data and use the best parameters in the blind period moving forward. This will expand the measurement of robustness.

Good luck!

@Alex, I understand your general point of view. You say: <...You somehow confused predictability, volatility, data resolution and randomness. > Not at all, I am fully aware of every aspect of each.

You express predictability as the: <... ability to generate rules that have positive mathematical expectancy.> There is the crux of the problem. The more a quasi-random price series is quasi-random, the less one will be able to find a short term positive expectancy. Using a stochastic differential equation (SDE) to express a price series as in: dP = µdt + σdW, we can separate what could be considered the trend from its random-like component. One could as easily have used: P(t) = µt + Ʃϵ to represent the price's regression line with its error term tending to zero (Ʃϵ → 0). This in turn leads to define a martingale environment with E[P(t+1)] = P(t) stating that the most probable outcome for the next price move in the series is P(t) itself: the no change scenario.

Knowing that there is a long term trend, we do after all have a couple of centuries of data to make this point. We both would agree that this is fine up to the right edge of any chart. The long term trend will describe what happened, not what is coming. One could always “predict” that there will be continuation, termination or reversal, but it does not mean it will happen.

As for the random-like component, there is no predictability available. In this sense, it would mean that predictability can only apply to the µdt component. Otherwise, you would be predicting the unpredictable which can still be attempted, but there, the term should be change to either a guess or a gamble on what might be, nothing more, no measure of significance.

My question would be: give me a “hard” probability on the next 100 trades that should be taken on a 100 stock portfolio. If there is “any” short term predictability, meaning that one can assign a probability measure for each stock, at a statistically significant level, then one should just follow these “predictable” outcomes to increase their portfolio value.

As a side note on patterns, I have seen a lot of people identifying a head and shoulder pattern after the fact, but I have never seen anyone out there identifying one before it happens, meaning before it starts, and be right.

I displayed the following chart in this forum before:

.

http://alphapowertrading.com/images/divers/SDE_long_term.png

Some simply don't want to see what is implied in such a chart. Last year, I did an article describing it:

.

http://alphapowertrading.com/index.php/papers/160-the-drift

In the example, I use IBM over a 13 year period starting in 2000 (3,316 trading days) and got the following numbers: µ = $0.0274, σ = $1.89. If you plug this back in the above equation, you get: dP = $0.0274dt + $1.89dW. This translates to less than 3 cents per day as its long term upward drift which is drowned in ± $1.89 of background noise, or within 2σ = ± $2.78 if you prefer.

The point being made is that in the short term, the long term contribution to the mix is minimal, I would say “peanuts” as shown in the above chart. The problem becomes even more complicated when looking at the future where µ and σ are now unknowns and themselves random functions.

People making short term trends are designing trends of mostly market noise. It's like drawing trends on coin flip series. You will find them, can draw them, but a the right edge of the chart, they will be of little help.

Looking at the right edge of the chart, the IBM SDE could be written as: dP = $0.0274 * 3,316 ± $1.89dW. Or: dP = $90.85 ± $1.89dW. And there, it is not the $0.0274 upward drift per day that matters, but more the fact that you waited 3,316 trading days.

Some will always want to make the short term bet when all they had to do was sit on their butts to win as if by default.