Order execution is the 50% of the trading strategy. Thanks for sharing...

@Piotr, how do you define "Good" logic ? Is there a "strategy" with "Bad" logic btw ?

@Alex, exactly, thats what I mean that you have an adequate understanding of the "logic" but no (ZERO!) understanding of how it will perform against the other participants in the market unless you run it in the live market !

I 100% agree with Alex. Besides, you would have had to run your model on the individual tick data collected of each broker in order to make a more fair assessment. A broker like Global Prime Au did not go down at all, but Alpari, FXCM, etc had price freezing, etc during the turbulent times.

Backtesting correctly with tick data to simulate past movements is still the closest and 'safest' way to properly simulate your trading strategy, catch bugs, etc.

How about some circuit breaker logic in the execution module and some kind of watchdog?

No matter what is promised by brokers or expected from a risk management system there are times where execution is just impossible and It does not matter if one is systematic or discretionary. The story will be in the January numbers but for CTA's holding Swiss currency futures it was mission impossible trying to execute their protective stops.

I would tend to disagree with @William's and others' point of view that backtesting is now dead.

One could say, for the very same reasons, that all of technical analysis is also dead, since none of it would have spared one from the outcome. And as @Gus said: <...it was mission impossible trying to execute their protective stops.> The price moved much too fast to accommodate many. The equivalent in stocks would be the '87 crash or the flash crash of more recent history. One can put the 90% drop in FXCM today in the same category of “rare” events that do happen. It's not the first and it won't be the last.

We all know that there are “black swan” events and outliers in this “speculative” game. And therefore, one should try to protect against such events. The first line of protection is not to put all your eggs in the same basket. If you have 30 to 50 positions in stocks, your risk of losing due to a sudden bankruptcy of one stock is a 3% to 2% drop in equity. Not such a big deal compared to a single position portfolio's 90% drop in equity as in FXCM.

I certainly wish to all proponents of the “backtesting is dead” mindset all possible and available luck since they will need it.

Hey Guy. Hope the new year is treating you well. I was waiting on someone to smack my back testing POV. I respect it was you. I was a back testing fanatic until I perfected the reactionary algorithms. I still back test but not to the extent I used to. Now the only back testing I do is superficial, just to make sure the trading parameters activate correctly, regardless of volatility or events.

My reactionary algorithms aren't effected by black swan events and outliers like predictive algorithms and standard methods are. Luck is something for individuals that waste their money on lottery tickets. I still diversify my positions but not because I have to but because I can take advantage of all of the great activity out there. You know me, I love being told that, "I can't do something".

I know that it will be years, if ever, before people will figure out the overall benefits to reactionary over predictive. So back testing for the majority of researchers will not die in their lifetime. Those researchers will wade through traditional techniques in order to form fit their algorithms in ever changing environments.

You are the voice of reason as always. I'm just not one to ever be satisfied with "what is expected or normal".

@William, love the contrast that you make between reactionary and predictive !

Piotr, have you thought about looking at market/broker prices for out of the money options for a view on the risk of a large price move.

While hardly dead, back testing remains flawed. The vast majority of applications may permit assessments of developing strategies and tactics. However, the only useful final test is real time with real capital. As time passes the Swiss franc incident will dissolve into the background with other extraordinary incidents. Tests will be run without recognizing or adjusting for the incident. The result will show the result noted within the original posting in this thread. A 59% will be recorded and the distorted result will be lumped into the study as a valid data point despite the fact that it was impossible to execute the order. I concur with William's remarks regarding testing. They are consistent with personal experience.

Regarding options, it is impossible to run meaningful tests to determine any probabilities or results. It is the nature of the beast that precludes success. Appropriate data is impossible to collect.

@Aaron

Brilliant!!!

Backtesting has many beneficial effects beyond the obvious PNL matching.

To begin, I need to make the point that backtesting in an FX environment has always been carried with way less expectations than in other more technology-efficient environments like equities or options for example.

PNL match is the most elusive and the smallest of the benefits of a research framework. I can list more interesting goals:

1. qualitative - backtest gives a feeling for the main factors in play - this is mainly for new strategies

2. differential - even if the final PNL does not match at all the production PNL, if a change in an algorithm or on a parameter produces a fall in backtest PNL, it is likely that the same change will affect negatively production PNL as well.

3. SDLC - some newer breed of platforms, like HBQuant's X1 research and execution framework (hbquant.com), are built with the ability to run the same strategy code in backtest or live mode. The benefit is that all the technology risks involved are drastically minimized because it allows for exhaustive/brute force testing of new code before it is deployed in production.

4. discovery - backtesting can be used to unearth new trades if the platform actively monitors for it. Again, this is if the platform was built with this sort of capability.

My opinion for what's worth.

@Aaron, there was nothing in the option prices to suggest that an event like that was possible.

For myself, I dont think backtesting is dead or anything close to that. Just be careful what you are backtesting.. Some markets are free to move, others are being manipulated, or better said "pegged". It is not the first time a peg is being abandoned and clearly no backtesting can show what will happen because that has not happened before. However, it does not take a great imagination to figure out that potential future price distribution is binomial - one set for pegged and totally different if peg abandoned. Clearly, markets underestimated the probability of that happening but it was not 0. I dont trade currencies but if I did I would have excluded it from my trading universe or reduce capital allocation and trade only when prices away from peg.

@ Henrique, Owen, Nikolay

A lot of interesting thoughts... thank you guys.

As regards to trading not totally "free market" instruments... the risks are different but it doesn't mean you should not trade it at all... depends on your profile. What I am looking for are some ideas on how to protect against execution problems of this magnitude ? These ideas are most welcome.

I don't trade currencies, but I'm curious how this trade brought down several big houses. My questions don't seem to have been asked in the media. Why would you be short on the swiss franc in the first place? It was pegged to the euro and since the euro was the weaker currency it wouldn't make make sense to short the franc instead of the euro. If it was done as an arbitrage trade, I'm not sure but possibly the Franc was trading at above its peg, then it would have been a suicidal trade. If a peg is failing to hold then it will break, no matter what a central banker says.

ok - not trading at all is strong, but probably as it relates to a quant strategy, it does. If you know you are dealing with a manipulated currency then no backtest will say what should happen if the manipulation is suddenly dropped - especially one like EURCHF which was put on in anger during the grips of a meltdown. That is why I would worry about CNY/CNH for quant trades - or at least I would bias a trade for USDCNH or USDCNY to be long USD - ya think the Chinese are going to let their currency get strong vs all the other peers by roughly pegging to the USD? Eventually, they may want to weaken vs USD enough to strengthen their exports and that decision could come at any arbitrary time.

I agree with Owen on CHF, as it's so manipulated gaps are always there.

Graeme, in the fx world when one names a single currency then by default it means this currency vs. the US dollar. Therefore "Swiss franc" means USDCHF, "the euro" means EURUSD and so on. All other pairs are called crosses, and they are always far less liquid than the "main" pair. In case of the franc it's possibly even more obvious, as USDCHF takes about 98% of daily turnover in all CHF pairs. So when one says being short franc this means being long USDCHF in the first place, and looking at the direction of the mid-term movement in this pair since the very beginning of the year may explain why there might be a skewness in the net position of clients of those dealers that went bankrupt. The rest is only poor risk management, or, let's name it more adequately, greed of those same dealers: when you can see that, for example, half a thousand accounts are about to get drained (and this is your source of income, remember it) then it's so tempting not to hedge the net position elsewhere but simply wait for another couple of days and pocket all money, right?

As to why trading EURCHF — well, exactly because of the discussed limitation on price differential between the two. Sort of a mean reversion during illiquid times and here you are. Only I doubt if it's possible to do unless you're allowed to send your own bids and asks to the market. And of course this strategy should be dead now.

Alex, you are completely 100% correct. "Short the swiss franc" is a figurative term, it would be carried out by buying futures on the USDCHF or futures on the USDCHF and EURUSD. But I'm not sure of the point of being literal here. My question was - Why? This trade bankrupted the oldest currency trading house in the world, and I can not actually find any logical reason why they would even be holding the position.

If it was a bet on the Euro/Franc dropping against the USD, then the sensible trade would be to (figuratively, Alex) short the Euro. And if the trade was an arbitrage trade then Soros would laughing at them harder than he has ever laughed.

Backtesting are, by and large, exercises that assume certain market frictions (liquidity,cost,etc..) and/or market conditions (peg). When these market frictions deviate significantly from the assumed ranges - results don't hold up.

But to say that 'backtesting is dead' is probably stretching it. If this is true, then one shouldn't look at charts/price history at all!

The Swiss episode hammered this point all over again: don't over-leverage. Do this and you will suffer at the most inconvenient moment. After all, extreme price moves aren't all that rare. These historical events should always be part of the backtesting process.

Backtesting can not take liquidity risk into consideration.

I think the headline here is a bit flawed. it's almost like a geologist saying that as soon as seismic activity produces spikes in their data, all geologists should stop forecasting earthquakes.... aren't those exact same spikes some of the most valuable data you can work with??

Jan 15th provided priceless data to use for simulations.

if "back-testing" with jan 15th data tells you that your overall strategy library/risk management (or whatever else you are trying to quantify) doesn't hold up, then that's the reality.

my first run of risk models is always over black swan events - that is a crucial test they have to pass to get approved.

I look at it this way, if my back-testing says: "by running strategy A, with X amount of credit allocated and market gaps 40-50% = i'm out of business", I would be happy I ran that simulation rather than being discouraged from doing my job.

Graeme - you can trade any pair OTC in large size for G10 currencies in the OTC markets. Just trade EURCHF straight up. Was such a nice short to have on for a tail event without much risk because it was very unlikely that T Jordan would lift the peg to 1.25 or something.

Question for ABT; what qualifies as a "black swan" event? What is the criteria? Seems that testing should be completely objective and oblivious to events however they are defined? Price is price and that is all that matters. Event recognition should not matter.

@ABT

My point is about the magnitude of an event. Actually I am very happy you have made reference to seismic activity - because in development of my algorithms I have among others worked with seismologists and as far as I remember any construction on seismic area is designed to resist only a certain seismic stress - and we are talking about human life not just money.

Maybe I should provide a little more clarification of my way of thinking. Any back testing of CHF strategies prior to Jan 15 would point to strong mean reversal. If you put on top of it any fundamental knowledge about Swiss economy and economic policies it would only reinforce this idea. It would require a certain level of genius to see it different way... and then it happened.. and the scale of the reaction proves everybody took the information provided by history as very firm. Now it is easy to make any contrarian comments but before one would most likely be considered crazy. So what is happening now with CHF back testing ? What are the hursts, roots etc now? Well it favors totally opposite sets of strategies but can one pick now what back testing points towards ? I am not sure because we probably all expect that the events of Jan 15 would not happen again anytime soon. So back testing on CHF would drive you directly to a disaster then and now. That was my point.

I understand that on top of what comes from back testing we should develop additional measures to assure we are not wiped out. And I did it because I got "only" 9.5% loss on my leveraged currency portfolio that day and I wasn't wiped out so the additional measures worked.

@ Larry and Piotr -

What considers as a "black swan" event? Taleb probably knows that better - but for us it's either something "unquantifiable" or tagged with so low probability that you don't take it into account directly.

However, these events don't just occur on their own, there is a kettle boiling somewhere, whether you look at May 6th, rolls over Fukushima or Jan 15th - there are signs of increased unknowns - which the market doesn't price efficiently. Based on ability and possibilities - either the human or the machine has to decide if it wants to participate in such landscapes when running a strategy.

Just reading these three replies, shows us that we look at "strategies" in three different ways. Whether we apply deterministic or non-deterministic models, making/taking models, or longer term I believe we need to gather data and see how it affects our modelling.

The points you raise Piotr are interesting, regardless of what kind of model you implement on anything CHF linked, there is a new landscape to adapt to:

- Liquidity is extremely thinner (at this point cannot be measured as a major).

- Trading costs are hundreds of percent higher for most market participants.

Note that mean reversal strategies are normally applied to "calm" markets - as the main logic behind it is that "things will go back to normal".

The ways I would implement the new data for such a model (just brainstorming here):

- Do my research/back-testing without the spike data.

- If it "works" - then I would start using the spike data to cover myself.

Here is the difficult part:

- The spike shows that you would drop 9.5%. How much of your earnings from calm markets is that?

- Is it worthwhile to keep picking pennies all day long if you have to hand-out dollar bills every now and then? Works for some models, not for others.

- Why didn't you get "wiped out"? Was it because you were diversified? Because you applied healthy credit rules?

- If you were diversified, run some models on what would happen if this happens in another market with greater degree of correlation to your core activities.

- What's the probability of such an event occurring again (not universally but for your specific strategies)?

- Implement a risk model that would take into consideration such an event occurring again (basic way would be to take into account a 40% drop, inability to execute orders for X minutes, etc)

- Take into consideration how a sudden 40-50% spike in one market would affect your overall capital, considering your activities on other markets.

The most important lesson we have all learned (regardless buy- or sell-side) is that diversification is key to control credit usage and "risk of ruin" and that this data is immensely valuable for every developer, regardless how we build strategies.

ABT, always interested in how others view the phenomenon. The interaction keeps me grounded and checks any creeping arrogance when conducting research. Your message resembles the check list applied here. Thanks for taking the time to share the thoughts.

@all - i'm not a headhunter - i'm looking for good c++ coders for a march - may (initial contract) - if interested - ping me.

C++ (high level).

Previous experience with financial algorithms good but not necessary.

Good understanding of the whole programming stack.

English communication at a good level.

Ability to work with different teams at different technical level.

based in London - but you can work remotely.