I think the problem is strictly related to how we compute the risk of our positions in the market. It's something like a black swan. The best we can do is to compute risk not as an average of something (standard deviation, etc.). We should consider the worst possible draw-down as a measure of risk.

I agree with Marcello, but I would like to add that the optimization is the "key point".

We can measure the risk, but in the next step (stage) we should (must) optimize the risk with our predictions, expectations, aspirations and normally with our available fund.

Despite everything, I believe that unreal expectations killing traders.

Many times traders see the risk (they measured the risk correctly), but they "deep" believe that something will happen or will not happen, although almost it's not possible.

You both make excellent points. It reminds me of what I used to tell new traders - if you are praying, you are over leveraged.

It is when the trader knows to expect a certain percentage of trades to result in losses but can't deal with any particular trade losing that bad decisions are made. Once you gain sufficient confidence in your model through actually trading it with live money, you do tend to see things in terms of simple probabilities.

I once saw over 2 million dollars lost in 3 seconds and on several occasions have seen hundreds of thousands of dollars lost in a few minutes. Maybe that gave me the perspective that however badly I mess up, it isn't anything compared to some of the things I have seen.

For me personally the problem has always been in processes which take much longer time, and therefore which cannot be even closely compared to a "deer in headlights". I can't recall last time when I made improper calculations of risk/exposure, however the main problem has always remained a change in the whole market regime, and this is something that happens not in a day, a week, or even in a month, but may take a year or more. Quantitatively speaking it leads to series of losses caused by a large number of negative scenarios which were not accounted for in the very system logic. Thus any calculations which could possibly have been done in advance simply start to make zero sense. I learnt the lesson in 2012-2013. Now I even humbly believe that I have a kind of a semi-quantitative, semi-qualitative method to predict changes in market regimes, which allows at least to properly manage the exposure. Time will tell, but so far, so good.

To the best of my knowledge it's one of the key problems for quant traders: changes in market regimes take so long, that normally are disregarded. Scott, many thanks for raising this important topic for discussion.

@Alex - It would seem as though while you don't face the "deer in the headlights" moment, you may in fact, face something much worse. After a series of losses that are right on the edge of the expected win/loss ratio, you could face a loss of confidence in the numbers themselves and the methodology that produced those numbers. The mathematics of determining the probability of a particular win/loss set is relatively well known but at the edge, it is painful and prone to producing self doubt and second guessing.

I feel this is an important topic because at some point as trading professionals, regardless of our trading horizon, we learn to deal with this issue or we leave the profession (voluntarily or involuntarily due to a blown out account).

As always, your kind words are appreciated.

Scott — painful, yes, and not only for traders. Investors tend to quit the game way before, despite them typically having agreed (or at least they think they have) to all terms and conditions, including the worst possible scenario.

However what I'd like to emphasise here is that in my humble opinion no mathematics based only on price time series (in whatever sense) and any metrics of the given strategy may help answering the key question whether it's only a temporary drawdown or the end of life for this strategy.

Just re-read my own post and thought I need to correct it a bit. When I referred to "a series of losses caused by a large number of negative scenarios which were not accounted for in the very system logic" I wanted to say that the way such a loss was achieved matters, not the statistics of losses. For example, it is very possible that the maximum known number of consequential losses is never exceeded, and nevertheless in longer terms the strategy loses money. Or even worse — stagnates, oscillating around zero.

what I got from the above discussion so far is that "fog of war" = risk

Then the eternal discussions about risk - how do you quantify and predict it, how much you take etc has ensued ...(nothing new there)

We got to remember though that traders get paid for taking risk - period. That's it my friends - you are getting paid (if you can extract any money from the markets at all of course) to "inhale" the fog of war ... on behalf of others ....

@Alex - when I say the mathematics of determining whether or not a series of losers are within the expected range I am referring to the expected distribution of wins and losses vs anything to do with price series. If I have an expected win rate of 95%, what number of losses in a row have to occur before we are at a point where we have to check the basic premise of the strategy. I think we are saying the same thing only slightly differently.

@Evo - In my case, I am not talking about the traditional risk associated with trading, I am talking about how one handles the fog of war. Some traders never really get comfortable with the natural ups and downs of trading but some go into what amounts to a full on panic mode if a trade goes sharply against them.

My question is - for those who have been able to get largely beyond that stage, how did they do it and what did they experience/see along the way.

We have used the OODA loop and applied it to trading . Also one needs to track the market over several time frames so that you cover all the time frames . Learning from the Octopus one needs to know one is a fish and one has to deal with the sharks on the FX market. Use the techniques of evolutionary survival and not rely only on algo's

@Vidur - I have been a fan of thinking about trading in terms of an OODA loop for years. If I could only pick one thing to tell a new trader, it would be to study that and fully integrate into their trading. Once you get a handle on it, it will give you a sense of calm in even the worst situations.

Plan your trade, trade your plan and have the discipline & patience to honour it.

That is why mathematics has to be used to solve for the unexpected....not just force/funnel people down a one-size-fits-all solution that you think you do know (trading as religion) One of the greatest things that ever happened to me was when I had gotten a very small allocation. I had to adjust and scale my strategy as much as possible to still be able to make money. But then there were times that movements happened that questioned my confidence. What you have to ask yourself is "how can I prevent this from happening, or lessen the blow?" So I had additional volatility filters in place put in so that I can adjust on-the-fly to handle market risk. And lots of people are tempted to simply fake the backtest result just to get their hands on an allocation rather than face the real possibility that there is risk of loss of client capital. Having support and a salary helps; although when you have to kill what you eat, it forces you to develop a robust ATS so you dont starve!

@Martin - who could possibly disagree with that idea? I bring this subject up because I feel it is important to let new traders know they need to be prepared for the mental chaos that can occur when things go seriously awry.

Keeping with the military theme - No battle plan survives contact with the enemy. Knowing what you should do and doing it can mark the difference between being a trader and being a former trader.

@Jon - I agree completely which is why I don't manage other people's money; not even close friends or family (actually especially those groups).

Scott Boulette, you have to start somewhere. It is difficult to scale without seed capital.

I completely understand where you are coming from though.

@Jon - most definitely. I took a different approach because in the end I am a really, really bad employee but a pretty decent trader.

following the price is the way we go about it, we have perfected this approach. we found a strategy that worked 12 years ago and have now been put to real production, our EA is finished we are making available 5 copies ONLY of the licence to the robot. nicolas@etradingtechnologies.com

A quote from Mike Tyson comes to mind: "Everyone has a plan 'till they get punched in the mouth."

Remember Maths can only deal with 6 real time variables in dealing with with complexity ( Hanah Fry Prof of mathematics UCL ...You tube ) You need to deal with at least 32 variables in real time icluding 4 time frames 2mins , 10 mins hourly and daily to understand FOREX MARKETS !!!!!

@ Vidur Yes, it can be quite difficult to keep up with that amount of information without having some sort of automated assistance.

Do some not use Stop Losses? Then one might also ask how to take advantage of the "flash" market. This last is what I have been looking at lately. Thanks, David

I'm quite skeptical as to whether mathematics would be able to protect us from financial meltdown or panics. We do not know what the real mechanism behind the bubble and panics are. As a recent example, the government decided to save Bear Stearns and not Lehman brothers. Official explanation would say that Bear was solvent while Lehman was not, who's to say that that's true? It all depends on how you mark your assets. Either way It's not something you can model mathematically.

Seems like the participants in this discussion really don't understand each other. It's clear that some talk about issues with very brief processes (say, seconds to days) where risks can and should be mitigated using quite a traditional set of tools (stops, etc.), while others try to emphasize the complete change in the market regime and even worse — the market structure, which completely destroys the whole classes of trading strategies, regardless of the money/risk management modes used with them.

Vidur, in order to understand the fx market you may rather want to learn the structure of the interbank market, including its infrastructure, typical behavior of its participants and of course regulations. You will see that not only you don't have to deal with 32 variables, but quite often you don't have to deal with variables at all.

@Alex, I quite agree with your assessment. I trade very short term so my only real risk is that I do something incredibly stupid out of panic. At this point in my career I have come to recognize prudent action from a panic response. If the situation feels somehow threatening, I can be pretty sure whatever I was thinking of doing is exactly what I shouldn't do.

Recently I was very nearly hit by a car while crossing the street; once I was on the other side, I started laughing to myself because I realized my heart wasn't even beating fast. I am not brave, just used to situations where you have to think clearly and quickly.

Alex can you give directions on reading material , sites which will illuminate me .Thanks Vidur

Vidur, no problem, here you are:

https://www.federalreserve.gov

https://www.ecb.europa.eu

https://www.boj.or.jp/en

www.bankrate.com/rates/interest-rates/libor.aspx

www.euribor-rates.eu

https://www.treasury.gov/resource-center/data-chart-center/interest-rates/Pages/Historic-LongTerm-Rate-Data-Visualization.aspx

https://www.sec.gov/answers/ecn.htm

https://en.wikipedia.org/wiki/Proprietary_trading

https://en.wikipedia.org/wiki/Volcker_Rule

http://www.bloomberg.com/news/articles/2013-12-11/banking-under-dodd-frank-takes-shape-with-volcker-rule-approval

Besides that it's always very useful to know what big guys do there:

https://index.barcap.com/Index_Products/FX

https://fxpa.org/wp-content/uploads/2015/06/fxpa-benchmarks-5-22final.pdf

http://financial.thomsonreuters.com/en/products/data-analytics/market-data/financial-benchmarks/currency-benchmarks-fx-spot-rates.html

And in general you won't believe how useful Google can be.

Thank you Alex

The fog of war seems to have suddenly lifted again Scott!

I just tried what seemed to me to be the obvious implication of Alex's final sentence:

http://lmgtfy.com/?q=%22fog+of+war%22+%22automated+trading%22

but the first page struck me as being entirely useless :(

Isn't (one of) the point(s) of "automated trading" to eliminate "do[ing] something incredibly stupid out of panic"?

"Robots" don't have emotions. Do they?

Hi Alex what are the heads of information you get from the regulations that makes trading less complex. ie the fixes during the day, Positions of the banks over the week end , capital flows , end of banks prop trading, bond aution results indicating Capital Flows ..... most of it retrospective information.. how does this help trading or is there inside information .... I hope I don't sound naive but appreciate your guidance Thanks

Vidur, it doesn't just help trading. It is the key price drivers in fx.

Jim, I am with you here. The only reasonable explanation that I might have is that most of trades placed in today's markets are still manual. At least this is what regularly confirmed through all my discussions with institutional traders, with a rare exception.

Trying to eliminate doing stupid things and actually accomplishing that goal are unfortunately very different things. Maybe I am the only one here who ever overrode their own trading system for what seemed like a very good reason at the time but in hindsight was for an incredibly stupid reason.

Alex - I don't suppose all those manual institutional traders shared their emotions with you did they?

Scott - Are you by any chance an ex institutional trader who can't quite kick the habit?

Jim, where did I say anything about emotions?

You didn't Alex, but I wasn't quite sure which of my points you were agreeing with. My overly concise attempt to clarify that seems to have muddied the waters even further! My apologies.

Based on your experience the institutional traders you mention never succumb to "The fog of war"? To quote Scott's words, they never "do something incredibly stupid out of panic"?

Even on a Black Monday or Wednesday?

Why, of course they do. In general their psychology is not much different from anyone else's. I'd even say that this "incredibly stupid" happens much more frequently than just on Black days of week.

I suppose the next obvious (but obviously rhetorical?) question in the context of this group is how might one go about programming an emotionless "robot" to see through the "fog of war" and make a modest profit from the activities of all these emotional discretionary traders with deep back pockets?

Jim, I can see no controversy here. If we needed to understand psychological patterns of behavior of an individual, then this would be quite a complicated task. But if we analyze mass psychology, then these patterns become much more obvious — because of a certain "averaging" if you want. I believe you heard that crowd is always as stupid as its stupidest member — don't remember who said that.

@Jim Hunt. With your tone and chosen words one would have to assume that you have the knowledge to construct an emotionless“robot” that makes a modest profit. I for one, am looking forward to some modest information of the results. Feel free to show results on 25 or more tradable symbols.

all the best

@James - My universe is obviously much smaller than yours!

In case it's of interest to you I can however reveal the results of a "superstitious robot" making a modest loss trading a single instrument for a single calendar year. A psychological experiment perhaps? Or a surrealistic one?

http://lmgtfy.com/?q=jim+hunt+superstitious+robot

@Jim, after reading several of the media articles and getting a brief understanding of your objective, I am the one that is impressed as I will list:

The very small lose at the end of the year was a total success.

The fact that you started with a method of trading the markets that developers usually do not look into until they have tested a few thousand

or more strategies.

I did my studies with cycles many years ago including the Lunar Cycle. I concluded that cycles are minor signals for my style of trading. But, at the same time I gained a ton of insight into the markets. The full moon shows a deep insight into how the markets work based in human nature. When the moon is full men become more aggressive and will take on more risk in the markets. Jonathan Kinlay recently wrote a article about a cycle he called the

Halloween Cycle of which in stocks was very powerful in the 19th and most of the 20th century. My name for this cycle is the 100 year cycle.

Describing my cycle, was simply to buy good solid stock’s + or minus one week of Oct. 15 and sell them + or minus one week of Mar. 15 and

then put your capital into CD’s until the next cycle. Typically over this time would provide about 200 to 300% more ROI than buy and hold.

Today, I use the term calendar events to cover all cycles. Calendar events can be very powerful with some styles of strategies.

If you decide to do any more test I would be interested.

all the best

James, maybe you could recall a report from one strategy that I published here about a year ago? It didn't use anything but just date/time, and did very well. So I am also all into cycles and calendars.

Thanks very much for your kind words James. As regards the media coverage, this is the one that caused me the most merriment:

https://youtu.be/UPW8zJmKlJI

Max never mentioned either Sid or his creators but nonetheless, and most unfortunately, Sid split his sides well before the end.

@Alex, I do not recall the report. Can you make it available to me? What I will be describing now is something that each and every person should

analyze. The result’s may have no benefit or very powerful.

Make a list of all U.S. observed holidays and mark the exact day of the holiday.

Now go back in time and mark the date that is 10 trading day previous to the holiday date.

Now mark the date of the next trading day after the holiday date.

This is your test period.

First note the number of long trades and profit of those trades.

Next the same for short trades.

Put the information into a percentage relationship.

Typically, what you will find over many years or say a hundred different symbols

is shorts are not productive.

Why is that? Again back to human nature. With a up coming holiday humans (read as retail public)begin to have a

more positive attitude = more willing to buy stocks.

So, how then does the invisible hand take advantage of this to make a profit.

As you analyze the test period on a daily bases you typically a nice dip down day(invisible hand buying........the average is about 7 trading days)

What are the odds of standing in front of a massive freight train(read as the invisible hand) after the dip down day with a short?

How close did I come Alex?

all the best

Very close. There are many more periodical/seasonal/cyclic patterns like that, in various markets and of various duration. My personal preference are intraday regularities in fx, but there are many more even in this market. If one understands the reasons which drive the price here or there in a particular market, then surely he is able to spot these regularities.

Not sure if I have that performance report still online anywhere, maybe I will need to upload it to somewhere as Linkedin doesn't support attachments or other media in discussions.

It's all about the "who, why, where and when"

@Jim I am an ex-institutional trader who decided I would rather make money for me than for a big institution (although I am not opposed to small institutions).

@Alex - Re your recent "education" post as well I guess, did you get much interest when you originally posted the performance report you mention?

Despite all the international "publicity", and the fact that he's open source, interest in Sid was noticeable only by its (almost) total absence!

@Scott - The smaller the better! (IMHO of course, and within the bounds of reason)

@Jim I have found trading at large institutions is not to my liking but have often thought that a small group of highly skilled traders/statisticians/technologists could build quite an impressive firm within a relatively short time frame. As with most things, all it takes is skill, luck, money and time : )

@Scott, I would be inclined to think such a group would be very successful.

Depends on how to measure the degree of success. If in relative figures (that is, percent returns on account) then most likely it will be considerable. If in absolute (just amount of dollars in the long run) — quite moderate. Ironically so far I personally met only one (!) really skilled trader or a technologist who could risk, say, 1M. All others didn't have such an amount to risk. Paradoxical, but true.

Jim, my posting of those results was quite successful: I received even two enquiries from guys who were willing to purchase that strategy for an incredible amount of money (about $100 if I remember it well).

In general, an illusion of "international publicity" multiplied by the amount of information leads to even further strengthening the role of traditional mass-media. In other words, after explosion of social networks it became even harder to promote anything on the web than before. I once wrote quite a lengthy and a bit philosophical post about it there: http://trading-as-religion.com/?go=all/vogue-big-data-and-big-hoax/

@Alex - in higher frequency trading, capital usually isn't much of a constraint. The primary issue is time. I could tell someone a dozen things to do to make an algorithm better (or even work at all) but each of those things would take a week to a month or more to accomplish.

Running multiple algos on different time frames? Write an internalizer to hand off positions internally among the algos where possible. Want to maximize a very short term scalping algo's profit target? Put out multiple profit exit orders and cancel all but the one that represents the optimal profit target given relative book size, speed of trades, delta and possibly percentile position over the last N seconds (or any variation of the dozens of measurements that you use to decide when to exit a trade).

What I described just above is weeks worth of work to even get the basics working correctly. Trading all day doesn't leave a lot of energy to code things that frustratingly enough, you know will make you money.

Costs : and buy or lease a seat .

Scott, if your development cycles takes only a week or a month, or even a number of months, I envy you. In my case it takes much longer, of course not to actually code, but to understand what goes on in the market (long term). For example, it took me literally 3 years (!) to come across an idea which eventually transformed into 5 lines of code. And this tiny beast works phenomenally well, independent of what changes in the the main market happen.

Alex - It is my turn to envy you; you often can't even initialize a parameter set in five lines of code. I was really talking about execution oriented strategies; strategies with a longer term view do indeed take a very long time to develop.

Even the most straightforward and obvious things we as humans can recognize are often very difficult to reduce to code.

Scott - Re your "impressive firm" suggestion, I think I can safely characterise myself as a "technologist" with less than Alex's $1 mio in the bank!

As you may have already gathered, I'm quite fond of collaborative projects, even "open source" ones! I was part of the original AlgoTrader v1.0 team, before it went "commercial", responsible for the market data/execution interface, plus a couple of example algos. As luck would have it I happen to have a fork of the project that runs Enterprise Java, MySQL and FIX on a headless Raspberry Pi!

Is there any common ground there? If you can live with Java I have even had discussions about a Quicker FIX:

http://trading-gurus.com/is-quickfix-j-broken/

I fear, however, that HFT may be a step too far for my RasPi/Java combo! Perhaps I should ask Alex nicely about his "5 lines of code"?

hypothetical

http://screencast.com/t/lFASvyjZ

http://screencast.com/t/6zWF6CFz7b

http://screencast.com/t/4Ler8VksO

all the best