Maybe:

1. Your risk moves beyond some pre-determined threshold?

2. Volatility filter

@ Sumit, John – Thank you for your input; I definitely use stops albeit held at the co-located server and additionally I have them monitor size before going live to avoid a 1 lot trading into 1200 on the book triggering a stop out. I also monitor volatility.

I am trying to avoid a lot of false positives while keeping the option of shutting everything down when necessary in the less than 2 second time frame and this is all the while avoiding parameters that can lead to either curve fitting or require continuous modification.

While my algos do not try to pick a local optimum, I am not in the business of giving money away by standing in front of a train. The primary issue is when some market participants have information I don't or the more likely scenario, they have information about an upcoming event deeming it important whereas I didn't and therefore do not have the event in my database of no trade times.

One thing I am considering is a filter based on the last price cleared from actual trades (i.e. that was traded through vs. simply having orders cancelled). For example, if there is an event and price trades 8 ticks with no retrace of a full price level then shut down. Obviously both the 8 ticks and the 1 full price level cleared would be parameters but the idea is something that would handle 2+ standard deviation book sweeps.

I think your approach is good. Logic based on twitter analyses will have too much noise.

@Deo - I agree; this is strictly from an algorithmic point of view. If the person watching the trade goes to get coffee and something happens, I want this to automatically shut everything down. I will likely have more than one action level.

Likely the first level is a pure alert and from there next level would be to cancel all orders, then cancel all orders and get flat with no more orders going out until a human intervenes to restart the algo.

If by algorithmic circuit breakers you mean psychic ability or even a crystal ball, you are going to have abort this idea altogether.

The stock market is too dynamic and algos are easily tricked by a savvy trader, just look what Sarao did.

What really matters at the end of the day is whether or not the market maker for that particular security is at his desk or on vacation .That’s the only way I think an algorithm could work very well, but will still have many limitations.

In a nut shell, every individual stock behaves differently according the majority operator for that stock at the point in reference. Many have tried to develop models in an attempt to standardize trading strategies, but this will ultimately fail in the long run as economic theory states, profits will always equal zero.

In other words you can try and be successful in the beginning but if you are holding a position overnight, the algo will only work during market hours so you have no control if it the stock opens down $20 a share. You can never full proof any trade like John Lazars reference towards Swiss franc. FXCM almost disappeared overnight because of it.

I just think this would not work (trust me I’ve thought about this a lot) as it assumes the market works in a vacuum. There is always some form of inherent risk when it comes to trading, even with an algorithm. Hope this helps.

@Nicholas - no psychic abilities are required for what I am talking about. I have no need to predict anything, I just want to detect that the market is either very unstable (as in flash crash unstable) or that it is no longer a two way auction because every trade is unequivocally on one side of the market.

Really, those are my only goals. I have found my single greatest risk never comes from the market, it always comes from me thinking I have something so nailed down another opinion can't help me. I have made that mistake in the past but luckily it is an easy one to correct. And if I can't do it on my own I have a wonderful wife who will gladly point out I am not nearly as smart as I think I am.

As to hold times, an algo of mine got "stuck" in a position for an eternity today - just over 3 minutes. I didn't intervene but I have to admit I thought about it.

I spoke to Stefan Teis of Deutsche Borse about this last year. His research reflected your own conclusions on the widening of the spread. He finally release his paper on this at the end of year (see below). Not so much new information for you, but may be of interest.

Link to the paper:

http://www.iwqw.wiso.uni-erlangen.de/forschung/iwqw-discussion-paper-series.shtml

It is paper no. 15 in 2014: "Herrmann, K., Teis, St. and Yu, W. (2014): Components of Intraday Volatility and Their Prediction at Different Sampling Frequencies with Application to DAX and BUND Futures"

Scott- I will try to find an article that I came across while working at Liquidnet that may help you.

@Philip - Perfect! Thank you very much, that is exactly what I was looking for. I always appreciate confirmation.

@Deo - Thank you; I appreciate all the help I can get.

I am now looking at a combination of the outright spread, the spread between the last trades at bid/ask and the spread between the last fully traded levels bid/ask. I will start capturing the data and once I have any statistical confidence in the approach that appears effective, I will let everyone know.

Not to hijack my own thread but these exchanges are exactly what I like about this group. Seeing another discussion about where EUR/USD is going next or the evils (or benefits) of HFT or whether FINRA is effective at reining in the bad actors in trading doesn't help me make money. Seeing these comments is what helps me make money.

Just off the top of my head, Scott, but I would lean on the options market for the instrument of focus. Institutions will buy the crap out of the direction they perceive a market pop, and if nothing comes to pass, draw spreads out of those long positions post hoc. But when their newsreaders start singing, they start buying.options (and selling longs). If you check out the CBOE Daily Market Summary, you'll see a great correlation to where the market *had been* for that day, and much less of a correlation to where the market may go tomorrow. Intraday is the thing. If you could hook up a zone alarm (_X_ strike, at _Y_ % above expected candle average), you'd have an instrument, AND a guesstimate of magnitude, in one fell swoop.

http://www.cboe.com/data/mktstat.aspx

And by the way -- "Yes!" to watching the bid/ask spread grow. Use that every FED announcement!

@tom - I didn't know economists watched spreads : )

What is this -- with the economist-in-the-room always being branded with the square peg for everyone's round-hole issues??? I dunno! I dunno....... {cue heavy sigh.....}

"I don't get no respect." Rodney Dangerfield, economics major at Faber College....

@Greg, in the 2014 paper I provided a link to before VPIN is discussed. Bottom-line, Stefan didn't find it useful in real-time.

http://www.iwqw.wiso.uni-erlangen.de/forschung/15-2014.pdf

@tom - I always think of you as a trader, I was just kidding about you being an economist. Oops, I guess I got that from your profile.

My bad, Scott -- I caught your tease and thought I'd cued up the Whiny-Snarky Font, but no-oooo.

{Cue Homer: "Stupid Whiny-Snarky Font switch..."}

Anywho -- I've been meaning to cue up a correlation study between the CBOE Daily Market Summary statistics (being primarily put/call ratios) and market moves on "Day n" and "Day n+1" and to my embarrassment, haven't gotten to it yet.

But it's on the white board.

(As is "Pay taxes" and "Shovel the walk.")

Hoo boy.

Tom and Scott, I think there is a really wide gap between someone (Scott) who has a trade space of about 2 seconds, and someone (Tom) who is talking about daily CBOE market stats. I do not want to upset anyone but you are plainly talking about very different situations and types of trading.

Scott I am confused as to what it is you are seeking with this thread. I mean, no one has access to the thinking and actions of all other players, you cannot game them all and win all the time. You currently have a working algo that stops trading when certain levels of specific imbalances occur. That is fine. However, you also seem like you want to extract every single penny of profit you may possibly get out of every trade. I do not think this is possible, not in an automated way and not in a way that will not expose you to very significant risk levels.

At the level you are trading and with the number of players gaming the market at that level, I think (personal opinion) it is best to err on the conservative side and just forgo a predefined set of potential wins or earnings. You can do this automatically with an algo, as you are doing now. What you cannot do is to try to second guess the algo and allow your personal feelings to intervene.

I would suggest an additional, and longer time interval, regular review of trading performance do. I would suggest daily intervals. You define parameters to determine good results, define a borderline range of results, and define a level of results that is unacceptable. You may use a broad measure of performance in these determinations, for example the % of losses or similar factors, your choice. When the daily review results are "good", all is well, there is no need to change anything in the way you are trading. When the results become borderline you may need to review and tighten your control algos. When the results become unacceptable it is time to go into paper trading until the situation clears up or until you revise your trading algos. I would suggest being conservative in your definition of unacceptable.

This is just a very broad suggestion that I thought you might like to use. Reading your comments I thought that you may be so involved in the detailed events happening in the 2 second trading interval that you might benefit by adopting a much longer level of regular review processes, something on the lines I described above, again you choose the details. All the best.

OC: Great post. Scott and I have been around the block a bit, and (although Scott forgot it here!) we joke frequently. But in there, too, I am seeing Scott asking a more-general question, and perhaps seeking precisely the sort of answer you provided above -- vis, 'meta-time-period' evaluations, for example. As well (and Scott, this is for further clarification to you), no, I was not recommending that he evaluate his intraday patterns with end-of-day data, but to signal him that such data exist, and that my *eyeballing* the data indicate persistently that put and call volume excursions happen in real time with market price excursions for the item-of-interest. Once the data were assembled, running a check and coding an alarm would be trivial.

let's say you have a system that sells into strength and buys into weakness and you have good stats that have shown you can capture three quartiles of the optimal dominate cycle. you have a solid base to know when something is out of boundaries. either set on the sidelines or employ an opposed alternative system which can complete the fourth quartile.

@Oscar - I have known Tom years and highly respect his opinion (and humor). If he tells me it is worth my time to check something out, that is good enough for me.

As to your confusion regarding what I am seeking; I am seeking ideas I might not come up with on my own and am more than willing to share what I know in return. I like to think that this is a group where the goal is an exchange of information. I prefer to think in terms of collaboration instead of competition.

To be clear, I am looking for ideas on how to determine that the market is unstable, is no longer a two way market or is about to be in one of those two states. I have a lot of things I currently do but that doesn't mean I can't improve by getting other opinions.

Thank you for taking the time to offer your suggestions; I appreciate it.

Thank you Tom.

@Scott. Yes, I agree with you.

@Tom - Apparently our posts crossed somehow. Nope, I didn't forget on the humor front and yes, I understood exactly what you were saying.

Actually the answer I am seeking is at the level of something I can code. Jenny gave me a couple of ideas I hadn't looked into before and several others reiterated ideas I had thought about but hadn't made up my mind on and some confirmed my existing views. A pretty good deal from my perspective.

@Scott - When the Hide Not Slide orders came into play we found that we were getting adversely selected. We did not have the capability to submit orders directly to the exchanges and so could not use the specialized orders required to play in the new game.

In order to deal with this we investigated various methods to determine the short term market direction so that we would only buy/sell when things were going our way. As part of that I looked into VPIN.

We found that the VPIN does have predictive ability, but only with very heavily traded tickers. This is because you need a lot of trades in a relatively short period of time to determine accurately where you are on the CDF. That position on the CDF is what tells you if you are entering a high volatility situation. We tend to trade very low volume tickers, so this was not very useful for us and we moved on.

@Greg that makes sense; thanks. I trade futures exclusively at this point; primarily CL and ES with some GC, ZN and the currencies (6E, etc.). Do you have any familiarity with using VPIN on those symbols.

It is likely a discussion for another thread but if you are market making, there are a some simple filters that will reduce adverse selection but at the cost of fewer trades. In many ways it is a function of how you want to manage inventory.

i have found that if "i" am going to kill something "i" must focus on one prey and not the heard. if "i" get distracted from from pursuing "1" target then "i" wind up with nothing. not for everyone.

@Mark - I am with you. One of my favorite quotes ever - Bruce Lee

I do not fear the man who knows 10,000 moves, I fear the man who has practiced 1 move 10,000 times.

@Scott, sorry I have no experience in the futures market so I can't help you with those symbols. However, I'd be very interested in any filters you can point me towards to reduce adverse selection.

scott - i know exactly what your talking about - rick saidenberg and myself were skeptical also of s/r as you know they seem to be self fulfilling because humans can not modify their behaviour to change.

It is a common idea to use market conditions to turn algos off.....You might also want to think about market conditions that can also be used to turn algos on. Or at least be alerted to make sure proper position size is being used for the current market environment. Using technology to monitor different types of market conditions can help you digest significant amounts of data more easily.

Scott, I think it might be worth considering a real-time measuring of widening spread and your 8 pip jump, happening at a same time, especially if the move is against you, and if both conditions are met, flatten. Would be good particularly with low vol strategies and hard stops.

scott - one reason some analysis works so well is s/r and % is absent of the pollutant "time".

@Mark - most excellent point!

You should never be working a limit order before a scheduled release. If your buying at the bid and the news is good then it will rally without you getting filled. On the other hand if the news is bad you'll get filled but immediately be carrying a loss. That's a terrible risk/reward scenario. Indeed Interactive Brokers offers a service that will automatically pull all your limit orders around scheduled releases. For unscheduled releases you can listen to websites like ransquark.com and tradethenews.com. Those guys sit there all day looking for these kinds of things.

Not sure if anyone has already suggested it because I only looked through the thread quickly, but has anyone tried to use the rate of data flow itself? I mean, number of ticks per second, sort of that. Works like a charm with fx spot data from most ECNs and similar live feeds, almost doesn't work with composite. Due to this very reason it should work with currency futures, and most likely doesn't work for ES as most events which may provoke unwanted volatility are at the edge between Globex and regular sessions.

Sorry if it has already been discussed and I missed it.

Alex, yes we use those frequency features, that is, ticks per sec, both for entries and exits or as a complementary signal to support other signals (increased signal quality). In terms of entries, you would want to measure the direction of the time (seconds) you use and be able to break-out or reverse.

Hmm. Never considered it as an entry rule. Exit — maybe, but not entry, especially in the breakout/reverse context. In my opinion the risk to be behind the market is too high. However maybe we simply talk about different things because I consider only conditions when the tick flow becomes really slow — and you can use whatever metric to decide it, stddev + percentile or anything. Sometimes even the hardcoded values work best, like if we can't get any tick within 5 seconds and time is like this then...

I write progressive algorithms... the opposite of this approach may help you filter out disadvantageous scenarios for your trading...

whether the tick frequency is increasing or slowing, both can be used. However, in my experience these tick frequency measures works best in combination with other signals, such as B/O, as complementary, that is, go... if both conditions are met...

Bjarni, to make this discussion more meaningful — what exactly data feed do you have on your mind when you say this?

spot FX tick data

This means nothing. Which source?

Of course this is meanignful only to people that trade this kind of security and use a very short trading interval.

This has no meaning because fx is not an exchange-traded market.

@Alex - no one had suggested using what I think of as "arrival rate"; I have that coded and working but getting the parameters correct is problematic. If that issue can be solved, it is the best overall solution for a circuit breaker that works with a strong market reaction.

One way to do that is a variation on fractals. Take the coastline of a bar (in ticks/pips) - for an up bar, open to low, low to high, high to close, then take the number of price changes (not book changes) traded during that same period (a series of ticks at the same price only counts once) and calculate the jitter in the period. If you keep an absolute average and a moving average of that, you will get a very good indication of the "agitation" in the market. The one issue is a sharp move will often produce a long coastline with little back in forth in prices resulting in an artificially low number.

@Bjarni - I missed your earlier comment regarding flattening - yes, that is how I use it.

Scott, interesting idea to calculate the jitter in the number of price changes. Have to give it some serious thought. As to my suggestion to estimate the arrival rate — normally it works as a leading indicator, if we observe serious pauses in the data feed then it's best to switch off certain systems, kinda that. I doubt if it can work after the event. Again, I'm speaking only about fx data feeds from ECNs and aggregators, not sure if it can be used (or even observed) in ES, for example.

It can be observed in ES and is useful but quite difficult to know what is a lot or a little. I use it to avoid a situation I think of as mada (mom and dad arguing). When you see that highly agitated back and forth in the ES, often one side is going to win and one side is going to lose - rarely do they just call a truce (those I know who use this always call the winning side mom since that mirrors the real world).

So it ends up being an indicator that something is up that I likely don't want to be a part of. As a counter trend trader, you want to avoid situations where price could move suddenly and significantly against you.

@Scott - So, we've established that the spread is useful, but it's one dimensional. It seems what you're looking for is a way to make it multi-dimensional in case there are issues occurring at time-slices other than just the Best Bid or Best Offer...

I'm a big fan of tick compression charts, because they smooth out the trend in respect to the change in flow rate of ticks. The problem with them is that each market has a different average tick rate, so you need to manage the compression as relative to the average tick rate for that market.

Then once you have a normalised tick-bars then you can start looking at the vol of those different dimensions... so instead of just time based volatility you have normalised tick based volatility. This way you should be able to get a different view of volatility across different time dimension to look for expansions that could cause problems.

Anyway, this is something I like, but I'm not playing your time frame. Hope it helps.

Scott, just for clarity and out of curiosity — which data feed are you using? Is it time and sales or anything of the higher level?

@Philip - you are absolutely spot on as to what I am trying to get at and what the primary issue is. It even extends to the issue of different portions of the day have different expected tick arrival rates. The last few minutes before the London close will be significantly different than the lull around lunch time (US East coast time)

I have a direct market data feed for CME (all its sub markets) and ICE.

@ Scott - now that's a very interesting concept. So yo don't average the ticks over the day... you average the tick rate and compress it for same time block on previous days. So you need to define time windows... and from what you're saying not an arbitrary window, but one based on physical events...

In our environment we'd do that by creating a gap-adjusted time-series of the compressed tick bars over the time windows from each session and stitch them together as an indicator. Well, technically that's how we'd do it, but we're not looking into DMA atm. Reminds me of my work years ago comparing days of the week and days of the month...

Monday and Friday's always the most profitable... Take care mate.

@Philip - I will keep you updated directly on how this goes.

how much time are you spending with this how much money is this costing you and how fast do you want a solution for it... send me a message joelackland13@gmail.com

The VIX is the best I know. The market averages have been shown to incorporate economic news within minutes. Hence, your algorithm has to be fast and you have to be careful to decide whether all the work is better than simple stops, etcetera.

@Joe - I have most of this working at this point. I am always looking for new ideas on how to find a better balance as to approach - not a lot of lag but not a lot of false shut downs. It is to avoid a sharp move that a human trader couldn't react to quickly enough or to avoid a situation where a known event is close but someone missed scheduling a shutdown (or more often going wide on the spread).

I have been in the business long enough to see a couple of million dollars go down the drain in a few seconds because a junior support guy forgot to update a database. In that particular case, the firm fired the entire chain of command from the CTO down to the guy's boss; interestingly enough the actual guy who forgot didn't get fired.

@Scott. The firm did the right thing, anyone can make a mistake, the people above the one making the mistake have the responsibility for finding out the mistake has been done and prevent loss. I bet this junior support guy never forgot the lesson.

@Oscar, watching that kind of money go down the drain in literally 3 seconds isn't something you ever forget; it was chaos to put it mildly. The head of risk control, the head of trading, the CTO, just about everyone on the floor was in a panic and screaming at each other trying to determine what had just happened. The best news is it wasn't one of my algos (thank heavens) and so I could sit back and just watch.

@Scott. Yes, my friend, and the moral of the story is that that kind of trading can be dangerous. You can wipe out a lot of capital in seconds because of minor errors or glitches.

@Oscar - now you know why I am so obsessed with every single detail of my algos and specifically this thread.

@Scott. Yes I understand. I also understand that you are burning yourself like a candle. Who can live with this kind of stress? I am not trying to offend you, I am trying to help you.

@Oscar - Not to worry, I have periods where I work 70+ hour weeks and I have periods where I don't have as much to do. It is a combination of things not the least of which is the MDP 3 data feed protocol cut over dictated by the CME and all the headaches that go along with that.

But seriously, thank you for your concern; I truly appreciate it.

As a finance professor I want each of you to know that I really respect what you are trying to do. There is a major body of research going on in HFT that is being published in the top 3 journals: Journal of Finance, Journal of Financial Economics, Journal of Financial Research, and Journal of Financial and Quantitative Analysis. Sometimes it helps to see where academic studies are finding both statistical and economic significance. What you are trying is hard work. We all know that.

@Dr. Scott Brown (@Scott seemed like it would cause confusion coming from me)

Thank you, I will definitely check those out. I read every academic paper I can get my hands on and have found a lot of articles that have very real practical value.

I would prefer to call you Dr. Brown; I mean that respectfully but was concerned it would come across as snarky when written.

I will try to remember to come back here to post anything on HFT algorithmic trading here for that I find. Let me know if that is exactly the information you are after. I subscribe to each of the four and read each that comes in for my own research.

For instance...

This highly regarded team finds that day traders in Taiwan make it small stocks around earnings announcement. Not HFT but worth the read since they studied the actual trading records of a half a million people over many years.

Barber, Brad, Yi-Tsung Leeb, Yu-Jane Liuc, and Terrance Odean. 2013. The cross-section of speculator skill: Evidence from day trading. Journal of Financial Markets 18. 1.24.

You can read an old version of it on SSRN.com here free ...

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=529063

Could program in same 5 day window around announcement dates based on this. Out otherwise.

Pay close attention to their market micro-structure analysis of the order placement strategies of the winners relative to the bid and ask.

@Dr. Brown - I really appreciate that. My particular area of interest is in market microstructure and even more specifically, tracking trades and order placement of market makers and commercials (especially the ones that are not obvious) and using that information to predict market moves (at a micro/tick level).

And another I found today in the course of my work:

https://www.pinterest.com/drscottbrown/post-earnings-announcement-drift-stock-investing/

Don't worry about slippage when liquidity is ample according to our work here: SLIPPAGE AND THE CHOICE OF MARKET OR LIMIT ORDERS IN FUTURES TRADING http://www.researchgate.net/publication/46539020_SLIPPAGE_AND_THE_CHOICE_OF_MARKET_OR_LIMIT_ORDERS_IN_FUTURES_TRADING

@Dr. Brown, thank you very much.

If you are trading at the 1 minute level you have to pay attention to how you are hammering or getting hammered by the bid or ask. That is why I said to look hard at the Odean work on the Taiwan exchange above.

Don't believe anything you hear from anybody until you find validation in your own trading. Experiment new ideas with small amounts of capital.

Treat each of your ideas like a new division in your family financial empire. Reward profits and cut capital to losing ideas as CEO and family financial steward.

Don't trust any of your ideas at first: test and under-trade in size and frequency.

I am an academic researcher in market micro-structure from my doctoral dissertation in finance at the University of South Carolina because of my background as a long term futures trader. The CBOT sponsored the work because I figured out a way to test if slippage really was a form of inflicted pain from the floor. I showed that it was not. Traders get as much good slippage as bad but they tend to forget the good. Hence the urban legend that the floor somehow picks off our trades is not true. There I go peeing in the punch bowl again at Wall Street cocktail parties. LOL.

I am long term not short term. Hence I did not take the time to decipher how the profitable Taiwanese traders are doing it. But my intuition is that there is something interesting there on the HFT programming level. I think their passive traders are probably market order placers and aggressive traders are limit order placers but I am still not sure. They got the data from the Taiwanese exchange. You could also try contacting one of the four professors on the study but be very respectful of their time. Academics in the end are a sharing lot with students. Keep me posted and I will try to keep you posted. :-)

@Dr. Brown - my hold times are normally well under a minute but can go as long as 3 or 4 minutes (that normally is not good news). And as you suggest, I always test promising ideas live with real money at the smallest size suitable for whatever it is I am testing.

You are certainly HFT. The 5 day window around earnings announcements is clearly interesting in equities. It is also important to notice that Keynes was never profitable in futures or currencies with his fundamental analysis. That said I know a currency trader who does very well timing off of central bank announcements and a few key macros like non-farm payroll. He sets up well within an hour of open and initiates. He times his entry as a day trader but is willing to hold profitable positions for months if the trend goes that far. The problem you face is that you are modeling all this with rigid programming as compared to a highly trained and flexible conscious and subconscious mind such as that of my friend. I will keep posting on anything statistically and economically significant for timing at the near instantaneous level. If you can clear define events proven profitable in recent literature you will have a good path. That is the non-academic reason I stay current with the academic finance literature in the top 4. I know you do too I am just recapping to help this community.

@Dr. Brown that is definitely appreciated. I do think you might be surprised how prevalent significant trading experience is in algo traders. I speak regularly with a trading colleague who has a PhD in Mathematics and we collaborate quite a bit.

One of the primary reasons that works is while his quantitative background is far superior to mine, I have a decent enough background in that area but I have the practical advantage of knowing that certain things work because I have seen them work for years and have strong reasons to believe they aren't likely to stop working any time soon. He is a very good trader in his own right and just today we were speaking about how difficult certain situations are to capture with pure logic and numbers such that they can be programmed.

There is an entire body of algos that do basically what a human does, just faster, with no fear and with only the rare mistake. You may want to check out Mark Brown's thread on What is Perfect for another view on the subject.

I know you are at that level from our prior in this thread. That was what I meant with the "I know you do..."

This is a fascinating as to whether HFT can really extract positive alpha consistently over the years for one trader.

I am in my prime as an academic. I have to teach doctoral students. I am probably reading a much broader range of literature in finance than any of your group because of the nature of my responsibilities. Hence I will report back even if I suspect you might already know something.

Of course clues are not always in plain sight.

For instance notice the heterogeneity in the traders in Barber and Odeans, et. al. piece in

Barber, Brad, Yi-Tsung Leeb, Yu-Jane Liuc, and Terrance Odean. 2013. The cross-section of speculator skill: Evidence from day trading. Journal of Financial Markets 18. 1.24.

It is not an algorithmic trading article but it analyzes traders who operate in that time frame and frequency. Hence that is why I brought it to your attention.

And it could well be that there are 100 HFT traders taking money from the nearly half million who lose trying to day trade stocks, at least.

That leads me to suspect that you or I could literarly write down what works to make money in the market right here in day trading stocks and 99 of a hundred would get it wrong. And it would not be due to market inefficiency.

It would be due to heterogeneity in day trader skill. I have come to believe this intensely over time.

This is just another studies that seems to me to confirm this. Marshallian economics is prevalent in finance.

It focuses on the marginal investor such that the individual characteristics are forgotten. Prominent experimental economist Glenn Harrison used to call this "scientific" approach "throwing the baby out with the bath water."

Yet here we have a study that says that most people are klutzes when it comes to operating on their own.

Clearly experience and skill is the big factor in the 1 in 900 who make day trading stocks work. I also talk with another Wall Street group behind the scene loaded to the brim with Ph.D.s in math and economics.

Despite your confidence in those programs to model successful trading behavior I do not buy the razor clean fit.

Could you drop the link for the Mark Brown thread here for me and everybody else?

I am extremely interested to read it. I know how smart all of you guys are --- and how stressful it gets. For those about to HFT, we salute you. Thanks! :-)

This is the link to Mark Brown's thread What is perfect - it is in this same group

https://www.linkedin.com/grp/post/1813979-6004422844358934528

@Dr. Brown - it is funny you mention that someone could post the exact specifications for a trade and most wouldn't be able to execute it properly. I actually made an offer once to provide anyone who sent me a pm the specs on a working strategy. I received a lot of pm's and even stayed in touch with a few people because of the offer but the vast majority never got anywhere and some were pretty close to hostile when it turned out I had not personally delivered an automated teller machine to their home office.

Never worry if I have seen something and never worry I will wonder why you thought I wouldn't already be aware of/fully understand... I got one of my best insights from a guy that has a very difficult time putting a string of coherent thoughts together but the one time he managed it, he was so spot on that I was stunned, not because he thought of it but because I hadn't.

That simple widening spread is a good tool to detect simple finance number times (like unemplyment..) I have done an indicator for that, which calculate average spread (non integer) so very easy way to filter out those. As using myself very short bars (2 sec) I have volume filter and time filter (as not so sure that spread widening is hapening all the time) in use too. Plus one ema based if price is totally too far away from "standard" (very fast movements).

It is crazy true, isn't Scott? At the end of the day we have to be willing to pull ourselves out of bed and work all alone in a home office as HFT traders or long term traders.

Then when we find a strategy that works we also have to be willing to work the strategy.

That is why I personally believe that the 899 of 900 people who fail in Taiwan each year do so because they come to these markets to get something for free.

And NOTHING here is FREE!

We have to develop our own approach and follow our own light. But it is amazing how the 899 out of 900 spend so much time looking for the answers outside of themselves that they can't see it when it si in front of them. That is how it was for me at the beginning.

You are right that you can program the characteristics of any historical trade.

But the problem is that whether we trade HFT or very long term whatever strategy we use has to offer odds better than a fair coin toss looking forward. And even so against that slight edge is a massive stochastic backdrop.

At the HFT level arbitrage is a furious force that does not operate over the long term where wide variations from equilibrium can occur. I know that you need to be aware of these potential high probability scenarios as an HFT trader.

And I will watch out for them.

Earnings releases are a clear candidate. Credit default spreads are another. Non-farm employment is another Marko mentions just above --- thanks so much for that insight Marko.

I will look for specific references for you. :-)

@Dr. Brown thank you, I look forward to seeing what you come up with.

it's not that hft is like thief stealing from the masses. higher education's agenda seems to vilify anything that resembles profit as somehow unjust. in fact the hft i have been involved in and the only other hft i know of and have been involved in preys upon banks inabilities to sync exchange rates amongst themselves. not taking some widows money as some have portrayed. anytime anyone thinks outside the box the intellectual community immediately admonishes those thoughts.

it's a wonder those who have advanced mankind have survived "many didn't" the revelation. it simply does not pay to be caught outside the box there are no rewards awaiting anyone who does so, and this fuels speculation unabated into a frenzy of fear. boo hft..

I am not vilifying anybody. The HFT market is not a lot different than the WPT.

You don't see anybody at the top in a tattered Budweiser T-shirt and a Playboy cap drunk and going all in while ogling off to the side at the shapely rump of the cocktail waitress.

But you see tons when you walk the casino floor during the day.

And contrary to popular belief the 899 don't jinx the dice when they bounce it out of the pit off the rail. Statistical runs can happen anytime over there.

But fair dice are random asymptotically.

These markets are massive modern day casinos where the odds do occasionally tilt to the astute.

But the astute are in it to win it. They study, practice and sharpen their knives before walking in.

The 899 are chumps who heard somebody got rich at it and came in for a sniff. All is fair in the markets and war. :-)

@Mark spot on! One of my favorite lines ever is something I said to a CEO once when I was doing some consulting.

He told everyone he wanted them to "think outside the box" as seems to be so common these days. I immediately offered him a suggestion to a major problem and he said "interesting idea but we can't do that". So I just looked at him and said "oh, I thought you wanted me to think outside the box but you really meant only as long as it wasn't outside your box". To his credit, he thought that was pretty funny.

Thinking out of the box is the best way to get fired when working under poor management. If you lose you get fired for the loss.

If you win you upstage your boss.

Congress and those CEOs behind the real estate collapsed should be vilified. I find it a striking social statement that hundreds did jail time in the RTC crisis yet not one was served for trail in this last.

Card counting profits are hard earned. Unlimited poker pots are hard earned. Trader's profits are hard earned.

But nobody profits trading inside the box. Again, however mahogany paneled you want to make the market it is a casino. Good luck you guys.

I will post here what I find.

About 15 years ago, I thought of a similar "shut off order execution" mechanism but based on any indicators of bad internet connections or intermittent or increasing latencies to servers. Never did that though.

This is split into two comments because it is too long to fit into a single post.

I thought it might be helpful for me to summarize the ideas outlined in the comments along with the issues associated with each approach I found actually viable in live, real money trading. Overall, I use circuit breakers in three broad categories:

* Indication that the market is anticipating an event i.e. a report is about to be released

* Indication that the beginning of a fast move is underway

* Confirmation a market move is underway (or alternatively, normality has resumed)

The first is the most important for me and the genesis of this thread as I have many ways to measure and react to the other categories. Market making and the issue of adverse selection is a well developed area; there are many sources for information regarding this, both in the academic literature and in practical applications. Discussing market making and the issue of adverse selection would be interesting but is worthy of a separate thread. However, in practice, this is highly proprietary and thus little real information is likely to be made available. As to market moves (trends), there are seemingly boundless approaches to knowing and capitalizing on a significant market move so I won’t go into yet another treatment of the subject.

All of the approaches have core issues of determining thresholds for action and false positives. I find using a parameter or some sort of deviation from an average works fine but is best coupled with testing for a book sweep and determining it is over (somewhat complicated currently but is relatively trivial with CME’s MDP 3 protocol) to avoid a good percentage of false positives.

The one approach everyone seems to agree on is bid – ask spread and that will definitely provide an early indication of an imminent event. Overall this is a good approach but potentially a lot of code if implemented such that you avoid the core issues mentioned above.

A variation on this approach is to use some combination of current top of book, last traded level and last cleared level (i.e. fully traded through). The side(s) anchored by a trade can lag but are hard (if not impossible) to spoof. One other alternative is to regularly calculate the cumulative size on both sides of the market and if it differs significantly from your baseline (either a parameter or an average of some sort), or if there is a clear imbalance between the two sides, an event is possibly imminent or underway. Of course, this approach is limited by spoofing algorithms unless you have a mechanism for spotting those as well and therefore, down the rabbit hole we go (a folk saying in the US).

My recommendation is to avoid the temptation to simply subtract the bid from the ask and compare the result to some number (lots of false positives are almost guaranteed); remember during a CME match event (in this context, a book sweep) you get trade messages but not book messages (by definition).

For detecting a market move is underway, I have several mechanisms but I use two primarily. The first is the coastline approach I mentioned earlier in the thread and is the most accurate. I calculate a coastline of a period (any search of fractal and coastline will give you the basics) and compare that to an average. This has the issue of lagging on both ends; by definition you have to have a period which produces a lag proportionate to the length of that period, you have to have an average to compare it against (this requires a “warm up”) and you have an equal lag to know the market has returned to normality.

The second is a simple arrival rate of ticks and/or book changes (either top of book only or full book). This will identify jitter in the market but little else. This could be somehow combined with volume but I have not been successful in using volume for this type of scenario (I do use volume significantly for other purposes).

Last there are a huge number of mechanisms for confirmation a market move is underway but calculating retraces against the primary market direction relative to a profit objective (or measurement period) works very well for me but is quite complicated code. If in the last P measurement periods (could be anything but normally time or volume), there was no retracement of N ticks or there was no retracement of X percent, you are at great risk of fading the move which is not to say, you should join it.

Thank you to everyone for the their excellent comments and both ideas and confirmation of known approaches. I am still open to any new thoughts on the subject but felt responsible to summarize what I have found works so far.

Please feel free to pm me if you would like to discuss a particular aspect further but don't feel it would be of general interest to those following this thread.

Nice summary, Scott.

@Tom - Thank you. There were so many great comments/ideas that summarizing them seemed a good approach to making it easier for others to get the information into actual code.