There are two answers to the question. If you have a lot of money then the market is predictable. Otherwise is random.

@Antonio

So simple and so true.

I know economics is not an exact science. But really? Asking if markets display random behavior implies that markets are unintelligible. This is patently false.

At times the market does appear random, but fact is most buy-sell volume is executed by the major funds. And most of those are based on some very intelligent rationales. So I find the market more deliberate than random aka chaos; albeit some market makers can sway a market but only for short time. The only time I read the market as chaotic is when decisions to buy and sell appear to be based more on emotions than a rational plan.

EMH and its counterpart, the random walk (like a Wiener process) are both oversimplifications of a very complex system. And often enough I get the impression, that opinions about future directions try to guess the mean while leaving out the prediction error ranges. But there are so many participants (humans and alogs) with so many different ways of making decisions (emotions, logic, news, patterns, money becoming available, etc.) and different timeframes of influence of these decisions (from single ticks up to years), that by picking some specific combinations for predictions one might get very good results or very bad (then looking like being random).

OTOH we sometimes see a predictability, which is nothing else than our own mind making it up due to its evolutionary grown way of interpreting the world.

It may not be that random. Matsui et. al. in their research have used a new high-resolution nonharmonic frequency analysis. As a consequence, they have succeeded in visualizing the various periodicities of stock prices. The periodicity fluctuates gently in many periods, but they have confirmed that it fluctuated violently in periods when a sudden event occurred. They expect that this experimental result in combination with previous research will help increase predictive accuracy and will aid long-term prediction.

The full paper can be downloaded here:

http://www.lifescienceglobal.com/pms/index.php/jrge/article/download/1020/pdf

The question assumes that participants of the discussion have a common understanding of a "random walk". The answers indicate (to me) that assumption does not work and "random walk" is substituted by "random" or associated with "unpredictable" or opposes to "predictable". Some of the replies look to me as answers on the questions "Are markets predictable?" or "Are markets unpredictable?" Maybe the question should be one of these.

Let us give a word to an authority in what is "market walk" (Kolmogorov A.N., Zhurbenko I. G., Prokhorov A. V. "Introduction to the Theory of Probability", Moscow, Nauka, 1982, Chapter 4. Symmetrical Random Walk.). At zero time a particle (read price) is at position 0 (read initial price). On each step it adds to or subtracts from the initial value 1 (+1 or -1 read as minimal price increments such as 0.25 in ES-mini or 0.03125 in Treasury Bond futures). If the probability of the increments +1 or -1 is 50%, then the evolution of the sum in time is referred to as "symmetrical random walk", SRW. If chances are not equal, then it is "non-symmetrical random walk", NSRW.

In order to prove that something is not something, it is enough to find a single example. In both random walks the price increment can be either +1 or -1. It cannot be 0, +-2, +-3, etc. Analysis of price increments shows that price changes 0, +-2, +-3 and greater by absolute value are ordinary found in price time series. Then, prices are not "random walk". And what?

SRW has many properties not observed in prices. For instance, with probability 1 the sum can reach any value including negative numbers. But prices are never negative. If prices would be NSRW, then their directions would better anticipated. I want to say that they would be random walks but more predictable. And so on. Without specification of which "random walk" is in scope discussion loses concreteness and we can naturally answer "Yes" or "No".

If the price is not a random walk it does not yet mean that it is predictable. There many other stochastic processes.

Best Regards,

Valerii

Valerii, good to read from you again.

Now a question: How do you address motive force? For certain, markets do NOT move by themselves, but are driven from outside influence by human activity which is driven by humans affected by both logic and emotions.

1) At what point or level does one view market action as random vs. motivated?

2) Is it back/white or a gradient?

3) Is it random due to a lack of significant (quantum) price jumps?

4) Is it non-random when motivated toward quantum jumps in price?

Theoretically anything is possible including negative prices. I don't think there is any law preventing a market maker from displaying negative prices. And laws can change. So here is what I think is my only problem: can I find a way of selling at a higher price anything I buy? (The assumption being I want to make money and not to lose it)!

I cannot predict or manipulate the present in order to create some type of future. And this means, according to my definition of "randomness" , that markets are always random.

Theoretically anything is possible including negative prices. I don't think there is any law preventing a market maker from displaying negative prices. And laws can change. So here is what I think is my only problem: can I find a way of selling at a higher price anything I buy? (The assumption being I want to make money and not to lose it)!

I cannot predict the future or manipulate the present in order to create some type of future. And this implies, according to my definition of "randomness" , that markets are always random.

Frank,

Your very reasonable questions emphasize primitive character of a symmetrical random walk, SRW, comparing with day to day market observations. It has very limited structure in order to mimic prices.

For instance, if one direction dominates during a time interval, then this could be more significant than it is expected from p = 1/2 chance of an individual price step-direction. SRW gives you a freedom to reset this probability p. It is not rich enough to give you other customization options One can set it higher. Then, it would become non-symmetrical random walk, NSRW, which has very different properties comparing with SRW.

However, we also know that markets' directions change and this would require reducing p at some moment. We can think that p is a function of time p(t). This model will be more flexible for fitting prices. Then, from a knowledge of prices one can solve a back task and calibrate p(t) - find such a function, which will minimizes deviations between the model and market. However, if this function cannot be extrapolated to the future, then in terms of predictions it becomes less useful.

Does it mean that SRW is absolutely useless for analyzing markets? Not exactly. The main reason is that SRW and NSRW have such properties, which are also observed on the market. For instance, many trading price patterns are formed on charts of SRW. Does it mean that when we see such a pattern in real prices, then it is a result of its SRW behavior? Actually, presence of a pattern cannot give answer. What would be more informative is to ask how frequently the pattern is formed. If it is a frequency following from SRW, then there are more chance than this is a result of SRW. But if not, then something else is responsible for its appearance.

As for the specifics of human beings, then we also have a stick with two ends. A crowd of people on a street can well obey the laws of a Brownian motion, even, we have all spectrum of desires and motivations. And the same crowd can become a strong force, if it gets a more definite direction.

Antonio,

Prices can be negative not only in theory. Here is an example. I have something, which I badly do not need and it occupies space in my garage. I'd be happy, if one will come, who needs the item, and take it (zero price for him or her). They do not come. I decide to prepay for the effort. Then, for him or her the price is negative.

Best Regards,

Valerii

Not revealing what the garage item is creates unnecessary intrigue. A robot trader.

Valerii, You want to get rid of that Robot Trader because is not good enough or you just don't like robots?

Antonio, the former (not good enough).

The market does not take a random walk. At very short time frames, the movement may fit the random walk model, but that does not imply that there is a rational basis for concluding that the theory holds.

At larger time frames, hourly and higher, there is absolutely no evidence that the market takes a random walk.

The way I understand the random walk is a market where all currently available information is aggregated into price instantaneously, so that changes come only from new information that arrives to the market at random times following a normal distribution, and the returns on day t+1 is the price of day t+0 + a random amount that is independent and identically distributed.

I have a few problems with this idea...

1. volatility crums. Days of high volatility tend to be followed by days of high volatility. So daily returns are not independent not identically distributed. This was observed by mandelbrot in the 1970's and many others since then.

2. if all information is aggregated to price immediately, it means that all participants are looking at the same information. This is not possible, since many participants with different time horizons base their decisions on information that is very different and often contradictory. A pension fund that accumulates a stock over several weeks doesn't care for order book information that may be useful for a day trader for anything other than to fine tune execution, likewise a day trader doesn't care if the fundamentals point to a given stock going up over the next 8 months, by then he would've taken many many long and short positions.

3. patterns in the data. the market data exhibits fractal characteristics (self-similarity and self affinity) in ways that are not present in a pure brownian motion.

Pablo,

Little by little.

"... large changes tends to be followed by large changes - of either sign - and small changes tend to be followed by small changes ...", p. 418 - Mandelbrot, Benoit. "The Variation of Certain Speculative Prices", The Journal of Business, Volume 36, No. 4, October 1963, pp. 394 - 419. For stressing this fact for cotton prices he credits this to Hendrik S. Houthakker. The term "volatility clustering" has been coined later.

Using the word "volatility" hides possible interpretation because of a need to define volatility. Is this a variance of the distribution of price increments? Is it a historically estimated sample variance of price increments or asset returns (logarithms of neighboring price ratios)? It is a process or parameters of a calibrated ARCH (auto regressive conditional heteroscedasticity) model or one of its variations?

At the same time a change of a price has unambiguous expression deltaP = P[i + 1] - P[i]. A price change for n > 0 steps is P[i + n] - P[i]. The latter is an algebraic (with sign) sum of all n increments. An ordinary task in the probability theory and theory of stochastic processes is to deduce the random properties of the sum based on the random properties of summands - increments.

Ironically, if we take a single smallest price increment, then we cannot "dissect" it on a random and non-random parts. Similarly we cannot do it for the sum. If we have a greater price change for n steps, we cannot say to which degree it is random or deterministic. We cannot say, whether it is a result of "random coincidence" (theoretically a coin can drop 100 times showing head) or a "trend".

One way to separate the effects is to compare this and other changes with those following from a process with known random properties (Brownian motion, or symmetrical random walk, or non-symmetrical random walk, more complicated processes with given structure of autocorrelations or non-linear dependencies, stochastic processes with parameters changing in time or changing randomly). In order to compare it properly we need repetitions (process realizations). Those must be obtained under the same conditions. This is where market is not very friendly - its conditions change. Therefore, we come to a need to define an "individual random object". Mathematicians were intensively moving into this (actually three) direction during the last 50 years. In my opinion, the price time-series of a E-mini futures contract such as ESH15 is such an object.

Best Regards,

Valerii

Thnx Valerii, I always learn a lot from your comments.

More than random, I view the market as a chaotic* system where:

- we have many different actors that influence price, we don't know the initial conditions of each of this actors or the rules they use to move from one state to another.

- we cannot walk back to determine the impact into price of the actions of each of this actors (non-linear system)

- the price has alternating periods of order and disorder, which lead to the formation of fractal patterns. I adhere to the investment horizons theory to explain the why of these periods and the reasons behind the change of states.

* while chaotic is technically just another word for random, it also implies a different kind of randomnes than the normality implied in say, a random walk... since this is a randomness more similar to say the weather, where we can make probabilistic forecasts of future conditions.

Pablo,

Many terms are used intuitively in these discussions. Some of them are reserved in special literature.

"Chaotic" associates with two (related) behaviors. The term "chaos" (certainly not the new word and without familiarity with a few important works on dynamical systems) was coined by James Yorke.

1. Li, Tien-Yien, Yorke, James, Period Three Implies Chaos, The American Mathematical Monthly, Vol. 82, No. 10, December 1975, pp. 985 - 992.

He has described iterative functions (I name them iterals of functions), where tiny changes in initial conditions create very different future trajectory. The functions are deterministic. However, predictions easily fail. The trajectories are not random, unless one sets equality sign between inability to track initial conditions exactly and randomness. This leads us to a traditional question: Is randomness something objective or simply a result of our poor knowledge.

Another reservation associates with the Kolmogorov's idea that randomness is absence of regularities. A sequence is chaotic, if it is "complex". This allows to measure complexity of a sequence of numbers by measuring entropy of its initial increasing segments. However, the entropy is defined a bit differently than in the classical theory of information.

2. Kolmogorov, Andrey, N. Logical Basis for Information Theory and Probability Theory. IEEE Transactions on Information Theory, Volume IT 14, No. 5, September 1968, pp. 662 - 664.

3. Kolmogorov, Andrey, N., Uspenskii, Vladimir, A. Algorithms and Randomness. Theory of Probab. Appl., Volume 32, No. 3, 1987, pp. 389 - 412.

The term 'fractal' is often used in these discussions in association with "self-similarity" - a form of invariance with respect to changes of time scale. However, Mandelbrot defines fractal as an object with fractional dimension, p 401:

4. Mandelbrot, Benoit, B. On the geometry of homogeneous turbulence, with stress on the fractal dimension of the iso-surfaces of scalars. Journal of Fluid Mechanics, Volume 72, Part 2, 1975, pp. 401 - 416.

The latter definition is useful because it gives tools determining an embedded dimension of a sequence of numbers, such as the correlation integral:

5. Grassberger, Peter, Procaccia, Itamar. Measuring the Strangeness of Strange Attractors. Physica D, Volume 9, 1983, pp. 189 - 208.

The following sections

"5 A Comment on Attractors and Fractals", pp. 10 - 12;

"13.4 Computer generated random walk vs. a-b-c-process", pp. 38 - 39;

"13.5 Computing the correlation integral", pp. 39 - 41;

"24 A Comment on Randomness", pp. 75 - 82

in already mentioned "Optimal Trading Strategies as Measures of Market Disequilibrium" (free download without registration http://arxiv.org/pdf/1312.2004v1.pdf) contain further references and can be viewed as introductions to these topics. There are also results of applications to futures prices.

Best Regards,

Valerii

For some reasons after pressing "Add Comment" the literature reference numbers were changed from 1 - 5 to 1 - 1.

'is randomness the result of pure knowledge'

i think that it is the result of imperfect knowledge. since there are many things we dont know about the actirs that participate of price formation.

like how many are they & their current state (specially tricky when you consider the ones who are out of tge market but looking) or the rules being followed by each if them to enter/exit (state transition rules)

the interaction between all of them results in chaotic behaviour, if we knew the state and rules for every single participant maybe price could be model to be deterministic ceteris paribus, or at least to follow a random walk caused by new information reaching tge market at random times.

Pablo, Brokers and Exchanges can have a lot of knowledge about what's going on i.e. they can know exactly what their clients are doing because they have at their disposal all the data necessary to do so. To what extent they share data among themselves to maximize the pain inflicted on players is something that we'll probably never know. We know that big players collude to defraud their clients and whenever one is caught they just settle any dispute giving a few bucks to the regulator. Settlements just add insult to injury because we all know who ultimately we'll pick up the bill.

Antonio.

A broker will have knowledge of the positions being held by its clients, not by those from competing brokers.

A centralized clearing firm (like CME's) will have knowledge of the positions of all clients.

In both cases they have no knowledge of the clients that are currently out of the market but waiting the right conditions to enter into the market (in either direction) nor do they know the rules that are used by each participant to enter one way or the other, or exit.

so even in their case the knowledge is imperfect.

A point of view, where randomness is a result of our poor or imperfect knowledge, is well known: "it is random because we do not know the reasons and if we only would, then ...". It is close to the Laplace's determinism already discussed in these threads. However, it leads to an extreme conclusion, where nothing can be changed - everything is developing in accordance with the reasons, randomness appears only because we do not know something, who is going to sink will not find another end of life, and so on. Einstein until the end of his life could not accept "randomness of quantum mechanics" and thought that a theory from the future will bring and recover temporary missed causality.

In contrast, one accepting randomness as an objective property supposed to find a basis of this objectivity. What makes it objective? Is there something already well known to human beings, what can serve as an explanation of "randomness" and make it "real" independently on our knowledge? I would not be satisfied by postulating existence of "randomness" but "explaining it".

As a mental experiment, I have taken a Laplace's deterministic system and asked what could make it random? I have prohibited using a random element because, then we would not explain but postulate it. I also would not be satisfied by chaotic properties, where small initial changes cause big deviations soon. This, again, could be interpreted due to imperfect knowledge of details.

I have found one property - one explanation.

Best Regards,

Valerii

no it does not and anyone who says so has just shown you what they know - nothing.

Antonio Rosario

Antonio Rosario

Computer Programmer / Researcher / Trader

Pablo, If brokers are competing with other brokers to get more clients is not relevant to the point I am trying to make. I am saying this because the money they make comes from the same source. Sharks don't feed on other sharks and if they were smarter instead of working alone they would join forces as some other animals do e.g humans.

And to defraud clients you don't need to know what's going to be their next moves. You just need to know their current positions and make sure that they are not wining positions i.e. just need to make sure that the market moves in the opposite direction.

Bookies like to tell us that they have no conflict of interest with their clients. They like to think they are just an interface that connects two sides. According to the same logic I could say that any bet I make is not between me and my bookie but between the bookie and the supermarket where I regularly buy food. To say that a deal/contract between two parties is after all not between those two parties is plainly silly.

Frankly, I have had a problem with the term, "random walk" ever since it came out as a book title. My friend, Arvin and I have argued over it years ago.

As a more discretionary trader who respects algos and those that write them, the lesson of my experience is simple: it tells me that the adage which states that for every action there is some kind of reaction is always in play beyond physics; likewise there is a cause for every effect. Granted, that's not a rocket science statement, but it does imply that quiet and chaos have root causes, and even drift has or reflects a purpose.

We assume minor price actions are no actions, though; even insignificant; and I think that presents an issue with the entire definition of random walks. Point: multi-dimensional laws of human nature are always at play here with markets, whether the market hours are open for transactions or not.

Doesn't matter who agrees or disagrees; with all respect, it is just my opinion until further notice. See, the math definitions, although very intelligent and logical on the surface, still has not given me a meaning sufficient to base or change my next trade.

In my opinion equity markets have a long term upward bias (nothing new here) and as you begin to reduce your time frame...that is move from monthly bars to weekly, to daily then minute bars the data gets closer and closer to being random. Certain time frames are more suitable for trend following strategies and some are more suitable for counter trend strategies. I am not sure that I answered your question other than to point out that market data has certain characteristics that can be exploited given your trading style....but all in all, the market is pretty close to random otherwise there would be a lot more successful traders than there are.

Frank: "We assume minor price actions are no actions."

Jeremy: "... equity markets have a long term upward bias ... as you begin to reduce your time frame...that is move from monthly bars to weekly, to daily then minute bars the data gets closer and closer to being random."

We have the chain of market prices P(t0), P(t1), P(t2), P(t3), ..., P(tn-2), P(tn-3), P(tn). For one at the time t0 and thinking about the time tn, the price difference is P(tn) - P(t0). One does not have to be a mathematician to see that P(tn) - P(t0) = [P(tn) - P(tn-1)] + [P(tn-1) - P(t(n-2)] + ... + [P(t3) - P(t2)] + [P(t2) - P(t1)] + [P(t1) - P(t0)]. Therefore, at any moment such as tn, the price P(tn) is the initial price P(t0) plus the sum of all price increments. One may like it, do not like it, ignore, or accept. It does not matter.

If we assume that "minor price actions are not actions", then to me it means zeros. The sum of zeros being added to P(t0) will leave it unchanged. We never come to a different P(tn). The assumption does not work.

If we say that "equity markets have a long term upward bias" and then insist that "move" to bars of lower time frames "the data gets closer to being random", then this looks to me as a statement attempting to eliminate in short term bars, what I see as a contradiction too. Indeed, with understanding that the price is the algebraic sum (what can be simpler than A + B + C + ... + Z) of all price increments added to the initial price, we come to the following. Somehow, a bias existing in the sum is eliminated from terms constituting the sum. I could assume something opposite: the terms have biases but those are of different signs and after summation the total bias for the sum is zero. But it is suggested that we should believe that sum has a bias but its components do not.

Hm?! Of course, minor price actions do matter and the price is nothing more then algebraic sum of the changes added to an initial price. The bias (drift or positive mathematical expectation) of a sum is not eliminated in smaller price increments collected in full from the same large time interval. Other properties change when we come from the sum to individual components. There were not named but exactly they create impressions expressed in such contradictory statements.

In a separate discussion I present the chart of S&P 500 from April 1, 1950 to December 29, 2014 and estimate a positive drift 0.13 per day for the entire ~65 years interval. From January 1, 2000 to June 30, 2002 the drift is very negative -2.68. From January 1, 2008 to June 30, 2009 the drift is very negative -2.38. One, who would buy the index with positive long term drift (bias) 0.13 based on the historical estimate of 50 years (1950 - 2000), would take the capital in 2000 for more than 13 years out of work into red losses (more than two times in ratio and twice).

Happy New Year All!

Best Regards,

Valerii

Minor typo: "P(t0), P(t1), P(t2), P(t3), ..., P(tn-2), P(tn-3), P(tn)" should be P(t0), P(t1), P(t2), P(t3), ..., P(tn-2), P(tn-1), P(tn).