- Formulation

=======================

During this stage we formulate a hypothesis that describes an event in the market.

- Formulate the hypothesis in written words

----------------------------------------------------------

During this step, we word out the hypothesis.

Describe the market conditions, or theories that provide a foundation

to the idea.

The hypothesis should be stated as a description of a specific market event.

What do we know about the event? what is unknown?

Can the event be described in terms of prior conditions?

- Formulate the hypothesis in terms of data

----------------------------------------------------------

The goal of this step is to define the hypothesis in terms of the data that we will need in order to test the hypothesis.

How much data will we use for the experiment?

Which securities/assets?

For what time range?

- Forecasting the event.

========================

If our event is defined in terms of its outcome, we must define the rules to label the historic data in terms that we may use to label the data automatically.

Then we need to perform analysis on the data, to explore different alternatives for generating the signal, and train several AI's to compare their results.

AI training

-----------------------

During this stage we will train an AI to forecast occurrences of the event on historical data. The goal of the AI is to identify the prior conditions for the event, and is not necessary if the event if already defined in such terms.

To prevent hindsight biases, the data must be separated by dates, so there is no overlapping timestamps on different data groups. This holds true for every data partition that we do during the experiments.

Training data:

Have the AI look at the training data, make predictions, measure the accuracy of the prediction against the labels and adjust the parameters.

Repeat.

Quiz data.

Every few hundredth iterations on the training data, run the predictions on the quiz data and measure the accuracy. Do not adjust the parameters based on the quiz set!

Test data.

This data is used only once at the end of the training and the results are reported.

Studying the event's outcome.

========================

For this step we must have an event that is defined in terms of its prior conditions. If we built an AI then forecasts coming from the AI are the prior conditions, and during these tests we must let the AI identify the occurrences of the event, without learning (to prevent over-fitting)

Profile the event

-----------------------------------

During the stage we will observe the outcome of the event and generate trading rules based on those observations.

We will use an event profiler during this stage. This is a tool that allows us to compare the statistics generated from many instances of the event.

The event profiler will allow us to make forecasts and predictions on what we can expect of the event's outcome in terms of risk/reward.

Trading test

=========================

- Generate trading rules

----------------------------------------

Up to this point we've worked only on the description of the data.

From what we learned by describing the data we formulate a trading plan.

What is the profit target?

What is the maximum loss we will tolerate?

What is the probability of the trade being a profit?

Should the entry be made in 1 shot, or in steps?

We will generate a set of rules to follow when a the conditions prior to the event are met.

Back test

------------------------------

We simulate executions of the trading rules on historical data and make adjustments.

In order to simulate market conditions we "play the tape" of the data in chronological order feeding the data to the signal engine and the backtester in the same manner that they'll receive during live trading conditions and we simulate executions on the data based on the trading rules.

We may use an AI to fine-tune the trading rules.

Genetic algo's are a good fit for this type of job.

Whether we adjust manually or through an AI, the backtest data must be divided in the same manner as the AI data to prevent overfitting.

On the training data, we iterate finding adjusting the trade rules to optimize the results. We run the quiz every few hundredth iterations and run on the test data once we consider that we have a strong set of rules.

- Walk Fwd test

-----------------------------------------

During this test we use the set of rules obtained during the back test and simulate executions against live market data.

The AI identifies the trading event, and we enter/exit based on the rules.

- Live test

-----------------------------

Test the algo in live markets and grow its volume slowly to measure the effect of slippage and control its risk.

* Conclusion

=============================

At this point the idea should have matured into a trading system.

What follows is the risk management and observation of the system, keeping in mind that even-though the idea seemed to be solid at the end of this process, it is still open to falsification, by means of changing market conditions and we must be ready to discard it, or at least push it back into testing, when that change happens.