EXPERT · CHAPTER 02
Controlled Strategy Testing
The learner can convert an idea into a precise rule set, choose a fair baseline or control group, isolate one variable at a time, account for sample size and statistical noise, score performance beyond win rate, incorporate realistic transaction assumptions, document tests so they can be reproduced, and distinguish a meaningful result from an artefact of repeated testing.
From idea to precise rule set
Turn a research idea into a testable algorithm.
A controlled test requires exact rules for signal, entry, exit, position size, market, timeframe, costs and outcome window. The rules must be unambiguous enough that two independent people following them would produce the same trades. This is more demanding than describing an idea in ordinary chart language.
Entry rules require exact timing
Define when the signal becomes actionable.
A signal based on a daily close, for example, can only be acted on after that close is known. The entry must specify the next open, same close, or another execution time. If the test assumes entry at the signal candle’s close when the signal is only known at that close, it may be unrealistic.
Exit and outcome rules must be specified
Define the exit before testing.
Without a pre-defined exit, a backtest can be adjusted until it looks good. Exit rules include time-based exits, price-based stops or targets, trail conditions, and what happens if the exit falls on a gap or missing data. The outcome horizon must be fixed before results are observed.
What is a baseline?
Understand the role of a comparison benchmark.
A baseline is a simple reference used to judge whether a strategy adds value. Common baselines include buy-and-hold for the same market, an equal-weight index, a random entry generator, or the average return of all candidate periods. The method’s results must be compared with the baseline, not judged in isolation.
Control groups isolate the idea
Use a control group to test the specific hypothesis.
A control group holds everything constant except the condition being studied. For a pullback method, the control might be a random sample of periods with similar trend structure but no pullback. Comparing the two groups isolates whether the pullback itself matters.
Matched samples reduce hidden differences
Compare cases that are as similar as possible.
A matched sample pairs each signal case with a control case that has similar volatility, trend state, timeframe or market environment. This reduces the chance that differences in outcome are due to hidden differences in background conditions.
Change one variable at a time
Understand controlled variable isolation.
If a test changes entry rules, exit rules, timeframe and sample all at once, the cause of any performance difference is unclear. A controlled test changes one variable while holding the others fixed. This is slower but far more informative.
Sample size and statistical noise
Recognise that small samples produce unstable results.
A strategy with only 20 trades may appear excellent or terrible due to random variation. Larger samples reduce the impact of single unusual outcomes, but they cannot remove uncertainty. The test should report the number of trades, not just the total return.
Win rate is not enough
Look beyond simple hit rate.
A strategy can have a high win rate but lose because average winners are small and losers large. Profit factor, average win/loss ratio, expectancy, maximum drawdown and return consistency provide a fuller picture. A low win rate with large winners can be more profitable than a high win rate with small winners.
Risk-adjusted scoring matters
Compare returns with the risk taken.
Two strategies may have the same return but very different drawdowns. Risk-adjusted measures such as return-to-max-drawdown, Sharpe or Sortino ratios help compare strategies fairly. A high return with violent drawdowns may be less useful than a lower return with smoother equity.
Transaction assumptions: costs and slippage
Include realistic trading costs.
Commissions, fees, spreads and slippage reduce real returns. A backtest that ignores costs may overstate performance by a large margin, especially for high-frequency or illiquid markets. The test should state and apply cost assumptions before evaluating results.
Fills and execution timing must be realistic
Model how orders would actually be executed.
Not every price shown on a chart is obtainable. Gaps, thin liquidity and limit-order fills can affect results. The test should assume conservative execution, such as next-open entry for close-based signals or paying the spread, rather than ideal intraday prices.
Reproducible tests require documentation
Record enough detail for independent replication.
A controlled test should document data source, universe, point-in-time treatment, signal and exit rules, cost assumptions, code or spreadsheet logic, and any parameters. Another person should be able to obtain the data and produce the same results. Reproducibility is a core part of controlling bias.
Multiple comparisons and data mining risk
Recognise that searching many variations creates false positives.
Testing dozens or hundreds of rules on the same data increases the chance that some will look good by random chance. This is multiple comparison risk or data mining. The more variations tested, the more sceptical the researcher should be unless those variations were pre-specified or corrected.
Repeatable controlled testing order
Apply the full controlled-testing checklist.
Use a consistent sequence: start with the pre-registered hypothesis, define exact rules, choose a baseline and control comparison, isolate one variable at a time, record sample size, apply realistic costs and execution, score with multiple metrics, document everything, and avoid mining variations. Treat the result as provisional evidence, not proof.