EXPERT · CHAPTER 01
Hypotheses, Bias & Sample Design
The learner can turn a chart observation into a clear, falsifiable research question, define the exact population and sample being studied, recognise selection, survivorship and look-ahead bias, avoid data leakage, choose fair outcome definitions, and design a review process that controls bias before testing a method.
Questions come before methods
Understand why a clear question precedes tool choice.
Before testing a chart-reading idea, define the question precisely. A vague claim such as “support works” is not testable. A better question specifies the market, timeframe, condition, rule and outcome being studied. The question determines what evidence would support or challenge the idea.
A hypothesis must be falsifiable
Distinguish testable hypotheses from unfalsifiable claims.
A falsifiable hypothesis can be shown to be false by data. “Price always reverses at support” is falsifiable because even one clear counterexample would refute the absolute claim. “Price sometimes reacts at support” is too vague to test. Testing requires a claim precise enough to fail.
Operationalise the idea into rules
Convert a concept into observable, repeatable rules.
“Trend pullback,” “support bounce,” or “momentum stall” must be translated into exact rules before testing. Rules include how the level is drawn, what counts as a pullback, when the observation starts, and how the outcome is measured. Without operational rules, different people will test different ideas under the same name.
Define the population and sample
Know what market is being studied.
The population is the full set of cases the conclusion should apply to. The sample is the subset actually tested. For example, all liquid US large-cap daily charts from 2015–2025 might be the population; 200 randomly selected tickers are the sample. The sample should represent the population.
Selection bias in choosing charts
Recognise how choosing visible examples distorts conclusions.
Selection bias occurs when the charts or cases chosen are not representative of the wider population. Choosing only successful examples, only currently popular instruments, or only periods with strong trends distorts what the method seems to achieve. A fair test selects cases before outcomes are known.
Survivorship bias in historical data
Understand how delisted or failed assets distort backtests.
Many historical datasets include only assets that survived to the present. Failed companies, delisted instruments or closed funds may be missing. This survivorship bias makes strategies look better than they were because the worst outcomes are absent from the sample.
Look-ahead bias
Recognise when future information leaks into past decisions.
Look-ahead bias occurs when a test uses information that would not have been available at the time of the decision. This can include using a full period’s high or low before the period has closed, using revised fundamentals, or drawing levels with the benefit of hindsight.
Data leakage is subtler than look-ahead
Identify indirect information leakage.
Data leakage occurs when information from outside the intended historical decision window enters the model. This can happen through normalisation, indicator settings, smoothing, index inclusion rules, or even the choice of the sample period after seeing results. It is subtler than obvious look-ahead but equally damaging.
Fair outcome definitions
Choose outcome measures before testing.
A fair outcome definition specifies how performance is measured and over what horizon. It should be chosen before seeing the data to avoid favouring a particular result. Common measures include average return, hit rate, risk-adjusted return, maximum drawdown and consistency across periods.
Random chance and base rates
Compare results against what would happen by luck.
A method may appear useful simply because many patterns exist and some will look predictive by chance. Base rates and benchmark comparisons help reveal whether a result is unusual. Without a baseline or a random comparison, performance numbers have little meaning.
Control for the time period and regime
Avoid judging a method from one favourable market period.
A method tested only during a strong bull market may seem effective because most long ideas rise. It may perform poorly in flat or falling markets. A fair sample includes multiple regime periods, and the research should state which regimes were covered.
Document assumptions before testing
Record every assumption to make the test repeatable.
Data source, timeframe, tickers, point-in-time survivorship treatment, costs, slippage, signal timing, outcome horizon and benchmark must be documented. Another person should be able to reproduce the test from the written record. Undocumented assumptions make results hard to trust or repeat.
Avoid hypothesising after seeing results
Recognise post hoc explanation as a bias.
After seeing results, it is tempting to invent a story for why a method worked or failed. These after-the-fact explanations can feel convincing but are not evidence. A disciplined research process states the hypothesis and expected mechanism before collecting evidence.
One test is not a conclusion
Treat early evidence as provisional.
A single positive result, even from a well-designed test, is not proof that a method has an edge. Sample variation, data quality and minor rule changes can alter outcomes. Robustness checks across samples, periods and definitions should follow before a conclusion is formed.
Repeatable research design checklist
Apply the full method before testing.
Use a consistent sequence: state the question, make it falsifiable, operationalise the rules, define population and sample, check for selection, survivorship, look-ahead and leakage, choose a fair outcome and benchmark, document assumptions, and plan robustness checks before collecting results. This separates research from confirmation.