EXPERT · CHAPTER 04
Research Synthesis & Personal Evidence
The learner can compare studies with different designs, weigh evidence without treating all sources as equal, separate statistical from practical usefulness, interpret uncertainty honestly, reject weak ideas without regret, document findings so they can be revisited, and build a personal evidence-led review process that improves over time without promising certainty.
Evidence comes in different strengths
Recognise that not all evidence deserves equal weight.
A single anecdote, a small backtest, a well-controlled out-of-sample study and multiple independent replications provide very different levels of support. Evidence synthesis means weighing findings by their design, sample, controls and reproducibility rather than counting the number of positive stories.
Compare studies by design, not by conclusion
Evaluate how a result was produced.
Two studies can reach opposite conclusions. The better study is not necessarily the one you prefer but the one with fewer biases, clearer rules, more representative samples, realistic costs and honest limitations. Compare the method before comparing the outcome.
Sample and period differences matter
Account for different data sets and eras.
A method tested on one market or decade may not compare directly with another tested elsewhere. Differences in sample composition, period, volatility regime and costs can explain conflicting results. Synthesis should name these differences rather than assume one study is simply wrong.
Statistical significance is not practical usefulness
Distinguish detectable effects from worthwhile effects.
A result can be statistically significant but too small to matter after costs, risk and effort. Practical usefulness asks whether the edge is large enough to survive realistic friction and justify the capital, time and emotional load involved.
Effect size and confidence intervals
Report the size and uncertainty of a result, not just a yes/no.
A result is more informative when reported with its estimated effect size and a range of plausible values, such as a confidence interval. A wide interval means less precision. A narrow interval around a meaningful edge is stronger than a wide interval around a tiny one.
Replication matters more than a single result
Prefer findings that have been independently repeated.
A single test, however well designed, is not final. Independent replication on different data, by different people or with different software reduces the chance that a hidden error or random artefact created the result. Replicated findings are stronger evidence.
Negative results are evidence
Treat failures as useful information.
A negative result—when a method does not outperform its baseline—is not wasted effort. It helps rule out weak ideas, prevents future losses and clarifies what conditions do not support the method. A personal evidence base should include failures.
Rejecting weak ideas cleanly
Know when to stop pursuing an unsupported method.
A weak idea may have no detectable edge, an edge too small to survive costs, extreme parameter sensitivity, poor out-of-sample performance or dependence on a narrow regime. Rejecting it avoids sunk-cost thinking. The goal is not to make every idea work but to find the few that survive honest testing.
Documenting findings for future review
Keep a retrievable record of what was tested and learned.
A useful research record includes the question, rules, data, sample, costs, result, limitations and the decision made. This prevents repeating old tests and allows later review when new data or market conditions appear.
Personal evidence log
Build a running record across many tests and observations.
A personal evidence log accumulates controlled tests, chart observations, regime notes and outcome reviews. Over time it becomes a reference for what has and has not worked in your own process. The log should include both positive and negative findings to remain honest.
Combine evidence without double counting
Avoid treating related findings as independent confirmations.
If two studies use the same data, the same indicator family or the same period, they are not fully independent. Treating them as separate confirmations gives a false sense of strength. Synthesis should note when evidence overlaps and avoid double-counting the same underlying signal.
Updating beliefs when evidence changes
Treat conclusions as provisional and revisable.
New data, different regimes, changed costs or failed replications should update a prior conclusion. Updating is not weakness; it is part of an evidence-led process. Strong prior evidence should not be discarded after one bad week, but material new information should shift the assessment.
From evidence to action with uncertainty
Use research to inform decisions without pretending certainty.
Evidence can guide position sizing, regime filters and whether to continue studying a method, but it cannot eliminate risk. A positive edge is not a guarantee. Action should include risk controls and honest expectations, with the understanding that even well-tested methods can have losing periods.
Building a personal review cycle
Create a repeatable process for evidence and review.
A personal process includes scheduled reviews of logs, re-testing of methods, regime notes and decisions to continue, pause or reject ideas. The cycle should run regularly enough to catch drift but not so often that short-term noise drives changes.
Repeatable research synthesis checklist
Apply the full synthesis process.
Use a consistent sequence: gather the evidence, compare study designs, note sample and period differences, separate statistical from practical usefulness, check replication, include negative results, reject weak ideas, document findings, update beliefs when needed, and act with risk controls. The output is a current evidence statement, not a permanent proof.