A backtest is a tool of proof and a tool of deception at the same time — the very data that establishes an edge can manufacture one if it is misused. The six traps below do not produce bad results; they produce beautiful false ones, which is far more dangerous, because a bad result is rejected immediately while a beautiful one is believed and funded.
1. Look-ahead bias
How it sneaks in. The strategy uses information not available at the moment of decision — today's close for a same-day entry, or a data point that was only published after the test date. What it does. It turns an ordinary system superhuman instantly: a near-straight equity curve and a Sharpe above 3.0 with no economic reason behind it. That cleanliness is the tell — a result too smooth to be real. The fix. Compute the signal on today's data, execute at tomorrow's price — without exception.
2. Survivorship bias
How it sneaks in. The test runs only on the companies in the index today; every firm that went bankrupt or was delisted has quietly vanished from the sample. What it does. The sample is pre-selected for winners — estimated returns rise by two to four points a year, and the worst outcomes simply leave the calculation. The fix. Test on indices and ETFs with continuous history; never hand-pick tickers with hindsight.
3. Overfitting
How it sneaks in. Parameters are tuned until performance peaks on past data — 47 days instead of 50, a threshold of 1.83 instead of 2 — each value chosen because it happened to work. What it does. The system memorizes the noise rather than the rule: it excels in-sample and collapses the moment it steps outside. More parameters mean less data standing behind each one. The fix. Few parameters, round numbers (50/200), and performance that holds across a whole neighbourhood of settings, not one lucky point.
4. Multiple testing
How it sneaks in. Hundreds of combinations are tried on the same data, then the best one is presented as a discovery while the failed attempts are quietly folded away. What it does. Out of 100 random strategies, about five will look "significant" at the 5% level by chance alone — which is exactly why the honest bar was raised to t > 3.0. The fix. Count your attempts, state them, and raise the significance bar in proportion to how much you tried.
5. Ignoring costs and slippage
How it sneaks in. Returns are computed on theoretical prices — no commission, no spread, no slippage — as if the market were waiting to fill you at exactly your quoted price. What it does. It kills high-turnover strategies first: a system that flips daily can lose its entire edge to costs, while one that trades ten times a year barely feels them. The fix. Make cost an explicit line item in every test; measure the edge after it, never before.
6. In-sample testing only
How it sneaks in. The rule is built and judged on the same period, with no held-out window — the student is examined on the answers already memorized. What it does. This is why published anomalies decay about 26% out of sample and 58% after publication: more than half the paper edge was never really there. The fix. Split the period — two-thirds to build, one-third held back — and never revisit the held-out third twice.
The six at a glance
| Trap | How it sneaks in | The fix |
|---|---|---|
| Look-ahead bias | Uses data unavailable at decision time | Signal today, executed at tomorrow's price |
| Survivorship bias | Tests only on the names that survived | Continuous-history indices/ETFs; no hindsight tickers |
| Overfitting | Params tuned to peak on the past | Few round params; performance that holds across a neighbourhood |
| Multiple testing | Best of hundreds shown; failures hidden | Count and state attempts; raise the bar (→ t > 3.0) |
| Costs & slippage | Returns on frictionless theoretical prices | Cost is a line item; measure the edge after it |
| In-sample only | Built and judged on one period | Split: two-thirds to build, one-third held back |
A backtest is not a verdict
Notice what every fix has in common: it makes the test harder, never the result prettier. That is the whole discipline. A backtest does not prove that a strategy will win — it proves only that it has not been refuted yet. A beautiful result is not evidence; it is an invitation to doubt, and it earns your trust only by surviving all six of these traps and coming out the other side still standing.
The next part turns to the other half of survival — not whether the edge is real, but how to size it so that a bad run doesn't end you before the edge has time to pay.