Learn
What makes a backtest honest?
A backtest replays a rule over years of past markets, trade by trade, before you risk a dollar. Most backtests online show one number from one period. Here is how to tell a real one from a curve-fit.
Short answer
An honest backtest includes costs and slippage, counts every losing trade, is checked on data it was never built on, still works when settings are nudged, and beats a shuffled or random version of itself. It also shows results year by year, because one great year can hide many bad ones.
Five tests
| Test | Question it answers |
|---|---|
| 1. Unseen data | Does it work on years the rules were never built on? |
| 2. Two market regimes | Is it profitable in different market conditions? |
| 3. Nudged settings | Does it survive every setting moved ±20%? |
| 4. Double costs | Is it still profitable with twice the commissions and slippage? |
| 5. Fake market | With the same price moves shuffled, does the edge disappear? (It should.) |
What these tests catch, from our Myth Tests
- One good year: the 10AM setup lost money every year from 2017 to 2025 and only looks good because of 2026.
- Fragile settings: our “fixed” 10AM version made +$40,245 on paper, but nudging one setting turned it back into a loss.
- Baselines: the 8:30 range model made +0.12R per trade, the flipped signal lost −0.21R and a random side made nothing. That is what a real edge looks like.
- Shuffle tests: the 5-minute candle rule beat 97% of 1,000 shuffled versions.
Why it matters
On August 1, 2012, Knight Capital ran untested code on one of eight servers. In 45 minutes, 212 customer orders turned into more than 4 million trades in 154 stocks. The stock fell 33% that day, and within a year the firm was sold. A system run with real money before it’s proven is the most expensive kind of test.
How Compass was tested
Every Compass signal went through the same five tests on the homepage. Every live signal passed at least 4 of 5. The one experimental signal came out roughly break-even, so it now only takes its strongest setups and is labelled unproven until it holds up on new data. See the full results table.
Questions
What is a backtest in trading?
A backtest replays a fixed set of trading rules over historical market data, trade by trade, to see how the rules would have performed. Results are hypothetical.
How do I know if a backtest is overfit?
Check it on data the rules were not built on, nudge the settings, double the costs, and compare it with a shuffled or random version. An overfit strategy usually fails one or more of these.
Why do backtests and live results differ?
Backtests can miss real fills, slippage, gaps and outages, and they are designed with hindsight. Including costs and testing on unseen data reduces, but doesn’t remove, the gap.
Keep reading
Cite this: Trading Decisions Lab, “What makes a backtest honest?” (Oct 24, 2026). https://www.tradingdecisionslab.com/learn/what-makes-a-backtest-honest
Simulated backtest. Hypothetical or simulated performance results have certain limitations and do not represent actual trading; see the full disclaimer. Past results don’t guarantee future results. Educational only, not financial advice. Futures are leveraged and you can lose more than your deposit.