What a Backtest Can — and Can't — Tell You

A backtest is one of the most useful tools in systematic trading — and one of the most dangerous. Useful, because it lets you test an idea against years of history before risking a rupee. Dangerous, because a beautiful equity curve is remarkably easy to manufacture and remarkably easy to believe.

At SKARSH we treat every backtest result as a hypothesis, not a promise. Here's how we think about what these tests can genuinely tell us, and where they quietly mislead.

What a good backtest can tell you

• Whether an idea is even worth pursuing. If a rule can't show an edge across a long, varied history, it almost certainly won't find one live.
• How a strategy behaves in different regimes. Trending vs. choppy, calm vs. volatile — the shape of returns across conditions matters far more than the headline total.
• Its risk profile. Depth and length of drawdowns, and how long recovery takes, tell you whether a strategy is survivable in practice — not just profitable on paper.

Where backtests mislead

Most backtest failures aren't fraud — they're subtle, and they flatter. The usual suspects:

• Overfitting. Tune enough parameters and you can fit any past perfectly. That curve describes history, not the future.
• Look-ahead bias. Accidentally using information that wasn't available at decision time — a future close, a revised data point — inflates results in ways that vanish live.
• Survivorship & data quality. Test on today's index members and you quietly exclude the companies that failed. Corporate actions, bad ticks, and gaps all distort.
• Ignoring costs. Brokerage, slippage, impact, and the price you actually get (not the one you saw) can turn a "profitable" system into a losing one.

A backtest tells you what would have happened if the world had been exactly as your model assumed. It never was.

How we keep ourselves honest

Because the ways to fool yourself are well known, our process is built to resist them:

• We reserve out-of-sample data the strategy never saw during design, and we care more about how it holds up there.
• We run sensitivity analysis — if a small change in a parameter collapses the result, the "edge" was noise.
• We model realistic costs and fills, and when uncertain, we assume the worse fill, not the better.
• We read the trade log, not just the summary. A result is a draft until the individual trades have been audited and make sense.

Even after all that, a strong backtest can still fail in live markets — regimes change, and no history contains the future. That is precisely why validation is only half the job; the other half is hard risk limits and real-time monitoring once a strategy is live.

The takeaway

Treat any performance number — ours or anyone's — with healthy skepticism, and ask how it was produced. A backtest earns trust not by being impressive, but by surviving every honest attempt to break it. That discipline, more than any single result, is what we think separates durable systems from good-looking ones.

Curious how we validate before we deploy? Let's talk.