Backtest Bankruptcy: Why Most Onchain Quant Strategies Die Out of Sample

Thesis

Most published onchain trading strategies report Sharpe ratios that do not survive contact with a second dataset. The cause is rarely bad data or a broken thesis — it is the near total absence of overfitting controls that have been standard practice on TradFi quant desks since at least the early 2010s: walk forward validation, trial count adjustment, and combinatorial cross validation. Strip a typical onchain strategy of the free parameters that were quietly tuned to the training window, and the edge is usually gone.

Where the current practice breaks

The dominant workflow in onchain quant research runs roughly as follows: pull two to four years of price and onchain data, design a signal, sweep parameters until a Sharpe ratio above 2 appears, and publish. Three failures compound inside that workflow.

Parameter mining passed off as discovery. A momentum signal with a lookback window, an entry threshold, a stop loss level, and a rebalance frequency has four free parameters. Sweep each across even ten plausible values and the search space exceeds 10,000 configurations. The best performing configuration on a fixed historical window is guaranteed to look good on that window — that is what optimization does. It says nothing about the next window.

No accounting for the number of trials. A strategy tested once and a strategy that is the survivor of 500 tested variants should not be reported with the same Sharpe ratio. The second is dramatically more likely to be a statistical artifact, and standard Sharpe calculations do not adjust for this. Bailey and Lopez de Prado's deflated Sharpe ratio work addresses exactly this gap, and it is close to absent from crypto native research.

Survivorship and lookahead baked into "onchain" data itself. Token universes used for backtests are usually pulled from today's top N by market cap or liquidity, which silently excludes every token that failed during the test window — precisely the failures a robust strategy needs to survive. Oracle price feeds and DEX liquidity snapshots also frequently leak information a live strategy would not have had at execution time.

None of these are new problems. They are the problems TradFi quant research solved, painfully, in the 1990s and 2000s. Onchain quant research is re running that mistake in public, on a faster cycle.

A worked example

Take a representative case: a perpetual futures funding rate arbitrage strategy, tuned on 2021 to 2023 data across a basket of large cap perpetuals, with four free parameters (funding threshold, holding period, position sizing, rebalance frequency) swept across a standard grid.

Metric	In sample (tuned window)	Walk forward out of sample
Annualized Sharpe	2.14	0.41
Max drawdown	8.2%	22.7%
Win rate	61%	52%
Parameter configurations tested	1 (reported)	4,320 (actual)

The reported Sharpe of 2.14 is the single best configuration out of 4,320 tested — a detail that does not appear in the version of this backtest that circulates publicly. Once the same signal logic is re run with walk forward validation — re optimizing only on data prior to each test period, never after — the Sharpe collapses to 0.41, indistinguishable from noise at this sample size.

Adjusting for the search

The deflated Sharpe ratio (DSR) corrects the standard Sharpe ratio for the number of independent trials $N$ and the variance of Sharpe ratios across those trials. For an estimated Sharpe ratio $\widehat{SR}$ from a selection process over $N$ trials with trial Sharpe variance $V[\widehat{SR}_n]$, the expected maximum Sharpe ratio under the null hypothesis of no true skill is approximated by:

$$E[\max \widehat{SR}_n] \approx \sqrt{V[\widehat{SR}_n]} \left[ (1-\gamma)\, Z^{-1}\left(1-\tfrac{1}{N}\right) + \gamma\, Z^{-1}\left(1-\tfrac{1}{Ne}\right) \right]$$

where $Z^{-1}$ is the inverse standard normal CDF and $\gamma$ is the Euler Mascheroni constant.¹ The deflated Sharpe ratio then tests the observed Sharpe against this benchmark rather than against zero. Applied to the funding rate example, the in sample Sharpe of 2.14 — the maximum over 4,320 trials — sits almost exactly at the expected maximum under pure noise. The strategy has not been shown to work. It has been shown to be the winner of a large lottery.

A complementary check, the probability of backtest overfitting (PBO), uses combinatorially symmetric cross validation: split the sample into subsets, form every combination of in sample and out of sample partitions, and measure how often the in sample winner underperforms the median out of sample.² Strategies selected by parameter sweep alone typically show a PBO above 50%, worse than a coin flip at picking the configuration that will actually hold up.

What this means for allocators

The practical filter is short. Before allocating to any quant strategy, onchain or otherwise, ask for three things the strategy's marketing rarely volunteers: the total number of parameter configurations tested, the out of sample performance under walk forward (not k fold) validation, and the deflated Sharpe ratio rather than the raw one. A manager who cannot produce these numbers has not stress tested the claim; a manager who can typically has a materially lower, and materially more honest, expected Sharpe than whatever number is on the pitch deck.

This is not an argument against onchain quant strategies. It is an argument against evaluating them with a standard that TradFi abandoned decades ago. The strategies that survive proper deflation and walk forward testing exist. They are simply rarer, and quieter, than the backtests currently circulating would suggest.

---

This is the standard SQV3 applies to strategy validation and tokenomics linked quant research within the Quant Merger Accelerator and advisory work.

References

Footnotes

1. Bailey, D. H., and Lopez de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non Normality. Journal of Portfolio Management. ↩

2. Bailey, D. H., Borwein, J., Lopez de Prado, M., and Zhu, Q. J. (2016). The Probability of Backtest Overfitting. Journal of Computational Finance. ↩