Learn · Multiple testing

Multiple testing in quantitative investing

Reviewed against the platform's code on Sep 23, 2026

Multiple testing, or data snooping, is the problem that arises when many strategies, factors or parameters are tested on the same data: some will look statistically significant by chance alone, so each result must be judged against the whole search, not on its own.

Why it matters

Test a hundred useless strategies at a 5% significance level and about five will pass. Without correction, the literature and the backtest folder both fill with false discoveries.

How it works

Corrections include White's reality check and Hansen's test of superior predictive ability, which test the best strategy against the full set tried, and step-down procedures such as Romano and Wolf's, which control the rate of false discoveries.

How Opulence Alpha applies it

Opulence Alpha's strategy statistics include a reality check for data snooping and step-down multiple-testing procedures, and the publication gate takes a maximum reality-check p-value as a required threshold.

The Promotion Gate →

Related concepts

Deflated Sharpe ratioThe deflated Sharpe ratio adjusts a strategy's Sharpe ratio for the number of strategies tried and for non-normal returns, giving the probability that its true Sharpe ratio is above zero.Backtest overfittingBacktest overfitting is when a strategy is tuned to the noise of past data, so its backtest looks strong but its live performance decays. How to detect and avoid it.Promotion GateA Promotion Gate is the validation step a strategy must pass before any portfolio may use it: overfitting, data-snooping and robustness checks, where a check that could not run counts as a failure.
All concepts →Sector behaviour in every market regime →

Questions

What is data snooping?
Reusing the same data to search many ideas, so the best one's apparent significance is partly an artefact of the search.
How do you account for multiple testing?
Test the winner against the whole set of ideas tried (a reality check), or control the false discovery rate with step-down procedures.
Why is a t-statistic of 2 not enough in finance?
Because thousands of factors have been tested on the same markets; after correcting for that, much higher thresholds are needed.

References

  • White, H. (2000). A Reality Check for Data Snooping. Econometrica.
  • Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics.
  • Romano, J. & Wolf, M. (2005). Stepwise Multiple Testing as Formalized Data Snooping. Econometrica.

Educational content about research methods. Not investment advice.