What is the difference between in-sample and out-of-sample data?

In-sample data is everything used to build a strategy: to fit its model, tune its parameters and choose between versions. Out-of-sample data is kept apart until those choices are final and is then used only to measure the result. A strategy's in-sample performance describes how well it fits history; its out-of-sample performance is the better guide to how it may do on data it has not seen.

Why does reusing a holdout ruin it?

Because every look that changes the strategy leaks information from the holdout into the design. Test ten versions against the same holdout and keep the best, and the holdout has become a second training set: the winner's score on it is now biased upward, like any in-sample score. A clean holdout is scored once, after the choices are final; if the result prompts changes, a new, untouched period is needed to test them.

Does a good out-of-sample result prove a strategy works?

No. A single holdout is one period, which may have suited the strategy by chance. Out-of-sample tests also use only part of the data, so they have less power than in-sample tests to detect a real but small effect (Inoue and Kilian, 2004). And markets change, so a strategy that passed can still decay. Out-of-sample testing guards against one kind of error, mistaking fitted noise for skill; the final test is live results recorded after the decision.

How does Opulence Alpha keep the search away from its holdout?

The holdout is cut before any search runs, and candidates are scored only on the search days. In the walk-forward replay, no search stage can name a set of days: every candidate goes through one runner that scores the folds' test blocks, and an automated test records the days of every simulation a search requests to check that none falls in the holdout. The winner is replayed on the holdout once, and the parameter-rounding check runs on the search days, so the holdout is not spent on a second decision.

FAQ · Out-of-sample testing

What is out-of-sample testing in backtesting?

Reviewed against the platform's code on Sep 26, 2026

Out-of-sample testing judges a model or investment strategy on data that played no part in building it. Data used to fit, tune or choose it is in sample; data held back until those choices are final is out of sample, and its result is the fairer estimate of how the idea will perform on new data.

Why it matters

Every choice made after looking at the data, such as a parameter, a filter or a start date, can fit part of that history's noise, and fitted noise looks exactly like skill. In-sample results therefore tend to overstate what a strategy will earn, and the more variations are tried, the larger the overstatement is likely to be (Bailey, Borwein, López de Prado and Zhu, 2014). The gap is measurable: McLean and Pontiff (2016) studied 97 published stock-return predictors and found their returns 26% lower between the end of each original sample and publication, and 58% lower after publication. Out-of-sample data is the one part of the evidence a search could not have shaped.

How it works

The simplest design is a single holdout: split the history once, build and choose everything on the earlier part, and score the later part once. With market data the split must follow time, never a random draw, and a gap at the seam keeps returns measured over a holding period from straddling it. Walk-forward testing repeats the idea: fit up to a cut-off, test on the next period, move the cut-off forward. The classic pitfall is reuse. Each time a holdout result sends the researcher back to adjust the strategy, the holdout joins the search and stops being out of sample; Dwork and co-authors (2015) show that adaptive reuse can overfit a holdout much as a training set is overfitted.

How Opulence Alpha applies it

Opulence Alpha's strategy validation sets aside the latest part of the history before any search runs. That holdout is the tail, never a random slice, and a gap of purge plus embargo days separates it from the search block; in a search campaign, the purge is taken from the strategy's own holding period. Candidates are scored on the search block only; in the walk-forward replay, on the test blocks of anchored walk-forward folds, where the history allows. The winner is replayed on the holdout once. The Promotion Gate reads the deflated Sharpe ratio on that block; an in-sample figure cannot replace it, and if no holdout could be cut, the check counts as skipped, which blocks publication.

Out-of-sample checks in the Promotion Gate →

Related concepts

Walk-forward validationWalk-forward validation tests a strategy the way it would have been used: fit on the past only, test on the next unseen period, then roll forward and repeat.Backtest overfittingBacktest overfitting is when a strategy is tuned to the noise of past data, so its backtest looks strong but its live performance decays. How to detect and avoid it.Multiple testingMultiple testing (data snooping) is what happens when many strategies are tested on the same data: some will look significant by chance. How reality checks and step-down tests correct for it.Promotion GateA Promotion Gate is the validation step a strategy must pass before any portfolio may use it: overfitting, data-snooping and robustness checks, where a check that could not run counts as a failure.

All concepts →Sector behaviour in every market regime →

Questions

What is the difference between in-sample and out-of-sample data?: In-sample data is everything used to build a strategy: to fit its model, tune its parameters and choose between versions. Out-of-sample data is kept apart until those choices are final and is then used only to measure the result. A strategy's in-sample performance describes how well it fits history; its out-of-sample performance is the better guide to how it may do on data it has not seen.
Why does reusing a holdout ruin it?: Because every look that changes the strategy leaks information from the holdout into the design. Test ten versions against the same holdout and keep the best, and the holdout has become a second training set: the winner's score on it is now biased upward, like any in-sample score. A clean holdout is scored once, after the choices are final; if the result prompts changes, a new, untouched period is needed to test them.
Does a good out-of-sample result prove a strategy works?: No. A single holdout is one period, which may have suited the strategy by chance. Out-of-sample tests also use only part of the data, so they have less power than in-sample tests to detect a real but small effect (Inoue and Kilian, 2004). And markets change, so a strategy that passed can still decay. Out-of-sample testing guards against one kind of error, mistaking fitted noise for skill; the final test is live results recorded after the decision.
How does Opulence Alpha keep the search away from its holdout?: The holdout is cut before any search runs, and candidates are scored only on the search days. In the walk-forward replay, no search stage can name a set of days: every candidate goes through one runner that scores the folds' test blocks, and an automated test records the days of every simulation a search requests to check that none falls in the holdout. The winner is replayed on the holdout once, and the parameter-rounding check runs on the search days, so the holdout is not spent on a second decision.

References

Bailey, D. H., Borwein, J. M., López de Prado, M. & Zhu, Q. J. (2014). Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society 61(5).
McLean, R. D. & Pontiff, J. (2016). Does Academic Research Destroy Stock Return Predictability? Journal of Finance 71(1).
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O. & Roth, A. (2015). The Reusable Holdout: Preserving Validity in Adaptive Data Analysis. Science 349(6248).
Inoue, A. & Kilian, L. (2004). In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use? Econometric Reviews 23(4).

Educational content about research methods. Not investment advice.