PBO explained simply: how likely is it that a backtest is overfit?
The probability of backtest overfitting (PBO) is the chance that the strategy version that looked best on past data does worse than the typical version on new data. Researchers Bailey, Borwein, Lopez de Prado and Zhu measure it by splitting history into many pieces, picking the best version on some pieces and testing it on the rest. A PBO above 50% means the 'best' version is probably luck.
Step by step
- Collect every version you tried. Keep the results of every setting combination, not only the winner. Overfitting hides in the ones you threw away.
- Split the history into blocks. For example 10 equal time blocks. Each version gets a result for each block.
- Pick the best on half, test on the other half. Choose 5 blocks as 'past', find the best version there, then rank that version on the other 5 blocks.
- Repeat for many splits. Do this for every way of choosing half the blocks (252 ways for 10 blocks).
- Count how often the winner falls below the middle. That share of splits is the PBO. Near 0% is good; near or above 50% means the selection is mostly luck.
A small example
You test 40 moving-average settings. Across 252 splits, the setting that ranked first in the 'past' half finished in the bottom half of the 'future' half in 160 splits. PBO = 160 / 252 = 63%. The best backtest is more likely to disappoint than not, even though its own equity curve looks good.
What to do about a high PBO
- Try fewer versions, chosen for a reason before testing.
- Prefer settings on a plateau, where neighbours also work.
- Get more trades and more market conditions - see how many trades you need.
- Paper-trade before real money.
Related measure: the deflated Sharpe ratio
Bailey and Lopez de Prado also proposed the deflated Sharpe ratio, which lowers a Sharpe ratio for the number of versions tried and for non-normal returns. Both ideas answer the same question: is the best result better than the best of many random tries?
Questions
- Do I need code to measure PBO?
- Yes, it needs every version's results by period. Without code, use the simple tests on our overfitting page.
- What is a good PBO?
- Lower is better. Many practitioners treat anything near or above 50% as a sign that the selection process found luck.
- Does a low PBO mean the strategy will make money?
- No. It means the selection step was less likely to be luck. Costs, markets and execution still decide.
Sources
- Bailey, Borwein, Lopez de Prado, Zhu: The Probability of Backtest Overfitting (SSRN)
- Bailey, Lopez de Prado: The Deflated Sharpe Ratio (SSRN)
Run the Strategy Review (free beta)
Upload a trade list; the review includes a luck check adjusted for the number of versions you tried.