Minimum number of trades required for backtesting results to be trusted

A useful rule of thumb (which I teach my students in week 3), which can be derived from the sampling distribution of a mean estimate, is that for statistical significance you need at least N data points where N:

N = 4* (s / m)^2

Where m is the average value and s is the standard deviation.

[This assumes that a T-statistic of 2 is significant, which is true at 2.5% significance for more than ~60 observations, i.e. you can be 97.5% confident that the true mean was greater than zero]

This implies that the more profitable your trades are (bigger m), and the more consistent their profitability (smaller s), the more confident you can be and the fewer trades you need.

Consider for example the following series of 100 trades: +$300, -$250, $300, -$250 ....

The mean is $25 and a quick visit to Excel confirms that the standard deviation is $275 [depending on whether we use the 'sample' or 'population' version of the statistic]. Plug into the formula;

N =4 * (272/25)^2 = 4 * (11)^2 = 484

So we'd need almost 500 trades to be at least 97.5% confident that our backtest results weren't just down to luck.

Of course this theoretical result assumes there are absolutely no issues with your backtest such as:

- overfitting
- survivorship bias
- data snooping
- source of return you are exploiting vanishing
- under-estimating costs

For this reason I'd generally multiply the figures above by at least a factor of 2, if not more.

GAT

 

Right, t-stat = sqrt(n) * sharpe, so n = (t-stat / sharpe)^2 ...
I love it when math just works

There are factors to go both ways. For example, if you have a strong prior you can be comfortable with a lower sample.

 

When you have no real trades, but just backtesting new strategies, how do you determine statistical significance?

 

Same maths applies, but you apply much more skepticism to backtests than to real traders (so more trades needed for significance).

GAT

 

But you don't have a series of profits to put in the equation. How do you estimate an average?

 

You get the series of profits from the backtesting software

GAT

 

Sounds a little like the chicken and egg problem, but I understand.

 

I watched the series and found them very useful. Lots of things he talks about we already knew, but some got new Perspective. Highly recommend to anyone who is interested in algo trading.

 

By "strong prior" I presume you mean "a good reason for shit to work"?

 

%%
Exactly, especially for something like SPY,QQQ,sqqq,UPRO...............
Reading can help also; polar bear patterns are much different from black bears/brown bears+ same with bull moose or bull elephants...................................................................[Edit ; by watching the weather report+ spending time outdoors, many years, any can gets some good hints.
BUT they don't call it weather predicting/LOL\its called weather forecasting]