Yes, good point. In addition to ensuring it generates enough trades, the historic data should also be as representative as possible of the market regimes you might expect to see going forward.
OK… I think I am starting to get my head around what the Chernoff Bound does/doesn’t tell us (sorry … it has taken me this long to get there! … a little bit of knowledge is a dangerous thing …): Here’s an example of how I understand it could be used … If you have a strategy with a win rate of say 55%, then the Chernoff Bound gives the minimum number of trades you will need in a backtest to be say 99% certain of getting backtest results with a win rate of greater than 50%. That is not the same as the minimum number of trades required to give x% confidence intervals on the difference say between the average trade measurements. Therefore it does not answer the question of how many trades we need in a backtest to be able to compare In Sample and Out Of Sample results statistically.