How much out of sample testing is enough?

I would like some expert opinions to throw against a wall with my amateur opinion. Is optimization on 20 years of daily data followed by 2 years of out of sample data enough? If not how much is enough? 20%? 50%?

SSB

 

20 years? Useless. Most markets have a "memory" of maybe 5 years at most. Concern yourself with the last 8-10 years at most. Markets today are very different from 5 years ago. Would you take a Ferrari on a dirt race track and expect to win? Each market environment requires a different vehicle to participate in it.

 

OK, even if I narrow to the last 5 years, what would you recommend for the out of sample period?

SSB

 

I don't do "out of sample". If it makes money in testing, I trade it, period.

I should add that I don't use daily data for more critical testing.

 

Daily data is crap anyway, I wouldn't trust 100 years of backtesting.

 

How much out of sample testing is enough?

However much you need, to feel confident. Fill in the blank: If I tested _____ months out of sample, then started trading my system, then experienced big losses, I wouldn't blame my test procedures.

 

the amount of out of sample data should depend on how many times a year your method trades. try to get a lot of trades in your out of sample data to get a better feel if the idea is holding up.

 

I don't have enough data to feel confident....ever. Then again on the other hand I've thrown away money on systems that tested well but stopped working the moment I started trading them....:eek:

Someone recommended using 1/3 of data to build the system, 1/3 to optimize then 1/3 as out of sample. That's actually a pretty good measure.

As for daily, I prefer intraday because it reacts much faster, but there's also a lot more noise which cancels out the some of the benefits of the faster action.

I prefer to trade some systems on daily and others on intraday. ...

SSB

 

30 is supposed to be the least number of trials required to prove any degree of significance in statistical tests.

so, i would use this number given the ever changing nature of markets. look at the last 30 potential trades.

 

I like at least 100 trades with at least 25 coming from the out of sample group and going back at least 500 trading days. This is for intraday systems; can't speak for daily as I dont do that.

DS