I have a question for the stats studs out there: If I developed a system that when backtested creates a smooth upward daily equity curve (i.e., >99% correlation) incorporating >3611 trades over 427 days, is it even necessary to do an in-sample/out-of-sample test on that? I can understand doing that when the in-sample results have a high beta and generates an erratic equity curve, but if my equity curve has a nice upward trajectory regardless of the time period I'm looking at, doesn't that already imply that the system is consistently profitable, or is an out-of-sample test still necessary regardless of the equity curve my in-sample generates? Thanks in advance for any insight!
On what instruments? On what timeframes / volume / tick ? Not to rain on your parade, and I have no idea what you are doing, but 427 days is meaningless IMO. Others may refute that.
It's not a matter of whether it's necessary or unnecessary, it's a matter of how confident you are that the feature you are exploiting exists in any data other than what you are currently testing against. For example, if I have a development process that starts with a bunch of data and finds a system that trades it perfectly, then I'm not going to have much confidence in it. I will probably want to see how it performs on different data. On the other hand, I could also imagine a situation where I had high enough confidence, based on the rigor of my development process, to take a system live after seeing only a few test results.
All my backtests use some form of walk forward analysis. Do not trust in sample backtests. Whenever I see a very smooth equity curve, the most likely explanation is some kind of mistake. Most common mistakes are look ahead and suvivorship bias, or unrealistic execution costs.
as a pseudo qualified statistician....there is no question there... analogous to randomly asking somebody "is this important"