Regarding the data-to-parameter ratio: I think since financial data contain a large amount of correlation, N data points are not really "worth" N data points -- as you point out, I think it's the independent degrees of freedom or number of principle components that matters. I agree with you that perhaps 7-8 DOF/parameters in the model is enough and beyond that it gets iffy. I like the rolling window training concept -- i.e., using only the recent data points for training. There is a trade-off of course between not having enough data (due to existence of large "tail" effects), vs fitting or over-fitting to irrelevant data from the distant past. I like to come up with trading rules that have some kind of behaviorial or fundamental mechanism or rationale behind them. By doing so, I think the chance of over-fitting becomes less because you are not just fitting your model against any pattern that happens to come up. Also by doing so, it is more likely that your model will remain relatively simple, and the number of parameters will naturally be smaller.
Thanks for this link! IMO, key to trading system design is as deep an understanding as possible of the "market mechanics" (or "market microstructure") underlying that price behaviour which makes the system succeed (or fail). With this understanding as a guide, my view is that "parameters" can be introduced as required AS LONG AS THEY MAKE SENSE IN TERMS OF THE UNDERLYING MARKET MECHANICS, and as long as they are not added just because the optimized result is better. This is how to avoid "curve fitting". The other point (made in the above link) about the lack of robustness of Type I systems is excellent. Thanks for sharing!
Interesting conceptual discussion at that blog post. Would have liked to see some data on distinguishing between the outcomes of his curve-fitting taxonomy, but it made intuitive sense.
I recently have run into the "window" issue regarding some quantitative relationships which retain their ordinal structure over time (X is always greater than Y) but not their cardinal relationship, since X and Y are not static, nor do they change at the same rate. It caused me to miss a very nice trade and now I am debating this very issue of how to integrate new data into the model. I think that I will go with all historical data to determine X and Y's relationship because my hypothesis is that over time the variations smooth out, precisely because X will always be greater than Y, so there is an upper and lower bound on the relationship. In other scenarios where there isn't an upper and lower bound on the quantitative relationship, using a window which correlated to whatever market regimes you were trying to trade would probably work better. Kind of thinking out loud here on this comment. Also, agree 100% on the notion that your rules need to reflect something you can intuitively understand. When I explain my rules to someone, they immediately get why those rules work, even if they are not traders. My model has 5 easily-explained parameters, only 2 of which are really core, while the other 3 are optimizations that take the strategy from being a profit factor around 1.75 to a profit factor of over 3.5. I started with only the 2 and have since added the other 3 as time has gone on and I started trading the 2-factor strategy in real-time and saw outcomes repeat themselves over time based on the values of those parameters.