Although I am a discretionary trader and not a systems trader, I have in my possession a book which may be of value for systems trading: Trading as a Business, by Charlie Wright (Designed for use with Tradestation) I got it from a friend a few years ago, and I think its only available directly from the Tradestation people It addresses the aspect of robustness by examining the best systems in different market environments. For example, I recall it discussing the use of a) moving average systems for trending markets b) oscillators for rangebound markets c) price patterns for volatile markets It states that no one system can work in all market environments but that, given a specific environment, certain systems have greater efficacy. I stopped hunting for my Holy Grail system / indicator many years ago. Market conditions can change very frequently intraday, so I use my discretionary assessment of the sort of market conditions currently prevailing, use a stop loss and take a bet ... I have no quantifiable system per se, but my P&L is generally in good health, and I use that as my performance quantification.
If you are testing your system on a subset of data that was not used to develop it, with similar results being obtained, you are satisfying one of the key requirements to validate it. Another major pitfall that many system developers find when they actually come to implement it is that they cannot get the types of executions the system demands, e.g. they have assumed too small a value for slippage. If you are trading the S&P Futures, there is a fine line between maximizing your profits and going broke due to a string of losses. Read about Optimal F in Mathematics Of Money Management by Ralph Vince. It tells you how to determine the size of account you need and how much to risk per trade to minimize the risk if ruin while maximizing your profit potential. Be warned however, that large drawdowns are often inevitable while trading one product to achieve maximum gain. Some people, including me, cannot take this, that's why I trade stocks with parallel trades.
ive been a system trader for over 6 years and in designing my systems i look for three financial figures 1)percentage win loss 2)profit ratio 3)ratio of win to loss. I multiply all three together and if that equels 1 or larger then I go on. I have to have over 450 trades in my population or I wont consider the population large enough. I then consider the sharp ratio and I consider the yearly return vs the largest drawdown. I I look to see if the annual returns are increasing or decreasing as time goes on. I will not optimize my variables often at the most twice a year. Im always looking at the largest drawdown and challenging whether i can live with that drawdown.
Just curious because you are using the word population: is it possible that you are also analyzing the markets and strategies from an evolutionary perspective (esp. evolutionary game theory) ? Your 3 financial figures pretty much resemble what is also necessary to analyze the strategies of life forms.
IMHO The important thing is that any value you choose for your RSI period makes money. Not just 6 and 12, but 1 though 30. It's kind of irrelevant which value you actually choose because it is based on the past and the future will be different.