After having a newly designed system, what are the proper processes for optimization and walk-forward testing and analysis so that it will not overfit? When do you consider your rules are not necessary for the system?
Back testing is BS. The only way to know if your system works is to trade live. You'll lose money half the time so it's all the same.
Ideally, backtesting a freshly "designed" system will confirm initial performance without having to spend time and money to trade it yourself. As long as results from backtesting do not propagate back to the design and rules, there's absolutely NO problem originating from backtesting at all. However, on what basis can one design a "technical" trading system without any kind of confirmations from historical data? Iterative development is key for explorative development, which may become the snake biting itself in the tail, which is fine - unless it starts swallowing. Potential problems may be created when updating design and rules due to backtesting results, thus initiating a feedback-loop of optimizing parameters and rules based on historical data. Then historical biases, statistical spikes and high dimensionality skews may creep into the design and rules, due to various lacks both in data and their interpretation. Thus naive optimization-efforts are often first time performed on the wrong conclusions. Forward-testing is not a silver-bullet though, when results from such also propagate back to design and rules, which is only natural if the system generates consistent losers over time. There's nothing magic about forward-testing, the same lacks and biases can be more probable as even less "fresh" data will be available per time. There's no perfect solution, but people usually do something akin to this: Always only test hypothesis/ideas with some fundamental / foundational logical basis to explain why it should work in the first place (ie. don't backtest all possibilities - which may end up overfit to imperfect data). Split the historical data randomly. One part for backtesting for updating design & rules. One out-of-sample part for validation of performance only. In time over many iterations, the same problems may manifest though, or problems due to qualitative differences in data between the parts. If you cannot explain why your system include a rule, you should consider its removal, in order to simplify as much as possible. Complexity works against general trading systems, but may be core to specialized trading systems (such as intraday trading). A general system should perform in almost any market, or similar markets and instruments. So should be tested against markets that trade similarly to the traded instruments. Know when to break these "rules" because you know why you can, but also handle the risks of doing so! This framework may help to order one's thoughts about how to go about being an actor inside feedback-loops of various orders, such as the markets and other systems.
This book is excellent. The Evaluation and Optimization of Trading Strategies by Robert Pardo It provides a great schema for understanding systems trading and how to properly backtest, the difference between overfitting and optimization, and how to run a walk-forward test. In addition it spells out how important backtesting is to calculating minimum capital requirements for running your system. For me this was the nail in the coffin for algo trading. Because there WILL be drawdown and you have to have the capital to accommodate that, your return on capital gets killed after you find out how much money it will take to run the system.
Thanks for your inputs, KMeriwetherD. BTW, have you read this book, "Design, Testing, Optimization Of Trading Systems" by Robert Pardo?
The system needs to be validated on Out-Of-Sample Data. Live Data can be found in a securities history, it just can't be included with In-The-Sample Data. - Andrew