Q Neal: That system would produce great results, and you know someone could sell it and call it a seasonal... Okay, seriously, give a quick-start version for evaluating a system. Committee: First, if you can't get the maximum total drawdown and Sharpe Ratio on thge system, then forget it. Second, use a system that works on a diversified portfolio of markets. I remember, when I got together with other systems developers, how we all laughed at the single market systems that have 95% winning trades and made millions of dallars. We laugh because we know how to optimize a system that gives great historical results but is worthless in real-time trading. It it is a good sysem, it can trade many markets. Third, never buy a system that is a black box. A black box gives you trades but no rationale behind the trades. The investor should learn what works and what does not. If he knows the system, he can then either improve it or integrate it into other systems he's developing. --- Trading Chigago Style (by Neal Weintraub) UQ
Q Doug Patrick: If a system doesn't produce good results over many time periods and market types, avoid it like the plaque. Let's say you've got the concept and it makes sense; now what? Now you work. As an example, the systems I use tool four years to develp and test, and they are still a work in progress. ... --- Same UQ
small newbie question : if you manage to build an overoptimized system doing great perfs on a significative set of data, then validate it on a larger set of unseen data, can we consider that the system is playable for real ?
Yes. You donât always need to validate over a larger set if the backtesting / optmization is done properly. All systems âdiscoveredâ through backtesting will suffer from some amount of hindsight bias. Walk-forward performance may eventually decay to zero, go negative, or oscillate unpredictably unless the system is continually revised. My personal experience with back testing and forward testing has revealed the need for robust design and self adaptation. Robust design Stick to plausible hypotheses that can be validated as non-random over long periods of time (many data points), over many markets, and as many trades or instances as possible. This will identify the most robust inefficiencies. Reduce the number of degrees of freedom (DoF) optimized in hindsight. This is very difficult explain, somewhat of an art. Some DoFs can be safe to optimize. Others can lead to over optimization and a short system lifetime. Number of parameters does not always translate to the impact of over optimization. It is critical to understand the nature of the parameters, and how much each parameter can falsely fit random (non systematic) market behavior. Donât cheat on simulated trade executions. Simulate market orders with real bid/ask quotes if you want to be safe. Use proper performance estimates. Design a system that can decide when to trade and when not to. Determining market direction can be as simple as the slope of a simple moving average if the system knows when to trade it. Use clean market data, and be sure your historical data is consistent in character with your real-time data. Otherwise, subtle differences in the real-time data will kill performance immediately. Run control experiments to identify biases and better understand how your systems work. Pursue novel indicators and ideas, less correlated to the actions public domain analysis. Performance variations will be lower. Self Adaptation Have the system continually adjust itâs own parameters, either using heuristics or using past performance to predict future performance for various parameter combinations. Paper trade a population of systems simultaneously. Essentially you are doing all of this already with manual back testing and forward testing. Better to automate the entire process. Better to have an architecture that lets you paper trade 1000s of systems in real-time instead of back testing them and hoping your backtest is proper. Some systems or system performance estimates can be predicted better than others (autocorrelation of returns, etc.). It is very frustrating to work with systems that have poor autocorrelation and require a long period of time to judge performance.
Diversification is also key, either by splitting size between uncorrelated systems, or by using a system design built on many "additive" non-correlating elements. I do both. I've had systems exceed their historical max drawdown (blow up), stopping out at a loss of 0.01 EUR/USD trading 6E, but were balanced out by other 6E systems that took the opposite position or others that profited on ES and NQ, leaving me with a much smaller loss, or sometimes net positive for the day.
Do we need to set any ranges for the parameters and constants of systems, individually and/or collectively, and how to define the ranges (with or without optimisation)?
It depends. Both self-adaptive and manually adaptive systems should not be over optimized by allowing too many DoF, or DoFs that bring too much diversity. Either constrain the parameters to a range (subspace) which is known to be robust based on longer term, multi-market studies, or allow greater ranges (greater DoF) but choose a population of parameters, each driving your system additively (diversified)... i.e. all your eggs aren't in one basket, so you can tolerate some parameters blowing up on you.
Definitely get "Smarter Trading: Improving Performance in Changing markets" and "Trading Systems and Methods" by Perry Kaufman. My library isn't very large. Most of my methods are original. I write a lot of code and test ideas constantly. My best advice is to learn to code up and test ideas accurately and rapidly, with good statistical significance, multi-market data and by using rolling back test trials (train in set / test out of set). Don't limit yourself to traditional TA methods, known indicators, or ideas that are intuitive or "pretty". I have been burned in the past by lack of statistical significance and over optimized methods. So do what you can to avoid these problems from the start