I have to note that your English was perfect in the previous post, so I have to take this as an act The goal for all system developers is a picture below, anything else is just a way to get there. Now, for the actual question of optimization and fitting. An ideal strategy is the one that's built on a very simple hypothesis and thus does not require any additional variables to improve the quality. However, as we all know, it's usually very hard to find something like that. So you end up taking an imperfect hypothesis (e.g. "stock that gone up 5 days in a row is likely to go down") and add knobs that you can twist and change the outcome. Some reasons for the addition of those knobs are totally OK. For example, if you have a strategy where you see good potential but the results are barely beating the transaction costs, you might want to add some parameters to reduce the trading frequency. Or maybe you have a strategy that trades across assets and you find that the relationship has some seasonality (say you are trading the spark spread), you might add a flag to avoid trading during specific times. There are situations where those knobs are misleading and dangerous. Let's say you testing a strategy that "buys the dip" in equities. While the strategy might have done very well in the last 5 years, you see a large drawdown in August 2011. So you add a variable that says "don't trade in August and September". The drawdown disappears, your out-of-sample tests look good. But what you have done is introduced a rare event bias that will eventually cause you some pain. My point (which I am finally getting to) is that in-sample and out-of-sample testing is just a model for the impact of free variables based on certain assumptions. You can come up with alternative ways of verifying the statistical impact of your free variable. For example, some HFT guys back-test modifications on the full dataset, but they would regress random subsamples of trades with respect to time and expect a near-unity slope (meaning variance/covariance relationship is perfect).
Hello sle, I apologize, I was not acting. Keep in mind this whole trading strategy development theoretically discussion is kind of new to me. I am learning as I work. I should have stated it better such as: your writings is above my current knowledge of system development. I honestly thought optimization is/was a good thing. I think it is, but what I am realizing or learning is that over optimization or overfitting is a bad thing because each parameter of the strategy is being fitted to produce max profits only for the sample period it was over optimized for. So when those over optimized parameters meet out of sample data or real time, the likely hood is the system will not produce equal profits as in sample data be cause optimized parameters was fined tuned only for in sample data. Does this make sense sle?
Thank sle for the clear explanation. The above quote is the stage I am. So here is my work flow process: 1. Watch the charts everyday 2. Find an trading idea that looks repeatable and simple and could possible make money. 3. Write the strategy on paper 4. Program the strategy 5. Click Back Test (In sample data only) 6. Evaluate results (normally all Red losses) 7. Optimize indicators and exit methods until see profit. 8. Walk forward the rest of the out of sample and hope I see continuation of profits. Is there something wrong with this process, I haven't made to 8 yet. It just seems to me Optimization is inevitable, just don't over do it. Keep the parameters loose. Everyone can fit loose pants. There is other performance and strenth of signal I have written down to check off for robust system. Also NinjaTrader has Monte Carlo Simulation I can use. Thanks