Pretty new to Automated trading and system development, but know enough to understand the pitfalls of curve fitting. I had an idea how to avoid this when taking a strategy live, just wanted to hear what people think, or their experiences if they have tried it. Obviously your looking for a robust range of variables when you optimize, but just to be on the safe side, would it be best to run a number of versions of your system, each with different settings, and splitting up your total position equally among them? Obviously not going to help if your system itself is crap, but could help smooth out your total returns, no? Just as one wouldn't bet their whole account on one trade or stock, why bet your whole system on one set of variables? Just a thought.....
It's pretty much diversifying your portfolio based on indicators. I can understand what you're saying, and I can imagine it'll help. I still felt like being contradictory--just must be the day I'm having If the system is curve fitted, then it very well may be "crap" as you say, and twice the number of magic numbers wouldn't help. Adding more magic numbers might just mean you're just curve fitted to more magic numbers than before. I guess the situation comes down to how much of penned yourself in to a set of parameters. If your magic numbers reflect a general trend then I feel there is some soundness to them. The situation may be that they don't exhibit as strong a performance as indicated, but they should still work. Trends also give you something to investigate further; if you found the principle behind a trend for some magic numbers, you could replace that variable with a formula. Yes, that may mean more magic numbers now... I have a situation here where in an indicator I had assumed input variable a would be greater than b for it to work. The optimizer I used didn't allow me to enforce that when feeding it values. So it gave it sets where a was less than b. It turned out that a > b never worked once. a < b has worked very well sometimes based on the other variables. So across a splay of numbers it is demonstrating something more like a trend. So in there I would expect some truth, but not necessarily can I expect the same performance if I just leaned on those numbers.
As long as your system has at least one independent variable it involves some type of curve fitting. If it varies that variable to maximize some objective function, like equity performance for example, then you have optimal curve fitting. The more variables you have the worre your system will be in terms of curve fitting because the easier it becomes to fit it to the data. Try system that have no variables at all...
Curve fitting is a waste of time, forget about magic numbers etc. Optimizers are primarily invented as snake oil by the sellers of trading software. Instead you should create an algorithm that analyzes an equity thoroughly for patterns of movement based for instance at time/ interval, volume etc and compare that with other similar analyzes.
Do you know of any such algorithms generally that don't require any parameters? I haven't heard of such stuff; even moving averages have periods to which they can be set.
Markets Change. If it works, it's robust. If it doesn't, it's curve-fitting. Simple enough? --- Whether something is robust or not, solely depends on whether the tendency of the market you're exposing has changed or not. That's simple too right? --- So (look at it as a bunch of ORs) 1. You expose a tendency that doesn't change. 2. You expose a tendency that can be tracked and monitored. (To calibrate/tweak the parameters) 3. You understand the tendency and apply the appropriate risk management schemes. --- In reality, there is no curve-fitting or robustness. It's just a single phrase people use as answer for their failed/working system. The problem with how most retail package optimizes is that they do not understand the difference between... exposing a tendency and applying what's been exposed. A trading system/model is an application of what has been exposed of the market. So if you run a parametric optimize of the system, you're going to skip through the exposure process leading to a failed system... Get it? It's like a relationship between a car and a driver. A system is the driver and he uses the car to go places. If there's a problem with the transmission, the driver can help reduce the impact of the faulty gear by driving carefully but it's never fixed until you goto the shop and have the transmission repaired / replaced. So... for a systematic trader, you're you're competing with a bunch of F-1 team (Hedge Funds) with top mechanics, strategists and top drivers to hit the finish line without breaking down your car (You don't have to win the race, right?). As an individual, you have to go out and do all the tasks involved in the race from building, fixing, driving and etc.
Price action does not require any parameters. Michael Harris has done serious work in the area of pattern discovery from price action using algorithmic approaches, such as data mining. His program APS Automatic Pattern Search does that and his methods are somewhat disclosed in his books. Try to find scanned copies around to get an idea. Also keep in mind that when dealing with price action patterns you do not have the problem of curve fitting but instead you must deal with selection bias. There are no freebies in the markets.
Your system itself must be steady. Let us suppose you have 2 variables and optimized them on historical data, got values 47 and 69 Of course it is a crap, because of you tried too many values. Also if you substitute 45 and your system becomes losing strategy, then these values are crap. Then suppose you have two peaks on the chart, with 4 & 6 and 3, 9 you may trade both simultaneously, it would diversify you a bit. But generally correlation between these two versions of one system will be around 0.80 which means your diversification is small. The best approach is to run your system on multiple non-correlated instruments. With multiple sets of parameters, as this will reduce your risks a little bit.
Interesting thread. Earlier, I had done a experiment based on the optimization issue and the link is as follows: http://www.elitetrader.com/vb/showthread.php?s=&postid=2078942#post2078942 http://www.elitetrader.com/vb/showthread.php?s=&postid=2090564#post2090564 Please have a look at the above and give your valuable feedback. Thanks