Hi guys, I've been trading with a discretionary approach for years and a couple of months ago I decided to build some systems too based on some ideas. But....I see that some systems work GREAT on some instruments (stocks or index) but don't do so well on others..is this normal? A system for every instrument? thanks
Simple idea: http://en.wikipedia.org/wiki/Selection_bias Thatz what u r doing. Very, very bad thing...
That isn't selection bias at all. My recommendation is this: if your strategy is ROBUST on a great history for an individual instrument, it may be taking advantage of a particular characteristic of that instrument. In that case, you are fine on specialized running a strategy on a single instrument. But I would urge you to try to recognize what this characteristic is. If you can't identify it, you won't be able to figure out if it disappears...
that is still called curve-fitting. it is more likely that the "trader" ran something on a bunch of instruments, and focuses on the ones that has "a great history."
Yes, it's possible that you can have an edge with one system that you would not have with other system(s). Welcome to the real world of trading.
Hence why I said it had to be robust over a large range of history, preferably including different markets. If he had a system that was good on tech stocks from 1997-2001, then that would be curve fitting. If he had a system that was good on tech stocks from 1990 to 2009, that may not necessarily be curve fit. It's all about how the system was designed. Again, it may just be taking advantage of a particular peculiarity of certain securities...
Or more likely, the peculiarities of the market at certain times in history. That is often why backtesting fails into the future. But backtesting IS good for predicting the past.
Again, hence why I am stressing that a) The backtest be done over a large range of historical market types and b) He understand what peculiarity he is exploiting so that he can recognize if the paradigm changes. In conclusion: backtesting is fine if you know how to employ it. But just like any tool, you first must know how to apply it.
but most backtests are one of a huge number that a "trader" applied. And they find one or two that "worked". That is still curve fitting regardless of your points. And few people do a serious walkforward test with some of the data. Without this, almost every backtest is irrelevant for trading the future. And even with successful walk forward testing, it still is unlikely to be of much longterm value. You either can learn to trade, or can continue backtesting, sniffing around for other people's systems to rent, or trying to make useless things (TA indicators, elliott wave, gann, fib, etc. etc. work.
I think we essentially agree. If someone is just back-testing random strategies over multiple instruments and time-periods until they find one that works, it is unlikely that it will work walking-forward since it would indeed most likely be a case of curve-fitting. If, on the other-hand, the trader has designed a system based on fundamental concepts he believes to be true, then back tests it over a large range of market cycles and finds that it works very well on some instruments and not well on others, he might be taking advantage of a peculiarity of the instruments. For example, he may develop a system that works on extremely liquid instruments, but not on illiquid ones. This is not entirely unreasonable. Again, back-testing is a tool. When properly employed, it can be a great addition to your tool-box. But just like any other tool, you have to know when and how to use it.