Quote from total_keops: . . . Not doing a forward test. This is refered to as curve fitting. absolutely true (I used to be a statistician).... Back testing is thoroughly worthless. That concept is sold by vendors who can't prove they're profitable. What counts is, the RIGHT edge of the chart, because volatility patterns and markets change. What's even more important, is money mgmt/position scaling/dynamic position sizes. -k
You may be right in some respects but there are some objections. I use backtesting not as a confirmation that a strategy will work in the future but to determine whether it has worked in the past. I think in that respect backtesting is not worthless. It becomes worthless when traders rely on it for strategy future performance rather than past evaluation. Would you trade a startegy that had a 90% DD in recent years based on backtesting? Another question: Would you feel comfortable trading a strategy that is not backtested when you know it could be?
Backtesting with optimization can't prove that something is not random... thus you can backtest for three years and blow up in three hours even... And backtesting with forward testing can't prove that anything is not random either...
Curve fitting has been notoriously inaccurate, a stochastic model would be much better. However, basic stochastic models are only so good until you get into the tails.
Thanks a lot for the link, intradaybill. The paper was interesting. Any idea about which software it was about?
I looked at the first section of the paper. I don't understand the author's point. He claims that these backtesting tools skipped certain entries/exits by mistake... what if it wasn't a mistake, but rather design? Why is it fundamentally wrong to implement a strategy that doesn't set stop/target losses until the *following* bar after an entry? I do the same in my code.
It doesn't make any sense to skip a bar when checking for a stop. Actually, this is not good trading practice. You may end up losing multiples of your stop-loss amount.
Well, there is still some debate as to whether stops themselves make sense. I don't see why anyone would take as dogma that it's simply fundamentally wrong to wait a bar before setting stops/targets. Especially if 10 years of back-testing results suggest it's a significantly better solution.
I have no idea since the author was careful to hide their identity. I can only guess but that is not good enough. It boils down that many people have been taken for a ride by companies that did not do basic testing before selling their software.
It is bad if the trading system designer was not intending of doing that but the software did it behind his back. That is the point. I think it is explained in that paper very well. I also think the difference is important between what you personally design and what you think you are designing because of some faults in the programs used. I think the difference is clear, to me at least. I believe stops not only make sense but they are mandatory for trading success. Traders who do not place stops (mental at least, you don't have to place the order in advance) sooner or later blow up.