Just how reliable is backtesting?

I have backtested the same systems on several backtesters and gotten wildly different results. Job number 1 is to find a reliable backtester. Back when I had Tradestation 2000 I found flaws in the language and the backtester that I couldn't live with so I taught myself C and made my own backtester facility that ran as subroutines called from EZLanguage. I tested it against manual backtesting until it matched the results and i was good to go. From there i proceeded to show myself that typical TA is ridiculous but hey...

 

Sorry to disappoint, but I've never heard of the book.

 

Most new inventions and renovating ideas come from a NON TYPICAL way of working or thinking. Which confirms your conclusion that typical TA is ridiculous.

 

Do you have enough observations? Have you graphed the historical drawdown phases? Did you conduct any optimizations? Be careful of Postdictive errors (backtesting using the known result to execute before the event). I built my original tests on 2000+ observations. Regards

 

I suspect he's not automating trading. That would explain why his backtesting results are different from his real trading results. Its a common thing to see...it is what it is and he'll just need to be more discipline in his trading to try keeping his real trading (application of the rules) as close to the rules he's using in his backtesting.

This forum is litter with trade journals as such...traders with discretionary application of a rule base trading system that's not an automated trading system.

That's ok but we need to understand and manage the fact there will be psychological inputs in the real trading that the backtesting has no such inputs.

 

To use anything for trading, without arduous verification and continuous improvement is ridiculous.

Kind of on-topic since we're discussing backtesting. However, it is interesting to hear about various pitfalls of backtesting. If you train your system on the same data used for backtesting/verification, the training data will of course more easily confirm the test without the benefit of statistical significance that you may get with out-of-sample data.

So the question becomes how can one ensure the best statistical significance for the backtesting to best ensure validity of solution development and test-results?

 

First you cant make a profit off a single MA unless you have serious price based logic going on aswell.

Likely covered, but your back testing with closed Candles, not tick data, it gives you a cheaty advantage, then it fails live.

I wasted 6 months chasing EAs and will never return and Im actually a programmer.

 

Something I did many years ago:

1. I backtested my system.
2. After that I started to trade.
3. After the closing of the market I "backtested" that day to compare it with my real trade.

Each difference was analyzed. What was the difference? Why was there a difference? What to do to: improve my backtesting, or improve my trading? Or improve both?
It was a continous repeating cycle of adjusted trading as wel as adjusted backtesting.

Trading and backtesting improved over time.

 

I think backtesting is useful but I think the approach by many is wrong.

One of the most common things I see is that many traders backtesting is done on data that's not from the same source as the data they'll be using real trading with real money. This is a bigger problem for traders not using automation trading in their real trading.

Another flaw with the trader is the lack of continuous backtesting for comparison to current real money trading results and the lack of documentation by the trader to explain the differences between the two.

I'm not a programmer but I sure you programmers have a way (solution) of dealing with that particular issue.

 

At first sight this looks obvious. But for me it is not. All depends on the question: on what is your trading system based?
It can be price action,S/R, using indicators, a mathematical model...
So it depends on the behavior of your system in reaction to what the market does.

I started trading FOREX and switched later to futures because FOREX is a market with lots of cheating and manipulation. I always used the same system, although to me FOREX is different from futures. And in both markets the system performs good. I even did a small test in stocks and even there it works well. So I can test in one market and use it in a few others without any problem.
My model is mathematical, so quotes are all I need. And quotes are quotes, no matter what market i am in. Mathematical models don't care if it is Forex, futures, stocks or options, all they do is number crunching.