if backtesting works, what could go wrong in real trade?

Rogue Trader
_
Registered: Sep 2001
Posts: 39

_

02-05-02 06:50 AM
Of all the paradoxes that exist in trading, the notion of “Back Testing” could quite possibly use some “Back Testing” of its own. The literal meaning of “Back Testing” is rather self explanatory and actually defines itself with little explanation. “Back” is referring to past occurrences and “Test”, is a trial of a model.

Regardless of the level of sophistication, rigidity, flexibility, or time frame of your model, you help to define it and ultimately accept it. You are simply testing a method against past occurrences and analyzing the results. Conventional wisdom supports that the result of such testing can be optimized, and
when applied to the future, produce a positive expectancy. None the less, one obvious fact still remains. The market, which is what your modeling into, remains a variable. Upon further scrutiny though, a less obvious change also occurs. Your perspective. Unknowingly, you have begun to reinforce in your mind what should happen as a result of your positive experience in literal backtesting.


Once satisfied with your results, your back tested model will still require one more trial. A trial by "The Here and Now.” You have learned the literal definition of back testing above. It is now time for “You the Trader” to learn the traders definition of
back testing. In the traders definition of “Back Testing”, “Testing” simply means to put on the trade, as dictated by the system of your invention. "Testing" remains constant and mechanical as a function of your system. “Back”, on the other hand, is actually a variable.


Now with everything to your satisfaction and your methods in place, your back tested system triggers a trade and your day of discovery begins. This is what you can expect to discover. Everything you thought your system to be, wished it to be, hoped it to be, ever dreamed or believed it could be is absolutely irrelevant under a trial by “The Here And Now”. Go back and read that again.


Enter “Back”, the variable of the traders definition of “Back Testing”. Under a trial by “The Here And Now”, once entered into a trade, you are given but 3 choices. But wait a minute. This is strange. None of choices are related to the literal definition of back testing, or the positive "perspective" from which you saw your trades resulting in. “Back” (traders definition) becomes to mean.....

1) Back Off 2) Back Down 3) Back The Hell Out.

Back Off: This refers to reduction of trading frequency.

Back Down: This refers to reducing your share or contract size.

Back The Hell Out: It means just that. Neither you nor your system are in the flow of the market. Get out and re-enter when
conditions improve. Both yours and the markets.


There in lies a paradox of back testing. From the traders definition of "Back Testing" you derive a finite and constant set of prepared responses on every trade, every time. From the literal definition of "Back Testing" you derive a deceptively skewed departure from "The Here and Now". You only see what should happen, not what could happen.


Now ask yourself this..... If the correct response to any trade is a constant, regardless of all the variable methods, signal generators, systems, and market conditions, where does one find a constant? The answer is quite simple. You become the constant, constant "WITH" the market.


It is one thing to say you trade the markets. It is something completely different to say you trade "WITH" the markets. If you have a bias, a belief, or behavioral response from the past, regardless of origin, of what the market will do, should do, or could do, you are not trading "WITH" the markets, you are trading your past perspective of the markets. The day you decide to trade the markets perspective in the moment, and not yours, is the day you will be trading "WITH" the markets.


Oh, by the way. There is one more definition of “Back Testing.” This one is provided to us courtesy of the markets. It is for all those traders who choose to ignore the traders definition of back testing. The markets actually re-structure “Back Testing” to “Backer Testing”. Backer testing occurs when you, the trader, have to go out and find new ”backers” to fund your account, the one you just blew up, as you chose to “test” your trading account for the sake of “Back Testing.” Literally, that is.

 

You don't get the same fills in the real world. Backtesting doesn't account for slippage.

 

backtesting is looking for a Dusty Springfield system.

How many times have we sat there and were flat going into a report and BANG, the ES is up or down 10 handles in a blink?

http://www.youtube.com/watch?v=GBwSN0Yw5l0

http://www.youtube.com/watch?v=ycbgHM1mI0k&feature=related

http://www.youtube.com/watch?v=0l-GpISGBFY&feature=related

Songs are like trading.........they are about what is in your head

http://www.youtube.com/watch?v=LjMcWXHOWoQ&feature=related

http://www.youtube.com/watch?v=JkhX5W7JoWI an old favorite

 

Or the Fed increases the discount window rate and in the blink of an eye gold longs are screwed.

Just because something happened in the past doesn't mean it will happen in the future. The market is dynamic. Back testing helps sure, but then some cure fit, over optimize and then can't understand why the same things don't occur in real time.

 

3) Backtesting is basically a method people try to prove a hypothesis. Unfortunately, probability says that your results can be almost anything.

Backtesters of course, only focus on "winners". Accordingly, they will focus on the few % of results that by definition, must appear on top. Unfortunately, this is almost always pure luck, not an edge. Say you backtest a 1,000 sets of data, and 100 seem promising

You must forward/live test all 100 promising backtests. Almost all will disprove themselves given enough time. And the same thing applies, statisticially, even your promising results set will yield a range of bad to good results. So if 10 of the 100 promisings look good, it is likely the same thing, as 10 of any 100 will be in the top 10 by luck. You must again live test the 10.

So basically, people are falling in love with luck rather than edges/skill almost all the time.

 

yes but ur hypothesis is already wrong in the 1st place. Why does Jim Simons hire statisticians instead of traders for his hedge fund? This is already a big clue.

 

Probably because of the changes in the market, you cannot dictate that even if you already have a system that you use.

 

Besides the problems some others posters mentioned and are more fundamental, it could also be that you did not correctly backtested your system or that your backtester has some bugs that distort the results.

I suggest you compare the trades you get in real-time with those you get from the backtester for the same period. If the entries are not the same, there may be problems in your backtester. Problems like the ones described in this article

http://www.tradingpatterns.com/Literature/backtesting/backtesting.html

 

Overfitting.

 

Give it a rest.