What's the catch?

Backtesting etc. IMHO only makes sense if it is done within the context of a statistical risk model. You need a sample and time-delimited set of data. However if you perform your analysis over a large time interval but trade over a shorter time interval your analysis may not have any relevance. Bottom line is that "backtesting" is not a simple excercise.

 

I know of several systems that I am pretty confident will make significant money over time but I don't use them. One reason is drawdown. I know they will take a 40-50% drawdown at some point.

Having said that, I think it is a good idea to at least monitor a couple of systematic approaches. I also find it useful to use a semi-systemized approach by using a couple of systemic filters on trades. I make discretionary trades but only in the direction of the filter.

Regarding the usefullness of backtesting, there are many caveats. One is that it is crucial to test on a data series that you did not use in developing the system. If you expect the system to be robust and not require constant tweaking, it is important to test it not just on some arbitrary number of data points but also to test it in different market environments. For example, say you are testing a stock index daytrading system. You might generate a lot of trades in the past two years and think it was adequately backtested, but would you say that period was truly reflective of the market going forward?

I disagree with the idea expressed here by one well-known guru that backtesting is useless. If a system can't make money in the past, certainly that tells you something and generally it is worth a big multiple of whatever you paid for the backtesting software.

Finally, I think there is a common sense component to system design that is often overlooked. It is nice to be creative but your system must have some rational relationship to market dynamics to be reliable. I read one well know developer's commnet to the effect that he always started by hypothosizing a relationship, then tested to see if anything could be developed from it.

 

The real dilemma with backtesting is the question of getting bamboozled by changes in long term price cycles.

For example, from 1982 to 2000 or so we basically had a benign inflation/bull market environment for stocks and a malignant deflation/bear market environment for commodities.

Any testing done using information from that long and protracted period will necessarily reflect the directional biases of the dominant cycle over the time period tested.

But what if the previously dominant cycle is dead and we now have mush in its place, or what if the cycles are preparing to completely flip over?

This could be a very big deal, because up bias environments, sideways bias environments and down bias environments can display dramatically different (system wrecking) characteristics over time.

It's only logical for the mechanical trader to backtest before risking hard earned cash- but the conclusions are potentially fraught with peril. You have to go way back for statistical validity, but the further back you go the more risk you run of getting obselete bias in your results.

 

I agree with CalTrader, AAA and Darkhorse. These are other catches and important remarks.

Surely a system that worked in the past won't necessarly work in the future. That's why you often read "nothing works all the time".

another important point overlooked with system trading. I defended several times on this forum system trading (and sometimes against the so called gurus).
This is the bottom line for me :
no trader should trade without a detailed methodology and complete plan.
furthermore, this method and plan should be tested (backward and forward).
I really, really, really don't understand the total arrogance of saying you can have an idea and go for it without testing (but a few forward tests) and/or without a complete definition of what you should do and when (a kind of program).

then, do you automate or not, do you call this system trading or not. I DON'T CARE !
you have a method, proven statistically in the past (focus on risk model, I agree so much with this remark, not trends. actually, half of our systems are not trend followers at all).

several of the market wizards, not using proper backtesting of their risk model, were WIPED OUT.
discretion is fine. and we do discretion too, always helped with software (statistical risk model again) and we do real system trading (load balancing several systems).
automation works too (ask timber hill).
discretion works too. it's not a battle between discretion and automation.
it's a battle between traders wanabe not serious enough to actually test their ideas seriously (they lose) and traders who know what are the risk parameters and precise plan they are supposed to do (some process this strictly by software, some partly software, some only with their brain..).

The problem with brain only discretionnary trader is MONITORING : it's hard to know if you are within the risk parameters of your method. if you are super good at remaining within these parameters and follow instinctively all your rules, then you become a super trader, a 1% or 0.1% trader.
Good for you. and I envy you.
But an 'easier' way is to delegate these skills to software and cheat genetics (!) and still be a 1% or 0.1% trader too.
While the 90% are just desperate and don't understand why they don't make it (it's all written above why not!).

tntneo

 

Well said tntneo

I know this is off subject but do you have any more info on what some of the Mwizards are doing now? More specifics on who blew up, who is still doing well etc?

I've always been curious where those guys ended up, I know the current status of a few but most are question marks. Would love to hear any 'where are they now' anecdotes/ interesting stories purely for curiosity's sake.

 

Without a doubt, it is discipline that prevents a great majority of traders from consistently making money. I suffered from a unique problem when I first started trading. I had a great system that was profitable just about every-time, but the problem was it only gave one or two signals a day (sometimes three). Since I was daytrading, I would really grow impatient and suddenly took on this, "I've got a good feel for the market right now, I'm going to trade!"

So, I would trade outside my parameters purely from boredom. The worst thing that happened was that I made profitable trades doing this, so conditioning rewarded me for going "outside my system."

Needless to say, you may luck out for a few days, weeks or maybe even months -- but in the end, statistics will catch up with your "fly-by-night" trading habits and then you get slammed.

I eventually realized that a majority of my losses occured when I didn't stick with my system, so once I got the discipline to just sit there and do nothing, I was rewarded with consistency.

I'll take consistency over a huge winner ANY DAY.

aphie

 

In system testing you need a statistically signifigant number of trades to generate results worth considering for trading. Using intraday data, you can generate a lot of trades with only a few months of data.

But, you also need to have coverage of all types of markets, up,
down, sideways, volatile, quiet, etc.

I keyed into TS6 a fairly simple volatility breakout system about a year ago, and tested it on 15 and 30 minute bars. Looked pretty good on alot of the volatile Nasdaq stocks, and as I recall, the QQQ as well. I sort of put it aside, thinking, nice... but, the TS6 data just goes back to November of 2000, and there's really not enough here to start trading on.

Now, TS has accumulated almost 2 years of 1 minute data for backtesting. I started fooling with this system again a few days ago, and with 30 minute bars in the QQQ, it's pretty impressive.

It uses a close of the 30 minute bar above or below a volatility band to generate a long or short entry. Optimizable, but not too many parameters to tweak, same rules and parameters for short or long entry (ie. not tuned to direction of market). Catastrophic stop is rarely hit, although its hit more often if you let the system add positions. Adding increased net profit, also increases drawdown.

Anyway, I'm getting way off track here, and I will post a performance summary because I know someone will ask for it.

But, bottom line, how many trades and or months of intraday data do some of you system traders (successful, hopefully), think is necessary for dependable test results.

Results are based on trading a unit of $15,000 worth of QQQ's. 4 cents per share for commission and slippage (8cent/round turn), which is plenty enough in my experience. No compounding, same dollar amount on each trade.
Results on next post.

 

test results

 

mgkrebs,

Those are some impressive stats. I don't want to sound negative but traditionally, the S&P's have been death for vol break out systems. Also, I found it interesting that your win percentage is in the 60's. Most breakout systems are lucky to be in the 40's. Good profit per trade as well. It might be a good investment to get some older data and run the system on stock indexes from the early 90's to see how it does in a sideways market. But if it holds up, those are great numbers.