I'm designing some automated systems, and would like to know what others use to verify the validity of the backtest results. Clearly there is nothing better than trading your system in realtime, but prior to this step, what do people use as go/no go tests? I use walk-forward tests and different timeframes. Then I test on different instruments in the same categories: ie different stocks for a stock system, different futures for a futures system. I generally test that the system works similarly long and short, however I have found something that works quite well long, but fails dismally going short. Would this fail your go/no go criteria? What other methods are used to validate backtest results? Adrian
Those are some difficult questions that have fuzzy answers. Some things to keep in mind though; foremost, the fundamental concept of the system must be sound. There are concepts that are tried and true and work no matter the product or timeframe. Does your system capitalize on a fundamental concept, or, does it use patterns or combinations of optimized indicators? Usually, a system with a sound fundamental concept works equally well on both long and short trades but tends to have a long bias (i.e. not as good on the short side due to the upward drift inherent with most stocks) - one should be able to flip the logic and tweak the short side a bit to get a tradeable model. Poorer (but not bad) performance in the short side is usually found in mean reversion based systems but should not be prevalent in volatility based systems. In terms of validation, what exactly have you tested? What is the systems frequency and what type of data and how much of it are you using?
Just about any system that is tested over a relatively long time frame will do better with a long bias, because of the market's upward drift (as mentioned by Mike). That being said, you could try to filter the system to operate with more of a long/short bias, depending on the type of regime you are in. Identifying that is a challenge: but it's been mentioned on the boards here, and is fairly intuitive.
I'm testing an intraday futures system that takes a maximum of 1 trade per day. Testing over 90 day/1 year/3 year timeframes I get an annualised return on the longs of about 100%, however the shorts are around -40%. Way too big a difference to be the long bias. The last 2 years are a great test, given the massive decline in 2008 and the appreciation in 2009. However my long side shows a nice equity curve, relatively unaffected by the shocks. The short side is just increasing losers and drawdown. My post was primarily driven by my lack of indentification of the short trade failures. Generally this is an alarm bell that my coding is wrong somewhere, but it's a simple system, and the code is correct. I guess I'll just start trading it in sim and test it out. Thanks for your input. Adrian
What good reason is there to assume that market behavior is the same in updrafts as in downdrafts? Volatility is higher in bear markets for starters.
There is no rigorous basis as to why a system that works for one stock should work well for another stock. If it works well for the stock you want, why should you care if it works well for another stock? And if it works well in backtesting frt hundreds of stocks, this tells you nothing about the stock you will trade. Again, there is no rigorous basis for such assumptions. Some systems work well long-only and some others short-only. IMO. maybe you have been cheated by some authors who claim that systems should work well in both directions. Remember, most trading book authors have never traded anything. EASY. Just trade your system with a reduced position size for 3 to 6 months. There is NO cheap way to validatate systems. All methods that rely on historical data or paper trading suffer from the same shortcomings. Trading is an expensive business.
How many years of data total and how many trades? Walk forward testing does have merit but should not be used on its own. What does the return distribution look like for the entire sample space? Are you trading this on a portfolio? If so, how did you select the portfolio? Do you have another portfolio you can test it on? Are you looking for a specific behaviour in a specific set of products, or , are you looking for behaviour across the entire futures universe? I'd be careful of the first point since your portfolio selection can introduce bias.
Thanks for the information. It all helps. The system is futures only, so there is no surviorship bias. After having spent more time on the system, it further indicates that good assymetrical systems exist (well in terms of my backtest results that is!). Next step is test in in sim for a few weeks to validate the backtest results, then implement live with a smaller contract size. Thanks all for the feedback. Attached are the backtest stats. Adrian