just went through some backtesting of a new system and had to pull out my little book of back testing check points and realised I left out an important one: Let system profit = profit and B&H = buy and hold return for same period as system test then : profit > 0 AND profit > B&H + total commisions incurred. This is VERY important and most people seem happy when profit>B&H but that is clearly STUPID.
use however much data you want. It's all subjective anyways. Past performance is no indication of future performance. Different time periods have different market conditions. Hindsight is always 20-20. You could always, knowingly or not, fit your ideas to the data and make them fit. So you test it for a month and most days it works but you have some draw downs, are the profitable days the norm or the exception. Did you use data in a time period of a huge bull market where anything made money. So really it's up to you. Don't let anyone else tell you how to do it.
i received some excellent comments here and just to update.. i revamped the steps i take in system building /testing. -create a buy/sell signal -test on ~ 1/3 of my data -comparing results w/ out of sample data. (curve fit is ez to see here) -choose settings that test best w/ out of sample data -fwd test (watch paint dry) oversimplified a bit, but you see my basic premise anxious to hook 1 of my systems up. we'll see... thanks again to all who posted
as i fwd test, (which to me would be a month or 2 for verification), i create new signals, new systems. i think i have some good ones. i'm afraid i'll have a hard decision deciding which system to go live with. in another month, i'll have 50 more systems. do you have a suggestion? maybe i should stop making systems. lol
There are always good reasons to explain why a system might work or fail. However, there are effects that are not predictable even with the best models and theories (e.g., dip buyers). In that case, it takes common sense and trading acumen to decipher between anomaly and "real" causality. My point is: Testing in sample/out of sample data is meaningless if the underlying principle can't explain the effect in the first place. Once your family of systems passes that test, this is basically how I choose alerts generated from several dozen systems: - Define the acceptable profit target and max drawdown - Forecast future mean return and standard deviation (ARIMA methods, etc) - Portfolio optimization to determine best mix of systems with same return but less risk - Repair/rebalancing (ideally in real-time, if you know how) Most importantly, you must use log returns in your analysis ( log(p2/p1) )
thanks for your comments. i'll keep ploddin' along. some of the suggestions are currently beyond my reach i expect. but, i won't forget them.
was a good week. made progress. my systems are becoming more and more viable. ( entry signal more refined) is it this simple? create or find your edge = signal. apply proper money management = exits retire focusing more on higher frequency systems (~ 2-5 trades/day). just feel i can trust the result more. and drawdown is easier to contain. although the stats might look better on others. my data limitations doesn't allow me to test the slower trading systems as i would like.
still here think i'm ready to go live actually, but will fwd test ~ 2-4 more weeks for good measure. i feel like i've come aways from where i started. (initial systems were scrapped..or i should say, improved). the system i've connected to IB simulation hit 7 straight from yesterday afternoon thru today. ~ 900$ for single contract. so, i'm very encouraged. several other systems i have look good as well. will be hard to decide which to go live with.. bb *my setup is functioning well w/ IB. no major glitches so far. i've been tempted to move to Multicharts.. would be a simpler setup i think, but will wait.