Wish to open a discussion on the ill fated and well orchestrated ways to back test trading models. To successfully gain an idea which might provide and edge and then translate it into code is daunting enough. The ability to properly back test is very elusive let alone produce an outcome which is replicable in real trading is a near impossible feat. This continual failure is non prejudice it will destroy the wealthy and poor, the idiot and the savant, the young and old all alike. There are no qualifications which mankind has labeled or obtained which will secure the right to succeed in developing a successful trading system. What is your contribution to avoiding failures and pitfalls, your confession. Also it would be intresting to share what works and why. I will start off the conversation by saying that for many years of great promise and greater failure. I looked in the mirror and asked myself if I had wasted my time and money chasing a myth which was only profitable to vendors who perpetuated the dream. More on my personal experiences. After much failure in the area of back testing. I gave up and went to work for a person who provided the resources that I simply did not have. Armed with a well funded operation at my disposal, there was no excuse now not to figure this out. I was in a position where we had to reverse engineer a system to meet the expected criteria of our investors. As if it wasn't hard enough to build a successful model, we had to tailor make one to fit an expected risk profile. This set me back probably ten years, but taught me things I would have otherwise never discovered. === Rule 1.) As much as I try and mechanicalize the process of optimizing models, there is always a bit of artwork which seems to be necessary. Example typically the most profitable parameters will be chosen by novice builders. I think this is because if they don't pick the absolute best parameter the model will fall apart. I often like to see a wide variety of parameters that the model can be set at which provide decent profits. Rule 2.) I mostly like a decent profit factor and a good size average trade net profit. However I find it almost impossible to even get that far looking at the stats of the model if the net profit is not at least fifty percent or greater than the gross profit. I have seen for example models that make one million gross but only net three-hundred thousand. I would rather see the net profit at five-hundred thousand or greater. This tell me the model has a chance in the real world, and this criteria eliminates many models. Rule 3.) This is where there is a clear fork in the road between what makes money and what people want. People want to make money but take no risk doing so. Less risk models make less money and there is a hidden risk of not making enough money fast enough to outrun ever present inflation. Since I have learned to embrace risk rather than hide from it. I set out some criteria that must be met for clients trading to ease their fears. This is where the real challenge comes to model around perceived fears. The greatest fear that should be heeded is that of not making money. So rule three is to identify who this model is for and ask what do they expect out of it. It is not good enough any longer to just have a profitable model. That's yesterday, now there are very stringent and sophisticated ways to dissect a model. You need to be prepared for long exposure under the microscope. So be truthful with yourself and know that you have something that is real and can withstand peer review. Rule 4.) I always test using single lot contracts unless it's something of very low value and then I use 10 or multiples of. This allows me to better track actual system performance rather than rely on some leveraging scheme to make profits. So what I am saying is to trade the minimum amount of contracts you can during testing and keep the position size always the same for base testing. Then after you have a solid model add in the leveraging schemes else you may be fooled by randomness.
Great post. Allow me to rephrase your rules to make sure I have them correctly. Correct as necessary. Rule #1 - get a robust system, one that is profitable over a decent range of parameter values rather than a narrow range. Rule #2 - get a profit factor at least equal to two. Rule #3 - if you're going to be a wussy about risk, get out of trading and become a granny-style buy-and-hoper. Perhaps this is just my own personal philosophy overriding your rule. Rule #4 - use a fixed position size for backtesting to establish positive expectation before moving on to optimizing leverage (aka optimal position sizing). I'm ambivalent about Rule #2 but agree with the others, especially #4. I have a discussion on Kelly sizing going on right now in Trade Management if you wish to join us.
Great post. Thanks for sharing! Another rule: Before you move on to creating a trading strategy to test, convince yourself (using Excel, or statistics, or whatever ...) that the edge you wish to exploit is really there in the data (i.e. in the price data, or whatever data you are using) ... only if it is, then build a strategy to test. The edge should be there and identifiable before you start defining entries and exits to exploit it ...
OK I'll bite. How do you determine that you have an edge before you test a strategy based on the edge? The main point of strategy testing is to determine that a real edge is there, or am I wrong about that?
Let's say your edge was that you suspected a break through the prior day's High or Low usually led to a break out that could be chased for a reasonable trade. What I am saying is that before you look at coding a strategy that looks for a break out and then enters a trade, and adds a target and stop to that trade, there is other analysis you can do ahead of that ... ... perform a statistical analysis of the price action at all such break outs and determine if price subsequently moves in the direction you suspect. If it does, then go ahead and code up a strategy and move to the next stage of testing. If it doesn't, give up there and then, and move on. The point is that you might still code up a strategy and backtest to get what look like good results, but if there is no statistical basis for the edge you have identified, then the risks have increased that you are just curve fitting (because you have just matched price action with entries and exits serendipitously rather than as a result of any insight into market action).