Can you guys tell me what your experience has been in designing systems that are less dependent on precise parameters, i.e. avoiding curve fitting so that the in-sample performance predicts out-of-sample results? I have been optimizing based on Ulcer Performance Index or Net Profit, but I'm not entirely satisfied with the results of either of those. I also tried Sharpe Ratio and K Ratio, but those seem to be worse than UPI. Any thoughts?
I think your design goals might be suspect... what type of system are you working with? If you're dealing with a directional, multi-day system, then optimizing for net profit might be a mistake. The reason being is that the optimization will focus on the larger winners (i.e. the tails of trade distribution). I would avoid system parameters that emphasize catching those tail trades. If you don't mind, can you post a distribution of trades for non-optimized and optimized results? That will likely tell you if you screwed something up... you'll likely see a less normal looking distribution with over-optimization. By distribution I mean histogram with individual trade PnL on the x-axis and frequency on the y-axis. Mike
This is an intraday futures trading system. I have attached an example of UPI optimized versus non-optimized results (forward test). I think the results look reasonable, but I guess I'm just interested in trading views on your fundamental approach to optimizing, i.e. what your favorite criteria for evaluation are.
Yeah, those results look fine. You might want to consider using R or Excel to get a bit more granularity/analysis per those distributions. Visually they look ok, but, in general I would like to see something fairly close to a normal distribution (numerically) with a nice peak somewhere to the right of 0. Generally, I use return / Max drawdown as my guide. The better this ratio, the better the scalabilty. Have you tried to optimize for this value?
Thanks for your response, Mike. No I don't think I ever used maximum drawdown to optimize, but I tried Recovery Factor. The problem I had with that is it tends to narrow the possibility of trades to a very small number, where the maximum drawdown was zero or one tick, something like that. I don't use exhaustive testing because I use a tick database and if you have more than one or two parameters it takes forever to optimize. I use CMA-ES to optimize. I used to optimize based on net profit and then pick the best combination of parameters by comparing net profit, average profit per trade, profit factor, maximum drawdown and recovery factor. Then in the course of my research I found out that a lot of traders liked to use Sharpe or K-ratio, but those don't seem to work as well as UPI, which is what I'm using to optimize now. The UPI criterion makes sense to me because it seems to be a good measure of consistency, better than Sharpe or K-ratio.
FYI, Amibroker has CAR/MDD as one of its optimization selections. When I develop a system (i.e. single concept), I look at the parameters you mention as a whole; they're all useful. The next step is to incorporate that system into a portfolio of systems and that's where the CAR/MDD ratio becomes very useful. For example, two systems combined may produce a lower Profit factor (since one is meant to hedge the other), but, the overall CAR/MDD will be vastly improved... meaning one can scale up with *some* comfort. Mind you, the systems have to be decent on their own right, but, this approach (Sys A + Sys B, where conceptually the systems oppose each other) allows for more consistent returns. I guess what I'm getting at is that I don't optimize that much on single systems, I optimize for an individual systems' effect in a portfolio of systems.
Thanks, Mike. I haven't used CAR/MDD much, but I'll take a look at it. Because that's based on annualized returns, do you think that would give a reasonable evaluation for an intraday system? When you said you don't optimize for single systems, how do you set your parameters, or are you using some kind of adaptive model that adjusts on the fly?
+ I have to admit this post summarizes well what one will learn after reading 10 books on the subject. Do you also optimize for optimum performance across multiple markets? To me optimization is not the important issue. I'm worried when my combined systems fail to perfom well when applied to similar markets. For example, when a portfolio of systems with excellent forward test results in S&P 500 Index fails to perform well in Russell 2000 Index.
Hi Ron, Sorry about missing this question earlier. I thought about it and meant to post something back a while ago. It slipped. Anyhow, you mention something pretty interesting. I predominantly test on the SP500 components (mainly because I have all the data for these), and will often do a gut check on mid-caps and small caps. The discrepancies I've seen are usually a result of liquidity. My systems need good intraday volume and decent ATR... So the more active equities will always do better... But, I too would be worried if one of my systems showed drastically different results on one index versus the other... In fact, I have thrown concepts out because of this issue. Fundamentally, what I like to see is systems that simulate better on lower volume equities. My take is that lower volume = higher spreads & less derivative activity = less efficiency. So if the system is doing something good, it should simulate better on less liquid portfolios. Of course the killer now becomes slippage & easy to borrow shares... so real world trading of such portfolios is usually not realistic (at least in my case). That's my theory anyway... not an easily verifiable idea, but, it seems logical to me.