Backtesting Metrics

Discussion in 'Automated Trading' started by DustyFoot, Dec 31, 2010.

  1. ronblack

    ronblack

    I also like the metrics you mentioned. Good suggestions. The important thing is that you provided a reason for looking at those ratios. Most people throw suggestions without justification. For example a profit factor of 5 may be a terrible thing if DD is excessive. This is in reference to the other poster who recommended the profit factor.
     
    #11     Dec 31, 2010
  2. ronblack

    ronblack

    Do you have to say anything about Pardo's book or you will keep trolling forever?
     
    #12     Dec 31, 2010
  3. nLepwa

    nLepwa

    Sure,

    The rolling pf gives you the distribution of your profit curve. From this distribution and the trade frequency you can compute profit and drawdown (and anything else you like) using monte carlo.
    It is the only statistically correct approach as measuring profit and DD directly from the equity curve gives you only the estimate from this one run that was realized in the past. The future values might vary widely from your estimate if you're "unlucky".

    You need 15+ years simply because most mean-reversion strategies have a huge regime shift around 1997-1998.

    Ninna
     
    #13     Dec 31, 2010
  4. nLepwa

    nLepwa

    Then you clearly don't understand what the profit factor is...

    From the pf and trade frequency you can compute the maximum DD (within your chosen confidence interval).

    In other words, if your pf (or trade frequency) is good enough, the DD can't be excessive. The DD depends only on the pf and the trade frequency.

    Ninna
     
    #14     Dec 31, 2010
  5. Being wrong and being rude. An interesting combination. I didn't know answering dubious questions was now considered trolling. I could just ignore you, but then some newbies might be misled.

    I loaned my copy to a friend working for an investment firm on some strategies of their own.

    However, the most important thing I recall learning (I believe from this text) was how to avoid "curve fitting" by dividing your historical data into two non-contiguous sets. One set to develop your strategy and the other to confirm your results on data it had never seen before.

    Also of value was learning to understand the value and proper use of some of the metrics.

    I don't recall if this book was the inspiration of my four step strategy qualification process.
    1. Back Testing
    2. Paper Trading
    3. Small Money Trading
    4. Serious Money Trading
    Every time I make a modification to my strategy, I go back to at least to step 3. If the extent of the modification is significant or if small money trading does not show good results I will go back to steps 2 at least. Sometimes all the way back to step 1.

    Sometimes I abandon the change rather than go back, but I never move forward unless the results are confirmed as an improvement.
     
    #15     Dec 31, 2010
  6. Samsara

    Samsara

    Thank you for the insight.
     
    #16     Dec 31, 2010
  7. I have the Pardo book. Interesting stuff and I would think it the definitive guide on proper backtesting procedure for any automated strategy. For myself, however, I don't create fully automated stuff so it was a useful read, but that's about all.

    As for some of the metrics. It's the usual stuff.

    # of trades
    P/L
    Annualized P/L
    % winning trades
    largest win
    lenght of largest win
    average win
    length of average win
    same as above for losses
    standard deviations of averages
    max drawdown
    lenght of max drawdown
    longest winning run
    longest losing run
    blah blah bla

    What I do, to keep it simple, is just track number of wins, losses, averave win, average loss, # of trades, # of trades per day, pf...

    I then put these into a kind of do it yourself monte carlo excel sheet that I created that allows me to run several iterations of a thousand trades and see what the worst case scenario might be.


    As for needing 15+ years of data to test an intraday strategy, I call bullshit.

    You'd first need to know just how often the strategy actually trades. Let's say that you have a strat that generates 30 trades per day. So, what? You need 100,000+ trades to know whether it works or not? That's just nonsense and belies a lack of understanding of statistics.
     
    #17     Dec 31, 2010
  8. nLepwa

    nLepwa

    The trade frequency is irrelevant if you suffer from regime shift.
    I have mean-reversion systems with thousands of trades per month that perform extremely well from 1990 to 1998 and then crash.

    1998 really seems to be a turning point for a lot of mean-reversion strategies. Since you don't know when the next turning point will come, backtesting back to 1997-1998 gives you a nice overview of how things could turn out...

    If your strategy exploits some kind of mean-reversion even remotely (and most intra-day strategies do) I would suggest to backtest atleast 15 years.
    Personally I backtest 25 years. And I expect my strategies to work for (atleast) another 25 years. Trading isn't a get-rich-quick scheme.

    Ninna
     
    #18     Dec 31, 2010
  9. IIRC he not only lists the metrics, but discusses their benefits and limitations. This permits the strategy designer to make informed decisions on which metrics to use.

    I'm not one for a completely automated system. I'm a retired software engineer and I know my limitations. I always want my brain and fingers between the trade recommendations and their execution.

    Totally automated or not, IMO, Pardo is a good reference in learning how to evaluate your strategy.
     
    #19     Dec 31, 2010
  10. Ninna,

    With an intended lack of respect to those you're addressing; you're talking to a few clowns in this thread. The range of belief systems held on ET can be quite diverse, unfortunetely, when it comes to these topics, most choose to betlittle information if it goes counter to their limited experience base.

    Point in hand, no one should trade a strategy that doesn't survive in all regimes, no matter the frequency. It's about confidence in the model. If one is putting on large positions, it comes down to confidence in the process one took and the method that resulted.

    The posters you're addressing don't even know what a regime shift is...

    Mike
     
    #20     Dec 31, 2010