My back-testing is too good to be true. What am I missing?

Discussion in 'Commodity Futures' started by TradeSparrow, Dec 5, 2010.

  1. Nice try, but I don't think anyone here wants to buy your "system". Ha-ha.
     
    #11     Dec 7, 2010
  2. The Sparrow Trading System prices at $1 billion US. Yes, it's just that good! Oh, and we only accept payment in unmarked bills.
    Anyway, I've already explained my strategy for free in the OP. So there. :p


    If anyone is interested in the Reddit post, you can find it here.


    Thanks LeeD!

    1) This point is really helpful. I think I understand what the Bid/Ask prices represent. I just assumed I should be using these prices to decide where I enter and exit.
    What price do you think I have my orders execute at? Mark? I can also link my stop and limit prices to Last.

    2) I'm not sure I understand.
    I use Think or Swim, which allows me to set triggers. For example, if my open-position order is executed, a close-position order will be immediately created.
    I forward-tested the entry strategy over several days with success. Today, I back-tested the entry strategy over the past 20 days using 5-minute charts. Again, it worked very well.

    3) I'm including commissions. :cool:


    Bone, that would be great.
    However, should slippage be a big concern? I use mostly limit orders to enter and exit. For exiting, I keep a back-up stop behind the closing limit order. Hopefully, if price skips over my limit order, the stop will prevent too much loss.
     
    #12     Dec 7, 2010
  3. LeeD

    LeeD

    It is usually safe to assume buy orders are executed at the next aks price and sell orders are executed at the next bid.


    Given we haven't seen the spreadsheet, suppose for the sake of example you backtest on 45-minute bars. If the system enters and exists a position during the same bar, you might be making overly optimistic assumptions about the price action within the bar. FOr example, you might assume that you buy near the bottom of the bar and later sell near the top while in fact the price "intra-bar" could have been drifting down and the modeled sell price would not be reached after the buy price.

    If on 5-minute bars you don't have executed both buy and sell orders within the smae bar, then the point above is probably irrelevant.

    Also note that 20 days backtest probably offers too few trades for any conclusion.
     
    #13     Dec 7, 2010
  4. #14     Dec 7, 2010
  5. Interesting results. I wrote some code the other day for bootstrap resampling motivated by the blogging of Michael Harris where he checks the results of a few funds (here is the link to his blog).

    My resampling algorithm seems to work well and I think it has no bugs. I used it to check the returns of the system TradeSparrow posted and the result is that the probability they come from a distribution with a zero mean is 5 in a million (p-value)! In other words, the probability the results are random is extremely small.

    So either he is pulling legs or he has got something there.
     
    #15     Dec 7, 2010
  6. Intradaybill, I feel a lot better about these results now. Thanks for doing that.
     
    #16     Dec 7, 2010
  7. Sorry for not being clear. All I meant was that if your profit per lot traded is below some minimum threshold (say 1.5 ticks per lot), there is a possibility that a majority or all of your profits are coming from what I would call 'unattainable volatility/liquidity', which is basically micro volatility that can only consistently be harvested in the 'perfect' environment of many unrealistic back test suites. Either that or there could be errors in your historical data. I've run into both of these problems before.
     
    #17     Dec 7, 2010
  8. I see. You're saying if I am making profit of tiny price changes - one to two cents - probably the model is unrealistic.

    Well, that's great news. My profits generally come from larger movements over the course of many days.
     
    #18     Dec 7, 2010
  9. As long as your average trade profit per lot is significantly higher than one or two ticks. You should be fine in that sense.
     
    #19     Dec 7, 2010
  10. Samsara

    Samsara

    I didn't look at your spreadsheet, but you said you're clicking through charts to backtest?

    Also, one risk I'd consider worth noting is the run-up / draw-down while holding an open position, not just close to close.

    Good luck and congrats if you truly did put something together in tune with the market you're trading. It's always good to see someone find a modicum of success.
     
    #20     Dec 7, 2010