backtesting is stupid

Discussion in 'Strategy Building' started by jonp, Jul 15, 2010.

  1. This is even better... here's Jack on William Eckhardt (who is still running several funds since the 1980s and when the S&P is shown to scale with his equity curve the S&P is so dwarfed it looks more like part of the x-axis):


    Did you think Eckhardt was making any money those 18 years??? Some of your posts read like you think he is making a lot of money or that he is a person to use as a standard.

    Making a point or two a day and not compounding it beats Eckhardt and his turtles.

    Here in ET, people are doing quite well unknown to you apparently.
    Comparing the pool extraction and its applications to Eckhardt is really humorous.


    http://www.elitetrader.com/vb/showthread.php?s=&postid=2445881&highlight=eckhardt#post2445881
     
    #61     Jul 17, 2010
  2. Whether you like it or not....everything you do in assessing your life is a product of "backtesting" (experiences)..


    Simple example if you are that way inclined at 19, you should be :D, ...girls...

    Just like parameters and indicators...they are all good until they go wrong...the trick is finding the right one...can only be done after assessing previous experiences and making the appropriate adjustments to succeed...ala "backtesting"

    The hard part...ensuring that they (girl, system) stays the right one...it requires work...

    :D


    NiN
     
    #62     Jul 17, 2010
  3. Golly! Guess I'll quit doing it, then!
     
    #63     Jul 17, 2010
  4. alfobs

    alfobs

    Arthur,
    I hope to keep same frequency of posts in the future. I really do not see a reason to join a thread unless I see some absurd or unproven statements based on emotions. Anyway I'd be glad to dialogue but not sure about a subject.

     
    #64     Jul 17, 2010
  5. Dacamic

    Dacamic Guest

    True that.
     
    #65     Jul 17, 2010
  6. Thanks. I am sure ET will provide you with a steady stream of provocations to keep you posting.

    Let me quote your pithy post:

    "IMHO, backtesting is the only way to get at least some confidence in a system before you start using it. So backtesting cannot be stupid for this simple reason. In fact a deep backtesting (with sufficient amount of data) can cover most of the market moves to prove a system on robustness, and therefore to take care of those cases in future. If somebody failed with trading after backtesting then it happened because of over fitting and lack of variety of market conditions in the historical data. Less indicators, some sound logic behind your algorithm (which must be no more complicated than should), implemented money and risk management will help to make backtesting valid. For instance my one year backtesting of 15 min bars have supported me for last 6 month without any significant deviation from backtest performance results. Also in my opinion forward testing does not have any advantage over properly organized backtesting process."

    What I would like to discuss is: "a deep backtesting (with sufficient amount of data) can cover most of the market moves". Let me give you an example from my own practice. I trade NQ intraday from a one-minute chart. I used to backtest on only the last three months of the most recently expired contract. And apply the results to the current contract. As the current contract got about half way through, I might start testing it. But in both cases I became a bit uncomfortable about the small number of sample days, and switched to testing the continuous contract, which EasySignal provides six months of. My weekly reoptimizations are yielding parameter sets which change sufficiently slowly that I am comfortable with the results. But when I read your post I imagined that perhaps you are going for longer history streams. Can you comment? Thanks.
     
    #66     Jul 17, 2010
  7. I don't think I can agree with this in practice. I understand what you're saying, but I consistently find a specific type of knowledge I can only get from watching a system calculate and operate in real-time; there are things I didn't notice, forgot to add, etc. It's the only way I know of to apply my own overall knowledge above and beyond the specific things I coded.

    I suppose someone could be good enough at this they could eventually not need to do it, but if so I don't think I'll get there.
     
    #67     Jul 17, 2010
  8. Alfobs may or may not speak up, as he says he only responds to abject stupidity, which your post isn't. My guess is that he didn't mean to exclude observation of how the system works in real time. I think what he meant is that so-called forward testing merely leads to more delusion on the order of selective hindsight: "I don't know why my account is down because it looks to me like it works." I had just such an experience that you speak of yesterday. My system said to take a profit, and being a good little robot I did. The trade went on to make twice as much. And of course I knew that when I reoptimized the system it would take some of that unexpected largesse. It did. Watching your system at work is a bit like arguing with yourself: "Who the fuck coded this abortion up?"
     
    #68     Jul 17, 2010
  9. Taped across the top of my main monitor on the desk: "Your code sucks."
     
    #69     Jul 17, 2010
  10. Yours, too? I thought it was just me. I have had essentially the same codes for years, and yet I still tweak them. Of the twenty-eight running on my screen, only one hasn't been touched in the last six months.
     
    #70     Jul 17, 2010