Is Walk-Forward (out of sample) testing simply an illusion?

Discussion in 'Strategy Building' started by pursuit, Oct 17, 2017.

  1. pursuit

    pursuit

    Oh, so you lied because you thought I'm "manipulating" you. LOL you need a psychiatrist, bud.
     
    #61     Nov 1, 2017
  2. sle

    sle

    I find that there are two separate degrees of discussion.

    There is theoretical “what if and how about” which most professionals engage in all the time without holding back much. A chat along the lines of "in my experience, cost of risk is a better way of smothing out your transaction frequency than hysteresis, especially in strategies with multiple legs".

    Then there are detailed "how do I make money" type of conversations that you could only have with the members of your team. Anything that involves your specific factors, instruments you trade or anything that can really dilute your alpha falls in that category.
     
    #62     Nov 1, 2017
  3. userque

    userque

    Yes, I understand what you are saying, but even with this example...generally speaking...it still just depends, imo. Just for hypothetical...

    You quote, "in my experience..." in your first example.

    If that experience was the result of lots of research, computer time, coding, testing, thinking, money, blood, sweat, tears, lost family time, lost social time, and led to some supposedly unique edge (or edges); then I doubt "most professionals" would willy-nilly reveal the results of 'their experience.'

    So, I see what you're saying, but 'it depends' upon the exact facts/circumstances, imo.
     
    Last edited: Nov 2, 2017
    #63     Nov 2, 2017
  4. sle

    sle

    Well, I have not and would never bleed or even cry for any of my employers :) However, I can speak from decades of professional experience in quant trading, both as a book runner on a dealer side and a PM at a couple funds. If there is one thing you learn is that there is nothing new under the sun. My portfolio has over 30 active strategies at the moment and I can’t name anything “unique”.

    By no means I would share the actual details of my alpha generation, but I have done in-depth conversations with coworkers or friends regarding more general ideas like risk metrics, measures of significance or various “how-tos” that are not specific to my own strategies. That my experience.
     
    #64     Nov 2, 2017
    userque likes this.
  5. userque

    userque

    Neither would I. I never referred to an employer.? :)

    There're always new things...or new ways to combine old things.
    https://en.wikipedia.org/wiki/Timeline_of_mathematics

    I agree. I never said I wouldn't discuss anything at all. I said "it depends"...same thing you seem to be saying. :)
     
    #65     Nov 2, 2017
    sle likes this.
  6. maler

    maler

    Your skepticism is understandable.
    However, if done correctly, out of sample testing is useful.
    Sometimes it is easier to get a feel for things if they are exagerated.
    Say two people come to you and claim they have a method to play the lottery.
    The first guy shows you his correct predictions for last week lottery.
    The second guy predicts the numbers for tomorrow and they turn out to be true.
    What model will you use to play the lottery?
     
    #66     Nov 3, 2017
    userque likes this.
  7. I think that out of sample/forward testing guarantees nothing. It can provide an added level of confidence to those who are about to deploy a new trading system but it cannot guarantee that a future sample will generate greater alpha than a historic sample. One subject that does not get talked about nearly enough here is how to define edge death/erosion. I know plenty of people here might just say "historic max draw down peak to valley will decide if my system is active or not".

    Step #1. Finding a profitable historic "edge" and hope that some sembalance of that persists into the future.
    Step #2. MAKE MONEY or at least break-even. Don't blow up!
    Step #3. Did you exit this system gracefully and with some money or did it blow up and take all of your profits with it? Yeah I know any quant reading this would just say "run a bunch of non correlated systems at the same time and who cares if any one individual system blows up? I writing this post with the retail trader in mind, someone who probably runs one or two trading systems max.
     
    #67     Nov 21, 2017
  8. ironchef

    ironchef

    Can someone kindly explain to me what is walk forward and out of sample testing? I don't understand the terminology.

    Back testing segment 1 and 2 vs back testing 1+2 combined segment, so what is out of sample, same data set if they are consecutive?

    Thanks.
     
    #68     Nov 24, 2017
  9. ironchef

    ironchef

    So, there is really no hope for us small retail traders? If we cannot find new strategies how can we beat the professionals with vast resources with the same strategy they use?

    For all the professionals, if all employs similar strategies, how can you make money? Unless you take them from us? With this logic, we retail traders should give up trading?
     
    #69     Nov 24, 2017
  10. sle

    sle

    I think this is something that deserves a separate thread (or even a whole book).
     
    #70     Nov 25, 2017