Is Walk-Forward (out of sample) testing simply an illusion?

Discussion in 'Strategy Development' started by pursuit, Oct 17, 2017.

  1. Macca1

    Macca1

    Have you actually tested this out for yourself? Or are just putting forward a hypothesis?
     
    #41     Oct 28, 2017
  2. Not an illusion at all if done properly with close to zero degrees of freedom.

    First, before doing any WFA, a trader needs to understand why, and how their model(s) gives them any kind of competitive advantage in the marketplace. Also make sure they're executable in real-time before moving into the analysis phase. Assuming doesn't cut it. Real money needs to be put on the line.

    Second, unless i'm marketing to investors, I don't give a shit about Sharpe, Sortino, Treynor and whatever metrics so called quants use, and I don't need R, MatLab, etc... to do effective WFA. A custom built Excel sheet works just fine. As a lone wolf, I can only keep track of a limited amount of variations/models.
     
    #42     Oct 28, 2017
  3. userque

    userque

    What you've described is not walk forward analysis/optimization.

    True WFA is robust. I know I'll have to explain/simplify...so I'll just get to it:

    1. Optimize over days 1-50.
    2. See how it worked on OOS (Out of Sample day 51. It must be out of sample because day 51 didn't exist during the analysis/optimization of days 1-50).
    3. Day 51 closes.
    4. Optimize over days 2-51.
    5. See how it worked on OOS day 52.
    6. Repeat.

    What you did say, and I agree with, is that optimizing by holding data out for *validation* is, many times, not much different that optimizing over *all* the data.

    Let me simplify further:

    WFA *tests* on *true* OOS data. In the above example, our hero optimizes *after* the close of the dow/nasdaq/etc. markets (4 pm ET)...but before the next market open.

    Then, the next day--after the next market close, our hero sees how well his forecast did.

    Repeat.

    It doesn't get more robust than proper WFA.
     
    #43     Oct 28, 2017
  4. sle

    sle

    There was a very nice study done by someone at DB (I think) that shows how your "percieved" Sharpe grows over a number of optimization passes. It looked pretty scary, IMHO.

    My approach is "hypothesis -> study -> first pass strategy -> live trading in small size -> improvement based on real results". In most cases, the causes for failure are aspects that can not be accounted for in paper trading (fills, borrow, information delays).

    Also, I am almost never ready to deploy a strategy unless I have a solid fundamental hypothesis regarding the source of alpha. There are no free lunches, only cheap lunches or stolen lunches. If it's the latter, I'd like to know what I am paying and if it's the former, I'd like to know who I am stealing it from.
     
    #44     Oct 28, 2017
    digitalnomad likes this.
  5. pursuit

    pursuit

    As you correctly my point is about traditional optimization/backtesting not the "WFA". As you describe WFA is basically "rolling window" optimization.
     
    #45     Oct 29, 2017
  6. userque

    userque

    Yes...also known as "sliding window" optimization.
     
    #46     Oct 29, 2017
  7. pursuit

    pursuit

    Well... then it's SOLVED. All we need is to use "sliding window optimization" and our system will always work live and will work forever. /s
     
    #47     Oct 30, 2017
  8. userque

    userque

    You're welcome /s
     
    #48     Oct 30, 2017
  9. pursuit

    pursuit

    How have your systems tested with WFA fared live? Was performance in line with testing? If so, for how long?
     
    #49     Oct 30, 2017
  10. userque

    userque

    Note: I can't be manipulated into revealing information I wouldn't normally reveal. I'm a pretty good Texas Hold'em player.

    This is your thread. I haven't started a thread revealing my data. If you want me to compile data for you, I can quote you a price.
     
    #50     Oct 30, 2017