FORUMS BROKERS SOFTWARE
Home
 
    Forums > Technically Speaking > Strategy (System) Design > Walk-Forward Testing and Optimization


Reply
 
Thread Tools
Old Mar 7th, 2012, 02:33 PM   #1
trader3cnd
 
 
Join Date: Apr 2008
Posts: 136
Does it make any sense? Has anyone used it with success or is it just another buzzword of the software industry?

IMO an optimized system - is an optimized system, whether that is done stepwise or not. More importantly, I could think of some cases where walk forward optimization would produce worse results than optimizing over the whole sample.
    Quote
Old Mar 7th, 2012, 05:39 PM   #2
jcl
 
 
Join Date: Jan 2012
Location: http://dictionary.of-the-infinite.com
Posts: 407
The answer of your questions is yes. Walk forward optimization makes sense, is necessary for successful strategy development, and produces worse test results.

When you develop and optimize a strategy, you don't want good test results. You want realistic results that reproduce real trading. Backtests can't do this. This can only be achieved with walk forward tests.
    Quote
Old Mar 8th, 2012, 04:40 AM   #3
intradaybill
 
 
Join Date: Feb 2008
Posts: 2,962
Quote:
Quote from jcl:

The answer of your questions is yes. Walk forward optimization makes sense, is necessary for successful strategy development, and produces worse test results.
Produces worse test results? I'm not sure what you want to say.

Regardless, nobody is going to take your word for it. Do you have some examples or you're talking under the influence?
    Quote
Old Mar 8th, 2012, 04:50 AM   #4
jcl
 
 
Join Date: Jan 2012
Location: http://dictionary.of-the-infinite.com
Posts: 407
Quote:
Quote from intradaybill:

Produces worse test results? I'm not sure what you want to say.

Regardless, nobody is going to take your word for it. Do you have some examples or you're talking under the influence?
I'm always talking under the influence of the mind ;).

I have plenty examples, as I've meanwhile used WFO for about 25 strategies since last October. Here's one example, with a short WFO tutorial:

Link Deleted Spam

There are two reasons why a walk forward test often gives you worse test results than a backtest or a parallel out-of-sample test. First, it eliminates most overfitting effects, and second, it tests in a slightly different market situation than you had when optimizing. Both effects make your tests more realistic, which usually means a worse test result.
    Quote
Old Mar 8th, 2012, 08:53 AM   #5
goodgoing
 
 
Join Date: Jun 2009
Posts: 383
Quote:
Quote from jcl:

Both effects make your tests more realistic, which usually means a worse test result.
If "realistic" means worse, would that make "reality" mean disaster? Because in reality there are unknown effects that make things even worse.

Why do you think optimizing during a choppy market period and then trading during a trending market is any good? It is a bad idea. WFO is a bad idea in general. A parameter optimization session should include a wide variety of market conditions. Like most things in trading, some guy once mentioned the method in his book and some other guys continue talking about it. Most of those who use it in actual trading fail. This is the truth.

There is an interesting ongoing discussion in this forum:

http://groups.google.com/group/adapt...f68c5ea2?hl=en


Some people who actually work with systems claim OOS testing is useless.
    Quote
Old Mar 8th, 2012, 09:59 AM   #6
jcl
 
 
Join Date: Jan 2012
Location: http://dictionary.of-the-infinite.com
Posts: 407
Quote:
Quote from goodgoing:

If "realistic" means worse, would that make "reality" mean disaster? Because in reality there are unknown effects that make things even worse.

Why do you think optimizing during a choppy market period and then trading during a trending market is any good? It is a bad idea. WFO is a bad idea in general. A parameter optimization session should include a wide variety of market conditions. Like most things in trading, some guy once mentioned the method in his book and some other guys continue talking about it. Most of those who use it in actual trading fail. This is the truth.

There is an interesting ongoing discussion in this forum:

http://groups.google.com/group/adapt...f68c5ea2?hl=en


Some people who actually work with systems claim OOS testing is useless.
Ok, there are some valid arguments but also some misconceptions.

The effects that amount to a difference between testing and trading are well known and based on relatively simple math. WFO is one method to deal with those effects. There are several other methods, but they are more complicated or less effective.

Some years ago, algo developers believed that strategy parameters should be optimized for a long time span and cover as many different market conditions as possible. Then the strategy was considered stable. As it turned out, it was not. Strategies optimized this way tended to become unprofitable in real trading after a remarkably short time.

No matter how many market conditions you want to cover with your parameters, you'll always encounter a completely new market situation in real trading after a while.

You can do a little experiment: Take one of your profitable strategies, and optimize and test it in the traditional way with a simulation period of 4 years, 6 years, and 8 years. You will almost always find that the 4 years test is the best, 6 years is worse, and 8 years is the worst - no matter in which year you start the simulation. This should answer the question about the usefulness of WFO.
    Quote
 
Reply
Thread Tools

Forum Jump



   Conduct Rules   Privacy Policy   Sitemap Copyright © 2014, Elite Trader. All rights reserved.   

WHILE YOU'RE HERE, TAKE A MINUTE TO VISIT SOME OF OUR SPONSORS:
Advantage Futures
Futures Trading & Clearing
AMP Global Clearing
Futures and FX Trading
Collective2
Automated Trading Services
CTS
Futures Trading Software
dom993trading.com
NinjaTrader Consulting
eSignal
Trading Software Provider
FXCM
Forex Trading Services
Global Futures
Futures, Options & FX Trading
Interactive Brokers
Pro Gateway to World Markets
JC Trading Group
Direct Access Trading
MB Trading
Direct Access Trading
NinjaTrader
Trading Software Provider
optionshouse
Option Trading & Education
Rithmic
Futures Trade Execution Platform
SpeedTrader
Direct Access Trading
SpreadProfessor
Spread Trading Instruction
thenut-trader.com
FX, Gold, & Stock Signals
TradersStudio
System Building & Backtesting
Tradier
Equity and Options Trading
Trading Technologies
Trading Software Provider