i think you mis understood me. My point was that it is my belief that your system should under go the following and you should post the performance from the following before any judgement should be made wether it is tradeable or not, otherwise there is no point. 1. How ever many parameters your system is using for the performance report, then do not change any for the test. Keep same rules and parameters set to the same values through-out the complete test period, otherwise we are fitting to suit so to speak. 2. Test period to be 10 years of data, with each year upon year showing net profit all from that same set of rules. 3. Comissions are in. 4. All fills from backtest exceed price by a tick before they are counted. Tradestation has a feature that allows limit orders to exceed price, and for entry i have coded my own version so that all my backtested entry fills are exceeded past my entry before it is counted on performance report. This way i know i would have been filled, and also filled at realistic price. And as i only trade spy and google i know if limit order exceeds then there is always enough volume to know i would have been filled trading those symbols. If you do the above and post performance report that would be interesting to see. Many thanks
I understand what you mean, but can not fulfill your request. I have no system that would run with the same rules for 10 years, and I don't think that such a system exists. Or if it exists, I haven't found it yet. We're developing parameter-free systems. Those systems permanently adapt their rules to the market, so the parameter optimization process is a part of the strategy. If I would use a fixed set of rules instead, the system would probably still be profitable within the past 10 years, but that would be just luck, and I would not trade it. As to commission and slippage, a test must simulate realistic trading conditions. A fixed 1 tick fill delay is normally not realistic. For commisson and slippage you must measure and use the parameters of your broker. We're using the current FXCM parameters for our backtests.
If I understand your systems correctly, I wouldn't call them parameter free. They are more like locally fit models I think. The potential pitfall of locally fit models is that you have bad estimation of the parameters and there is no statistical significance of those estimates. The problem with longer timescale fits or global fits is non-stationarity and lower frequency supercycles. There are no easy answers as always.
Hi, who is "we" by the way? If the optimization process is part of the strategy then your system involves hundreds or even thousands of parameters. There is a "simple" test to check if you are effectively optimizing according to the TYPE I - III Curve-Fitting theory. 1. Start the first step of the walk-forward optimization (WFO) using the last 2/3 of your data sample. For example, if data starts at 2001 and ends at 2009, start the WFO at 2004. 2. After the first step, go back and add the misisng data to the start of the sample and run the WFO again. 3. Do you get the same entry points as in step 1? If not, you optimized in TYPE-I mode (worse type).
thanks for your reply. So are you saying that if i had a backtested system that was profitable every year upon year going back 10 years without changing the settings, the results would be coincedent?.......I would argue and say it was a clear postive expectancy system and a clear edge if i could produce a system that was every year profitable going back 10 years, surely that is the goal of us all?. Lescor, a profitable trader has used the same systems for years now and he makes decent returns every year so far since 2001, and again his returns are only 1-2 cents per share on average, which shows edge does not need to be large.
I may be mistaken, but Lescor has mentioned that he uses multiple strategies on multiple time scales, and he uses his discretion to decide which ones to use. He also notes sometimes that a certain type of strategy stopped working for a while so he stopped using that one etc. This is indicative of the non-stationarity of the markets. I believe Lescor comes up with strategies thinking about the mechanism (buyers sellers emotions reactions, etc) which means they are likely significant patterns even if one could not prove them to be so. I think it's the non-stationarity that will doom any single strategy but a collection of them might work for a prolonged period of time.
Allow me to show you some perspective! See attachment. Same pair as you EURUSD. Commissions, Slippage included. Entry based on 2 parameters, same pips stop loss, same pips profit target for all these years (independent of market volatility!). Elementary level maths involved in strategy (class 8 student can do it, no engg maths needed). I developed this system on Oct 2002 to March 2010 data. Developed without any walk forward optimization, because I was 'sure' that I am capturing a real market effect, and didn't want to unnecessarily optimize the thing. Developed using an off the shelf trading strategy software, no independent software development. At leverage 15, annualized performance 70% p.a. without compounding on backtest period. Max DD 60% in 2008 (Bear Stern day), 2nd worst DD 40% (2005). DD seen in most other years 15%.
Same system as above. Completely out of the sample performance - Real forward performance on never seen before data - Sep 2010 to Jan 2012. Performance beat the historical averages by a mile - because I hadn't over-optimized the historical dataset while developing the strategy. For FORWARD trading period, At leverage 15, annualized performance 130% p.a. without compounding. Max DD 30% in 2011. EDIT: I removed attachment from this post and the one above it. Sorry, but I am too paranoid. I think there was way too much information in those attachments that I do not wish to lay out so openly on the internet. Btw, I completely agree with the idea that a combination of multiple strategies/multiple instruments will give more confidence compared to a single 'simple' strategy even if it has worked for 10 years in past and continues to work.
thanks for posting 10 year. well, for me it has always been my personal goal to find systems that are consistent over the long haul. I can easily find settings that work for a year or two but i always throw them away. My goal is to find a postive expactancy over a long enough sample. This way it shows system is real and not coincedent. And in terms of multiple non corelated systems, i too believe in it completely as it blends and smooths drawdowns. Thats why i have 6 systems to date that all individually year upon year profitable going back as far as i can. This is my personal critria for choosing systems. It gives me belief in my systems and to be this is the whole idea and purpose of backtesting. Anyone can pull up a system that is profitable for a year or so, that is very easy to do in tradestation. As for lescor pulling a system. Yes he states that but he always states that he has used 2 of his systems since he started, his bread and butter ones. Also his opening strategy is not as favourable as it once was but he still trades it year in year out so it must still have an edge. He states he will always track a strategy even after ditching it. This all shows me that some edges do last longer than we think, as long as our timeframes are long enough to profit.
"We" is a group from game programming that went into trade system research and wrote the Zorro program last year. I do not really agree with the 3 types classification in the link you posted: if a strategy overfits or not normally depends on the strategy design, and not on whether it optimizes the entry point or the exit point. In your example with the 3 steps, sure, WFO will generate the same entry points again. Every WFO cycle uses only price data from inside the cycle, so it should not matter if you start the WFO in the middle of the data or at the begin. But this does not say anything about a possible overfitting.