better way to do Out-of-sample test?

since your glasses seem to have been misplaced: "Most of my strategies that perform well in-sample will be falling like a rock out-sample." ---> EVERYONE...

Most of EVERYONE'S strategies will fall like a rock. Because most theories do not work once you use real data - out of sample.

Your statement

---? I have a system I developed in 1993 and it still makes money. --- ignoring the fact you provided no proof, provided no metrics, and badly misuse "makes money" as even just holding a stock index since 1993 made money, and do not even say whether you traded it the entire time since 1993, or with how much money (in other words, you are naively all wet)

Likely proves what I said. For one, you did not say ANYTHING about your other systems, so it is likely MOST of YOUR systems FAILED. and you only discuss "A" system, which basically reinforces what I said - MOST systems fail for EVERYONE.

Keep up. Your underwear is showing...

 

Don't really know what you are trading ... but how about including another period of extreme volatility in your in sample data ? Say 1999 to 2001 ?

 

Are you a psycho or what? Why are you so offending and insulting with other people? Do operators of this website understand that it is psychos like you that chase people away with their insulting attitude? You are probably an idiot who has never traded a single $ or who blew up in a matter of 10 days and then takes his anger here on other posters who try to learn and help other people.

What I can do is say screw you to you and never come back to this forum again. I will not take insults from idiots like you. You should really consider visiting a competent psychiatrist to receive treatment.

 

I think Nanex / NxCore will sell you exactly this for maybe around a few $K. Not in text format, but accessed through their API, the same one they use for their real time data service, which I think is quite good.

 

I meant the two extra parameters (in-sample and out-sample sizes) used in WFO lead to a lot of variation in test-performance. You will see that different sizes lead to very different performance.

Therefore, you have to have one more layer overlay optimizing these two parameters, which leads to another layer of curve-fitting...

 

If you trade large lots you should look at back testing with CME's datamine historic BBO and Level II DOM data.

 

Very good point! I am testing FX, for some reason, my data only go back to 2001. hmm, I should probably look for more data to include 1999-2001 in the training sample...

 

In this thread and other threads I am beginning to hear rumblings from traders who are building dynamic hybrid strategies that they require different testing approaches. For example Pocket Change states:

“Unless your strategy is dynamic adjusting to market changes of course you are going to have different results.”

Or Vikana states

“I eventually settled on a different way to optimize. In my opinion the point is not to "optimize" but to "stabilize" and I therefore approach the problem as follows….”

I have spent most of my professional life programming and testing IT software. Since I retired to trade full time in 2002 (after 37 years of IT) I have developed and tested hundreds of different strategies and found some of the old maxims of walk forward testing useless for my type of dynamic hybrid swing trading strategies and the time intervals I trade in.

I now believe we are entering a different era in retail software programming and testing. Many of the simple trading strategies that were first coded in the 1990s to trade simple set ups have evolved and given way to dynamic hybrid strategies that are well aware of price, volume and volatility metadata and can adapt to changing market conditions.

Many retail traders like me are injecting “qunat” or artificial intelligence into strategies to make them adaptive. When I did this I had to question whether traditional walk forward testing was still relevant given the nature of the programs being produced. Much of testing today is more to finding a stabilized strategy to trade with (rather than an optimized strategy) as Vikana stated.

But I still notice most of these discussions on testing in ET still revolve around 1990’s straight set up strategies that are static and require a multitude forward testing periods and validation before they can be put into a production environment making profits or they will be just another “curve fit.”

I believe we need new testing methods to accommodate this changing environment. But to some in the earth is flat group what I have written is hypocrisy. For them testing is fixed and rigid. They may be right. For trading the ES the old ways may be the only way. But I live off other testing methods for trading stocks and these new methods work for me and that is all I care about.

So I’m curious how many of you have gone beyond yesterday’s simple walk forward testing procedures?

 

"Metadata"?

 

The term metadata comes from IT meaning data about data. In trading I use it to refer to the data conditions that are taking place during buying and selling. For example programming into a strategy what the correct volatility conditions, price direction, high or low volume, etc.. that must be in place for a trade to execute.