There is only one method that I trust, which is to painstakingly gather the tracking data yourself and evaluate the model errors periodically. Re-calibrate and make the proper model adjustments when necessary. Its an iterative process. Basically you become like a fireman, putting out the "fires" in your model as and when they occur, until they become less frequent and stabilize. If it doesn't stabilize over an extended period of time (say a few years), then its likely the model is flawed, and then you may need to start from another foundation.
If you segment sample say monthly or quarterly, even a handful of outliers can effect outcome. Adding current data regularly and segmenting, reduces lag of what is then dynamic results. I used set-up probabilities for about 2 years, long ago. I updated daily and maintained the all-time, yearly, quarterly, and monthly probabilistics.(I know, that's not a word. LOL). I used the monthly for intraday trading. It was a pretty easy to see when the edge was disappearing. In the end, I determined the internal and external stimuli of the market(s) was way too much for me to identify and observe to properly include in the data set, and I moved on. Today I do use Volume/Volatility probabilities, which updates daily and is based on only the front month futures contract intraday data. It's very low-lag dynamic. It is not a trade trigger. It is for trade maintenance only. I have no need to observe or identify specific internal or external stimuli as it all shows up in volume or lack of.
ATR! Handy little beast. While recognizing that markets go through unannounced regime changes ("mood swings") on regular bases, it's still instructive to monitor your losers and winners for things like time-in-trade and esp. points earned or lost. There will be a sweet spot in there, that will take more $$ from the losers than it will take from the winners -- and you'll end up "dollars ahead".
"Overfitting" is an abused term. It means to so specify a model that the model has no use outside of that data used to establish it. In practical terms, in real life, that's really hard to do. What is more, the solution (to market/data regime change) is a production/performance feedback loop which everyone should be doing anyway. (As you are doing right now, a priori.) See esp. tiddlywinks post (#12 in this thread.) Way solid. Oooo! And freestyle's post (#11.).....
Something exactly paralleling this happened when I reviewed my trend-following trades a few years back. I looked at the losers and identified the criteria which they lacked or which were weak in comparison with the winners and I filtered out all the apparently weak trade entries. To my frustration when I ran the revised strategy, I found I had filtered out so many small winners along with the losers that I now had a losing strategy. So I reverted back to a policy of taking every confirmed trend, whether it really has a score like a winner or not from recent price behaviour. Then I let price work it out - if its a loser the stop-loss gets me out, if its not making great progress the TA tells me to get out, and if its a winner I pyramid it. Much happier now.
here is my method for finding holy grails. take as much as data as you can and find the yearly, monthly, daily average high / low range as a benchmark. then omit all data that is above this average and patch the remaining data together. then with this data build a reversion to the mean system. set it aside for now. not take the remaining data and build a momentum system for that patched together data set. then using switches -, -0, =,+1, + etc. leave intact the reversion model as you define when to turn on and off the momentum system. basically you have a reversion to the mean primary system operating inside of a larger picture momentum system. why do i do this? because you can not create a one size fits all method that will endure. you either compromise the reversion or momentum but you will never compliment both together.
Probabilty involves the estimation and a lot of quantitative analysis, Quantitative analysis involves gathering the data from the winner, lossers and rivals from the market that will helps the company to understand the results. Your artical is also based on this type of infomation but are less professional.
There are many types of models that implement what you have described, typically mean-reversion jump-diffusion or threshold models, or two-state regime switching. In two state regime-switching, transition probabilities can smooth out your 0-1 switch. Similarly, in mean-reversion jump-diffusion, you can assign time-varying parameterization of the mean-reversion factor or jump-diffusion factor (or alternatively a "momentum" or trend factor) to become price dependent (on a threshold) to achieve what you have described. You are only limited by your mathematical mechanical imagination ...
i actually think they are quite scarce, at least i have not seen any other than the one i published. talking about them is one thing, having one that works is another.
Sir, I agree with you totally that having something that works is never an easy thing, and did not mean to imply that truly original work is prolific, such as your refereed publication. If everything was easy, why would one bother to snoop around forums keeping tabs on state-of-the art like a paranoid thief in the night?