Yes, I mean there's a data snooping bias, but if it's clear as night and day and statistically significant... are we back in the realm of bias, or not? Is an out of sample test enough to avoid "cheating", so to speak?
One methodology that you could implement is based on simple Sharpe ratios. Suppose you decide to enter your mean reversion trade, because your ex-ante Sharpe is greater than some arbitrary target. If your realized Sharpe, once you're in the trade, is significantly lower than what you expected, you cut the trade. Repeat this at every sampling interval. Refine the two Sharpe thresholds and you can have something that's relatively systematic.
Kind of; there still is a data-snooping bias with out-of-sample observations. If you must though, try not to "optimize" any variables, or seek out a favorable sample. It should verify/disprove your original hypothesis.
sampling error and everything that follows. how do portfolio managers deal with that type of risk? there are a few ways
In my tests of mean reverting strategies, I've found that price based stop losses almost always hurt results and time based stop losses sometimes help, but in the long run not by much. There are other ways to manage risk besides stop losses. Namely small position size and diversification of positions and strategies.
This is my current approach, but I'm a small time trader attempting the retail automated route for the first time. So while diversification usually helps, the occasional sting is still felt. On Wednesday, I had a basket of a few vs many, in which the many stopped hedging the few because the one-in-a-few had a broker upgrade that blew it out against me a full 4%. Arguably, the basket construction was poor. But even in this scenario, the deviation from the mean was a slow and steady operation; although, on that particular day, there were not very many opportunities for diversification to offset the loss. Similar things happened today, although diversification took the sting out of it. But in each and every case where there was an error, the departure from norm was generally a steady and sustained event.
And how many times was there a generally steady and sustained event that reversed as intended, to recover at least some of the loss and in some cases turn a profit? The problem is that once a trade goes against you, what's the best move from that point? I find that many trades that go against me, sometimes significantly, turned around for a much smaller loss or even profits. You can use stop losses to reduce your risk on a given trade, but I haven't found a way to use them to improve the risk adjusted return of my overall strategy. I wish I could and believe me I've tried. The rare but significant moves against me are quite painful. I take solace in knowing that testing shows my strategy as is to be superior to the alternatives I could identify and knowing that most traders would not be willing to sit through such moves.
Don't frown, double down When you're in pain, say "hit me again" Stop out if you've puked on your keyboard
Many of them do come back around for smaller losses, but somehow I am not content with this. My experience shows there to be two cases: i) There's heavy volume and a sustained push against me. I can see the market accumulating and bouncing on the VWAP/TWAP, the correlation to the indexes drops dramatically, heavy moves on SPY/ES are nowhere near being mirrored, etc. ii) The volume isn't heavy, but someone is passively distributing/accumulating and impeding the movement. Both cases are dangerous. In case 1, you have no idea how explosive the move will be or how damaging it will be. In case 2, you have the problem that your hedge no longer works, and you're at the mercy of the market's volatility in your hedging instrument or basket. With case 2, I think this can be mathematically resolved in that the disparity in realized volatility is something that might be a signal/detectable. You can dispatch this awareness to another strategy engine and profit from it, maybe even through another asset class. With case 1, I don't think it's wise to let the explosive move run. The problem becomes qualifying what's explosive. I tend to think a manual override taking the loss makes more sense. I was thinking to augment the system with some sort of predictor on whether the explosiveness is really dangerous.