Damn you, out of sample trades!

Discussion in 'Strategy Building' started by logic_man, Aug 22, 2012.

  1. Does the ratio of the maximum win to the maximum loss tell you the same thing or something similar? On the optimized version for the ES, the biggest loss is 8.75 points and the biggest win is 30 points. Unoptimized, it's 25.25 and 40.75, so the ratio tightens up a lot in the unoptimized version, even though that set of trades has the biggest overall win.

    Honestly, for me, having lived with the model for nearly 3 years now, I take it as proven, to the extent it is possible to prove something in trading, that there is a directional edge. My main issue is finding the marginal areas of improvement I can. That's where I run the big risk of curve-fitting. The actual unoptimized system is almost assuredly profitable, but I am always striving for more.
     
    #31     Aug 22, 2012
  2. dom993

    dom993

    No ... there are tons of trend following strategies, with win% way under 50%, but avg-win >> avg-loss. Most of them do not have a directional edge, meaning with a target = stop they will have a win% ~ 50%. Their edge is essentially in the trade management, and using a timeframe in which the market trends often & long enough.
     
    #32     Aug 23, 2012
  3. gmst

    gmst

    LOL, seems you got some raw deal from some IL guy recently :)

    Nevermind though, I agree with your basic sentiment. 5yrs working, developing and refining a model and actually trading it on a 20k account is more valuable to me than a 1 yr out of the sample trading using 1M, if the effort in developing a good model hasn't been there. It takes time to become a good system developer.
     
    #33     Aug 23, 2012
  4. Have you gone back to run a backtest over the period of these 18 out of sample trades? Will give you a good measure of how close to "reality" the previous backtests might have been.
     
    #34     Aug 25, 2012
  5. sle

    sle

    Out of curiosity, why would you deliberately design a strategy to have negatively skewed return distribution, even if the mean is positive?
     
    #35     Aug 26, 2012
  6. He wants a higher win%, and setting a stop prevents his account getting blown up due to a report coming out when the system is in a trade.

     
    #36     Aug 26, 2012
  7. themickey

    themickey

    Hi Logic man.
    You can talk to other traders until blue in the face, most won't get it as it is your custom system and works only in a manner you understand.

    There are however a couple of cardinal rules which apply to everyone.
    Trade very small (as small as possible) when starting.
    Let the system run for a while and be patient. Still keeping the small size.
    Incremently increase your size as your confidence and bank grows.

    Back testing is not everything it is cut out to be.
    Just like your system may contain errors, so can backtesting.

    If you are not extremely pleased with your system prior to initiating the first trades, don't trade.
    Traders who are experimenting on their system ie haven't fully worked it out prior and are trading are nuts.
    You must know you have a profitable system first, second, prove it by trading very small.
    Everything will follow on from that imo.
    Talking to traders who are clueless about your system is a waste of bandwidth. May as well talk to a circus owner. :)
     
    #37     Aug 26, 2012
  8. But the out of sample trades have been over the last couple of weeks and are trades I've taken, so there isn't any other data to backtest on during that period.
     
    #38     Aug 26, 2012
  9. It actually was not deliberate. This "system" is based on the entry and initial stop movement logic of a broader intraday system. I've separated the "entry and initial stop movement" component of that trade from the "intraday swing" component. I found that for many values of my model parameters, the "intraday swing" component of the trade was unprofitable, but those trades did actually traverse the distance from my initial entry price to the initial stop movement price. Thus, I reasoned that while those trades were not suitable as intraday swing, they were suitable as trades to "scalp" the distance between the entry price and the stop movement price.

    Let's say price action says enter the ES long at 1400 and the price at which I can move my initial stop is 1403. I am now looking at "how" price moves from 1400 to 1403 and determining if it is worth holding the trade or not. The odds of price making it from 1400 to 1403, under certain parameter values, are extremely high, but the odds of the trade then going on to be closed out above 1400 are low. So, I would find myself in a situation where I would enter at 1400 with an initial stop at, e.g. 1385, and then the ES would reach 1403 and I could raise my stop to, e.g. 1398, but then the ES would fall back to 1398 stopping me out with a 2 point loss overall. In this new scenario, I would enter at 1400, exit at 1403 and re-enter at 1403.25 if that additional parameter value were such that it favored the trade ultimately concluding at a price greater than 1400. This is based on the hypothesis that the way in which price moves from 1400 to 1403 can provide clues on whether price will continue in my direction beyond 1403 in such a way as to make the likelihood of price moving in such a way as to enable me to manage the trade to a close higher than 1400 higher.

    Basically, there is a "good" way for price to move from 1400 to 1403 and a "bad" way for price to move from 1400 to 1403. Overall, the likelihood of price moving from 1400 to 1403 is that 90%+ rate that I mentioned at the start of the thread. But, since the initial stop set at the entry at 1400 is often quite distant, when it does get hit, it leads to that skewed ratio. There is also a "good" way for price to reach 1400 in the first place, but even when price reaches 1400 in the "bad" way, it is still very likely to reach 1403. It just isn't likely to continue on in such a way as to end above 1400, at least not the way I manage a trade.

    One of the primary benefits of this is that trades I would have completely passed on due to the lower likelihood of them ending profitably under the "intraday swing" system can be entered and exited for a small profit under the "entry and initial stop movement" system because even those trades for which I have a good idea that they won't end up profitable as "swing trades" will usually at least hit that initial stop movement price. So, instead of completely passing on the trade and avoiding a loss, I take the trade but exit at the initial stop movement price. I still avoid the loss (most times) but I gain a little on the trade.

    Hope that makes sense. At least, it makes sense so that you understand what I'm trying to do, even if you disagree with me trying to do it.
     
    #39     Aug 26, 2012
  10. I did think that going right to the "curve-fitting" argument as to why the first few trades didn't meet my expectations was a little premature. If you look at the threads I've started in the past, they almost always involve asking how to avoid "curve-fitting", so it is really my primary concern at all times, especially since, with only a few parameter optimizations, I'm able to get such good historical results.

    One of the ways I think about curve-fitting in the bad sense is how stable the parameter values I optimize are. If I would have chosen the same parameter values as optimal 6 months ago that I choose now, that implies a level of stability I want to see. In the case of the ES, where I have the bulk of my data, I can definitely say that is true and that, if I had thought up this idea 6 months ago and chosen the optimal values for the parameters, they would be the same as today. In the Euro and Crude, where I have less data, I can't really say, but I do know my system very well and one of the things that has always been true of it is that it rarely signals a trade which does not at least reach the price at which I can move my initial stop. The reason that happens is because it is very sensitive to momentum and that momentum typically carries to the price I anticipate it reaching as a minimum. Like everything in trading, though, even it reaching that minimum price is a matter of probabilities. It just so happens that that probability is extremely high, but when it doesn't happen, the losses are much bigger than the wins. In the end, though, if it has a positive expectancy, that's all that really matters. If I had a system with the opposite characteristics of a low win rate, but winners much larger than losers, I would trade that, too.
     
    #40     Aug 26, 2012