Home > Technical Topics > Automated Trading > Curve fitting

Curve fitting

  1. Everyone says that you shouldn't curve fit a system perfectly because it will fail. So why not perfectly curve fit a system and then just do the opposite of that system?
     
  2. Because failure does not mean a guaranteed money-losing. Instead it’s going to be neither here nor there and going to slowly bleed transaction costs.
     
  3. Development of a strategy's rules is good, writing unrealistic rules that could only have been justified by hind-sight is delusional.

    For example, you take stock long trades triggered by some buy signal. Development would be a filter to disregard buy signals received while the stock's index is bearish. Curve-fitting would be a filter to use smaller position sizes on losing trades - these could only be identified after the event.
     
  4. It might not fail completely. Just give you huge drawdowns.

    When testing a curve fit system it might give worst case 25% drawdowns with average 100% yearly returns. Sounds excellent.

    But in real world trading, because the system was curve fit, it might actually give 75% drawdowns and only 25% return.

    If you attempt to trade a system with 75% drawdowns you will eventually fail in your execution due to psychological issues. Even if you automate such a strategy you will eventually turn it off and fail.

    If you attempt to fade the same system you will also eventually fail as the system is net profitable. If the system went into drawdown straight away and you faded the drawdown and made 75%, that will all be lost + more when the system recovers.
     
  5. Indeed. The trading problem is a problem of consistency. The more you study, the more you realize the lack of it in the markets. Longer-term frequencies may help, but still the future is fundamentally unknowable for most. If the system is consistently losing, you can just reverse it to win. However finding a consistent loser is just as hard as finding consistency in winning.

    I've started thinking any system I make will be wrong for some period, and need to mitigate that. This is due to sheer improbability of predicting 100% correctly, or even anticipating or following correctly, without some larger overall fail periods.
     
  6. That's not the definition of curve-fitting. You're talking about peeking (as in, peek into the future).
    Curve-fitting is over-optimizing parameters for maximal risk-reward, no peeking involved.
     
  7. Curve-fitting with relevant inputs is good.
    Curve-fitting with irrelevant inputs is bad.
    Over-fitting with either is bad.

    The desired skill lies in (1) identifying the relevant inputs and (2) not over-fitting them.
     
  8. Everyone is wrong : system fail not because of perfect fit, but because the fit itself is no substitution for finding fundamental rules of the market!

    And you are wrong too because opposit of wrong is wrong in trading at least
     
  9. What about applying technical analysis to the equity curve of the system?
     
  10. This is the actual reason:

    Curve fitting means you looked at the past and built a model on the noise during that period.

    It’s the same as throwing tea leaves to decide where the prices will go. Whether you follow the leaves, or do the opposite, you are still just making a decision from noise.

    you could ask an octopus what to do. It’s the same thing.

    Hope that illustrates why curve fitting or the opposite, has no predictive power at all!
     
  11. When people say some trading method or strategy will fail, what does it actually mean? If it means it will definitely fail, the difficulty level to know it for sure is the same as saying a method will succeed. Anything in between is just a random bet.

    Trying a ML (machine learning) method, curve fit or over-fit are common mistakes because it will mislead to a totally failed system, that is, a wrong design. Here the word "failed" is a totally diff thing in ML from what a trader thinks.
     
  12. saying it will fail does not depend on the outcome. you need to think in probabilities.

    i dont think the word is used differently unless they are an extremely unsophisticated trader.
     
  13. A trading system equity curve is not a market price chart, TA wont work on it.
     
  14. Everyone says what they hear others say. Finding a perfectly curve-fit system is as hard as finding a perfectly not fitted system. The author of Fooled By Technical Analysis book offers a few lines of proof of this.
     
  15. This is in part the reason that Prudent Risk Management is really the only thing that matters in trading. Everything else is just noise
     
  16. Hello boza,

    In my opinion, there is nothing wrong with curve fitting a system perfectly using in sample data. Just make sure once you have it curve fitted how you want, you test the system again using out of sample data. Split the sample date 50%. 50% data for in sample and another 50% data for out sample. This is simple to do.
     
  17. That’s not curve fitting in the sense used here. He means overfitting.

    So you’re actually saying ‘there is nothing wrong with overfitting’, which is absurd
     
  18. Curve fitting is great for discretionary trading.
     
  19. Actually, this is definitely not true. You can keep increasing the ratio of free parameters to the number of trades in your backtest. Eventually, by doing so you can perfectly fit any dataset.

    Alternatively, you can fit on the full system and then use random sampling with a regression to understand the value of the parameters your introduced. It's probably a better way of doing things, since you are not really leaving untested time periods.

    In any case, the practice is somewhat dangerous. Curve fitting assumes that you are introducing extra variables in the system with the aim of improving performance such as reducing the number of drawdowns. If you are dealing with a highly skewed distribution, you are going to have a fairly small number of drawdowns and your strategy will be naturally overfit even though it would appear that you have good out-of-sample results.
     
  20. Hello sle,

    I am not understanding your comments in detail, its a very complex writings.

    Let me ask. Is the below picture I drew the goal of all system developers? I just drew this cause I just start this journey. If I test a strategy that come close to that picture, I am trading it. Please correct me if I am being to simple? Optimize in sample data, Walk Forward Out Sample, Forward test, Go Live, Make money. Is that not simple enough? I could be wrong, I don't know. Still learning as I go.
     
  21. I may have failed you in this regard B1S2. Look for my journal update this weekend. *hides*
     
  22. I have to note that your English was perfect in the previous post, so I have to take this as an act :)

    The goal for all system developers is a picture below, anything else is just a way to get there. Now, for the actual question of optimization and fitting. An ideal strategy is the one that's built on a very simple hypothesis and thus does not require any additional variables to improve the quality. However, as we all know, it's usually very hard to find something like that. So you end up taking an imperfect hypothesis (e.g. "stock that gone up 5 days in a row is likely to go down") and add knobs that you can twist and change the outcome.

    Some reasons for the addition of those knobs are totally OK. For example, if you have a strategy where you see good potential but the results are barely beating the transaction costs, you might want to add some parameters to reduce the trading frequency. Or maybe you have a strategy that trades across assets and you find that the relationship has some seasonality (say you are trading the spark spread), you might add a flag to avoid trading during specific times.

    There are situations where those knobs are misleading and dangerous. Let's say you testing a strategy that "buys the dip" in equities. While the strategy might have done very well in the last 5 years, you see a large drawdown in August 2011. So you add a variable that says "don't trade in August and September". The drawdown disappears, your out-of-sample tests look good. But what you have done is introduced a rare event bias that will eventually cause you some pain.

    My point (which I am finally getting to) is that in-sample and out-of-sample testing is just a model for the impact of free variables based on certain assumptions. You can come up with alternative ways of verifying the statistical impact of your free variable. For example, some HFT guys back-test modifications on the full dataset, but they would regress random subsamples of trades with respect to time and expect a near-unity slope (meaning variance/covariance relationship is perfect).

    [​IMG]
     
  23. Hello sle, I apologize, I was not acting. Keep in mind this whole trading strategy development theoretically discussion is kind of new to me. I am learning as I work. I should have stated it better such as: your writings is above my current knowledge of system development.

    I honestly thought optimization is/was a good thing. I think it is, but what I am realizing or learning is that over optimization or overfitting is a bad thing because each parameter of the strategy is being fitted to produce max profits only for the sample period it was over optimized for. So when those over optimized parameters meet out of sample data or real time, the likely hood is the system will not produce equal profits as in sample data be cause optimized parameters was fined tuned only for in sample data.

    Does this make sense sle?
     
  24. Thank sle for the clear explanation.

    The above quote is the stage I am.

    So here is my work flow process:

    1. Watch the charts everyday
    2. Find an trading idea that looks repeatable and simple and could possible make money.
    3. Write the strategy on paper
    4. Program the strategy
    5. Click Back Test (In sample data only)
    6. Evaluate results (normally all Red losses)
    7. Optimize indicators and exit methods until see profit.
    8. Walk forward the rest of the out of sample and hope I see continuation of profits.

    Is there something wrong with this process, I haven't made to 8 yet.

    It just seems to me Optimization is inevitable, just don't over do it. Keep the parameters loose. Everyone can fit loose pants.

    There is other performance and strenth of signal I have written down to check off for robust system. Also NinjaTrader has Monte Carlo Simulation I can use.

    Thanks
     
  25. hehehe....
     
  26. Anyone notice how they spelled "Franklin" wrong on the bottom of that C-note?
     
  27. No haha, made you look!
     
  28. by yourself a fooking loupe!
     
  29. quadruple heheheheheheheheheheheh.....
     
  30. Is this thread a real NY fooking joke!
     
  31. Let's understand "curve fitting" and "overfitting". Say you write an algo that automatically writes Shakespeare's "Romeo and Juliet". This was your goal for the algo, and it does so with minimal input and parameters, thus can also be thought of as compression algorithm of sorts: Ie. "the knowledge" of how to recreate the perfect response is embedded in your algo. But is this "enough" for all possible use cases?

    Later, when you continue to run the algo, would you expect it to successfully write the sequel: "Romeo and Juliet II"? If not, maybe the process of your algo is different from the process of Shapespeare (the market)?

    So "curve fitting" or "overfitting", fits your algo's output to your goals, but doesn't really achieve longer term goals like future profitability, because they might fail miserably going forward out of sample. Indeed, there may be many reasons it may fail, both due to the method (algo) but also because of external factors (Shakespeare's death / fundamental market changes). Some things are just impossible to perfect, but maybe "good enough" will do?

    Say you failed out of sample, and spend 100 attempts at making your algo. Your "out of sample" is now "in sample" due to your very process of creating one algo iteratively. So the algo can still fail going forward into the future, even though performing wonderfully both in sample and out of sample!

    Because the nature of backtesting and optimizing is to "curve fit", performance is likely to be less going forward. There's nothing wrong with "curve fitting" per se, but when results fail, if it's because of "overfitting", then it was bad. Though, you might also see short-term performance exceed backtesting as well (unf.likely), the random nature of the markets makes it hard to see what works and why.
     
  32. Basically, the problem is one of System Identification for which there is a vast literature in control systems and econometrics. If you really want to have an answer to your question then you need to be asking people who have understanding of system modelling, developing, testing, performance analysis. You are unlikely to get that on these forums.
     
  33. Thank you Simple for your explanation.

    This is a good explanation.