I have struggled for years with developing profitable strategies. Many times I have found what I thought was the "holy grail" only to crash back down after further testing/live trading. My recent threads being no different. However, with a new year approaching, I am back at it again. This thread is meant to be a continuation of past threads. A chance to start anew with a new methodology. I will be building a new strategy based entirely on patterns discovered through a custom neural network. You can read more about how I got to this point and what this method of pattern searching is about on my blog: <a href="http://www.thestrategictrader.com">Strategy Design</a> This thread will be used as an outlet to get feedback on my methods and hopefully improve them. Building strategies based on patterns in this way has endless applications. Small tweaks to the inputs produces 100's of new patterns. Hopefully as a community we can determine good inputs to produce profitable patterns. On my blog I will distribute the actual pattern files as well as the engine that runs the patterns. Before I can distribute the files, I have to make a few more modifications to protect some of the IP that I do not wish to release into the wild. However, enough will eventually be released so that the community can follow along with what I am doing.
The first attempt at using this method can be viewed in this post: <a href="http://thestrategictrader.com/technical-analysis/trading-strategy/rsi-pattern-strategy/722">RSI Pattern</a>. There were over 220 statistically significant patterns generated. The patterns were generated using the following inputs: AddParam(RSI(3,1)[0],"RSI3 "); AddParam(RSI(7,1)[0],"RSI7 "); AddParam(RSI(14,1)[0],"RSI14 "); AddParam(RSI(21,1)[0],"RSI21 "); AddParam(RSI(30,1)[0],"RSI30 "); AddParam(RSI(45,1)[0],"RSI45 "); AddParam(RSI(60,1)[0],"RSI60 "); AddParam(ADX(3)[0],"ADX3 "); AddParam(ADX(21)[0],"ADX21 "); AddParam(ADX(45)[0],"ADX45 "); AddParam(RSI(ATR(5),3,1)[0],"RSIATR5 "); AddParam(RSI(ATR(14),3,1)[0],"RSIATR14 "); AddParam(RSI(Volume,3,1)[0],"RSIVOLUME3 "); As you can see the focus clearly was on RSI based inputs. With a few inputs for measuring trend strength (ADX) and volatility (ATR) added into the mix. The best pattern produced had an expectancy over $41 per ES contract when simply exiting after 5 bars which is enough to remain profitable even after liberal slippage and commissions are applied. Imagine what this pattern could do if the exit criteria made more sense? Any suggestions on inputs to try next?
Full marks to you, frostengine for openess and willingness to go about this in such a collaborative manner. May luck and good fortune be on your side ... I am not familiar with pattern searching as a means of looking for an edge, so forgive my beginner's question... ... however, it seems to me that if one's starting point is that you are going to have a computer try one pattern permutation after another (or one set of strategy parameters after another), then you are very vulnerable to the result being simply a fruitless exercise (from the point of view of future trading profitability) in curve fitting. No? Unless you understand why a certain pattern might make sense (and unless you understand this IN ADVANCE ideally), it is very dangerous to focus on the "n" pattens that work after discarding "N" (where N >> n) patterns that didn't. "Survivorship bias" is the enemy here, surely?
I don't want to be critical, but guess I'm going to be. I would be asking myself 'do the things that I'm feeding into this NN make any sense'? For example why the RSI of anything? Why the RSI of an ATR? Why one length of indicator over another? Otherwise you're simply curve fitting nonsense. ML should be used to explore a specific idea, it does not generate ideas out of random inputs, not ideas that mean anything anyway. So my suggestion would be that you need to formulate a trading idea first, an idea that has some grounding in market domain knowledge and observation, then try to explore that idea by formulating inputs for your ML which would quantify the idea. Not just a random array of indicators.
Abattia: Curve fitting is a real concern with this type of strategy building. However, I am highly confident in the safe guards I have in place to guard against this. Keeping in mind that curve fitting is not necessarily bad. A pattern is nothing more than identifying past actions that have the ability to tilt future price action into your favor. By identifying what results are possible due to random chance (ie some patterns will be profitable purely due to the law of averages), and by having enough data points you can avoid the dangerous forms of curve fitting. Other safe guards include not trading on the same time frame as the data you train against. I am trading on 5 minute bars, but the patterns are being trained against 3 minute bars with over 200,000 data points. The patterns the system has been able to generate has remained profitable not only on the ES but on nearly every instrument I test against. This level of robustness points to the system learning an underlining pattern in the way the market moves instead of what is generally referred to as curve fitting (in the negative sense).
You state in your blog: "Keep in mind, because this is only one pattern, the number of trades is a bit low. Because of this we also validate the pattern against the Dow Mini (YM). This allows us to feel more confident that the pattern is not simply curve fit to the ES data." The two contracts are highly correlated and thus confidence is increased also in the case that the results are due to selection bias. The low number of trades for a period of 10 years in 5-minute data may be an indication of curve-fitting. You should have in excess of 1,000 trades in the in-sample and in excess of 300 trades in the out-of-sample. I have been using similar software for a few years. Keep in mind that there is nothing new with what you are doing and there are programs available that do this for quite some time. Maybe you are re-investing the wheel. While this is a good exercise the opportunity cost is high. Finally, the systems you described are subject to what Michael Harris refers to as Type I curve fit. Read here for more details. These type of systems have very low of success rate in real trading.
ronblack, The experiences I have had in the past is that when you have a curve fitted strategy, simple changes such as: 1. Changing length of bars (from say 5 min to 4 min) 2. Changing instrument, even if highly correlated Would produce horrible results. This is because the good results you see is due to the system getting lucky and picking the best trades for your data set. Then when you make a sight adjustment it all falls apart. Concerning a Type 1 curve fit, looking at that article a Type 1 refers to optimizing on both entries and exits. This strategy only the entries were optimized. Exits were left static as a very simplistic "exit at 5 bars". As far as re-inventing the wheel. I have tried software applications such as these in the past and they all fail for one reason or another. In a lot of instances produce overly curve fit strategies like you are referring to. This is why I built my own, I am dong something a bit different in how the network learns and interprets the data which fixes these problems for me. I do not believe any system out there does what mine does in how it learns.
If the entry is changed then 5 bars after that results in a different exit price and that makes it Type I still. I also think it is a good exercise but there is opportunity cost. It is a lot of work. If you have the time then it is good for you.