Exactly. Machine Learning is a great toolset to do stuff faster and discover faster algo when it comes to computers. Like if the trader knows what to look for ( the patterns ) then machine learning can help find it in a less amount of time than it would take for a normal scanning methods. Excellent post btw.
Just want to give my 2 cents on neural networks in trading. I use one neural network (along with 6+ other mechanical systems) to day trade etfs. Here is a screen shot of one of my best models. It has just three inputs with a one day time horizon (I prop trade with no overnight positions). My advice to new traders is skip NN's for now, and develop mechanical systems first. Much less chance of curve-fitting.
Actually that does not necessarily hold true anymore, take for example the very basic classification algorithm to detect handwritten digits. In this convolutional network one essentially knows nothing about any relationships, one has no idea about how each pixel in the image of each of the training data sets relates to the other. The network successfully builds those associations all on its own. So, the analyst does not have to know at all what to look for other than specify the goal and appropriately structure the network and think about how to label the training data.
Interesting discussion. But I would caution making the analogy to handwritten digits. One problem with the comparison is that the digits have an underlying and differentiating structure that is stable across many samples. Most of the textbook examples of success have underlying data that is fairly stable across samples and not very noisy. Real market data is very unstable and very noisy. I don't think deep networks will solve this problem either. One challenge is to either modify the underlying raw data or alter the existing architecture(s) to deal with the inherent noise and instability of market data.
You have to watch out with the term noise. Sometimes what appears to be noise is really a signal. I do not advocate the use of moving averages as the input for neural nets - you can be removing useful information.
Excellent point. Most of your standard technical indicators have very close to zero predictive capacity when used in predictive analytics (NN, SVM, etc.) Why? Because as you imply they were designed to hide and summarize information. You can easily validate this. Create two predictive models in your favorite analytics application. Training the first model using RSI and a couple of other similar famous TAs. For the second model use the input and intermediate work variables that were used to create the RSI and other TAs, but do not provide that model with the final value. Note which model has higher predictive capacity on out of sample data. Pretty amazing isn't it?
You can easily solve this issue by accurately defining your goal and labeling trading data sets accordingly. You could for example train an algorithm to classify your time series under investigation and predict whether such will exhibit trending or mean reverting properties over a future time span. As long as you define a discrete outcome even a simple classification algo can get you quite far.
Sometimes you are forced to sample data or you may even prefer to do so. Due to computational limitations you may not be able to map a tick based time series onto the input layer's perceptrons. Most researchers (and you can verify this in most papers on time series usage as part of deep networks) sample time series data by, for example, feeding in compressed time series (1 minute or hourly bars,...). The results are actually much better than raw tick series because the meaningful information content in this particular example does not lie in tick dynamics but in the time series properties within the compressed series. In that an indicator is really not that different from compressed time series. My point is that it really depends exactly where you think the information content lies that you look an algo to build meaningful relationships from.
Not an easy solution at all as you are over-simplifying the problem. Even if you discretize the data for classification, the problem of stability will still persist. And it's not trivial to properly define and discretize the data sets to give good - stable - results. To put it in some perspective, a well known hedge fund (heavy on AI and PHD level talent) sponsored a contest for some of the best practitioners of Machine Learning to predict financial time series data using any ML type algorithm (NN aren't used that much nowadays, either). Very few participants could do much better than simply predicting the mean return over the out of sample period.