Technical Analysis vs Quantitative Analysis vs Machine Learning

Have you considered streamlining your design? ML is not going to magically discover money making algorithm when you throw at it all possible numbers generated from time series together with fundamentals, news and everything else in between. My suggestion would be to focus and narrow your project after getting more working experience in financial markets. Some of the red flags in your design is blind reliance on publicly available FA (Enron comes to mind) as input where it is known that publicly available data have very little predictive value. Similarly you cannot use news for their face value since market reacts erratically to seemingly similar events generating the news. As to indicators you can experiment with them but you have to realize that many of them are variation of the same underlying analysis method. This will not be able to help in better separation of subpopulations in data. Interestingly you plan to mix all good and bad inputs together hoping that it will somehow work without having solid hypothesis and expectations. The best thing that might work for you is the enthusiasm and desire for success.

 

Thanks for your response. I am always open minded to idea/thoughts.

In the next few days left of my Xmas leave, I am focusing on streamlining my design, making something that people call Minimum Viable Product (MVP). The specification of this MVP as follows:

1. Able to extract ES future historical data and market data from IB via Java API, then store the data into PostgreSQL (I don't like IB data quality, will get a new data feed later, still researching)
2. Implemented one popular TA indicator, which will be calculated as frequent as possible based on real time data feed
3. Be able to place, update, cancel order, the order will be strictly limited to 1 contract, either long or short. So my position would be either 1 short, 1 long or 0.
4. Optimising the parameters of the TA indicator above.

After this MVP is done, then refine/tweak the TA indicator, or add more TA indicators, and FA indicators, etc....wrap into ML, add QA for risk management

How the market reacts will have to be tested by applying best practice ML experimentation (with regularization, cross validation etc).

In post#14 this thread, I mentioned 50% of the effort is understanding the financial markets domain, 25% understanding data and indicators, 15% engineering & coding and only 10% is machine learning (probably Quantitative Analysis is also included in this 10%).

In ML, for example, the model can predict a binary outcome (price increase more than 10% as 1, not(price increase more than 10%) as 0), then machine learning select which indicators or variables or features to use in the model to minimise the loss_function or to maximise the utility function. Bad indicators may not be selected at all.

In summary, the complete plan would be
QAforRiskManagement(
MLforExperimentationPipeline(TAforIndicators,(FA+NLP)forMoreIndicators)
) = Consistently Profitable Algorithmic Trading System.

 

If you want to lose money fast try ML. I am telling you that after 20 years in HF business.

 

Would you please elaborate further why using Machine Learning make me losing money fast?

 

If you do it wrong, you lose money.