One of the most honest but above all accurate posts in a long time on this site. Most would be well advised to invest their efforts in other areas for much higher returns financially and emotionally.
I will have to try this automated trading things for a year, either success or fail, it is ok, I still have my day job. If success, I will quit my day job, if fail I will stop trading.
They sure do. I used to work for a guy who averaged close to two handles on the ES for several years, he'd trade infrequently, but "in size". tradingcomputer : don't get me wrong but you are setting yourself up for failure. If you think you know more than a trader who used to be a musician because you've taken a machine learning class, that arrogance will cost you money. When you start trading you'll be against traders who know more about markets than you do, so you'd better step out of your comfort zone and learn the basics first.
Johanmul, Thank you for your advice. Here my honest self-assessment of my skill set: Machine Learning: intermediate to expert Algorithm: expert Programming: intermediate to expert Data Processing: expert Trading/markets: basic, I believe I need to learn a lot in trading/markets, there is no way I can apply my other skill set effectively without knowing the trading/markets knowledge in depth. Many years ago, someone who claim that he is a machine learning (ML) expert, told me that he does not to understand the domain, in this example fraud, to apply machine learning. He failed, as the features that he built --as inputs of ML algorithm-- are too generic, not using any domain knowledge. In depth understanding the domain --in this use case-- trading/markets is paramount to the overall success. Machine learning, algorithm, programming, data processing are just supporting tools. First, need to understand the trading/markets domain in depth, then formulate the problem to be solved (that is asking the right questions), finally applying the right tools correctly to solve the problem (the answers).
@tradingcomputer: I know I am little late to this thread. But I do want to share a few thoughts. (1) Read @Ghost_of_Blotto's post again. Re-read it; then, re-read it again. If you still want to move forward continue; (2) As a newbie, take @garachen's advice "stick with low frequency - maybe a dozen or fewer trades a day"If this hold time (or longer hold time) fits your personality, move forward otherwise ... (go back to (1)?) I have been running a fully automated system for the past 2 months. I had 'manually' traded the system for about a year before that. I used ML to "mine" my trade setups -- took me about 2 years, working full time, to formulate the research problem, implement, and validate the trade setups. My current automated trading setup (home grown, self coded) consists of: a broker interface for Market data (FIX coded in Haskell); a broker market order interface (FIX coded in Haskell); A messaging bus (RabbitMQ) Trading Strategies (Haskell); Redis to hold detailed monitoring data from trading strategies. Web server to serve the day's trade signals generated by the Strategies, and other data held in Redis (Node.js with node-redis) Monitoring (Monitis -- server monitoring, RabbitMQ monitoring, network monitoring, broker interface app monitoring, and trading strategies app monitoring. Monitis has an ios app that is very handy). My FCM provides me with the ability to look at my order book and make modifications if need be over a web interface (in case I get failure alerts from monitoring application and I know there are open orders working in market(s)). So, I didn't have to recreate it. A couple of thoughts on how I approached my research, which I think might be of help to you. I looked at the problem as pattern detection in the context of a complex adaptive system. By framing the system as a CAS, I had the theoretical justification to look for patterns in the first place, and the knowledge that the goal is not to find similar instances of patterns but to extract a pattern class from similar pattern instances. It turned out, it was not easy to even 'encode' patterns. I finally settled on techniques from Bioinformatics to encode patterns, use probabilistic clustering algorithms to group 'similar patterns' (yes, not a supervised learning approach!; did consider MDPs/reinforcement learning but couldn't succeed in formulating a research statement). Going from here to finding a trading strategy was a process of iteration -- going back and forth between raw data and the similarity sequences within a cluster. Then comes the easy part of testing the strategies in an out-of-sample data set to determine statistical validity. However, I want to emphasis that the statistical testing done is almost useless given the assumption of CAS. In a CAS, the system evolves. This means, the distribution of a class of pattern 'holding' will not be Gaussian -- it could be clustered, have power law distribution, or in the worst case, be total unstable. So, however good the out-of-sample results were, I had to take it with a grain of salt. All the best with your research -- don't underestimate the effort required. Regards, Monoid.
You've got the perfect set up for the home trader, inclusive of strategy and approach. You really should consider selling your platform. There are tons of algo traders using IB who would pay for a package like that.
It sounds superficially appealing, and I've thought about it. But there is more work involved than you might think. The code would need to be rewritten, documented and fully tested to be a commercial product. There would be ongoing support and upgrades. All in all it's something close to a full time job. If I wanted a full time job I would return to my former career (systematic hedge fund manager) - I'd make a lot more money than I would selling software. So it doesn't make any sense for me at least. GAT
Thank you for this post tradingcomputer and all commenters. I learned and realized a lot from reading them. The only contribution I can add is inspiration. The fact you put four hours after the working day into making your goals a reality is simply put impressive. It inspires me to up my own game. I manage realistically on average two hours after a work day and I see myself as dedicated. With that determination combined with your skills any obstacle will literally be a small bump in the road. You will make it happen, 101% sure of that. That's all of value I can add, but if only it gives you energy I think I contributed with some small thing /Robert
I re-read @Ghost_of_Blotto's post, especially on "the information is not in the data" Thanks for sharing the setup. I am looking into RabbitMQ, Redis and Monitis. By the way how much does Monitis cost you per month for your setup? Would you let us know which company is your FCM? By the way, what is your Sharpe ratio? I am working on two approaches: 1. Deep Learning with millions of hyperparameters to optimize 2. Feature Constructions followed by multiple Rule Learning algorithms combined within Ensemble Learner setup. All models will need to go through regularization process to avoid overfitting. Both approaches produces models, the test will be out of sample test, followed by forward test of paper trading, then forward test on real money 1 emini contract - only a few trades per day, then increase then number of contract and frequency of trades. Thanks, I don't underestimate, I understand this is very hard. Even for someone with a PhD in Machine Learning and 20 years industrial experience this problem is challenging. Regards, Trading.Computer