Hi All, Re data read/write performance; here are my 2 cents. I think this is a very relative topic, and only relative to this thread since Chris is making a true trading class library 'platform'. What really matters imo is 'where' you are going to host your database, and 'how often' you are going to read/write data to it. Based on these dynamics driven by trading style - you have a plethora of options open to you. From local databases/file structures to cloud blobs / tables or relational database. Each can be appropriate (if not optimal) based on these underlying dynamics, and are somewhat relative to the complexity/scale of the platform design. It becomes a sliding scale at the end of the day, but some general patterns emerge. Use memory not a database/file for high frequency trading. Ensure the client/processing end of your trading system is hosted in the same location (data center) as any stored data. Balance your tick data time frames against requirements and associated processing loads. (there is no sense bottlenecking your system CPU and I/O unnecessarily) How many instruments do you also intend to trade? What kind of reliability / SLA factors do you need? Analyse hosting costs / initial setup fees / ongoing costs relative to your IO frequency and size - and incorporate this into your technology selection and hosting location choices. Then, you can worry less about different individual platform/service performance metrics, understanding that at the end of the day, if you can afford performance - you can buy it, and it is only getting faster each day; and focus on what matters 'most' (the trading strategies) which if profitable, will allow you to buy/account for any future desired/required data i/o performance requirements. This can be summarized also by concept of 'cost of entry', as for example to build a per tick high frequency automated trading platform is obviously going to require more investment/resources/complexity relative to a system which only stores and process hourly or daily data. So whilst there is not perfect system I think, I would recommend to anyone building any form of automated trading is to ensure you are using cloud based services and not local user/system run-time applications/files/databases. I see many great 'little applications' people have built that will never scale appropriately at a code level (even if they are profitable or look pretty) because they were not built with a platform architecture mindset. Cloud services allow you to be A) more modular in your designs and B) provide optimal stability / performance / redundancy scale-ability. I see so many blogs of python script hack job trading 'code' that it brings me to tears that they (and the MT4/5 community EA people) don't understand the value of a system that can reboot, update, and have an internet outage gracefully.
Sounds like a great project! Curious as to how exactly you incorporate the Microsoft Cognitive Toolkit into your algo discovery.
Hi Fan, thanks for you up late at night question Yes I hope to build a private community here and take this project all the way to the bank so to speak. Below is some description of CNTK (available for c#). It is Microsoft's 'version' of Tensorflow and Google Deepmind etc, or the underlying components used (imo theorizing) behind such online services as Azure Machine Learning (web drag and drop etc/prebuilt webservices etc) LSTM is in essence the 'algo' in itself. It is a 'type' of neural network architecture which is both very 'new' relatively speaking to the scene but has also been identified by the industry (as far as I can tell) as yielding the best results for time series analysis when it comes to machine learning 'models'. After all it is predicting the price quite accurately after tweaking. However in order to make money from a prediction - you need many more layers of logic applied to your order entries/exists. This is where my final 'strategy' incorporates using 'best of breed' variables from both the machine learning world and traditional statistical analysis / market indicators in order to make a decision. If you are interested to learn about just how cool CNTK is I highly recommend checking out Bahrudin Hrnjica's blog @ https://bhrnjica.net/ He has a PhD and is a professor and has valuable information and code examples that can be experimented with / extended / learn from and thankfully he works in C# (most who use CNTK currently using ugly python(not suitable for enterprise platform architecture implementations)). P.S. The logical extension to the platform design I have described is to extend the trade and account management layers (and potentially execution) to DQN model. This is much more complicated when it comes to trading as one must design 'agents' and an architecture to manage them, however I have learnt a lot on this topic recently and believe that I have the right agent definitions and variables to work with. I have some 'basic' code for DQN using CNTK in c# from a microsoft researcher but this project would probably need more funding and people to achieve such an outcome in reasonable timeframe. The hardest part is finding people to work with which for example I could hand that DQN code template to (them already understanding my LSTM) and have a rational conversation about it with - both understanding the terminology and underlying implementation requirements. Oh.. i forgot to mention. DQN would be like proper AI, not 'machine learning' which imo is just glorified iterative statistical analysis. The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes neural networks as a series of computational steps via a directed graph. In this directed graph, leaf nodes represent input values or network parameters, while other nodes represent matrix operations upon their inputs. CNTK allows users to easily realize and combine popular model types such as feed-forward DNNs, convolutional nets (CNNs), and recurrent networks (RNNs/LSTMs). It implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers. CNTK has been available under an open-source license since April 2015. It is our hope that the community will take advantage of CNTK to share ideas more quickly through the exchange of open source working code.
I will take a look at the blog you referenced. The project you are working on does sound like an epic undertaking as you mentioned. The approach I am taking is to only build software for specific problems that are not adequately solved by existing solutions. IMO, there are solutions which do a pretty good job with charting, live trading and metrics. I am evaluating AlgoTerminal right now and it can solve those problems adequately. What is cumbersome and time consuming with most platforms is finding Algos that "work". That is the problem I am working on where my platform can generate code to run the algo in an existing ATS. I much rather spend a few hundred bucks a month on an existing ATS solution than build one from scratch. That being said, you may have specific requirements not met with existing platforms. If that is the case, I would be curious to know what those requirements are.
Hi again Fan, In one word, I can answer by saying 'flexibility'. One can think of platforms like AlgoTerminal as a constructed layer within which you can work. Don't get me wrong, working with such platforms can be more than adequate but as you question - 'what more could I possibly need'. Indeed such platforms (by their inherent business model) are design to make it 'easier' for people to implement automated trading strategies (even in c#!). Well I can think of many, news feeds, sentiment analysis, AI/machine learning, even basic things like exchange rates. Due to their business model such services (unlike some brokers) simply cannot keep pace with the rapid rates of change/advancements 'required imo' in order to continue to trade profitable in a market where over 60% of the trades are already automated. In short if you have the option/flexibility to do 'the same thing' independently and not rely on a 3rd party, then in my experience in life - whilst perhaps more difficult - usually the better long term option. Speaking as a professional software developer however, you can almost form a basic hierarchy of auto trade solutions out there; 1) fully 100% custom built 2) an custom assimilated and compiled class / reference libraries derived from multiple sources/repositories/examples 3) an open source or proprietary broker / strategy/analysis 'code base' and or associated API service 4) web based service 'scripts' or client side application (MT4/5) 'hosted' scripts. Since custom built would require genius level coding skills and trading knowledge, or corporate resources, lets assume this is out of selection even though it will offer highest degree of flexibility. Next is a collection of classes or analysis functions/methods which have been obtained as a result of 'research and development' - effectively tailored to the requirements of your design. This requires a lot more time and effort to assimilate this 'IP' into your 'working platform stack' - however is 2nd best for providing flexibility. 3rd place goes to defined libraries and or service machinations, as the way they work within functional boundaries - this will restrict your to ability to 'truly' evaluate 'algos'. I will concede that such platforms make things like backtesting much easier but honestly, you are much better off simply using a broker that has these services offered directly even if that is desired functionality. As you say there are many algos available but how to find a profitable one? My advice on this topic is there is no silver bullet, which is why you will find the 'best' quant traders out there doing linear regression against multiple algorithms running in parallel to pick best one to run at different times. Also with regard to backtesting - whilst a valuable tool, I think more focus should go into real live trading on practice account in conjunction to an 'agile' or iterative monitoring and development approach. The least flexible is of course web service that will run a 'script' (pseudo code) for you, or tie you down to implementation 'within' a client side application which for automated trading is just a dumb idea for so many stability / functionality restrictions. KR, Owen
I prefer a hybrid approach of 2 and 3. The key is to find a solution in 3 (off the shelf) that allows for 2 (customization and extension). That being said, most of the off the shelf platforms have not met my needs. I have personally chatted with several people on this site who have fully functional solutions for algo trading that they mostly built from scratch. While impressive, I also know the level of effort put into building those platforms and that is time I do not have. Anyhow, not sure how active the OP is on the site but looks like you have some great coding skills and could likely recruit someone to your effort.
Yeah I hope OP is active, but i'm guessing others may contact also. You know its a interesting thing the politics behind creating such a thing. In reality it is effectively a license to 'print money'. As such I am already cautious about who I work with / sell code to or collaborate with. For example I asked a friend if they wanted to invest in the platform for a small one off fee (to help me start funding 'actual serious go live account with sufficient capital to cover all the instruments being traded (after all I am just a broke entrepreneur trying to fund an actual unrelated product release)) I know his story in wanting to make money etc for his kids future etc etc, and since I know my platform is profitable - I proposed to provide all upgrades also to strategy files etc. I explained to him as best I could the system, and all he can ask is 'what is the % profit daily/weekly like in the emails we used to get'. I don't plan to reply because I now understand that that is 'too easy' an option. Nobody ever came to me offering guaranteed results, saying here is free money with no risk/effort required. I now reply by saying ok, so do the calculations yourself from my signals if you can be bothered. This is from someone who actually enjoys helping people. I am guessing this is the same dynamic many people on these types of forums have picked up on. Where are all the people making money and what tools/codebases are 'they' using and why they are so quiet. (well to anyone appropriately aligned reading this (my) exception to the rule and looking to become part of this exclusive hidden class of traders hopefully reply haha) All I can say is stick with it an align yourself to people with similar goals. Also from your approach you should probably take a step back with a whiteboard in terms of 'what matters' and then base your coding around this. 3 providing 2 does not exist by rule of nature I just described. We live in an interesting age. Personally with advent of quantum computing / laws banning them for personal use etc etc. I think this type of trading has a definite time limit, beyond this it is AI and systemic breakdown of traditional systems. Think of it like this , if someone releases to public the perfect algorithm and simple method of application, this will spread like wildfire. Very soon its strategy across many players will 'MAKE' the market. It will do this up to a point, much like the time to sell is when interest is at its maximum; however beyond this - the market will then quickly adapt, and said strategy will no longer function as it once did. You may find it is actually less code than you think and if you know what you are looking for its just a search away. for example indicators. why be limited by one single api computing them for me or the range they have available. better to source from open source libraries and compile your own library of 100's to choose from. Also, how much % of their exposed features will actually be useful? One tip I can give, is just to stress the real-time monitoring 'in conjunction' with your applied strategy 'rules' or signals. before i created the 'signals' dashboard it was very difficult to perform any meaningful tweaking of indicators relative to different timeframes and get them in meaningful alignment. Whilst not 100% necessary - this visual type feedback is very helpful to determine what combination of indicators/settings/processing variables works best. such a dashboard is really small amount of work in reality. I guess i should clarify that my approach to 2, is to KISS, and only add things you actually need to help achieve positive returns reliably. Everything else, including many of the features / subscription services offered, and including things like charting etc is really not necessary but just a value add on top. Just don't get bogged down in platforms/libraries. Formulate or adopt/adapt a strategy of your own and then build the platform 'around it'. This will be way less work than you are imagining in terms of actual effort. Another tip would be don't rely on other peoples recommended settings. Always experiment with your own values in order to morph them to your needs. So understand that again with these online platforms, may of the functions or template recommendations are just that, recommendations. So whilst someone might recommend 70% training/20 evaluation and 10% dataset split as a industry recommendation (understand they are reciting what they were told) and whilst in 'general' these things are true - may not always be accurate to the model outcome you are trying to produce. so for example in above machine learning data if one were to use this 'recommendation' for any price movement outside known boundaries no matter how much past historical data you have - your adjustment is going to up to 30% lagged in time. Lets take recent oil over 80 scenario... it will fail epic-ally.