Hi, Hoping to ask for some advice on an automated trading system setup please. I’m trying to decide whether to use an off the shelf backtesting and live trading service (such as Ninja Trader) vs a mix of open source systems (such as PyAlgoTrade and IBridgePy) and was hoping for some advice from more practiced hands. Short version of the question: I’m a little confused as to how flexible some of the off the shelf systems are at using external/custom metrics and trading (interval) frequency. After having a less than amazing experience with a developer trying to build a custom solution I’m looking at off the shelf software. Is anyone aware of a backtesting and live trading software system that is: standalone software that can be hosted on AWS (or a VPS), can live trade with interactive brokers, make live trading decisions based on DTN Iqfeed data (rather than IB feed data), backtest rules (event) based on DTN Iqfeed data, incorporate custom metrics within backtest & live rules, and operate on a 1 sec ‘heartbeat’? Longer version of the question: Some considerations and requirements for the setup I’m looking for: 1. I’m not a developer by trade. I’m learning Python but I don’t want to wait 6-18months to become adept enough to put live a trading system that I can trust enough with real money. From a systems perspective, whether it’s an off the shelf or open source system, I’m going to need to hire a developer to help configure a system that I can live trade with for the first time (and then evolve the rules myself). From this perspective I’m leaning towards an off the shelf system as I won’t need to worry about bugs as much, and upgrades are presumably easier to manage than iterating open source versions. 2. From a rules perspective, I’m confident in being able to write my own trading logic that runs in a system (when to buy, when to sell etc). 3. I’m a Product Manager & Business Analyst by trade. I can write system design documents and requirements, and have hired developers via upwork in the past (so open source powered systems are feasible). 4. I want to (must Requirement) keep my rules private and only use standalone software that I can host on AWS (which I’m very familiar with). This rules out cloud based systems like quantconnect 5. I’ve already hired a developer who has automated the retrieval of 1 sec & 1 min OHLCV data for 500 US Equity Stock symbols from DTN Iqfeed, and created some custom metrics that I want to be able to use within a trading rule. An example metric would be ‘count of stocks that closed the interval higher than it opened’ (which would be used in conjunction with other rules such as moving average crossover). I’ve been using jupyter notebooks, pandas and matplotlib to run analysis and hypothesize some rules that I now want to backtest and hopefully put live. 6. I’d prefer to (should Requirement) continue using DTN Iqfeed for making live trading decisions as the data quality is good and I’ve read it’s better quality than most broker feeds like interactive brokers. 7. I want to (Must Requirement) use interactive brokers as my style of trading needs low transaction fees and I eventually want to expand to more markets. 8. Back testing must be event based so I can easily flip from backtesting to live trading (or ideally paper trading before live trading but not essential) 9. I’d prefer (should Requirement) any software to use Python as the rule coding logic as it’s more transferable (rather than being locked in to a language that only works on one system). 10. Must be able to trade on a 1 sec ‘heartbeat’ (rules out zip line as a backtester which is 1min intervals) As a next step (if no joy from this forum) I was intending on emailing various trading software providers to ask if they can satisfy the above requirements, but as I’m sure many will say they are the best, I was hoping to either learn from this trading community or be educated on my requirements if they are too restrictive. Thanks in advance ChainsawDR
Thought experiment: man who wants to be a doctor says he doesn't want to wait for med school and starts treating patients. What happens? The 6-18 months is just the dues you pay, don't think anything else of it. Unless you have a low beta trade that you do manually, it'll take you a long time just to figure yourself out. Ah that explains it. Do it manually and find out how close you are to being right. 1 second or 1 min doesn't make a difference unless you think you have an edge in pricing. Which I am like 99% sure you don't. Not relevant. You have a trade that you've researched using data with Jupyter/pandas/etc. Write a notebook that gives you signals. Follow those signals manually and see how well you do. Out of 500 stocks, maybe you'll have 30 that meet your signal. You can use ib_insync in your notebook to make the trades. Once you start seeing success, you will start to automatically invest in the right things regarding automation. The advertising makes you think that automation is the thing that will make you money. It is, but it is also the process through which you go to develop new trades that does it. You don't need to automate to figure out if your process works.
Why can't you use off the shelf products for back-testing, but use different platforms or build your own for real time? Unless your rules are so complicated that it would be a challenge to implement them twice. The only requirement I would have is that both must use same data source so that forward testing matches back tests.
I’m definitely open to using different platforms for different aspects (sorry if that didn’t come across). Using the same data source for both is sensible/logical. Does anyone have recommendations on software (that would sync well with Iqfeed & IB, and operate on 1 sec? The Iqfeed data is being stored in MySQL so I suppose I should think less of it now as Iqfeed connection but rather being able to connect to MySQL data? The doctor scenario isn’t really comparable (but thanks for the advice). Time is money/such a thing as too much preparation/sometimes you’ve got to operate on an agile basis - I don’t think I should need to be a full blown Software development Engineer before getting started - I’m imagining that there must be software frameworks out there that can take care of the architecture/API connections and allow traders to just code the rules that a system applies (I would have guessed that I’d be a typical customer for a piece of software like ninja trader?). For added context, I work full-time 60hr weeks at my day job at a FANG company and I’m aware of my own strengths and limitations: I can’t take time off to manually put on trades, I can’t dedicate X hours a week to learning Python or C++ and have to accept that I’ll have to learn this on the go (hence requiring backtesting to help mitigate risk of throwing money away from novice coding bugs). I’ve researched the input data sources (DTN Iqfeed) and brokerage (IB), I’m now looking for advice on software that can help tie these together as there’s so many options out there. Many thanks in advance ChainsawDR
One of the main reasons for posting here is that reviewing off the shelf software online is proving troublesome to find software that’ll meet the must have requirements, Eg Multicharts: “can’t use the same code for backtesting and live trading”. Seems like that’s asking for trouble unless it’s just a case of copying the same code into another part of the system. The other comment about needing to resend orders to maintain order positions didn’t sound great either. From here: Amibroker: “You need to partially rewrite the backtester code to make it reliable for live trading” - this also kind of defeats the point of wanting an infrastructure that works out of the box. From here: http://www.quantifiedstrategies.com/my-thoughts-on-amibroker/ Ninjatrader: seems to be very forex and futures focused (equities not mentioned on their homepage at all and struggling to find reviews on backtesting and trading with IB with NT) If there are any recommendations for software that’d work well with Iqfeed & IB that’s be greatly appreciated (so far Amibroker seems the best bet)
I use Amibroker with AFL/javascript and perl script. All my trades are done manually. The stock market is hard beast to tame even with IT wizardry. Not sure how true is this story.
If you can't take the time off to get gud, then just dollar cost average your presumably large FANG salary into the various index ETFs
You can try out our opensource project OpenTrade. I have read through your post, it supports most of your requirements. This is an open source OEMS, but also an intraday algorithmic trading platform. It supports C++ and Python alpha both. The same code for backtest and live. By default, it is tick-by-tick event based. If you need 1 sec or any other interval bar, you can write your own indicator with its customized indicator extension like this. BTW built-in support of IB order entry and BPIPE/IB market data connectivity.
I think that’s sound advice mcnoob, thank you. For me, I don’t think I’d be able to rest without taking it further - eyeballing data, testing rules, iterating, learning how to do that next step. I’ve been on this path for ~18 months (reading lots, listening to podcasts on every commute, doing some discretional trades, taken courses to use Linux, basic Python, pandas/dataframes and data science techniques). I definitely don’t want to stop on the path, but was hoping there might be some software that’d allow me to just focus on testing and putting live rules and not having to worry about coding infrastructure. Quantconnect got pretty close to perfect but I found the lack of ability to research to be tough (it feels odd to go straight into testing rules without first spending 10x the time analyzing the data to hypothesize the rules first), and the privacy aspect was a dealbreaker. Tbh I think I’ll end up taking a path of paying ~$6k for an SDE to put together the infrastructure that meets the requirements and some sample basic rules, then focus on learning on the job Python as I iterate the logic. Hopefully I might get some promising results in the first year and build some confidence for switching to it full time (at which point I think I’ll be able to do the hard graft study on the coding and evolve the system as you’ve recommended). Thanks for taking the time to help out a noob
That sounds reasonable. The reason I built my own infrastructure was because I did not want to be held back by another Dev teams decisions. It's been a good decision so far but I think I would have preferred to use MT4/5. I just didn't trust the investment would pay off. I can always transliterate MQL functions.