Hi all, I am in the beginning stages of starting a company to provide tools for automated traders. Generally speaking I want to automate as much of the research, development, and production path as possible. I'm targeting sophisticated quant traders (equities for now since I'm familiar with that) who have a good understanding of trading technology, have or want access to the best data, want to work at a higher level on trading strategies, and save time and money on accessing and analyzing that data. I'm looking for feedback as to whether I'm completely misguided, generally on the right track but need some help, or if I'm targeting a need perfectly (great, where's your checkbook?). Here's what I have so far, mostly low level stuff and I want to expand into more research tools down the line: Feed handlers. So much boilerplate and the bane of everybody's existence. I have auto-generation for feed handlers which takes the protocol specification and some protocol-specific rules and spits out (super fast) C code. It can obviously target anything with little effort. ETL. Again so much boilerplate crap and nobody wants to do it but it's a huge time sink, converting between formats and updating all this code every time the exchange publishes an update to their protocol or adding a new protocol. Again given a spec I can automagically transform between binary / ascii / sql and load without any boilerplate. Here's what I'm working on: Custom / in-memory database implementation. Crunch terabytes of data in seconds, 'nuff said. Disk-based SQL implementation. A high-level language for trading, which allows you to query historical data to create signals and specify algorithms. I think a StreamSQL+LINQ-like language is the way to go, with special primitives for trading operations. Signals are described declaratively in terms of reductions over streams of events (functional style), as well as combinators like merge / split / set dependencies between signals. Obviously performance is a problem with general purpose CEP / LINQ languages so I'm compiling to a style of C with deterministic memory / performance profile. Maybe FPGA down the line but that's a lot of work. This way you can have your cake (abstraction) and eat it too (performance). With this I hope to achieve the pipe dream of having a unified environment for prototyping, backtesting and production. Lastly I hope to separate the concerns of signal generation from order management. In other words you do research on the signal and then plug it into an order / risk management framework which suits the needs of the particular strategy. Miscellaneous dumb things like: Being able to quickly look up the definition, commentary and explanation of a field in the specification. I can't tell you how much time I have spent reading specifications and understanding them, so many hours of my life I am not getting back. I hope to save other people this pain. Access to data exactly as it's in the spec. With operators to easily define equivalence relations between specs. Normalization is evil because you lose data / can't predict what's being done. Obviously I want to provide standard normalization tools so if the user doesn't want to do that work they don't have to but also give access to the data as defined in the spec with no changes. So the resulting service is basically, no-hassle data management, a nice database / query service, and seamless deploy to live trading. All hosted, I don't think people really want to deal with managing their own servers / operating systems anyways. Down the line I want to add graphical tools, and also guided tools for mining data (with specific heuristics for trading) but right now those are out of my reach. Here's what I've turned up in terms of companies which are doing similar things but just not quite the right fit: https://www.quantconnect.com/ http://www.spiderrock.net/ http://www.softwareag.com/corporate...kets/algorithmic_trading/overview/default.asp https://www.onetick.com/ Feedback? Thanks
I have already developed pretty much everything you described. Plus full UI for the front end, and in addition distributed compute instances which allow me to spin up hundreds of compute instances for backtesting/profiling parameter optimization purposes or to run concurrent backtests with parameter/symbol/timeframe iterations. While I laud your idea and believe you are much more well versed in hft space I am not sure it will address the needs of anyone. Think about it, your solution and the technology, described, will not even compete in the tier 2 or tier 3 segment of most hft, latency wise. Having said that the question then arises why having to go so low level on many of the components you walked through? Furthermore, it looks like you focus with your ideas on US markets. HFT in most markets outside the US is on the decline or outright prohibited/discouraged via sky high stamp duty or cancel/modify charges and that includes many markets in Europe as well but particularly Asia. I would even argue that the best times for hft in the US are over and believe that companies will not compete for latency anymore in even 2-3 years due to a mountain of upcoming regulations. I think the focus/future will be on a more intelligent way to develop trading strategies and not the trading/execution of those. AI (deep learning,...) will be huge and I am not talking about dumb indicator optimizations that pretty much all commit curve fits. I am talking about the requirement to handle unstructured data aside pure pricing data. I am, for example, currently working on the extraction of Tweet data and finding edges in information gleaned from such unstructured data. There will be huge implications from data intelligence applied to financial trading. I would not say hft is dead but something else is about to become the focused and that is data intelligence and deep learning and I think it can be integrated into a financial trading and development platform. Just my 2 cents... Btw, Quantconnect via Tradier is a horrible idea to start with. A firm that sells order flow catering to "quants" and algorithmic trading? Isn't that an oxymoron?
Thanks, I value your input, as always. Having already built a lot of these components, I am sure you can see how most firms keep reinventing the wheel over and over again? And that it would be a real cost saver for most firms not to have to develop and maintain the same basic technology. I am not sure where you got the impression that it will be not latency competitive. My understanding is most tier 2 / 3 firms are at single digit or low double digit microseconds wire to wire. What is your understanding? Is it low level? Thank you for your suggestion to focus higher level. You have to start with the basic building blocks. I do want to move into AI (which I call guided data mining), once I have the basic infrastructure hammered out. Selling order flow to quants is kind of an oxymoron . If you think about it, though, isn't that what IB does?
I think unless you program FPGAs you might already by definition qualify as second tier today, no? That was my impression so far at least. Of course if I misunderstood your original post and you look to support start up hft firms or hedge funds with low latency feed handlers or execution capabilities then that changes the story. But I do not see the need on the retail side as explained. With low level I meant hardware near programming, C or slightly more abstract, C++.
@hft_boy Sounds intersting. Personally I'm always interested in such implementations. I like your idea of standardisation and no hassle data-management. I do believe that volpunter is right here. The future focus and challenge in the mid range is towards sophisticated AI and I think also in trader interaction/integration and ease of strategy development with integration of multiple also "social" data source. Hosted solutions are definitely the way to go, but it must have some depth ... quantconnect is way to superficial. From my personal experience: Don't focus to much on ultra low latency because at some point you are going to loose the focus on "real demands or requirements". About your in memory database: Did you already start with the implementation? You may soon realize or already realized that, in a simple implementation, IO is going to be your bottleneck, so intelligent data sharding/segmentation is the way to go. Have you thought about which "customers"/traders to target? Cost, price range ...
from my location, the trading firms like to reinvent the wheel or use an off the shelf product like TT, optioncity, orc ... What you're proposing seems to target the firms that write their own stuff and they always think they are really fast