I was hoping someone was feeling charitable and could give me a rough idea of how the programming logic behind high frequency automated strategies work; this is partly out of idle curiosity but also so I can better understand how various trading applications work. I'm not interested in strategy logic, just how receiving market data and checking for signals is prioritized / triggered. Ultimately, I'm curious about how this is properly done on a large scale (say 1000s of symbols) but I think I need to start with baby steps and basic concepts. I currently use Amibroker to autotrade and that will loop through all symbols and check for signals on a fixed time interval. Could be wrong, but I'm guessing that the HFT space is entirely event driven rather than endless looping. Having said that, I can see the one potential advantage to the looping approach is that you evaluate the latest market data when you're ready for it (whenever each loop executes) rather than getting a backlog of tick events that you're signal checking code may not be able to keep up with. I guess that's the gist of my question: -If you're using a tick event driven model and you receive 5 tick events in the time it takes to process the strategy logic once, how is that typically handled? Does it: 1) Finish executing the code in response to the first tick then ignore the 2nd,3rd, and 4th and go to the 5th (most recent)? 2) Execute the code in response to every single tick (2nd,3rd,4th,5th) and have to play catch up? If it's #1, then I can't see much of a reason to ever use the interval approach...at least not on a single symbol -- that much isn't clear to me yet on a large multi-symbol portfolio but that's a question I'll save for now. Appreciate any insight on the above.
1) connect to market data and consume 2) distribute that data over a middleware( 29west, tibco, homegrown) 3) write lots of bots that consume off middleware and generate signals/orders
Thanks for the post rosy, but I'm having trouble deciphering an answer to my question from what you wrote (and the link). Any way you could dumb it down a little within the context of my example? Again, I'm just trying to understand how things work on a practical level.
Have any pictures that explain how a backlog of tick events are handled (as explained in my original post)?
Probably most used approach: - create a new object (consumer or whatever you choose to call it) for every symbol tracked - let the objects work independently - react to every new tick received immediately This usually isn't time critical if coded properly (with modern CPU power). I assume that most HFT algos work - on tick basis (not some arbitrary units like 1min, 15min..) - interface directly to some trading API (like FIX or brokers proprietary APIs) Possibly the question arises from working with a platform like Amibroker which adds extreme amounts of overhead (for the CPU, goes unnoticed for the user) for rather simple computations.
Thanks uexkuell, this is helpful. If I understand you correctly, the tick backlog isn't generally a concern since modern CPUs can execute the strategy before the next tick arrives. As a way to learn, I'm trying to verify how that would work here with some simple math but I need help with a couple assumptions/data points. To run my strategy logic in Amibroker it takes ~ .002sec per symbol. No clue if that's good or bad -- I'm sure it's highly dependent on complexity (mine is quite simple), but any ballpark idea on how long something simple, well written, and standalone would take? Just looking for a rough idea to run some numbers with. Again if I understood correctly, if one were trading 1000 symbols, they would create 1000 threads. How does this work get distributed across cores? As an example, I have an i7 950, so 8 total cores. Am I correct in assuming that one core would be dedicated to receiving market data and updating the quote arrays, leaving the other 7 to execute strategy logic in response to tick events? If so, then only 7 tick events can be processed simultaneously? Sorry for the novice questions - it's tough to find information on this that addresses the subject in an ATS context but isn't way over my head.
1. This is very slow. If an ATS is written in some proper programming language (like C#, C++, Java, VB) a not too sophisticated strategy would need only some microseconds to process a tick with modern CPUs. (Not milliseconds like in the Ami example) 2. Not necessarily. My first answer was intentionally vague (didn't use the term "thread"). There are many possible approaches. It is quite possible to process all the messages to the single objects in a single thread (which probably many programs will do). Then the CPU would just run the appropriate code of the symbol object that should be addressed (which holds all the status information from the last ticks) when a new ticks arrives. In other words not necessarily a need to bother about threads, distribution on different CPUs and so on. Again a high level platform (like Ami) gives the impression that processing is slow and difficult. When writing code for the processor things work so much faster and there is usually not so much need to bother about performance.
Ah okay, with that type of execution time I see what you're saying. I pulled these numbers from a consolidated tape report based on Q3 2011: -Average Daily Peak Trades/Second NYSE/Arca/Amex: 41,701 -Average Daily Peak Trades/Second NASDAQ: 23,225 the link in case anyone is interested: http://www.google.com/url?sa=t&rct=...q7C5Ag&usg=AFQjCNGr3MZbHkDMamd5JFfL3n0-5jvVQg So, unless I've bungled the math, if one were trading every major exchange listed security (~8,000 symbols?), on average the busiest second of the day would only require a tick to be processed in <= 15 microseconds [1,000,000 / (41,701 + 23,225)] when using a single core in order to keep pace. Not sure if by 'some' microseconds you meant something in the neighborhood of 15, but regardless, considering it's based on trading the entire listed market I'm sure something 10-20x slower would be fine for practical purposes....not to mention the option of going to multiple cores. Generally praised for its speed, Ami is a lot slower than I thought in this instance.
Are you referring to changes in the order book for a given symbol or managing order submissions/fills/etc?