Finally completed my tick compressor I was interested in making it much faster than the fabled TickZoom where the inventor claimed that it could process 11 million ticks in 8 seconds on a quad core machine. Well.. my algorithm compresses 3.75 million ticks per second in debug mode. It compresses the ticks into 15 minutes bars. It is developed in C# 4.0 => 30 milliion ticks in 8 seconds - and that is performed by a Dual Core: Processor Intel(R) Core(TM) i7 CPU Q 720 @ 1.60GHz, 1600 Mhz, 2 Core(s), 4 Logical Processor(s) Take a look at the screenshot, I actually tried to optimize to the best performing parallel processing threads, it's is around 49 I will keep posting on this project.
by the time you GET the ticks, the big boys are already trading against them. try working on a time machine heres my prototype
By a mistake I didn't mention, that it was for historical testing only In real time, it's only able to process 1.5million ticks per second, well at least in theory. This theory is based at my own tick plant and tick pipeline into the tick engine. Unfortunately I don't have access to a broker that is able to deliver real time data in such speed Anyway... the speed is quite interesting and should give other developers a good hint about what can be accomplished by C#. IMHO everything more than 50 million ticks per minute is more than enough for both back testing, forward testing (whatever that is ? *LOL*)) , and real time processing of tick data.
Define "Compress" please? are you refering to 1.5m ticks of data hitting the socket and your app reads off the socket, decode, and process the data? How many products/symbols is your feed? How big is one of your ticks? is it a single price change (i.e. bid price change = a tick, bidquant change = another tick, etc)? If your canned data consist of only a single product and ticks are refer to the same price level, it can cause scewed results as you get much more cache hits than misses.
On top, please note that basically TickZoom would be faster now. Because the number was quite old and processors get faster
Um... looking around the chart, isn't the optimal number of threads around 7-8? The absolute best performance is pretty useless since after around 7-8, it's all basically just random variations?
Hi mickmak A compression is defined in my architecture as an aggregation over a frequency of some type. For instance, as we all know "time", a compressor then takes the frequency, i.e. 5 minutes and each bar then consists of the ticks that matches between bar start time and bar end time. In my POC there is only one symbol, but more symbols will not affect the performance, max throughput will still be 1.5million/ second. A tick is described as a trade, trade price can either go to the sell side og buy side. The tick format is : datetime, price, size, please take a look at the attached tick sample. Not sure exactly what you mean ? Please explain