Appropriate design pattern for recording instant tick data in memory

Discussion in 'Automated Trading' started by rohan2008, Dec 11, 2014.

  1. vicirek

    vicirek

    It is getting too complicated for search of one out of 40. You can do it many times over and many ways at once and it will not matter in terms of performance.

    Check if search is even necessary. Many API assign index to your request or have other specific integer ID attached to returned contract data and you may use it to directly index to your struct using just that. Check the documentation because I have no experience with Rhytmic API.

    The best way to address cache performance is to have continuous layout of data in the memory instead of pointers to objects scattered all over main memory. Other than that there is little you can do to affect processor heuristic in prefetching data.

    Keep in mind that compiler will pad structs if necessary to align fields and the sizeof can be different than calculated by hand (compiler options how to pad structs are often available). Your solution creates unnecessary large struct.
     
    #11     Dec 14, 2014
  2. second that, OP, what are your latency requirements. Recording should never be a latency priority. The lookups more so. But what exact specs do you have?

     
    #12     Dec 15, 2014
  3. rohan2008

    rohan2008



    RAPI doesn’t have the integer ID for tick data; it does have id for other data such as order reports etc.

    Ok, here’s the issue at hand: I have some low/medium frequency strategies that rely on one min candles. I trade in higher timeframes. I am sensing that I can develop a few algos that can examine real-time tick data and reduce some entry risk for my existing strategies… My issues is how fast my algorithm should execute in order to input all the ticks that I get from the market without a lag? Note that I am not too worried about latency involved in responding to the ticks though (that’s an another topic).

    Assuming that I subscribe to tick data for 40 contracts (20 current & 20 forward; no options) and lets say, I get about 20 million ticks per day. Now, if we assume that 80% of the volume happens within 5 hrs in the morning … so that’s (80% * 20 million/(3600 *5)), which roughly comes to about 1000 trades a second… which gives me roughly about 1ms to save per tick… which is quite manageable (Please correct me if I am way off here). However, once in a while I do see short-term instant volume spikes (e.g., news events, I once say 500 contracts sold for NG in one sec) and so I was wondering how fast should my algorithm be in order to input all the ticks. So, if I use 1000 trades/sec as my average base line, what kind of instantaneous volume spikes would I end up seeing? The original question was a result of this issue :)

    So, folks, please enlighten me here since I myself don’t have much experience with tick data. What is the average trades/sec number do you normally notice during the day and what kind of volume spikes do you observe once in a while. How many micros do we end up having in order to successfully input a tick without generating a lag.

    appreciate your suggestions....
     
    #13     Dec 15, 2014
  4. I still do not fully comprehend your question; are you asking about the storing mechanism of the incoming ticks and related latencies? Or are you asking about retrieving the stored ticks? In case you refer to storage, you do not need to care about latency sensitivity when it comes to storing the data. You store a buffer in memory and periodically persist chunks of data in memory. Does not matter whether you deal with 1 million ticks a day or 50 million.

     
    #14     Dec 15, 2014
  5. rohan2008

    rohan2008

    The former; within what time should we store an incoming tick in order to avoid a lag on the trading system side... for 1000 ticks as my rough math suggested, its about 1 ms, which is quite reasonable. But do we see any volume spikes in the tick data that it can even go up to say 10,000 ticks for a few seconds which gives me 100 micros. Even this is manageable for futures. STL might be more than sufficient for this and if need be, go with the contiguous single threaded array approach if we want to get it down the nano sec level. Guess I got my answer... but any info about volume spikes that you notice can be helpful. thanks.
     
    Last edited: Dec 15, 2014
    #15     Dec 15, 2014
  6. I recommend you segregate the concepts of storage vs processing of ticks as part of your strategy engine. Your strategy engine should optimally be able to processing all incoming ticks even during volume bursts. If that is an issue then you may potentially "sample" incoming tick data (see Interactive Broker's technology for reference). That is obviously not a choice if you trade high frequency strategies where your model depends on micro market dynamics.

    Now, an entirely segregated topic is the storage of incoming ticks. That should not at all impact your actual trading framework. You want to store and handle the storage on an entirely different thread/task/process. Latency and performance is not critical at all in this area. You should concern yourself much more with persisting data in a safe and memory/space efficient way. Latency is not critical here.

    I recommend you define exactly which latency you are concerned with. Storing data in the most time efficient manner imho is not important.


     
    #16     Dec 15, 2014
    rohan2008 likes this.
  7. Butterfly

    Butterfly

    volpunter, don't try to use language you don't fully understand, it makes you look foolish, again !!! LOL

    jesus christ, what a silly amateur you make. Hopefully no company is foolish enough to pay you !!!
     
    #17     Dec 17, 2014
  8. 931

    931


    Maybe i cut context out, but can you elaborate on that?
     
    #18     Jun 28, 2020