can you give some code? It might be straightforward to you, but to people who haven't done this before, it is definitely not straightforward
It's hard to say without knowing where your data is coming from or what type of technology you're using. Suppose you're using interactive brokers with a SQLite database, python, and are trying to snap executions. It'll look something like this.. (the code is clearly incomplete and will not work). class Handlers: def __init__(self): blahblah def watcher(msg): yourQueueClass.addToQueue((msg.m_time, msg.m_orderId, msg.m_, msg.m_shares, msg.m_price)) class yourQueueClass: def __init__(self): self.__q = Queue() threading.Thread(target=self.worker) blahblah def addToQueue(self, item): self.__q.put(item) def worker(self): while True: c = dbConn.cursor() sql = "INSERT INTO execution VALUES (?, ?, ?, ?, ?)" item = self.__q.get() c.execute(sql, item) c.close() #then just do all the other crap necessary here You probably don't even need to deal with threads if your only objective is only to snap data. Multithreading makes sense if you're also doing other stuff with your program. That being said... If you're storing data and it's absolutely crucial to store the data live then that's the way to do it. However you're going to end up with an extremely high IO esp with tick data. The other was is to have your feed handler store the data into memory, then flush once it hits a certain threshold. Dumping everything out into a binary file. Then you upload everything into your db EOD.