A side note. I found this book on amazon (only available for kindle), that may interest some of the readers on this thread: Zen and the Art of Automated Trading: A Journey from Day Trading to Mechanical Trading" I'll provide a review as soon as I finish it.
Trading a single instrument is ok not to multithread, but I want to have multithreading to handle processing of multiple instruments.
Okay -- I don't mean to be rude or anything, but why do you need multiple threads for multiple instruments? Just loop through them. Otherwise you have to sync the I/O which is often the most expensive part anyways.
Because processing each instrument takes time, say it takes 10ms, so if I trade 10 instruments it takes 100ms to process them all. That means I am 100ms late on the quotes, so when I start processing quotes in the next loop some of the the quotes will be 100ms old, which is essential during fast moving market. I am not talking low latency, only mid latency. It also depends on the algo, of course. Such delay is okay for a longer term strategies, but for day trading strategies imo such delay should be avoided. You do not agree? P.S. you're right about I/O - it's a pain.
A self-published book from someone who appears not to have verifiable experience in the field (at least nothing googling turned up)... not to mention billed as "Part career-travelogue, part self-help book"..... ... this doesn't appear to be a serious book.
If your strategy can't survive 100ms delay, you are not going to trade the strategy in a retail setting anyway. (yes yes, threading is not that hard, nor is spawning processes; it's just not that necessary)
That's fine reasoning. I agree that if it takes you 10ms to process each quote, you could look into threading. But do you know that it takes 10ms per quote? That is awfully slow, especially considering that 1 ms on a modern computer is 2-3 million clock cycles. Think of it as about 10 million instructions. There are usually better ways to optimize than multithreading. I don't write much in interpreted languages, so I don't know how slow they can get. But presumably somewhere in there you're looping through the threads trying to figure out which one to give the work to. I don't know exactly what you're doing. But I'm guessing that the overhead of finding the thread, assigning work to it, locking it and the whole shebang probably takes more time than any of your quote processing does. The only way to really know is to profile it, of course. Also, quick Google indicates that Python threads don't actually work like they are supposed to. Apparently there is a 'Global Interpreter Lock', which means that (to prevent nasty lock and sync on the part of the end user), the interpreter only ever uses one CPU at a time anyways. Ouch. http://kurt.seifried.org/2010/05/31/python-performance-part-2/
OP is using Python, I develop using C++/STL/POSIX on Ubuntu, I am almost done with infrastructure and soon to start with strategies whenever I get time for this again. At least I need to calculate SMA and st. deviation on, say 1000 quotes, I haven't done profiling to estimate how long that would take. I'd love to get rid of multithreading however I am afraid I am going to limit myself to 1) the number of instruments, if I want to grow to hundreds of instruments; 2) the strategies, if I find a strategy that would take 1sec to do calculations that would be prohibitively long to run on multiple instruments. I think it's okay to have some delay for entry, but it should be as fast as possible for exits. Disclaimer: I've traded only with paper.
Updating an SMA calculation with each incoming quote is literally one subtraction, one addition, and one division; Granted, these are floating number ops, but the amount of instruction needed for multiple threads will be FAR larger! (std is a few more instructions) Know your algorithms!