by the way you are talking the whole time about multiple ES contracts. Are you trying to capture multiple future expiries?
Before you enter into complicated long-term contracts with dedicated servers, first simply try out a Google machine instance. You can get machines with up to 24 cores and literally over 100 gb of memory. A 4 or 8 core instance is way cheap enough to run some performance tests and spinning up a Windows Server 2012 instance takes literally 1-2 minutes.
Yes, not multiple contracts on the same thing, but rather multiple expires of the same contract and in other cases exchange supported spreads. Forgive my nomenclature.
Got it, thanks for explaining. Unless you attempt to capture the entire universe of listed options I would claim you should not run into any hardware limitations up to a thousand symbols including market depth changes if you implement it intelligently.
@volpunter No worries. I was hoping you'd chime in at some point on this thread. Thanks for putting my mind at easy about the server hardware requirements. As I understood the problem further I kinda came to the conclusion that multi processor servers and 128+ GB RAM set ups are really only necessary if you have a 10Gbs connection that is pumping out a lot of data. With a 1Gbs cap (and much less for practical purposes), I feel like a lot of cheap commodity server arrangements should be ample as you suggest. I looked at Google, as well as other "blue chip" providers like Azure and AWS, but I think that the server I'm using is going to be better for keeping costs low. There's no commitment and I'm paying monthly, so I'll probably switch over to Azure or Google if I run into trouble with these guys. My instance at this point is an 8 core Xeon at 2.4 Ghz, 8 GBs RAM, and 400 GBs hard disk for 80/month. If you happen to know of a better arrangement for less, do point me in that direction. I would be open to sacrificing some processing power for memory if that's an option elsewhere. Having an SSD might be nice too, because faster write speeds means less RAM necessary. Overall, I'm not super concerned about this server arrangement being suitable though. Let me know if you think otherwise. Best piece of advice you've offered me on this thread was the Tasks Parallel Library. I'm still playing with it, and it will probably take a good amount of work to get proficient with it, but it seems way more accessible than my previous attempts to learn about multithreading. My coursework next semester is going to introduce some parallel programming concepts, so it would be good for me to get an idea about them now and/or at work this summer. I don't know anything about multithreaded design, but I think the basic idea here is to let the concurrency of the TCP sockets do their own thing, then have a dedicated thread (or more? I don't know) working on the task of parsing the feed, and then one thread waiting for the queue to be populated with incoming data to write to disk. I'm clearly new here, so please chime in if there's a better way. Thanks for your help.
Tpl in c# is the way to go. Nobody in their right mind programs threads anymore. What you want to focus on is spinning up tasks and how to program async. As mentioned dataflow blocks (requires the loading of an additional assembly) will provide parallelization of workloads already standard. What would taken even a proficient c++ developer several days to get right you can get right and running with most likely way less bugs in c# in hours. Regarding your assumptions at the end of the previous post you are absolutely right. That is why the data flow pattern is so useful here. You have even within the same class a clear separation of concern and that on potentially different threads in a baked in thread safe manner.