So it was you who brought up ZeroMQ then I got it wrong about 2rosy. I found an article where someone compared the performance of OpenDDS vs ZeroMQ vs Boost. He claims ZeroMQ latency 170 microseconds, which is ~5882 messages per second. Am I reading it wrong or why do you think there is such a difference between his and your results? http://mnb.ociweb.com/mnb/MiddlewareNewsBrief-201004.html
Thanks for the feedback. The words you mentioned -- task processing, async programming and actor model -- to my mind are all just frameworks or ways of setting up your APIs. And of course it's important to set up APIs in a nice way. But that has nothing to do with their implementation, which is what I'm dealing with here. As I said, sockets are one way to go, and they are very elegant. ZeroMQ uses TCP and Unix domain sockets underneath, as far as I can tell. Which can be plenty fast. I am skeptical of 16M msg / s over TCP unless you have TCP_NODELAY turned off, in which case latency might be pretty high. At a high level, shared memory mapping isn't really any different from TCP or Unix domain sockets, except that it's all handled in user space (and it's faster). I also said that the hard part is writing a nice API to this thing so it looks like a socket, and so the calls are all agnostic as to whether it's another process on the same machine or on another machine. Cheers,
you are confusing latency with throughput. Latency is about how long it takes to get data from A->B. Throughput is about how many items you can send from A-> B per second. I stand by my throughput claims. You can actually squeeze even higher numbers out of ZeroMQ if your messages are <16 bytes (or if you come up with a better approach than I have implemented ;-)
At a high level, shared memory mapping isn't really any different from TCP or Unix domain sockets??? Excuse me? The two could not be more different. What are you talking about? By the way, in case you did not get my point: IPC messaging is utterly useless when being used on the same machine. Then memory mapping is preferable. IPC really makes sense when you send data between different machines over the network sockets. Not sure what was so confusing in my previous post. And I stand by my 16m/second performance numbers using ZeroMQ .Net libraries and C# on .Net 4.0. The performance is based on sending messages IPC over tcp albeit on the same machine (which can immediately be changed by registering the ZMQ sockets on different ports and hostname. By the way this performance includes the serialization and deserialization of 16byte messages.
You don't know what you are talking about, but to someone not in the biz you might appear like you do.
who loses messages? maybe on your $150 netgear router at target. lol. Get the infiniband switch and colo on every exchange. Otherwise don't call yourself HFT. if you still get dropped packets, learn to program.
just 30 seconds skimming over your past posts shows you try to pee on every pole there is. I have no interest in wasting my time to converse with people like you. Make your case and start making sense with facts otherwise consider yourself on my ignore list.
go ahead, run and put me on ignore, you are of little significance to me. I only state facts so you are only running from the truth. frankly, its not my job to go into details and give out any secrets to the business which may affect the firm I work for's profitability or my own future profitability. I think everything I've said has been extremely generous, general but valuable advice that if pondered on can save years of time going in circles.
Would one of you guys mind giving an explanation of what ZeroMq does and how it's used in an automated system that a fourth grader (me) can understand?
It is a message bus / pping system. The idea core being that you can not do everything in one process / machine, so passing messages between a communication / dispather system (that talks upstream to the exchange) and machines running the processing loops (possibly 1 core per algo, hot - i.e. in a spinning loop) is needed, and that must be FAST. ZeroMQ - MQ standing for Message Queue - is a middleware for passing messages.