Just wrapping up now, was a pretty interesting talk almost all about high performance computing and the evolution of their infrastructure. Some interesting parts: Everything was originally written in VisualWorks Smalltalk and then a majority of the functionality was ported to Java. In 2004, they started focusing heavily on performance where as before high productivity was more important. They were able to get average transaction time of 500ms down (eliminating DB writes was the first thing they did, duh. Now everything is aync journaled in binary format and maybe later put into a db.) to 2ms. They were very happy using the JVM (Standard implementation, they don't use the real time JVM), but to squeeze the last bit of performance out and to appease clients that blame all performance problems on Java, they rewrote only the matching engine in C++. The rest of the stack is still in Java + Smalltalk. They have many different teams of engineers working on different parts of the system and they have a dedicated performance testing team. Each development team is allocated a certain amount of "latency", for example the network guys get 5ms, the database guys gets 15ms, etc. They always determine composite performance and then specifically allocate. They do a lot of testing on weekends. They have nice investment in Amazon EC2 and use that to realistically do client load tests and simulate market conditions on their servers. He talked about a client who called them up and was complaining about receiving data 40 microseconds faster on server1 compared to server2. They spent a good amount of time trying to track down the problem. The issue was that servers are reset daily around 7pm and load distribution during low-volume times were favored on Server1, therefore Server1's JIT had been optimizing for much longer than Server2. Lastly he talked about high performance Java/C++ (dos and donts, object creation, collections, GC, etc) and how the NASDAQ has written large amounts of their platform for the JVM in base Java syntax with all custom rerolled libraries.
He said that if you want extremely high performance, you should design your application for this from the start and most importantly less is more. He didn't really go into detail but skimmed over a slide with these things (there's more, but I can't remember, most of the standard stuff I guess). C++: No exceptions, eliminating temporaries, know what code is being generated for you, virtual functions, profiling STL functions and replacing them with your domain specific version, stack vs heap, designing for locality, then a quick note on cache, branching (use a bit mask instead of branch if you can) Java: A lot of the same principles as from C++, use JVM 1.6 and profile between versions, object reuse, prefer primitives, avoiding things like bounds checking on array access, presizing Also mentioned OS. They used IBM AIX and now Solaris on Intel Nehalems. Infiniband with kernel bypass.