Some clarification is in order : "algo trading" is clearly an overused term. The 100K to 400K LOC servers I talked about were from my line of work : sell-side broker algo trading systems that do order management and DSA-style strategies a la http://www.ibb.ubs.com/marketing_campaigns/dsa/strategy.shtml. As an example of order management, think of what runs behind the scenes at IB's servers connecting to multiple exchanges, combining market depth to give you a consolidated order book / NBBO in real-time, executing your orders, etc; I was not referring to the kind of strategy that is often discussed on ET (opening range breakout, stat arb etc. This is my day job, and the technology really matters in controlling complexity, reducing bugs, and scaling up without a performance hit. Apologies for not clarifying the context!
TigerBalm, No worries. I understood it. I used to trade in-house with LEH for a short time and have a 1.5st hand knowledge of what's going on behind the scenes at a sell-side firm. Your "day-job" inputs are very valuable. Thank you for your post.
The issue I see with Clojure is that I have heard it said that the JVM is not well-suited to parallelism, so that you would not see the performance gains in a many-core environment that you can get with other functional environments.
I do not have experience with Clojure, but it runs on a standard Java JVM, which typically handle concurrency very well. I write multi-threaded Java middleware professionally, and the concurrent performance that a modern JVM can provide is exceptional.
See http://java.dzone.com/news/javaone-brian-goetz-concurrenc "The tools we have in Java 5 are good enough to find coarse-grained parallelism (usually at the unit of a user request) and spread it over a small number of cores (2-8). However, these tools do not scale up to many-core boxes, which will become increasingly prevalent. The shared queues and other infrastructure used by executors and thread pools becomes a point of contention and reduce scalability." Also it requires very skilled programmers to write thread-safe code in Java without horrible bugs and bottlenecks. Most of these problems go away in functional languages that inherently make no use of shared (global) memory.
Brian Goetz is great, his concurrency book is a must read. But I think he is arguing that the current libraries in Java don't scale well to multiple cores, which is why we need fork-join. The JVM itself scales pretty well to many cores, and a tool like Clojure would presumably be able to utilize the JVM as well as the fork-join framework.
A new virtual machine design (as well as language design) is needed for best multi-core performance. See http://geek.susanpotter.net/2009/04/scala-vs-erlang-debate-part-2-geek-off.html "The Erlang VM was built from the bottom up with lightweight processes (a.k.a. actors), therefore it is stackless and much better at lightweight concurrency. Erlang also has a built-in mechanism for load balancing these lightweight processes across all available CPU cores. This means Erlang has a massive advantage over JVM based functional programming languages at getting the most out of multi-core hardware, which is at the crux of the problem domain that software needs to address going forward. Scala vs. Erlang: Performance In terms of linear performance (sequential) Scala has been shown to beat Erlang in certain cases. However, concurrent performance consistently shows Erlang beating Scala, sometimes by wide margins on multi-core hardware. Not surprising considering what we learned in the concurrency sections above." Benchmarks are showing functional Erlang trouncing JVM-based Scala in multi-core environments. Now, I need to add that OCaml is not good at multi-core yet, but will be once the new parallel garbage collection is released. Erlang is already there. JVM-based Scala is not. Nor is Java itself. I think Clojure would be the same. Here is another article: "Multi-core may be bad for Java" http://www.devwebsphere.com/devwebsphere/2006/11/multicore_may_b.html
<i>No offence you commuters but if you cannot ride a horse none of these fancy cars help you. Technology is a trap. Simple ideas make money. I know people who still raise horses. I know many others with 10 cars who constantly commute poorly. Oh yeah, V8 engine cars transit for faster stupidity are here. That will help geeks to drive faster and get more tickets.</i> ... We've heard it before.
Fork-Join is idiotic. The whole point of multiple threads is to make use of the fact that you have the same virtual address space being utilized by different threads of execution. You either need this for performance or you don't, and you can use it correctly or you can't. Look at the i7 and Intel's quickpath architecture. Why would these people go through all this trouble to bring you integrated memory controllers and point-to-point connections if you're going to create disjoint address spaces? Even with SHMs, you are still looking at larger context switch penalties. What fanciful world are you people living in where these low-level details don't matter?