I was merely arguing with your broad generalization that it is difficult and expensive to take systems not designed to run in cluster/parallel and adapt them to such a processing architecture. The transition is trivial where inter-node communication is minimal. Just split the data and/or parameters to survey among nodes. You are probably talking about the opposite extreme with high inter-node communication requirements, something most intelligent system testers/traders would try their best to avoid from the start. High inter-node comm. is more appropriate for massive computational fluid dynamics or finite element simulations... 3D or 4D data, not for analysis of 1D or 2D market data unless one is doing heavy statistical arbitrage between 100's or 1000's of markets or something like that. Otherwise for most market data analysis problems you rarely need a high inter-node comm. architecture. I have designed my architecture to maximize memory locality and to naturally split the problem into multiple independent optimizations that can run for hours without any inter-node communication. That's a scalable and easy solution, attainable by anyone who understands the basics of algorithms and processing architectures, for both backtesting and real time trading. Again you generalize too much. As long as there isn't any need for extreme inter-node communication, it does not matter so much how the code changes. Just distribute the new code among nodes and implement a method to assign different data, markets, or parameters to each node. Problem solved.
Seems like you run a semi-homogenous group of linux machines with admins who are well experienced. It is non trivial for a user of one machine, who has to devote time to administer that machine, without the extra experience, once they have extensive customizations, hardware specific customizations (eg 4 LCD panel display setups), several applications, applications/tools only available in source code, etc. They have to manually update the kernel and libraries over time. They can't just switch distributions or computers that easily. Contrast this with Windows where the application and driver installs are fast and easier (assuming one uses quality hardware, not cheap crap) and system updates are automatic.
Well I agree with this statement. In our company this was part of the argument made by people to keep the status quo: look, if it is not broke then dont fix it. For our operations we viewed our relationship with $soft, $bm, and $racle as broken. I simply forced people to adapt - they were given some training. Now we are in a position that we have IMHO better quality systems at much lower TCO and development costs and everyone is at least as happy as before if not happier. When we added up all the costs - even including training we achieved a higher performance, lower cost environment. I dont see why an individual could not apply our experience to their own situation and lower their costs and improve reliability. Perhaps $soft and the others will change the way they do business and change their pricing and the way they treat their customers: I was not interested in waiting to find out ....
Not all classes of optimization problems and/or optimzation requirements fit into this category: a whole bunch do and a whole bunch dont. Problem solved only if the needs fall into your problem classification.
Interesting how we are both 100% confident that our choice (Windows or Linux) is the most cost-saving and time-saving side. We have both invested a lot of time learning these systems, acquired a great deal of admin techniques and the right tools for any problem. We may never agree that the other system is easier, because from our perspective, what we know is easier and faster which tranlates to less cost. Ten years ago I was arguing for Linux on the grounds you are, including initial cost, and obvious technical superiority. Now Windows and Linux are very similar in both regards. The cost of Windows is insignificant compared with the time spent administering either system. The fact that Linux is free is merely an immediate advantage, especially for large groups of machines like the ones you use. What matters most nowadays is the long-term functionality, stability and maintainability of these systems. This is very hard to judge because it depends greatly on the particular application and the competency of the admin(s). Probably itâs best for people to try both and also get two books, one on administering Linux and one for Windows.
We're not talking about all classes of optimization problems... just ones that apply to market data. My contention is that analysis of market data can almost always be done with a low inter-node communication architecture. Why? Historical and recently received real-time market data is static, not evolving in 3 dimensions over time like the physical simulations clusters and supercomputers are really needed for. Because the input data is 1D or 2D and static, you immediately cut down on inter-node communication for the input data, real-time or not. Secondly you can load blocks of market data into local node memory, evaluate the full parameter space, cache the (partial) results and repeat, on a single node until all data is surveyed, regardless of how much data there is or the number of parameters to test. Local memory requirements are minimized and there is no need for distributed memory operations. Other nodes can process other data, or different parameter spaces, but probably not both the same data and parameter space simultaniously because that may necessitate higher inter-node communication for the integration/accumulation. This a transpose of sorts for the typical load-all-data across all nodes, simulate, consolodate results across all nodes, simulate again which is used in physics simulations. It is wrong to assume market data should be processed like physics simulations.
Ten years ago I was arguing for $soft: In fact when the costs were computed it was the cheaper solution for a variety of applications. Today the situation is completely reversed: Software talent is cheap and getting cheaper ... Hardware and computing power is cheaper per operation than it was ten years ago etc. The thing that has changed are the costs and quality of the OS: Linux ten years ago was not a good solution for most computing needs. Today it is and it has a very good desktop solution. The costs of admining are not insignificant: this is FUD propagated by software vendors. Add up 1, 2, or ten fewer people over 5 years and that is not a small amount of money. Also, development is cheaper on Linux since it is more transparent and the vendor is not changing development API's and technologies every three years ....
I find that odd because ten years ago Windows was technically a piece of crap compared to Linux and other Unix or VMS systems, for which Linux was rapidly becoming a credible substitute for. True software talent, the kind needed to build and deploy profitable trading systems is still very hard to find and very expensive. Maybe Linux admins are cheap. I meet Linux users all the time and try to figure out why the actually like or need Linux. Usually it's because Linux is cool or free, not because they doing any software development or running a server. Linux was and is still great for software development and server apps. Windows is still preferred on the desktop, mainly because of the availability of user applications, and ease of installs and updates. Everything depends on the particular application and competency of the admin. The APIs for Linux systems evolve and go into and out of style just as often. Remember Xaw, OpenLook, Xview, motif, Tk, KDE, Java⦠each useful for certain things, or popular at one time, yet none of them offering the range of functionality and performance available with win32, even if win32 is a bit more messy. Personally I prefer Matlab's GUI support, which is fast to develop and OS independent, bringing able to tap all the quant advantages of Matlab in a single unified package.
Windows was far far cheaper for some applications ten years ago than unix or mainframe apps. The thing that has changed is the cost of labor. .. and true software talent for profitable trading systems is easy and cheap to come by ... the only folks that continue the argument that it is scarce are those looking for work. Windows is preferred on the desktop only if you are under mandate to keep it: All of our trading apps run under linux and we have had better reliability and lower software development costs since switching. Our total cost for our linux infrastructure versus Windows infrastructure is much, much lower and these savings roll on year after year while with $soft $bm, and $racle the only constant we can depend upon is continuing costs and churning changes designed to increase our costs. Gnome and KDE could go away as could any other environment.: all of our people are competent enough to handle the switch if it should become necessary and we dont expect it will, especially with national infrastructures being based upon these platforms.
Of course there were also apps that needed the scalability and reliability of unix or mainframe systems, which Windows couldn't do. Like I said, there are examples either way. You just contradicted yourself. Why would any programer able to *actually* produce profitable systems be "easy and cheap to come by"? They must like volunteer work, right? Where might one find these cheap and proftitable programmers? I'd like to hire a dozen. Yes, Bill Gates has a gun to our heads, right? I never realized we were talking about large or inflexible business environments. I'm glad Linux works for you. If I was in your situation with many machines and good admins, I'd probably experiment switching over some servers to Linux and go from there. I actually like KDE in some ways, especially because it's ported to Windows. I was merely pointing out that APIs do change on both Linux and Windows