J, I currently have six systems for many different types of instruments: equities, equity options, financial fututures, index options etc. Only one of them searches through massive data for opportunities. Your suggestion of using a cluster would work for this system, but for technical reasons it is not possible at this point. For the rest of my systems, I use some of the statistics during trading and other stats I just gather so as to find ideas that I do not currently consider. The computational demands wouldn't be much if I were considering just a few symbols. For my futures systems, there is very little computational demand. For my equity and equity options systems, the computational demands are great because of the sheer number of symbols and the combinatorial explosion. It is when you combine lots of symbols and lots of computation per quote that you need the fastest machine possible, or equivalently a cluster. nitro
...and it is in these cases where your software design and implementation makes or breaks the feasibility of your computations. It is very difficult - and thus expensive - to take a system that was not designed to be run in cluster or parallel and refactor/rewrite/redesign for a different computing topology. In most cases very little can be salvaged. Our software accomplishes quite a bit in a 6 hour run - which came down from an initial 11 hour run after we ported it to a parallel environment. This was only possible because we designed it from the start to scale to a cluster/parallel configuration. It was also designed to be as vendor neutral as possible.
Yeah you do run a massive quantitative operation. Are you having to prune your list of symbols or calculations to best balance computational limits with expected or actual profitability? I am in this situation. Even though I only consider ES, NQ, 6E and ER2 currently, I can theoretically diversify between any number of system variants per symbol. I can add symbols too. Capital isnât an issue anymore. Multiple systems can vote on the trading of a single contract per symbol if need be. The problem involves having to balance the number of systems and number of symbols with computational load (real time and non-real time setups) and management time.
But I'll disagree slightly on one point... Your argument is probably only correct for poor designs. Any sensible programmer nowadays designing a system for real time processing of multiple market data will naturally code it as a modular, multi-threaded or multi-process architecture, perhaps with TCP or UDP connections between processes and between the quote server(s), systems and execution server(s). Tasks should be segregated into quote aggregation, processing, execution and segregated by real-time-critical and non real time critical. This is what I did with my systems. One process handles the time critical timestamping and data logging. The actual system processes are non-time-critical and run at lower priorities. These could be migrated to other nodes tomorrow. Yet I never actually designed this with clustering in mind. I was just trying to get it to work on a single machine. Backtesting, if done properly with one's own code is even easier to adapt to a clustering topology. Often there can be little need for inter-node communication, such as the case of splitting an optimization problem across dimensions of parameter, market, etc.
Like I said, the futs systems don't have any particular computational needs. It is my options and equity scanners in realtime that do. For those, I do have to compromise what I look at since even a quad would get overwhelmed very easily, or by the time it got to analysing the opportunity, it would be gone. I am moving to more transient scans for equities and options now too, so that will add to the computational load on the machine. But the need to get these transient opportunities as quickly as possible will not be necessary for these types of scans. nitro