has anybody tried distributed computing for backtesting? I am thinking of NVIDIA GPU computing, which is quite accessible on desktop... Anybody had experience? How does it compare to distributed computing via building a network (the maintaininence of several computers is really troublesome)...
Brain usage helps you a lot. With some thinking you may get quite most of the answer sorted out without asking. * I do not think that Cuda style systems are applicable to most algorythms (and that is what Nvidia does). Problem is that the calculation of the indices may be accelerated, but all the trading simulation is harder. I think the benefit may be minimal. The main problem here is that most algorythms are not exactly computing intensive in absolute terms. * Network distribution may work, if one has a network to distribute (and most traders do not). Seriously, the range where this makes sense is really high - first you can upgrade your main system to a dual opteron (now 8, soon 12 cores) and take advantage of the processing power ALSO for other things. Getting a network (of mulitple4 core systems)... what you want to use them for otherwise? No system I know of supports either. I plan writing my own framework, and I may support the second option. Not the first, because It makes little sense to support CUDA and another langauge or strategies, and one can not rely on cuda eing there, and second, as I said, I think the benefit is not that great. Network - well, happens I HAVE a couple of computers lying around. But before I do that it must make sense - which I do not see (mostly because I do not really think parameter optimization is a good approach, and if I want to try it out occasionally, can wait. Helps that I currently have an 8 core server But at least I can tell you that I have around 7 servers running all the time anyway with low resource usage... CPU wise (pretty RAM loaded). In this case network distributed backtesting may make sense.
Whenever I get into CPU ressources scarcity I try to think of another way around. There will always be a too big problem for your hardware. One time I did a genetic algo instead of going throuh all the possibilities Another time just random parameters that I run as long as I want/can. You can start from the results you get and play around. It can give you an idea or a go/no go. Assuming you already optimised your code, you can look at wjere is the time consuming part and try to reduce it. There are so many ways. Also, the language/program you use can have a huge impact. When I have some real crunching to do I try to do it in C++. I dont know if the packages available (TS, ninja, Wealth Lab,...) are efficient for big problems and optimisation. I'm of no help to you but I can tell that for poor fellows, there are ways out without the big tech toys.
Why build a network when you can rent on-demand scalability with cloud computing services like Amazon EC2 or GoGrid.com?
Not for backtesting, yes for realtime computation. Distributed Computing is an order of magnitude easier than GPU computing at the current time. The programming tools and abstractions for GPU computing suck. I think MSFT is working on making it easier to program GPUs. Google bought out a company that did a good job of writing tools for this sort of thing, but they then turned the project internal AFAIK. For backtesting, if you have spare Pcs around, that is far better way to go since you are only wall clock bound, not realtime bound.
Mizhael, It doesn't sound like anyone has successfully used NVIDIA CUDA. In response to the suggestion of cloud computing, I've run some tests (after much installation effort) and found the results unimpressive. At UC Berkeley I have friends in the parallel computing lab (parlab) who tell me CUDA is actually relatively easy and extremely powerful. If you use Matlab, you could try this software, Jacket, which makes GPU processing plug-and-play: http://www.accelereyes.com/ I haven't tried it but it looks cool and it's only about 100$ (I missed the beta). Right now I don't have a gfxcard. Regards, Max
Wow, you gave me some very good pointers. I do use Matlab and I happen to have two computationally intensive tasks on hand: (1) Process high frequency tick data and run backtest on these gigantic amount of tick data; (2) Run gigantic size of Monte Carlo to get my probabilities. I will give the Cloud computing and the GPU Matlab interface a try... Great pointers! I happen to have the lowest end GPU, G8400. Originally I thought I have to convert my Matlab code into C, and then into NVidia Language, which is tough for me, given my short amount of time and lack of experience in low level computing... Now it seems there is hope...
Not only does Jacket look like a very nice program, that has to be one of the cleanest and easiest to navigate websites I have ever seen.