Is there a linux, open source, command line tool that allows me to backtest, optimize, walk forward a strategy/indicator? It has to be able to handle HUGE ammounts of data and many instruments at once without needing to keep it all in RAM. So I want to be able to say for example: ./backtest --symbols AAPL,MSFT,GE --time-frame 1m --indicator macrossover --parameters period,20m --units shares --size 100 --date-start 20011001 --date-end 2010101 You get the idea. It doesn't matter where it expects the data, I can get it in that format, .csv, mysql etc.
R comes close I guess but I am not sure it can handle very large data https://r-forge.r-project.org/R/?group_id=316
nitro, I once used R in an Amazon EC2 large instance to handle large amounts of data for a machine learning competition (predicting links in a social network). To be specific, I used the m2.4xlarge instance, which gave me 64 GB and 8 cores. The only issue I had was human : I did not optimize my code and my prediction algorithm ran for 26 hrs!! R 2.15.0 had no problems munching on my data. However, despite the newly introduced compilation to bytecode, it was quite slow than, say, a C++ application. I suppose this is why firms which can afford it end up going to kdb for the REALLY high frequency backtests. However, as I'm sure you'd know, the learning curve there is quite a pain and not worth it unless you can hire someone to do it for you (and if / when they leave, you'd be left with some weird looking symbols that mean nothing to you)!
AFAIK with R you need to read the entire dataset into memory, I definitely know that's true with zipline. Another issue with R is that whenever a process forks, it takes the entire dataset with it. E.g. You're operating on a 20 gb file, have the process fork 4 times then you'll be using up 80 gb in ram. I think your best option is going to be code something up yourself. I ended up doing something in python. Presumably your data is in some kind of flat file or database. Just read the data line-by-line and have your order types, logic, etc.. implemented in classes. It shouldn't be too difficult. You could also consider using Esper which is a complex event processor for either Java or C#, it's reasonably easy to use.
Absolutely beautifully done! I am impressed. Rarely do I get impressed by software, but this person knows how to code: http://gbeced.github.io/pyalgotrade/docs/v0.12/html/tutorial.html It even runs the workers on a cluster!!!! Major Kudos. Now I have to see if this thing can handle huge amounts of intraday data.
Thanks for posting! You amaze me sometimes with some of the interesting content you bring into ET. I'm gonna have a closer look. Question.. but then use different language to trade off of? I'm not sure how I feel about that. Do you use python to trade?