sorry if I contributed in creating such impression. As I pointed out I think R and Matlab have their applications. I was trying to assess what the OP was looking for and believe I gave the best advice that my experience taught me. Others may disagree and I respect it, I just tried to back up why I think R and Matlab are more tools of quant analysts and strategists (who rarely put on trades and manage risk) vs. someone wanting a set of tools that can be fully customized and expanded to also handle live execution later without having to change a line of code. Someone who just wants to profile couple ideas and does not have extremely computationally heavy and data heavy requirements should go with R/Matlab/Python. But when speed, latencies, throughput and complex strategies come into play then I would always go with C++ or C#. This is admittedly just my two cents and I am not saying this is the way it has to be because some friends of mine work at this or that fund who go a similar route, but I think it is pretty much established fact that the biggest exotics are functional language shops, then those heavily using interpreted languages while the vast majority uses compiled languages. I would rather go with what the majority of professionals use (given its within the budget) rather than trying to look for the easy route.
my data's stored in flat files (csv) indexed sensible for fast(ish) reads. my analysis is done with python, kicking out data parameters for use daily with excel. in excel, i run on tick events, have watch "it" run tens of millions of times a trading session, but governed down to 10-20k runs lately. speed's just not that critical, and i'm never going to compete on it anyway. i trade with ib. they have a c++ api, and i've done a lot of straight c programming, but prefer quick development / deploy to dealing with that. i'm competent at java, but again, trying to use newer better tools for development and trading.
i can think of no bigger waste of time and resources than attempting to compete with the big boys on speed / latency.
whatever works for you ...good luck with that. By the way this was not my point, but its ok English is not my native tongue or not everyone gets my points. Cheers.
I certainly wasn't knocking your comments. Your comments actually have been very helpful and gave me some 'things' to think about
Out of curiousity, are you saying that you can push 5 mil ticks per second through a full backtest? Say 6A for instances with five depth levels maintained per side and probably about 1.1 mil diff messages per day (just looking into a random file I've got from last year), ie fair a bit of activity to be considered when fully updating/ generating a full snapshot of the book at each event plus your strategy on top of it adding, cancelling, modifying away, keeping track of your positions, resting orders,... - this will complete in (less than) a second? I'm impressed! I know of a few funds that won't be able to come even close to that (though a lot is legacy related).
I trade fx, thus quotes are just bid and offer price updates, no depth (its irrelevant on the fx side). But the data is sent over the message bus anyway in byte arrays, so adding order book deltas should not create much more overhead. Yes about 5mil messages per second from the time the back test is started until the last tick is processed by the strategy. However, it depends on how many symbols the back test entails as more symbols will weigh heavier on the merge/sort algorithm. It also includes the updating of the OMS/PMS and Risk modules (OMS for child limit order book, PMS to keep track of open positions, Risk to manage positions in order to stay within risk limits). Please keep in mind that I coded everything from the ground up to run concurrently and I use a new actor model data flow library which seems to perform very well so far.