Before I embark on a crazy project like this, does anyone know of commercial or open source software that allows one to capture order book data and backtest on it? I'm familar with ninja, quantconnect, quantopian etc but none of these have such capabilities.
What sort of strategy are you trying to create from this? For futures products you could potentially do it in Sierra chart using ACSIL, however, their backtesting engine is not going to handle HFT strats, If that's what you had in mind. You would also need to have recorded the data yourself. SC don't supply historical order book data, but they will at some point.
I want to test some ideas I have but I will need to record L2 data and backtest on that. Just wondering if there is anything out there before I have to build it. Maybe quantconnect can do it by hacking their event handling.
Before you think about the software to accomplish what you want you should think about the data source. To my knowledge no data vendor supplies an order book feed but if you have a high quality data feed (with adds, modify, removes) and access to their historical data then you can actually build a limit order book yourself. You would need those data sets to start with. Depending on the exact data source this can be very pricy. Forget iqfeed or eSignal or all that retail stuff, their feeds are way too error-prone to be able to support the building and maintaining of an accurate order book. Unless the order book is absolutely correct you can forget backtesting data on it. (Disclosure, I work in this space)
Futures or Equities? If Equities, then good luck. You need direct feeds from each exchange. To the best of my knowledge, no retail "data" provider currently offer this. If you were serious then someone like dxfeed could provide it. If futures, then it's a lot easier, because of the centralized exchange. I wouldn't go building out your own platform/ backtesting engine just to test your ideas. Your ideas might suck. I'd only do that if you had a strategy worth deploying. If you are experienced in C++ and want to test ideas on historical order book data for individuals futures products, then Sierra Chart could cater for this. Sierra Chart can store historical depth data at multiple price increments. However, you have to have captured the data yourself in real time. https://www.sierrachart.com/index.php?page=doc/MarketDepthHistoricalGraph.php You could programmatically access depth data based on the last update at each time stamp. https://www.sierrachart.com/index.php?page=doc/c_ACSILDepthBars.php