I understand what you are saying, however if you are separating your backtesting algo from your live algo you might be introducing some issues without being aware of it. Perhaps you will use your approach for prototypoing?
Yes, for prototyping, if you will. Basically, I do 'research' (plotting, screwing around, wrestling with matching indexes) in MATLAB in a exploratory fashion. This is done mostly with discrete data and is separate from Backtesting. I totally hear you on the need to have essentially the exact same code running live trading as the code I backtested with. My Backtesting platform (work in progress) is designed with a 'broker' object and 'exchange' object completely separate from my code. This is done to a) make the brokerage and exchange functionality something I can change easily as I learn more/move to different venues b) insure that going from Backtesting to live trading is as close as possible to just placing my code in a for( ;; ) loop and commenting my sendOrder() function out to whatever the new equitable by is.
I like how you think, I wish you great success in business. Also, at your level, I think you need to speak to more reputable sources about creating your db for data or contact resellers of that data and snoop around for technical info to get more clues. Im not a db specialist at all, I do flat files Works for my game. Also for derivatives market, you might need to start looking at gpus at some point, because you might not be able to backtest enough in a timely fashion depending on complexity of your logic.
Good suggestion. Yeah, I just got a quote from Nanex on data and will be pulling the trigger regardless of whether it comes for Christmas or not GPU computing is really cool. I've played with CUDA a little bit, and have in the past wondered if there are ways to parallelize a whole backtest (not just certain operations) on a GPU, but ive never really tried. I'm taking a class on scientific computing next semester and another on numerical analysis for finance later, so hopefully I'll get to learn by implimenting some stuff in class.
Nanex is very good for some things, however for live equity data its no good no matter what they claim their efficiency is. Their servers are too far. You can definitely run many tests using cuda at once.
I'll be sure to look into the CUDA further. My current GPUs are pretty good GTXs, so they are solid for CUDA in fixed, but not as much for floating. I'm sure there's some way I can do this at school if I need more horsepower, because those TESLA GPUs for floating point scientific computing are $$$$$$. I'm still not even really sure what I'll do for live data. I have IQFeed for their breadth of historical data at a low price right now, but we'll see. I spoke to some of the IT folks at the clearing firm im looking at use and they seemed to think using a proprietary feed would just me down vs whatever it is they do. I'm not super on top of realtime data feeds because I'm not currently live trading. It's a ways away though
IQfeed is a decent option for sure, but once again, not for equities. They have servers in Chicago though as far as I remember.
Unless something has changed recently, IQFeed disseminates out of Nebraska only. Nanex is in Chicago.
I believe they do dessiminate from Nebraska, but it's of no importance to me because I'm really only interested in their historical data offerings. I will keep IQFeed for their wide range of historical data I can use, but as I've discussed on here before, I don't think they provide enough historical data to completely reconstruct the incoming feed, which is one of the objectives of my backtesting program.