Ah - this is trivial. Seriously. I am working on a storage engine now for NxCore CmeGroup feed. My test tape (2010-12-14, I use this as replay for my tests) has about 196754 symbols (unless I have an error there, then there are more). So far I manage to read and store them in about 6 time real time speed on a sql server.
The limitation of this test is most of these symbols probably don't have a single trade on most days... CME has tons of futures, which are traded once a month or not traded at all. There also options on such futures... Do you also happen to track market depth (second, third best bid/ask etc) as separate symbols?
No, level 2 is not separate symbols. I pull complete level 2, but these are on the same symbols. NxCore has a "MarketMaker" field indicating the MarketMaker responsible for a bid/ask, and they code L2 there (same symbol, MM from D1 - D10). We do not talk of CME here, btw. We talk of CME Group. This is CME, CBOT, NYMEX, COMEX, Clearport. The Test-Day has 871 million ticks stored (although I do generate in my adapter some doubles - the Level 2 changes are not properly recognized yet) and I had no problem with this.
That's not the challenge. The challenge is executing code on every time a tick comes in. Some of the issues: -code can do looping and re-calculation -code don't use all the cores( almost 99% of the platforms today ) -code call other things, directly or indirectly...like dlls that you have no idea what is inside. -code can have locking down there in the plumbing code and associated dlls( 100% of trading platforms have this locking problem ). So even if you they are using all the cores.. at the end almost everybody get stuck in the bottleneck of the plumbing code. For the .NET-only-solution-people: now imagine in the middle of that the GC decides to "reclaim some memory"!! :eek:
Actually I did. I run long term loading tests at the moment. Multi core, quite some calculations. Data load running 4 hours, memory consumption stable all the time, although I do generate and throw away tens of thousands of objects per second. Not a single variation in response time I can identify. The joy of the background garbage collector. Bad software is bad - but espeically today you sould spend some time preparing multi threaded code with little interlocks. A higher end processor has easily 4, more normally 6 cores these days, and not using them is nearly a crime.
Have you done any marketing or you are trying to gauge interest from posts like this? A friend who is a developer tells me that the trading software marketplace is totally saturated and has shrunk considerably in the last 5 years, I guess proportionally to market volume. Why do you think another backtesting platform is needed? 95% of traders do not know how to backtest properly, 90% do not know how to program at all. The only model that can work is the NinjaTrader and MetaTrader one where you give the platform for free and you try to make some money from the commissions. I have no idea how this works though. Programming takes time. A small bug can kill you. As a trader Iâm more interested in platforms that design trading systems in automated mode so I do not have to program. There are some programs for retail that do that. They spit out EL and code for other platforms. Two that do not do random permutations are called Trading System Lab and Price Action Lab. The first is horribly expensive for me, even the people I work for find it expensive. It costs as much as a BMW 5 series. The second is inexpensive in comparison, it works well but it is limited to patterns. I would like something in between and I think there is a good market for it if it can do patterns and indicators and if it will cost around 5K, no more. I think that is a good project.
By "institutional" standards it may even sound a beat on the cheap side... (sarcastic laugh) Did you have an opportunity to use TSL? Is it any good? I understand you would happily live without genetic optimisation and indicator precalculation if it was much cheaper.
While I'm not bind to any non-compete. I prefer not to reveal any personal information because I worked several years in the industry and I don't want to hint anybody. No website for this project yet. You are free to email me if you have more questions.
MHtrader, I'm not trying to burst you bubble, but you need to define what you are trying to achieve. Do you wish to create a product that suits your needs and improves your trading? Or Do you wish to create a commercial software product you can sell? If its number 1, then go for you life. If its number 2, then ask yourself how long it would take you to recreate amibroker, which sells for $140 bucks. The answer would be 5 years work minimum. Before you can sell a single copy. My advice is to spend your time creating a c# plugin that extends the functionality of ninjatrader. Something that offers hardcore users advanced features. You'll have a ready made user base because the product already has traction in the market. If your plugin is really good, Ninja might buy it. Alternatively, you can create custom indicators that you can sell on monthly subscription. If your plugin becomes popular, then use the revenue from subscription sales to build a full program later on. This way you won't starve. cheers, Runningbear