thank you, really appreciate that. I knew about sqlite but was not aware of hadoop for windows. Thanks
What are you going to read? Ticks between start date and end date og aggregated data? Explain a bit about the nature of your queries
For best performance tick data access and search has to be custom designed with pre-programmed fields matching specific requirements and computer memory based with no or minimal I/O operations. It has to be very simple and rudimentary approach suited for live market action. Commercial products are designed to please wide variety of customers and are mostly concerned with data integrity and safety with speed as secondary concern. They are not suitable for real time financial markets.
Actually that approach is not used by investment banks or hedge funds anymore. It has proven to be cost prohibitive to make it work right, it does not scale across asset classes or regions and it is not an open architecture that accommodates change (requirements, algos, strategies). Big firms today utilize either OneTick or kdb+ for their real time capture and historical tick database. Both of those databases are very expensive and not approachable for home-made ATS. For home made ATS and smaller firms, I have found that the TimesTen database is as close as it can get to kdb+ or OneTick (minus build-in analytics & statistics functions) for real time capture and retrieval of tick data. A fully functional development instance can be downloaded for free.
They probably found out that there is no money in analyzing past data (even I they are 1 second old or less) because markets are randomized these days and lots of money is spent to hide tracks.
You would need to write a feed handler. No off the shelf product, that I know, can push ticks into an x10 database. Unlike OneTick and kdb+ that are specifically designed for tick data, x10 is a more general in-memory database that can handle tick data (almost as good). Because of that, several third party tick vendors develop feed handler adapters for those databases only. From personal experience I can say that none of those third party adapters for kdb+ work great and introduce several risks. It is very common that firms that can afford a couple extra programmers, develop their own feed handlers anyway, regardless of the database choice.
Sorry I was not accurate. I realize that the Feed Handlers have to be written and then a program that knows how to talk to TimesTen to send it the ticks from the feed. What I meant is, what does the PERSISTANCE of the ticks? I assume that TimesTen is strictly an in memory db?
Yes, TimesTen is strictly an in memory db. You cannot have more data than your available memory. There is an agent running as a separate process which receives messages via TCP/IP and replicates into disk. This is done asynchronously (in other words, you may loose messages if your db crashes, but that is a 'feature' of all in-memory databases.). I have worked with in memory databases as a user and not admin, so I am not very aware of the internal workings, but I guess TimesTen offers different variations of persistence via configuration, which may influence performance. I have seen environments where persistence/replication has been disabled all together to gain some speed, and at the end of the day a 'save' command writes everything to a disk file.