MySQL, Linux 2.6.15 kernel, Athlon 2800+, ordinary IDE drive: 7.24 seconds to select * from a 700K row table of tick data.
I've seen this question answered many times before in various forums and the experienced programmers always respond the same. Don't do it. The ratio of speed of memory access to disk access has increased over the years. Disks have become relatively slower and slower as compared to RAM. Disk access is still measured in 1/1,000ths of seconds as you need to wait for the platter to spin around so that the disk head can read it. Random reading access is barely hundreds of times faster than it was when I started doing futures research more than 20 years ago. Disk size and linear disk access has increased over the years at the pace of Moore's law or even slightly higher. That's why we have 500 gig drives now and we had 5 meg drives in 1982. You can read in a large chunk of data very fast if it is stored as one piece. Unfortunately, databases tend to store their data in relatively small pieces. So a typical read of a large chunk of test data using a database would require many separate reads of the disk; each of these, in turn, requiring many 1/1000ths of a second. This can slow a large test down tremendously. If you are testing using intraday data, your tests will take a long time due to the sheer number of bars as compared to tests with EOD data. The last thing you want is to slow things down even further. The solution is fairly simple, just store your data in flat files with fixed size records. You can then easily determine where the data is for your next read and you can use simple data access routines to read the data. For example, if your data is stored in 32 byte chunks, you can read in data in some reasonable multiple of 32. You might read in 32 * 1024 * 1024 records at a time. This will be much much faster than trying to do the same with a database, perhaps 10 to 100 times faster depending on the database and its specific settings. If you want to get fancy, you can even do your reading asynchronously so the disk is reading in the next chunk of data while you are processing the current chunk of data. This will give you your disk reads for essentially no additional time. In short, the problem is simple enough that a database is overkill and will end up slowing you down an order of magnitude in the typical case. Even a fast database will be much slower than simple linear disk access. - Curtis P.S. I should note that specialized databases designed and optimized for this particular case could theoretically achieve the same performance as the simple mechanism I note above. However, I have not found that any of the commonly available databases have performance that even comes close to approaching this theoretical ideal. P.P.S. The memory mapped files suggestion by 21Centtrader above is another way to handle this problem. It is more dependent on the specific OS (i.e. Windows vs. Linux etc.) and gives you less control but may achieve similar performance to what I propose above.
MyISAM. My knowledge of RDBs is slight, so I'm not really the best person to be handing out recommendations.
mmap should outperform anything else (all things being equal) because the extra buffer copies from kernel to user space are avoided. As well as using C or C++, you should be able to use Java with the java.nio package for memory mapped I/O and it should be portable. I havn't tried it but it supposedly improves I/O performance. One advantage of SQL is that various applications understand it and ready tools are available for maintaining your data. If you roll your own, you will need to write your own code for this. If performance is sufficent, an RDB might still be a good choice.
Time series data is generally not very well suited to being stored in a RDBMS or being accessed using SQL. In the vast majority of cases, the data is accessed sequentially ("give me every bar after yyyy-mm-dd") and the overhead imposed by the database adds no value. I am not aware of any commercial systems that store market data in a relational database. Most modern operating systems offer some flavour of memory mapped files and I believe you will get far better performance storing the data in flat files.
Recently came across FastDB / GigaBase - both are open source. Haven't evaluated the DBs yet, but documentation indicates FastDB uses the main memory (RAM) to store information; while the GigaBase uses memory mapped files.