MySQL is almost certainly adequate for your purposes. Oracle would be better, developer version of which is available for free (install Enterprise edition, not Standard). If you move to higher frequency than one minute bars, such as message (bids, offers, cancels, executes, etc...) data, then kdb would be a better choice for that portion of your data. The 32bit version of kdb is also free.
Thanks Kevin, I'll look into Oracle, the platform is built with an event based architecture so there are a lot of messages being passed around using the Akka.NET framework. Persistence of aggregate states like orders and positions will be handled by a NoSQL document storage such as RavenDB. I like raven because it's .NET native, fast and has a very clean API.
Yes...I have used MySQL for time series data and it is quite slow. I am rewriting my trading software to read directly from flat files and the performance gain is enormous. Granted I am going from PHP/MySQL to Golang file I/O with flat files with a SSD so the comparison is not apples to apples. Anyhow, do some benchmarks and see what comes out on top. fan27
Wow. That took you quite some time to figure. And I don't say that in a condescending way. But I remember to have recommended flat files in one of my first posts to you after which I got insulted by the sql crowd on this site. Sql was, is and always will be a poor choice to deal with time series based data. Period
It did not take me a "long time" to figure out. My original system/website was built back in 2012 and the preceding years, way before your flat file advice was offered on the thread I started. Anyhow, had not done any new development for about 18 months until the time you offered very good advice to my Go thread I started recently
Right now I am just beginning to re-write my trading system in Go. I am working on a marketdata package that will load various types of time series data, aggregate to different time frames and link them together, etc. The package is currently written to handle CSV data but it can be extended to handle any other data source. It is open source and I will make available when it is closer to being done.
That's sounds great fan27, I'm not familiar with the Go language but I see it popping up. InfluxDB is written in Go it seems. Flat file CSV is how I've previously been storing my data... it works... but I'm not yet convinced it's the all round optimal solution. I'm sure the average hedge fund out there isn't directly working with CSV, maybe just when it's been download from a vendor if they aren't capturing their own. Here are a few links I've come across tonight DB-Engines Ranking of Time Series DBMS http://db-engines.com/en/ranking/time+series+dbms Management of Time Series Data (Thesis) http://www.canberra.edu.au/researchrepository/file/82315cf7-7446-fcf2-6115-b94fbd7599c6/1/full_text.pdf Developing Time-Oriented Database Applications in SQL (.PDF book) http://sql-info.de/sql-notes/developing-time-oriented-database-applications-in-sql.html Time Series DBMS fastest growing in popularity http://db-engines.com/en/blog_post//62
Sorry, I actually checked and I was referring and recommending a binary file data store and query engine, not flat files. Got that confused with someone else. My apologies.