Question to programmers... or system traders who uses real programming language for their real time trading... What do you guys use to manage data? SQL? Ascii? What do you use? Or what would you think is the most efficient way to store and manage massive amount of market data... I'm using GlobalServer from TS but now that it crashed and I had to re-install TS2000i and all the backups (which took me the whole night without sleep) , I'm seriously considering separating from TS. Well, any thoughts???
If you only look at speed, most programmers go for simple flat files, often binary files for speed and space. See here as well http://www.turtletradingsoftware.com/forum/viewtopic.php?t=980 If you need a database for reasons like querying and searching within your data, Aberdeen Group has recently done a study on Open Source Databases, comparing them to enterprise, ie. commercial databases. Very good results for SleepyCat, MySQL and PostgreSQL. See here http://www.aberdeen.com/ab_abstracts/2004/03/03040006.htm If you use Java and you need a database for maintaining data integrity while not loosing speed, look at Prevayler ( http://www.prevayler.org/wiki.jsp ). It claims to be more than 9000 times faster than querying Oracle, and more than 3000 times faster than querying MySQL through JDBC. Hope this helps a little, agrau
I'm a system trader using my own C# application. I am not so familar with DataBase stuff, but I always found it's extremely easy, flexible and useful to employ DataSet/DataGrid in.net framework library. The data format is XML. It's not the database with query function, however I understand you won't use query for market data. The market data structure is simple, timestamp, price, volume, right? http://www.codeproject.com/csharp/PracticalGuideDataGrids1.asp http://www.codeproject.com/cs/miscctrl/DataGridZen.asp http://www.codeproject.com/cs/miscctrl/cprogressdatagrid.asp?target=datagrid and more you can serach on this site. Feel free to ask me about C# stuff although I know little on SQL etc. Good Luck.
Thanks... I'm also a C# guy, and looking into QuantStudio. Well, I do a lot of tests under a very large data so I need to have a relatively speedy data management and access along with the platform. Well, thanks for the link. Please keep em coming...
Cool. Good to know you use C#. QuantStudio should be defnitely helpful. I think with QuantStudio and .netFramework itself, you can have it all. As I said dataSet and dataGrid looks enough and it's extra to install extra component such as SQL.
1) revise your backup strategy. For me to recover takes less than 15 minutes. I have two sets of backups: One "pristine" that has no data loaded and one "current" with last weekends backup and data loaded (am running 24 hours, use incremental data backups every 12 hours and swap it to another hard disk) 2) stay away from SQL based databases, 3) binary files are the most likely the best for the amount of data stored when using ticks 4) see if you can find a database system that has the data in memory (memory is cheap these days) 5) make sure to have a system that is "powerfail save" (used to be an expert in the massive database field) lost touch with the recent developments (last fours years out of the industry ) and am slowly familiarizing myself with C# what charting package / software are you looking at to use have you thought about getting an SDK and write a replacement for the GS? Peace
Actually reviewing QuantStudio.net again, this platform and C# looks perfect tag for data management and system testing etc. QuantStudio itself is written in .netFramework, probably in C#.
Use either a combo of SQL/Java/C# or a combo of Text/awk/C. Perhaps Excel is okey. But html/word stuff is definitely junk for storing data. It is a mistake that some brokers only provide data and report in html and word. Excel data is also inconvient and expensive to access from automated processes.
That is the difficult part, you can "log" changes in another file (in memory) and swap that to (another disk) rapidly. If power goes down then you may "re-apply" the changes. This is if you do not swap the main database out to hard disk very often or if the main database is slow to update the data on disk. (eg Oracle which has on average 8 disk writes for each update / insert) Alternatively you work with two databases in memory and then "rotate" between the databases, taking one off line for backup and then using the other, gets all very quickly very complicated. You may want also to take a look at http://www.codeproject.com/cs/database/MlVlHyperNetDatabase.asp ( have no personal experience with it and do not know the performance.) Hope this helps