Interesting. Do you know what the performance characteristics of Firebird Sql are in relation to MySQL, or Postgress, or even MS SQL as far as being able to read/write realtime data (bid/ask/size bid/size ask/last/size last) for tens of thousands of symbols? I am just wondering if a SQL store is even the right approach for this kind of app? nitro
It works fine if the data is not stored in relational way. I remembered that some guys did that at Enron for their market data history repository and you knew how slow it was to get a 30 yr forward curve out of database Store serialized arrays or objects in the DB. It's lightening fast on I/O and you deal with OO stuffs all the time. The SQL and ADO.NET will be just a few lines to get data in and out. You can use a file to store serialized objects for sure, but DB is less likely corrupted and easier to manuplate if you have a lot of data in and out. Basically you treat the DB as better file system.
I don't have direct comparisons available right now but you can easily do them yourself. I know that Firebird SQL compared to MS SQL is a little bit slower and also to MySQL. However, it's slower compared to MySQL only because MySQL still hasn't implemented stored procedures and triggers yet. If you serialize the data like trader.net suggested you won't run into any performance bottle necks and i will just run really smooth
Definitely in arrays on memory not files on HD. The intraday data array eats up big mermory, but the performance is the priority. Actually bigger the data become, more the memory access counts. You know why web search engine such as Google is so fast? The servers contains the information as cash on the memory not HD. Good luck.
richtrader, You are 200% right about needing database support besides a language when writing your own app. This is what I have been doing for the last 6 years since I threw my T*n and M*k stuff in the garbage can. I used C++, MSAccess and SQLServer. In fact M$ practically forced me into VB as the only "practical" way to deal with their db stuff. Boy oh boy! I am not going to even try to tell you about "the trouble I have seen all these years". I must quickly add that: (1) I never regret to have started my own app; (2) right now things run rather well with M$. Things like ADO now work about as they should have years earlier; (3) I gave up on the VB.Net and the C# stuff about 2 years ago as it made me vividely recall the bad years when M$ was learning how to do its tricks with the earlier gimmicks. From other ET threads I learned that other C# practicioners are telling us about a lot of nogo stuff. The problem is that you are stuck deep in the muck already before you get wise enough. I will never get caught a second time! richtrader, if you start now, lucky man! Go to linux! Database? Go either MySQL or Postgresql. What has been said about MySQL on this thread is roughly exact. But do you need all this other stuff if you want to make money by trading? I did replace SQLServer by superior Postgresql but I keep looking back at simpler MySQL - many people use it though. You read perhaps about my earlier post about Kylix. For more than a year I used this as a cross-development platform to bring my trading app to a safer more stable linux platform. As of late Borland seems to go slow with Kylix. I can't take any risk here. Am I glad I looked a bit further. I knew about Python but had never taken this really for serious. Now I know: GO PYTHON. (www.python.org). This is all I will say, few will believe me. Of course any implementation methodology attracts big crowds mouthing off about pro's and con's. Anybody who has gone a bit beyond this knows that it takes A LOT of hard learning before you can do anything sensible in any of these environments. That's all folks, I gave you away my best secret, nononsense
Hi dude, Before calling somebody computer illiterate on other ET threads, please do your homework! You mouth off about a lot of things that you don't know a f*cking b*ll off! You just posted the above sequel on this thread: "You know why web search engine such as Google is so fast?" Do you know what language Google is written in? Please tell us know-nothing f*ck-head! It's written in Python! Yours truly, nononsense