Hi guys - I'm a bit of a newb using databases so I'm hoping that someone can point me in the right direction. What I would like to do is build a central database of real-time price data which my various trading apps can all use to make their trading decisions (rather than subscribing to the external price-data feed from each individual app) I'm sure it's easy enough to take real-time price data from my data-provider and store it in a database, with a different entry for each stock. Is it then possible to trigger an event to notify each program reading that database that a data item (price) has changed? Is this a standard DB feature / is there a specialist product that can do it? If not, is there a simple alternative? Thanks for any thoughts FWIW I program all my apps in C# (.Net)
Google "SqlDependency" or "Asynchronous Sockets". You'll find sample codes and do what's appropriate for you.
thanks - looks like it is very possible to do what I want. Now just need to figure out the simplest to implement. The shorter my development time, the better!
a database will never work. look into a messaging layer like http://www.spread.org/ http://activemq.apache.org/ http://jira.amqp.org/confluence/display/AMQP or if your a strong programmer http://www.boost.org/doc/libs/1_37_0/doc/html/boost_asio.html http://mina.apache.org/ http://code.google.com/p/lidgren-library-network/
This is a better recommendation than mine. I strongly support and can't live without all the gadgets from Apache.
These suggestions are, from my point of view, pretty large-scale solutions, appropriate for an enterprise. Since you're using C# (.NET), Windows is a safe assumption? Fortunately on the Windows platform there's a simpler, and 'closer to the metal' solution. Use COM / C# events. These are by nature multi-subscription interfaces. Construct a simple application that reads your real-time data stream and then 'fires' appropriate events to its subscribers. This resolves for a server app with subscribers on the same machine to, in the simplest case, a callback function. For servers and subscribers on different machines DCOM is used which is based on sockets, TCP/IP, etc., fortunately all hidden from view. From your code's view there's no difference - you are simply providing handlers for a defined event interface. If you are using the MS Visual Studio IDE, it can 'lead you by the nose' to create a C# (server) app that exports an event interface as well as an app that can connect to that interface. ( I'll admit my experience with C# is limited in this case - I've constructed lots of objects with event interfaces in C++/ActiveX/COM/ATL/MFC over the years and the stuff 'works as advertised.' From the C# docs I've perused, that event mechanism is available in C# and appears to be even easier to use. ) Hope this helps I'm a newbie at posting on this forum, but I'll be glad to help if there are additional questions. PS - I use Interactve Brokers and they use exactly this mechanism to communicate real-time price data from their secure app ( manages logins ) that receives this data and then provides it to other 3rd party apps, i.e. AmiBroker, etc. They provide a documented ActiveX interface, including events, that might serve as an example of what I'm pointing to ...
You probably won't need it but I had to deal with receiving socket msgs over SSL. It's good to keep it in the back of your head if you ever have to send/receive msgs from remote servers. http://www.example-code.com/csharp/socket.asp or if you like to use generic stuff: http://msdn.microsoft.com/en-us/library/system.net.security.sslstream.aspx
great stuff - thanks very much to you guys for taking the time to lend some constructive advice. my main goal here is to create a simple solution which can be implemented as quickly as possible. ideally i would like to be able to send .net objects from one appliation (server) to other apps (clients). guess i might have to serialise the objects in some way? for obvious reasons, most of the tutorials on the sites use text messages in their examples. i might need to hook up 3-10 clients to the server - not hundreds - so no need for an enterprise level solution. i appreciate the advice to get back to basics and use the in-built netwroking solutions in c#. however, the idea of a ready-made api vs making my own networking objects definitely appeals (hope this makes sense). my c# skills are improving but I am definitely not a strong programmer so anything which will require advanced programming skills is not gonna work. i am sifting slowly through some of the docs on the many website links provided. i must admit to being slightly overwhelmed by the options. any further advice on the simplest route forwards would be great. i hope that doesn't sound too lazy - i am happy to work hard on this but always helps to get a running start. cheers!
Are you planning to test this system before you go to production? NOTE: I'm not necessarily referring to "back testing" just testing. If so you will go through a lot of pain. Why? You're speaking about putting real time data into a database. Before you spend any effort on this, try a simple test with any database you prefer. Create table with the commin real time data fields you want like bid/ask/last, time stamp, etc. Then write a small program to load a few million rows of random data. Measure how long it takes that program to run. (It will seem like forever). Then try to select all the rows out of the data based and simply write out the average of the bid or ask or something and measure how long that takes. If you want to save time, I tried it as 15 year expert at database performance tuning. The time is unacceptable. It can take over an hour or more to this processing. When I was at Verizon we did million record transactions at times which ran for 7 or 8 hours as overnight jobs. real time market data doesn't work in standard databases. In contrast, if you only use 1 minute data, you could make this work.
This is fine for real time data. Again, like my last message, any system built like this will likely need some testing if not to develop a strategy, to at least verify that all the calculations in the strategy are giving expected results. To do that over even a vew months required millions of ticks of real time data. C# events (because I tried this) don't work for that. It's because they are comparatively an extremely slow method for firing events when you do them in a loop 2 millions times. Try it. Set up a little C# program that does nothing but fire an event to another class. Then execute that 2 millions times and measure the results. It will take at least an hour or more. Then do the same program with a simple call back method instead of an event. Run it a million times. It will take on a couple of seconds. If you want to use a really fast system for testing and deploying a trading system then there's only one viable solution out there that I know of that passes the muster in performance.