Does anyone know how to create/develop a server-side data feed? Usually, we're concerned with receiving a data feed on the client-side. I had an intellectual curiousity to create a server-side data feed that delivers data every 5 minutes to the clients. Does anyone know how that is done? Sockets programming?
1) you server-side program would connect to data sources (exchanges via FIX, or something like wombat or reuters) 2) it then subscribes to tickers 3) it then publishes them onto a queue (tibco, rabbitmq, 29west, ...) 4) your clients subscribe to the tickers off that queue you could do it directly with sockets but its 2009
Check with your data source and make sure it's ok to replicate the data and send to multiple computers.
I hear ya. I have developed a basic "ticker plant" server as a means of replaying historical data that has previously been stored - in that sense it is delivering " original" data on demand. Clients connect via TCP/IP to this server and make requests or subscriptions for tickers much like you would connect to a real-time datafeed. To build this server I used a basic "protocol server" toolkit called Apache Mina. It's java-based. There are arguably better alternatives available that I might migrate to e.g. Netty 3. Naturally, I also developed a basic protocol - this is actually the first step. I should stress it's very basic and a work in progress. It can also send the messages to a JMS queue/topic so that a single subscription request can trigger a message stream that many clients can read. The idea of this is to facilitate parallel backtesting and optimization but there are other simpler use cases where you would want market data being fed to a queue/topic with multiple clients or "views" having access to that queue/topic. ActiveMQ as the JMS provider has been OK for me thus far. If you have insight into the protocols used for any commercial or open source data feeds and the client API then you can try to reverse-engineer what the server-side looks like. At least in terms of the API and the data format that the server produces. If your server-side data feed is simply relaying real-time data from another data source with some augmentation/transformations e.g. converting NASDAQ ITCH -> Proprietary compressed data feed format, then that is potentially quite straight forward (in principle!) You could look at ESB and related technologies such as Apache Camel, Apache ServiceMix or Codehaus Mule - I can't comment how these techologies fare in terms of latency and impacts on high-frequency data feeds etc. though. Lots of options for a home-made solution. As long as you don't need hard-core redundancy/failover, super low latency, a basic set-up is quite achievable for hobbyists. Good luck.
Byteme, Thanks for sharing your experience and insights. That's what I want to do. Replay historical data or original data across the LAN that feeds into my C# trading app. This is mainly an intellectual exercise. Perhaps I will look at RabbitMQ as a method of delivering and subscribing to data.
Just use multicast and model your design after how ARCA does it with their multicast feeds. You miss a packet or sequence number, and you get it down a retransmit channel. If you're using a feed for playback, just set up your network to bust out multicast data and don't bother with this TCP/IP server/client non-sense. If multicast is too much, put all the clients on a broadcast address and spit the data out. It's ghetto because it forces the CPU to process the packet on the broadcast address, but hey, whatever. Do what needs to be done.