yeah..i'm also curious -how the hell you are going to process L2 and T&S(mean change every field) for 1000 stocks? specially,when there is a specific groups of machines are designed to basically jamm you with bs quotes,doing it 20000 times per second. i can understand setup,where you have dedicated a PC for like 10-50 stocks,but not sure,if any data provider would allow this. and even if they do your room is going to look like this-
Actually no, it is quite trivial in most cases. See: * This overload is not on tons of stocks. * Most strategies do not do a lot with level 2. A standard higher end machine (even with one socket) should happily handle a million updates per second, especially if you destroy cross market causality and go multi core processing (which is perfectly fine as lone as you do not need repetition of execution in testing and / or do single stock strategies anyway, OR can bundle the threads to adhere to similar blocks as strategies do). You should be able to easily handle a million or two updates per second on a non-trivial machine (2 sockets).
Good post. But your cost analysis is irrelevant... Because a software engineer that can work with real-time data... Is worth at least $10,000/month... And doing anything intelligent with terabytes of tick data... Storing and converting all that into useful trading signals Requires AT LEAST one very talented, dedicated full-time software engineer... Who will disappear in 3-6 months with a better job offer... (Even though you are competing against people with 7 figure tech budgets).
To add to Nitro's post, more on the performance side of things: First, they have three or four distribution centers... I remember Chicago but forget the other locations exactly - I want to say Oakland, CA and somewhere in the Midwest... Nebraska or something. They do have plans to push data out of NYC (or the NJ data centers) but that is in the works and not live yet. I agree 100000% that they need to be able to push equity data out of NYC and to take it one step further they need to allow for their application to split up the feeds - so you can pull options and futures feeds from a Chicago IP and the equity data from a NYC IP. Right now I have three clients subscribing to NxCore data so over one 100mbit line I am pulling three separate NxCore feeds into three separate servers and three separate databases. A typical day of NxCore equity feed data is roughly 2GB. Really anywhere from 1.4GB up to 2GB but for planning purposes 2GB per day is a pretty safe number to use. A typical OPRA feed day is 6.5-8GB, for planning purposes assume 8GB a day. Also note that this data is compressed roughly 20:1 so yes... when you try to extract this all to a tick level format your database will grow by a multiple of 20. NxCore is limited to 25ms intervals. Each and every tick is recorded and sent to you - but timestamps are only in 25ms intervals. On the equity side you can have anywhere from a few thousand to several hundred thousand updates within that 25ms update time frame. Each message is not given an individual timestamp however they are numbered sequentially. Finally, like Nitro said, this is a really nice feature, they push ALL the data that you subscribe to you and it is your choice what to do with it. This means that you are getting a real data feed pushed to you versus having to use TCP protocol and ask for the data to be sent. In terms of latency I would guess that Chicago is about 25ms out from origination and then the end user is whatever their latency is from Chicago to their computer. The equity data needs to be sent out from the exchange over to their syndication servers - and then on to you. With regards to price, my opinion differs a bit from Nitro's - I think NxCore is a STEAL at what they charge. Compared to something like Activ or a Bloomberg feed the NxCore guys have a legit data feed for a very reasonable price. Everything in this world is "you get what you pay for" and with NxCore they are still working out extremely ultra-low latency feeds over the general internet. You will be getting the NxCore data with 25-50ms (just a guess) of latency whereas if you wanted to get an Activ feed or get a B-Pipe you would need a low latency point-to-point connection (thousands of dollars) as well as pay for the service and the exchange fees. NxCore is perfect for the high-end retail trader, small or startup fund as well as large institutional firms/traders sending orders over the internet or without low latency demands. Bottom line is the same as everything else - if you have to ask how much it costs you probably can't afford it just like if you are sending your orders over the open internet you probably have zero reason to complain about NxCore latency. If your strategy requires you to be colo and receive direct feeds then you should know better as well as realize that NxCore is priced well for what it is.
Thanks, this is more actual useful technical information... Then is available on the Nanex 90s web site. Still, the idea of deploying a $5,000 tape drive... In order to warehouse 40 Terabytes/year... Strikes me as counter to everything I've learned... About data analysis and trading in 25 years.