I have four of these NxCore feeds coming into the same location (about to have two more) and so far I've been just using their primary/archive function for writing data to the primary machine/database. I'm looking to stop using the NxCore automated feature and start using a script to push the daily tape file both to the main database as well as a copy to the local backup/DR/archive repository. I can do this fine on my own however what I am running into is that it is messing with the tape verification process. I think that the API or application thinks the tape was deleted off the primary disk vs. moved simply because I do the move via script vs. NxCore. So does anyone know what I need to add to tell the API that the tape moved and that it's verified (or to verify it from the new location)? This is all on the same local machine. Server 2k8R2 Ent, couple striped SSD's for primary disk and a simple 12TB raid6 array as the archive.
There is nothing you CAN do, I think. What I do is use the calendar (i.e. current date) and move / delete separately - keep everything local that NxCore expects to be there (14 days historical), move the rest off into an archive location. This means basically copy next day, delete 14 days later. Validate in the movement process with a PowerShell command that just runs the tape without doing any real processing. There is not a lo else you can do. Validation has no API, neither has historical download. I am in the process of setting this up together with an SCOM task (System Center Operations Manager).
Ahh... The primary disk that's pulling the daily tape is a small (64gb) SSD so I was trying to only keep that day's tape on the SSD and then since the I/O on the 12TB array is reasonable (~600mb/sec sustained) I was trying to dump as much as possible over to the RAID array. May need a larger SSD or a dedicated SSD to push 14 days worth of tape data to. Is that how you are doing it? I'm not sure that I ever learned what tapes you are getting but have you noticed any difference between equities vs. options vs. futures data (with regards to the archive and validation process)? Thx for the response, was a big help!
Well, I only get the futures tape So, no, I did not realize any difference. But: Why keep the tapeson an SSD at all - even when you get them it is good enough to put them on a disc But yes, otherwise you need a larger SSD. nxCore could need some more features - I also miss integration into performance counters and / or event log
So NxCore writes directly to tape? With tape seek times added to latency? Or does it have a real-time cache on SSD... That the API can access with near zero latency for recent quotes?
When we say "tape" that's what we are referring to as the daily files. Think old school when the ticker used to print on a paper "tape". You just pull a NxCore proprietary data file. Most clients actually write the tape files normally to disk as you suggest. I write the daily tape file to a SSD for one client because they use it for realtime (or close to realtime) stuff... that's about where my job ends and the trader's job begins.
That file would be many times larger than any SSD... And real-time trading would seem to be the whole point of an expensive data feed... So a user configurable, sophisticated cache would be compulsory... One that would include a symbol table. As a software engineer... My whole problem with NxCore is the information blackout... The word "tape" is not even mentioned on their web site... And their technical documentation does not give a simple overview... Their salesman would just tell me I can run it all on an iPhone. Their fancy algo charts for Zero Hedge are clever propaganda... But I'm convinced that very few people actually use NxCore... I'd be surprised if they have 1,000 customers... Because they just ignore basic issues every engineer would demand.
There is a overview of the structures here http://www.nanex.net/API_Explain.html, the word 'tape' occurs multiple times. I had heard that one of the things that people found irritating was that a subscriber model was not supported, in so much as you have to consume the whole exchange/market.
I believe you subscribe to specific exchanges and receive the entire exchange multicast. Tapes are just Nanex's syntax for their historic data files. If all you need is a few hundred symbols take a look at iqfeed... Same parent company. QUOTE]Quote from Craig66: There is a overview of the structures here http://www.nanex.net/API_Explain.html, the word 'tape' occurs multiple times. I had heard that one of the things that people found irritating was that a subscriber model was not supported, in so much as you have to consume the whole exchange/market. [/QUOTE]
You can get a trial with historical real data quickly and without any issues. There are over 10+ sample apps that you can use along with examples. Don't understand what is the "blackout" here.