FORUMS BROKERS SOFTWARE
Home
 
    Forums > Tools of the Trade > Data Sets and Feeds > NxCore Users - scripting question


Reply
 
Thread Tools
Old Jul 19th, 2012, 08:57 PM   #1
WinstonTJ
 
 
Join Date: Jan 2009
Location: NYC
Posts: 2,095
I have four of these NxCore feeds coming into the same location (about to have two more) and so far I've been just using their primary/archive function for writing data to the primary machine/database.

I'm looking to stop using the NxCore automated feature and start using a script to push the daily tape file both to the main database as well as a copy to the local backup/DR/archive repository. I can do this fine on my own however what I am running into is that it is messing with the tape verification process. I think that the API or application thinks the tape was deleted off the primary disk vs. moved simply because I do the move via script vs. NxCore.

So does anyone know what I need to add to tell the API that the tape moved and that it's verified (or to verify it from the new location)? This is all on the same local machine. Server 2k8R2 Ent, couple striped SSD's for primary disk and a simple 12TB raid6 array as the archive.
    Quote
Old Jul 20th, 2012, 01:07 AM   #2
NetTecture
 
 
Join Date: Mar 2009
Posts: 1,129
There is nothing you CAN do, I think.

What I do is use the calendar (i.e. current date) and move / delete separately - keep everything local that NxCore expects to be there (14 days historical), move the rest off into an archive location. This means basically copy next day, delete 14 days later.

Validate in the movement process with a PowerShell command that just runs the tape without doing any real processing.

There is not a lo else you can do. Validation has no API, neither has historical download.

I am in the process of setting this up together with an SCOM task (System Center Operations Manager).
    Quote
Old Jul 20th, 2012, 08:36 AM   #3
WinstonTJ
 
 
Join Date: Jan 2009
Location: NYC
Posts: 2,095
Ahh... The primary disk that's pulling the daily tape is a small (64gb) SSD so I was trying to only keep that day's tape on the SSD and then since the I/O on the 12TB array is reasonable (~600mb/sec sustained) I was trying to dump as much as possible over to the RAID array.

May need a larger SSD or a dedicated SSD to push 14 days worth of tape data to. Is that how you are doing it?

I'm not sure that I ever learned what tapes you are getting but have you noticed any difference between equities vs. options vs. futures data (with regards to the archive and validation process)?

Thx for the response, was a big help!
    Quote
Old Jul 20th, 2012, 08:38 AM   #4
NetTecture
 
 
Join Date: Mar 2009
Posts: 1,129
Well, I only get the futures tape ;) So, no, I did not realize any difference.

But:

Why keep the tapeson an SSD at all - even when you get them it is good enough to put them on a disc ;) But yes, otherwise you need a larger SSD.

nxCore could need some more features - I also miss integration into performance counters and / or event log ;)
    Quote
Old Jul 20th, 2012, 09:00 PM   #5
DeeDeeTwo
 
 
Join Date: Dec 2007
Location: Toronto
Posts: 623
So NxCore writes directly to tape?
With tape seek times added to latency?

Or does it have a real-time cache on SSD...
That the API can access with near zero latency for recent quotes?
    Quote
Old Jul 20th, 2012, 09:53 PM   #6
WinstonTJ
 
 
Join Date: Jan 2009
Location: NYC
Posts: 2,095
When we say "tape" that's what we are referring to as the daily files. Think old school when the ticker used to print on a paper "tape".

You just pull a NxCore proprietary data file.

Most clients actually write the tape files normally to disk as you suggest. I write the daily tape file to a SSD for one client because they use it for realtime (or close to realtime) stuff... that's about where my job ends and the trader's job begins.
    Quote
 
Reply
Thread Tools

Forum Jump



   Conduct Rules   Privacy Policy   Sitemap Copyright © 2014, Elite Trader. All rights reserved.   

WHILE YOU'RE HERE, TAKE A MINUTE TO VISIT SOME OF OUR SPONSORS:
Advantage Futures
Futures Trading & Clearing
AMP Global Clearing
Futures and FX Trading
Collective2
Automated Trading Services
CTS
Futures Trading Software
dom993trading.com
NinjaTrader Consulting
eSignal
Trading Software Provider
FXCM
Forex Trading Services
Global Futures
Futures, Options & FX Trading
Interactive Brokers
Pro Gateway to World Markets
JC Trading Group
Direct Access Trading
MB Trading
Direct Access Trading
NinjaTrader
Trading Software Provider
optionshouse
Option Trading & Education
Rithmic
Futures Trade Execution Platform
SpeedTrader
Direct Access Trading
SpreadProfessor
Spread Trading Instruction
thenut-trader.com
FX, Gold, & Stock Signals
TradersStudio
System Building & Backtesting
Tradier
Equity and Options Trading
Trading Technologies
Trading Software Provider