Registered: Jan 2009
Posts: 1947

I have four of these NxCore feeds coming into the same location (about to have two more) and so far I've been just using their primary/archive function for writing data to the primary machine/database.

I'm looking to stop using the NxCore automated feature and start using a script to push the daily tape file both to the main database as well as a copy to the local backup/DR/archive repository. I can do this fine on my own however what I am running into is that it is messing with the tape verification process. I think that the API or application thinks the tape was deleted off the primary disk vs. moved simply because I do the move via script vs. NxCore.

So does anyone know what I need to add to tell the API that the tape moved and that it's verified (or to verify it from the new location)? This is all on the same local machine. Server 2k8R2 Ent, couple striped SSD's for primary disk and a simple 12TB raid6 array as the archive.

Registered: Mar 2009
Posts: 1010

There is nothing you CAN do, I think.

What I do is use the calendar (i.e. current date) and move / delete separately - keep everything local that NxCore expects to be there (14 days historical), move the rest off into an archive location. This means basically copy next day, delete 14 days later.

Validate in the movement process with a PowerShell command that just runs the tape without doing any real processing.

There is not a lo else you can do. Validation has no API, neither has historical download.

I am in the process of setting this up together with an SCOM task (System Center Operations Manager).

Registered: Jan 2009
Posts: 1947

Ahh... The primary disk that's pulling the daily tape is a small (64gb) SSD so I was trying to only keep that day's tape on the SSD and then since the I/O on the 12TB array is reasonable (~600mb/sec sustained) I was trying to dump as much as possible over to the RAID array.

May need a larger SSD or a dedicated SSD to push 14 days worth of tape data to. Is that how you are doing it?

I'm not sure that I ever learned what tapes you are getting but have you noticed any difference between equities vs. options vs. futures data (with regards to the archive and validation process)?

Thx for the response, was a big help!

Registered: Mar 2009
Posts: 1010

Well, I only get the futures tape ;) So, no, I did not realize any difference.

But:

Why keep the tapeson an SSD at all - even when you get them it is good enough to put them on a disc ;) But yes, otherwise you need a larger SSD.

nxCore could need some more features - I also miss integration into performance counters and / or event log ;)

Registered: Dec 2007
Posts: 623

So NxCore writes directly to tape?
With tape seek times added to latency?

Or does it have a real-time cache on SSD...
That the API can access with near zero latency for recent quotes?

Registered: Jan 2009
Posts: 1947

When we say "tape" that's what we are referring to as the daily files. Think old school when the ticker used to print on a paper "tape".

You just pull a NxCore proprietary data file.

Most clients actually write the tape files normally to disk as you suggest. I write the daily tape file to a SSD for one client because they use it for realtime (or close to realtime) stuff... that's about where my job ends and the trader's job begins.

Advantage Futures

Futures Brokerage & Clearing

AMP Global Clearing

Futures and FX Trading

Bright Trading

Professional Equities Trading

CTS

Futures Trading Software

DaytradingBias.com

Professional Trading Analytics

ECHOtrade

Professional Trading Firm

eSignal

Trading Software Provider

FXCM

Forex Trading Services

Global Futures

Futures, Options & FX Trading

Interactive Brokers

Pro Gateway to World Markets

JC Trading Group

Direct Access Trading

MB Trading

Direct Access Trading

MultiCharts

Trading Software Provider

NinjaTrader

Trading Software Provider

OANDA

Currency Trading

optionshouse

Option Trading & Education

Rithmic

Futures Trade Execution Platform

SpeedTrader

Direct Access Trading

SpreadProfessor

Spread Trading Instruction

thinkorswim by TD Ameritrade

Direct Access Trading

TradersStudio

System Building & Backtesting

Trading Technologies

Trading Software Provider

Trend Following

Trading Systems Provider