 |
WinstonTJ
Registered: Jan 2009
Posts: 1947 |
08-06-12 01:44 AM
Quote from LivermoreRisen:I do wish I could find the historical tick data for all stocks and futures, indices, from the beginning of time until present. To any one who would share this information with me I would promise that the data given to me would find its way on the internet in some way so that it is accesible to the masses...
I'll tell you what... I'll give you 3 years of raw data for free. Stocks, Bonds, Futures, FX, Options. It's located at 32 Ave of Americas...
You break Windows file system limits for files in a directory once you are >25k... Where are you going to put 60,000,000 monthly tick files?
Quote from ogarbitrage:I can't even begin to imagine how many terabytes of storage that would require.
http://blog.backblaze.com/2011/07/2...g-more-secrets/
I have two of those that are 90% full. The chassis were modified to fit SuperMicro 16-port backplanes so they actually hold 48 drives (16x3) not the 45 as stated by Backblaze. They are technically 138TB each in RAID 6 but the usable space on each drive is only 2.7TB so the ZFS arrays are closer to 129-130TB each.
Going back to "the beginning of time" gosh I have no idea...
Keep in mind that's just disks spinning to hold the data. You start to get I/O bound around 4-5TB of data. If you are doing anything over a network you'll get bogged down at anything over 100-300gigs.
|
| |
|
Edit/Delete • Quote • Complain |
skunks
Registered: Jun 2012
Posts: 19 |
08-06-12 05:52 AM
Do you mean not only Times & Sales, but also Top Of The Book and Book Depth by raw data ?
I'd like to have 3 years of Times & Sales- not more then 3TB in compressed ASCII or 700GB in binaries.
|
| |
|
Edit/Delete • Quote • Complain |
cdcaveman
Registered: Aug 2011
Posts: 3514 |
08-06-12 05:59 AM
just wanted to pipe in here to NT.. love the platform i've used it myself.. i used my data provider.. which was Interactive brokers.. there are alot of different attributes about data that you should become aware of.. tick data seems expensive.. but they give you a ton of information and you pay for what you get.. now that being said... NINJA TRADER PEOPLE.. I AM DYING FOR YOU PEOPLE TO GET AN OPTIONS TRADING PLATFORM.. there is nothing like NT for stocks for options.. i love your platform.. Look at Think or swim and trade station... look at Hoadley financial add in for excel.. and come up with something awesome please thankyou..
|
| |
|
Edit/Delete • Quote • Complain |
PocketChange
Registered: Jul 2008
Posts: 2037 |
08-06-12 12:28 PM
Raw Market Message Data averages closer to 20GB / Day Compressed. OPRA, Futures, Equities etc. You'll need about 12TB just to store the compressed files and 4x that and a lot of I/O and CPU's to process the data.
It takes 5 min to 25 min to run through a day processing Billion + Messages. Take a look at Nanex, they are a decent source for historic whole market message data.
A better alternative would be tick accurate consolidations from the raw message data. Instead of processing a Billion + messages each run through a day be nice if someone offered just the bid/ask changes and related trade data.
This would reduce SPY as an example from 10M messages in a typical day into 10K consolidated price tick records. There is no need to process Level II historic liquidity and Time and Sales alone doesn't provide an accurate representation of "executable" market conditions.
Tick accurate Bid/Ask Changes and a snapshot of traded activity inside each price change interval would reduce the data sizes and processing times by a factor of a 1000 and provide a clear and concise view of the market.
Load these tick consolidations into SQL and you'll have a fast accurate back test and analysis platform. SQL Queries can return result sets in milliseconds versus coding a C++ 5M/s message processor that still has to churn through the compressed raw data files.
Quote from skunks:
Do you mean not only Times & Sales, but also Top Of The Book and Book Depth by raw data ?
I'd like to have 3 years of Times & Sales- not more then 3TB in compressed ASCII or 700GB in binaries.
|
| |
|
Edit/Delete • Quote • Complain |
skunks
Registered: Jun 2012
Posts: 19 |
08-06-12 05:45 PM
Quote from PocketChange:
Raw Market Message Data averages closer to 20GB / Day Compressed.
I agree. Book snapshot data is even 2 times larger, but it can be easily constructed using raw order add, cancel, delete etc messages.
Instead of processing a Billion + messages each run through a day be nice if someone offered just the bid/ask changes and related trade data.
That's a good idea. I've checked out IQFeed, but they provide only Times & Sales (without Bid&Ask changes).
ESignal is a better alternative, because, as far as I understand, they provide exactly the data you speak about without the need of extra programming in order to extract trades and bid&ask changes from raw data
|
| |
|
Edit/Delete • Quote • Complain |
cdcaveman
Registered: Aug 2011
Posts: 3514 |
08-07-12 01:06 AM
wow did you hit it right on the head... i'm sure there are some signals in the noise mess you leave over.. you'd most likely get lost in it and the cost is high considering..
Quote from PocketChange:
Raw Market Message Data averages closer to 20GB / Day Compressed. OPRA, Futures, Equities etc. You'll need about 12TB just to store the compressed files and 4x that and a lot of I/O and CPU's to process the data.
It takes 5 min to 25 min to run through a day processing Billion + Messages. Take a look at Nanex, they are a decent source for historic whole market message data.
A better alternative would be tick accurate consolidations from the raw message data. Instead of processing a Billion + messages each run through a day be nice if someone offered just the bid/ask changes and related trade data.
This would reduce SPY as an example from 10M messages in a typical day into 10K consolidated price tick records. There is no need to process Level II historic liquidity and Time and Sales alone doesn't provide an accurate representation of "executable" market conditions.
Tick accurate Bid/Ask Changes and a snapshot of traded activity inside each price change interval would reduce the data sizes and processing times by a factor of a 1000 and provide a clear and concise view of the market.
Load these tick consolidations into SQL and you'll have a fast accurate back test and analysis platform. SQL Queries can return result sets in milliseconds versus coding a C++ 5M/s message processor that still has to churn through the compressed raw data files.
|
| |
|
Edit/Delete • Quote • Complain |
| Receive
an email whenever a new post is added to this thread by subscribing
to it. |
|
|
|
|
