Intelligence test. I answered to a post of "with a trial oof Iq years of data are free". You do realize that free != selling?
Sorry, I answered on this statement. I though it's better to pay for that is useful for you then complain about non-usability of free stuff.
Please try to imagine some strange firm providing FREE book data. The entire sessions for US markets since 2007 are 6 TB of data in zipped format. Who is going to provide that for you for free?
I'll tell you what... I'll give you 3 years of raw data for free. Stocks, Bonds, Futures, FX, Options. It's located at 32 Ave of Americas... You break Windows file system limits for files in a directory once you are >25k... Where are you going to put 60,000,000 monthly tick files? http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ I have two of those that are 90% full. The chassis were modified to fit SuperMicro 16-port backplanes so they actually hold 48 drives (16x3) not the 45 as stated by Backblaze. They are technically 138TB each in RAID 6 but the usable space on each drive is only 2.7TB so the ZFS arrays are closer to 129-130TB each. Going back to "the beginning of time" gosh I have no idea... Keep in mind that's just disks spinning to hold the data. You start to get I/O bound around 4-5TB of data. If you are doing anything over a network you'll get bogged down at anything over 100-300gigs.
Do you mean not only Times & Sales, but also Top Of The Book and Book Depth by raw data ? I'd like to have 3 years of Times & Sales- not more then 3TB in compressed ASCII or 700GB in binaries.
just wanted to pipe in here to NT.. love the platform i've used it myself.. i used my data provider.. which was Interactive brokers.. there are alot of different attributes about data that you should become aware of.. tick data seems expensive.. but they give you a ton of information and you pay for what you get.. now that being said... NINJA TRADER PEOPLE.. I AM DYING FOR YOU PEOPLE TO GET AN OPTIONS TRADING PLATFORM.. there is nothing like NT for stocks for options.. i love your platform.. Look at Think or swim and trade station... look at Hoadley financial add in for excel.. and come up with something awesome please thankyou..
Raw Market Message Data averages closer to 20GB / Day Compressed. OPRA, Futures, Equities etc. You'll need about 12TB just to store the compressed files and 4x that and a lot of I/O and CPU's to process the data. It takes 5 min to 25 min to run through a day processing Billion + Messages. Take a look at Nanex, they are a decent source for historic whole market message data. A better alternative would be tick accurate consolidations from the raw message data. Instead of processing a Billion + messages each run through a day be nice if someone offered just the bid/ask changes and related trade data. This would reduce SPY as an example from 10M messages in a typical day into 10K consolidated price tick records. There is no need to process Level II historic liquidity and Time and Sales alone doesn't provide an accurate representation of "executable" market conditions. Tick accurate Bid/Ask Changes and a snapshot of traded activity inside each price change interval would reduce the data sizes and processing times by a factor of a 1000 and provide a clear and concise view of the market. Load these tick consolidations into SQL and you'll have a fast accurate back test and analysis platform. SQL Queries can return result sets in milliseconds versus coding a C++ 5M/s message processor that still has to churn through the compressed raw data files.
I agree. Book snapshot data is even 2 times larger, but it can be easily constructed using raw order add, cancel, delete etc messages. That's a good idea. I've checked out IQFeed, but they provide only Times & Sales (without Bid&Ask changes). ESignal is a better alternative, because, as far as I understand, they provide exactly the data you speak about without the need of extra programming in order to extract trades and bid&ask changes from raw data
wow did you hit it right on the head... i'm sure there are some signals in the noise mess you leave over.. you'd most likely get lost in it and the cost is high considering..