This is more of a discussion than a hardware / config thread (but I'm willing to make it up to everyone with the posting of nerd porn and how-to nerd porn to the 10-24tb range... only if 777 is good). All I can say is that 5-8TB just went from top tier hedge fund to basement warrior in the blink of an eye so that now top tier hedge funds are in the (wide) range of 10-36TB with the big boys averaging in the 24-48 range. Clearly I know what you do with this space... but do you? So most are looking for a quick 10-24tb of space so they can pull $80k-$250k worth of data and get started yesterday... The new requirements are legit but are they here to stay? Anyone have a clue at what I'm talking about? How much of a data scam/bubble is this vs actual trading - and I don't mean how many 1's or 0's are they adding to the feeds... I mean has risk really tightened and do they really need this new amount of data or have hedge fund managers lost it again...
Sounds reasonable to me. I work for a university and we are now generating 500+ gigs of system logs a day (600 servers, a few hundred switches, firewalls, apps, etc, etc). It wouldn't surprise me if over half of that growth is just in the last 10 years or so. 20 years of EOD data is smaller then 1 year of 60 second ticks. Stephen
Could it be that they store the historic price data on every tick on every stock for algo back testing? TB is cheap these days. Under $100 per TB. 50TB? Pocket change. This is not 1985... (when Back to the Future came out...)
Anyone remember their first computer's hard drive capacity? Mine was 20MB on a Toshiba laptop back in 1989. I had the option of buying the more expensive 40MB version but 20MB did me just fine for a few years
Mine was a compaq 286. It came with a 20mb and I was told that it would take a life time to fill the disk. Uh hu.
I use a tandy 1000 with a tapedeck to upload my charts... It takes about 5 minutes for one eod chart to get on my screen...