Goodbye! You won't be missed. Thank you! Great advice and resources. Mucho appreciato. Sincerely. LF.
I somewhat disagree with your post about subscribing to eoddata, the reason being is the only real upside I see to going to a data vendor for EOD data are as follows: 1) They handle delistings and symbol changes effectively and 2) provide corporate actions. You'll certainly be able to pull everything in one swoop from them, but the upside is nominal IMO. Ok so you need to pause your loop when pulling from yahoo, you'll likely be running this process overnight so it's not really a big deal I don't think. You'll just lack delisted securities from Yahoo, as well as symbol changes. However you can deal with symbol changes on a forward moving basis either create your own unique symbology or just use bloomberg's open symbology. Plus there's definitely value in this guy learning how to get data from random sources.
I know of only 2 places to get listed and delisted data - Premium Data, and CSI. Premium Data is affordable, and comes with the CUSIP number. CSI requires a commercial license costing $10K at a minimum. Downloading from Yahoo Finance doesn't take hours. My Perl script takes about 15 min for the download portion. One user in another thread wrote a C downloader using libcurl which downloads the data in less than a minute. Why choose a random site instead of one reliable site like Yahoo? Heck, Google can't even come up with a decent Finance site. If they can't, how can we expect any other site to be better - more reliable, more comprehensive, and FREE? Since Yahoo is a highly trafficked site, 40 API calls or 7,000+ data downloads won't be noticed, let alone the TOS suddenly being enforced with an IP address ban. I recommended eoddata.com because at some point, if you're maintaining your own infrastructure, you just want to get the job done easily and reliably. I've used CSI (hacking their Market scan file to get 10 years of data for every listed stock). I currently use Yahoo Finance for US stock data. I've used Quandl for US futures data, but it's often not available until 5 am the next morning. So I subscribed to EOD Data to get today's futures data today.