FWIW, I can still easily download a few thousand EOD symbols from Yahoo finance no problem. I have a set of about 1900 ETFs (csv/zoo format). But from casual inspection, there are lots of gaps and outliers everywhere. I think it is cleanable and can imagine a simple script to adjust for improper splits/gaps, by simply back stitching over at least large gaps (similar to stitching futures contracts). I think that outliers could also be reasonably cleaned by an automated script. One time consuming aspect would be visually verifying/validating and troubleshooting against some reliable dataset/data provider. I have data gathering scripts written in R and could possibly work on it in other languages. If any individuals have an interest in tackling it with a divide and conquer approach (at least for say the largest 500 ETFS for example)... let me know. At the very least, I can send you a small set of csv files to work on. Who knows how long the data will be available; this isn't really dealing with ongoing data, but at least you'd hopefully have a cleaner dataset to backtest over. I've seen similar issues with Quandl data, so I think a good script cleaner would always be useful. And as others have pointed out, I don't see a lot of historical ETF data providers out there.
The problem with gaps is, it's impossible to know whether the gap was real or it's a missed split. When you have a reliable database of splits then it's a matter or comparing prices on and before the split with the split ratio. But Yahoo for example has missed splits completely (RRD in 2016). The best one can hope for is an "acceptable rate of errors" and that requires combining and comparing of data, which is a pain. I see myself going the paid data way in the end as the time spent on data stitching isn't worth it.
Understandable. If there is a public data source (even a chart provider) to compare to , it can give you some sense of validation --One reason I thought divide and conquer might at least mitigate the problems (maybe so, maybe not) . If you find any good historical providers with ETFs (paid or not) please post.
Reuters datalink https://www.metastock.com/products/data/ $24.95 a month If you want esignal daily and intraday data subscribe to classic (delayed) for $55 a month and then use qcollector from www.mechtrading.com to access the historic data. It will keep the intraday file up to date automatically.
$24.95 a month or $55 a month is apparently too much for many elitetraders. Very strange for a site where the majority makes good money...
I did not know that 1900 ETFs existed. I limit my ETF selection to only ETFs with weekly options and I'm down to around or less than 100. Some of those ETFs must only have a few trades per day.
I've been trying to work through Yahoo's changes. My backtests are making wild swings using yahoo's new daily bars. Upon closer inspection, I noticed that some closing prices higher than the highs of the day. I suspect that the Open High Low values have been changed to adjusted open high lows, rather than the actual open high lows from before. Closes are still actual closes. I started wondering if they changed the dividends as well. Sure enough, some dividends are fractions of pennies. TLDR, yahoo's OHL bars appear to now be div and split adjusted. Dividends have also been div and split adjusted. Can anyone else confirm?
"Dividends have also been div and split adjusted", what does that mean? The new Close values are unadjusted but all other fields are split adjusted, the adjusted close being the relevant field. This is why the Close can appear higher. There are no dividend adjustments that I can find. Do you have symbol examples? Would be best if Yahoo didn't mess with the data at all and just supplied divs, splits separately. Oh well, one can dream.