Home > Tools of the Trade > Data Sets and Feeds > Looks like Yahoo Download API is now Dead - 11/1/2017

Looks like Yahoo Download API is now Dead - 11/1/2017

  1. Since Yahoo terminated their API back in the May/June time frame, I was still able to get web-based data to download into EXCEL from a query such as:


    The above web query would yield:

    spy 257.15 10/31/2017 4:00pm 0.4 257.18 257.44 256.8 60304781
    dia 233.63 10/31/2017 4:00pm 0.3 233.64 233.91 233.2 1772047

    Well, it seems that as of sometime today, this has become disabled as well. The above web-based query yields a:

    "Sorry, Unable to process request at this time -- error 999."

    So, assuming that this web service is dead in the water for good, can someone offer an alternative source (free or paid) that would yield the same, via a web-interface (or equivalent)?

    Thanks in advance.
  2. I was wondering if something was wrong with my PC. Now, it is confirmed the problem lies with Yahoo server.
  3. Demand a refund!
  4. I bet Verizon feels that way.
  5. Haha. You're being unreasonable. The data was free to start with. However, what Yahoo did wrong was to pull the plug immediately. At least give some warning so that the customers can look for alternatives before this nasty surprise. Now, everyone is left vulnerable and fighting in the dark against the market. bad bad Yahoo.
  6. You get what you paid for.
  7. I agree but for goodwill reasons even to non-paying customers, don't pull the rug under our feet without warning. Nobody likes that.
  8. Seriously? I'll repeat it. You get what you paid for. No business is "in it for the feel-good goodwill of the thing." If they were, they'd be called a non-profit charity, which is also in the business of making money.
  9. This is a blessing in disguise. Retail traders who rely on "free" Yahoo data are typically undercapitalized (else they'd subscribe to a real datafeed) and if this discontinuation keeps them out of the market, they'll be the richer for it in the long run.
  10. There was a similar thread on here a few months back. Look for it. Maybe Google has free data.
  11. I hear what you are saying but try to not be so negative. It is possible to "beat the machines", I even hear that there is a nice book out there on how to do it.
  12. Appreciate all the philosophy on this thread, but hear no recommendations on alternate sources of data (either free or paid) that could replace my original post.

    And shame on Yahoo! for not having the decency to at least post a message indicating that the service is shut down, as opposed to a cryptic error number.
  13. Anyone know if there is an undocumented GOOGLE equivalent to the following, that generates a CSV for multiple tickers in one shot:

  14. Everyone on ET squawks about phone-home risk with trading software -- "they might be trying to get my IP!!" -- but wants to get IP for nothing from Yahoo. Go figure.
  15. So, I downloaded AmiQuote to test, and can download from Google EOD (Yahoo broken obviously), and there appears to be no way to generate a single CSV with EOD data from a list of tickers. Anyone know if the ability to do this magically appears upon license purchase?
  16. You get multiple csv's? One ticker per file?
  17. In the trial I downloaded, I specified 2 tickers, and run the download from Google. I need to download on each respective ticker to get the associated CSV for it, but would like a single consolidated CSV with data from each ticker in it (that is, one row per ticker). Not sure if the unavailability of this feature is due to it being Unregistered. I just emailed their customer support, with that an a few other questions. I'll post back here, when I hear from them.
  18. Stock quotes shouldn't be "IP" and the recent crackdown from NASDAQ exchanges/owners smacks of market manipulation and unfair cover-up of public information.
  19. From Yahoo:


    [​IMG] Nixon

    Administratin' 'Hoo

    32m ago

    Re: Is Yahoo! Finance API broken?
    Hi All - I'm blocking new replies to this thread since it refers to an older outage.

    The new download issue which began 11/01/17 and returning an error 999 for most users is currently being investigated and we hope to have it resolved soon.
  20. Call your Congressman! Call the SEC! Call the US Attorney! Demand an investigation!
  21. -----
    @hayman - on the other hand admin = nixon keeps writing: "UPDATE: It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com Thank you. "
  22. There is already a thread that I posted in that covers this subject. I haven't looked at trying to scrape intraday data from yahoo but I can tell you that it is possible to data scrape daily OHLC data directly from the HTML from yahoo. You will need to do a bit of work to harvest this data but it is possible.
  23. You can also cut your lawn with a pair of scissors...
  24. LOL, if the scissors are automated and can run themselves automatically then why not? Scissors have less of a carbon footprint too so it is good for the environment. Please continue to share your valuable knowledge here my friend.
  25. I did see those posts, and I believe (and I might be wrong), that those users were directly using Yahoo! data on their websites, which is a breach of terms of service. Then again, the wording of "the service is being discontinued" is worrisome. Here's hoping that this Yahoo Admin just doesn't have a command of the English language.
  26. I've scraped in my youth, and although a fun programming exercise, the problem is that when websites change format (and Yahoo does this often), your scrape code usually needs to change along with it. I gave up scraping a long time ago, because it just wasn't worth the maintenance headache.
  27. Oh boy, I just tried to run a simple download query:


    and now I get the message,

    "It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com."
  28. The data from yahoo is JSON so it is basically structured data within unstructured data.

    If you scrape based on exact delimiters and their location then you will constantly need to rewrite your code like you said. If you treat the HTML like one big "string" then there is usually a keyword marker that can act as your "start of data". This is a way to keep your code more nimble.

    I don't deny that this is a headache and a possible waste of time. We are not on a "time management" message board. If you want to spend your free time doing it then yes it is possible regardless of what api services yahoo shuts down.
  29. Scraping Yahoo looks like a mess to me. Just tried. Then I tried google and this is the string you would look for: ["SB","Safe Bulkers, Inc.","3.42"

    ["SB" - is only there once - an unique entry. So, you would look for ["your symbol" Good Luck !
  30. Anyways, to answer this question from the OP of the OP...

    Get a broker with an included data feed. Or just pay a small fee for a dedicated data feed.

    End of line.
  31. Yup, just a pain in the ass. I only need about 50 tickers (stocks and ETF's) on an EOD basis, but this drives my portfolio, a couple of Wealthlab trading scripts (which are now dead in the water, since it's WL 4.0 - unless I convert from native WL to ASCII upload), and my semi-annual 401K re-balancing. Work that I wasn't planning on doing, but I guess this day of reckoning was about to come sooner or later.

    How much did Verizon (over)pay for this company?
  32. Just use Deriscope. It is 100% free and gets live prices with only 1 minute delay into Excel
  33. Why not put the quotes into a portfolio and download using the export link?
  34. Sometimes the simplest ideas are the best......great idea. I was going to build a DDE-based spreadsheet to use with my IB feed, but I like your idea; thanks for that! Didn't know that their portfolio feature was exportable like that. This solves a couple of my problems right away.

    Unfortunately, my Wealthlab issue is much larger, but not as urgent.

    Thanks again!
  35. Can't you get that from Quandl?
  36. You found me out
  37. Yes, but a little pricey for my needs. Thanks.
  38. Scraping is slow and wasteful. But well written code for scraping can handle some changes in code.

    I hope you realize that if you only update the latest day, the past data won't be split or dividend adjusted.

    Google's data is full of missing days and it's split but not dividend adjusted.
  39. Quandl free does not have ETF's. Try https://www.tiingo.com/
  40. I used Yahoo finance to pull certain data from them (stock price, dividend rate, dividend per share, and dividend pay date)
    I did this by using some code in VB in Excel that would allow me to update the spreadsheet automatically anytime I hit update all.

    Is there another site I can pull this same data into my excel spreadsheet?
    Preferably something that is free.

    Do you guys have any recommendations?
  41. I have no recommendations for on-demand web requests for live or delayed quotes to replace what Yahoo or others do.

    I use NxCore for my live intraday feeds and have a second feed of Norgate data for daily timeframe data.

    I supplement my systems with cross-checks from a number of sources. Why? Many events cause big changes in the prices of stocks. If those events are just due to normal capital changes/dividends to a stock, I need to know that. For example, I need to know if there is a split or dividend that means I need to adjust my stops.

    The quality of Yahoo and Tiingo is not good enough... Too many errors in their systems that generate false positive events that take too much time to validate (mostly invalidate).
  42. No problem, it can even be automated if you really want, but that involves working with the command line and a download utility called 'curl', it's a bit fiddly.

    With freebie data, it wouldn't be a bad thing to have two different sources worked out, then if one changes or disappears it doesn't leave you in the lurch right away.
  43. Actually curl works with alot of programming languages. No command line required . I use it with c++. The library is libcurl. There are other links that yahoo has that gives pure json data. I found one that requires no scraping or cookies. I have been wanting to post the link that I found and use but I worry if to many know yahoo make take that away also.
  44. Yeah, have a backup source is good. I will probably build a backup using my IB or TD Ameritrade feeds.

    FYI, there's an old command line utility called WebFetch.exe, which you can probably hunt around for, which could assist in the automation within a command line script.
  45. There is a fix for pandas data reader that scrapes the data from yahoo finance for you and stores it in a data frame. I've used it to acquire some data and it seems to work fine. Please find the link below.

  46. To use curl to download the export file it needs three things:

    1. A copy of the cookies that Yahoo uses. Best way is to install a fresh copy of Firefox Portable and add the Export Cookies add-on, then log into Yahoo. Then right click on the title bar, select Menu Bar then select Tools/Export Cookies from the menu bar and save as say cookies.txt

    2. The url that downloads the export file. In Firefox it can be copied with a right click on the 'Export' link and selecting Copy Link Location

    3. Installing curl and running it on the command line. Grab a copy from https://curl.haxx.se/, install, copy the cookies file into the same directory, open a command line and change to the the directory. Then run something like:

    curl "the-magic-url" --cookie cookies.txt --output yahooquotes.csv

    Hopefully that should do it! :)

    I guess Yahoo don't want to allow anonymous download as it makes it more open to abuse and denial of service attacks, even hacking.
  47. I've just produced a python script that can handle multi ticker yahoo data acquisition and thought I'd share incase this can help anyone. **The script requires the following packages which can be attained through pip: pandas / pandas_datareader / fix_yahoo_finance **

    The script saves each tickers data in a csv file in a folder named YahooData which is created in the same directory you place the python file. I have currently configured it to attain yahoo data for all S&P 500 companies but this can easily be modified to your own requirements by changing the list in line 42 such that it contains the tickers you require and finally comment out line 43.

    To run the script simply type "python AttainYahooData.py" in your command line ensuring that you are in the directory the python script is located.

    I've multi-threaded the process which speeds things up significantly.

    The data acquisition isn't always successful first time round so to retry type y when prompted by the command line. You may need to retry three or four times if you're downloading hundreds of tickers.

    The python script can be accessed on my google drive here:

    I have just modified the script so it will repeatedly try to fetch the tickers, without the need for user input, until tickers have been downloaded successfully. This will be better suited to automated systems based on yahoo data. Make sure you type the tickers correctly or else you will be stuck in a never ending loop :) (this version is named AttainYahooData_v1.py therefore use command "python AttainYahooData_v1.py" to run.

  48. and such programs:
    HSQuote / HSQuote Plus
    it is not sufficient?
    The creator of this HSQ is very polite and always helps with problems.
  49. Would you consider to create a modified script that creates a single csv file with one ticker data per line? Of course, if it is too much trouble, forget about it. I would do that myself but I don't have the skill needed.

  50. I can take a look into this when I have some spare time but could you please help me in understanding exactly what you require. Am I correct in thinking you want the script to fetch the most recent data available for a number of defined tickers and then compile it in a csv in the format below?

    ticker, open, low, high, close, adj close, volume
    A, xxx, xxx, xxx, xxx, xxx, xxx
    B, xxx, xxx, xxx, xxx, xxx, xxx

    If not could you give an example of what it is you want.
  51. I would definitely be interested in something like that.
    Ideally I would love to be able to pull Price, Dividend, and Upcoming Payment Date

    If this is something I could pull from a web query into excel automatically that would be great.
    I'm currently looking at https://iextrading.com/developer/docs/#iex-api to try and do this, but my knowledge is limited. I was using VBA code in excel to pull from Yahoo.
  52. Thank for your attention and for your answer.

    In the code box below you can see the actual content of the csv file I used to download from Yahoo every few minutes with Free Download Manager.

    This was the URL that created the csv from Yahoo:
    Then I have an auto updating query on external data in an Excel file that will get the data from the csv so every five minuts I can see the values updated.

    Now that Yahoo feature is dead, being able to recreate a similar csv in some other way would be awesome.


  53. To get the data into Excel I first created a "Query on external data" manually importing from the csv into a sheet named "tests2", then I went and see the query properties, gave it a name (in my case "PF13") and saved it and then I created a button on a sheet ("WEB" for me) that triggers this VBA sub routine:
    Private Sub UpdateButton_Click()
    Application.ScreenUpdating = False
        With Sheets("tests2")
              .QueryTables("PF13").Refresh BackgroundQuery:=True
        End With
    Application.ScreenUpdating = True
    End Sub
    Apart Yahoo errors every now and then, it's been working like a charm.


  54. Thanks a lot for discussions in this thread. Although none of the solutions solved the problem I have, it gives me valuable clues to finally allow me continue automatically download from yahoo financial site a list of tickers in a format that can be easily converted into csv format. For those who know how to program, it is easy. Otherwise, please do not bother to read the following. Here is what I do:
    1. use your browser either google chrome or firefox;
    2. create a yahoo account, and log in, and create a portfolio. You can add upto 1000 tickers into each portfolio.
    3. create a view to customize the display for the portfolio. for example, you can select: ticker, open, high, low, last, vol, time.
    3. you can manually download the web page into a single html file by selection "Save as: Web Page, Complete". (Wait this is not automatic yet. I will tell how to make it automatic if you continue to read). In the browsers's address bar, you could see the URL such as - https://finance.yahoo.com/portfolio/<your-fortfolio-id>/view/view_<your-view-id>, copy it to be used for later automatic download.
    4. After you get the html file, use the program "html2text" (free available for Linux/Unix platform) or use lynx -dump <the html file>. You get the text format of the quote table. From there, it is easy to convert the table into a csv format or anything you like if you know how to write a script (python, or anything you like).

    To make the above download automatic, that is, write a simple program (robot) to do downloading by simulating the clicks and keyboard types. Use the free tool automatic-save-as (from github.com) and xvkbd.

    The problem I cannot use curl or wget or lynx is they do not support full web page downloading into a single web html file (or I still do not find a way to use them correctly) even I added cookies.txt to start programs.
  55. Hello:

    Why does not a hyperlink to the watchlist created in excel work?

    Thank you very much
  56. I am new to this forum. My problem is I wanted to use Yloader. I have download and install and all is fine. But I am unable to convert the ascii or csv files to metastock format. I am asking for help from this forum. Thank you and god bless.
  57. www.databull.com may still get data into metastock. If ascii will do then Amiquote http://www.amibroker.com/products.html has more free sources than just yahoo. Otherwise you will have to get free data from an api.
  58. It's still possible to get the end of day data from Yahoo finance.
    1. Timestamp has been changed to unix format. It's important to submit the right time for start and end period e.g. GMT 4pm. Do a historical download and analyze the unix timestamp in the url.
    2. The new API must include crumb code.
    3. Get request must be with cookie.
    Some helpful chap has posted an Excel VBA script on how to retrieve the crumb and cookie at StackOverflow. You can adapt it to your script.
  59. There is another way to get this data without a lot of effort: .
  60. Not work (Excel 2016)