Yahoo Historical Data - Did they change the URL recently?

Discussion in 'Educational Resources' started by Rationalize, May 16, 2017.

  1. d08

    d08

    No, not at all. They seem to be fixing it yet again but now it's inconsistent. The website has one set of data, downloaded CSV another. At least they are doing something but...makes you wonder what kind of people do they hire, they seem to be struggling with the basics.
    Take a look at EFA.

    Pre 19th June dividend on website:
    19 Jun 2017,66.64,66.79,66.60,66.66,65.60,26706100

    Downloaded CSV:
    2017-06-19,66.639999,66.790001,66.599998,65.598000,66.660004,26706100

    Notice the switch in the 5th and 6th column. The correct adjusted close should be 65.60.
     
    #141     Jul 12, 2017
  2. inCom

    inCom

    I can relate to your frustration, I guess, which is the same as any of us who has been using Yahoo so far, but.. truth be told, their data was not perfect even before may. Glitches, missing samples and "null"s have always been there. Not to mention whole pieces of history that were missing if you downloaded today, magically reappearing the following day. Let's hope things will get better. I'm not defending them, but they are clearly passing through a transition phase, euphemistically speaking.

    Only, my point is: maybe other sources are not bullet proof either. Nasdaq.com, for example, gives still other numbers compared to Quandl. Which one is telling the truth? Would you be better off by choosing any paid service among others, if all of them gave different data? This is what I'd like to understand.
     
    #142     Jul 12, 2017
  3. d08

    d08

    Which numbers differ? Yahoo's adjustment formula is different from Quandl's a little. This is where the differences come from.

    I contacted them regarding the missing history (beginning of July) and it was fixed the next day, I also contacted them because June 30 was missing for almost everything just a few days ago (same for 2016). Fixed the next day. So they're very responsive.
    I doubt the paid sources are ideal either but I'll let others comment further.
     
    #143     Jul 12, 2017
  4. toonerdy

    toonerdy

    I am grateful to Dennis Lee for writing updated Yahoo historical data fetching code at https://github.com/dennislwy/YahooFinanceAPI , discussed at https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working/44050039 .

    Using his code as a guide, I have written the following shell script, which I hereby place in the public domain, that uses the grep, sed and wget programs for fetching Yahoo historical data for a stock symbol in comma separated variable (csv) format.

    Of course, I disclaim any responsibility for any defects, bad data, or any other results of your using this script. Use it at your own risk.

    Code:
    #!/bin/sh
    
    cookiefile=/tmp/saved-cookies.$$.txt
    
    if [ $# -lt 1 ] ; then
      echo "Usage: yahooquote-historic symbol" >&2
      exit 1
    fi
    
    symbol="$1"
    
    download_summary() {
      wget --quiet --no-check-certificate --no-cache --keep-session-cookies \
        --output-document=- --save-cookies=${cookiefile} \
        "https://finance.yahoo.com/quote/${symbol}?p=${symbol}"
    }
    
    extract_crumb() {
      grep CrumbStore |
       sed 's/^.*CrumbStore/CrumbStore/;s/}.*$//;s/.*"crumb":"//;s/"$//;s|\\u002F|/|g'
    }
    
    crumb=$(download_summary "$symbol" | extract_crumb)
    
    period2=$(date +%s)
    
    wget --quiet --no-check-certificate --output-document=- \
      --load-cookies=${cookiefile} \
      "https://query1.finance.yahoo.com/v7/finance/download/UVXY?period1=1&period2=1500194366&interval=1d&events=history&crumb=${crumb}"
    
    status=$?
    rm -f "${cookiefile}"
    exit "$status"
    
     
    Last edited: Jul 16, 2017
    #144     Jul 16, 2017
    truetype and Simples like this.
  5. Be careful using data from Google and other sources. Check for missing entries. Google misses dates over entire database.
     
    #145     Jul 30, 2017
  6. toonerdy

    toonerdy

    I made a mistake in the script I previously posted, that has the effect of providing historical data only before July 16th, 2017. I had left a numeric unix time code for a time during that date, 1500194366, in the final wget command in that script, instead of "${period2}". Here is a corrected version of the script.

    Code:
    #!/bin/sh
    
    cookiefile=/tmp/saved-cookies.$$.txt
    
    if [ $# -lt 1 ] ; then
      echo "Usage: yahooquote-historic symbol" >&2
      exit 1
    fi
    
    symbol="$1"
    
    download_summary() {
      wget --quiet --no-check-certificate --no-cache --keep-session-cookies \
      --output-document=- --save-cookies=${cookiefile} \
      "https://finance.yahoo.com/quote/${symbol}?p=${symbol}"
    }
    
    extract_crumb() {
      grep CrumbStore |
      sed 's/^.*CrumbStore/CrumbStore/;s/}.*$//;s/.*"crumb":"//;s/"$//;s|\\u002F|/|g'
    }
    
    crumb=$(download_summary "$symbol" | extract_crumb)
    
    period2=$(date +%s)
    
    wget --quiet --no-check-certificate --output-document=- \
      --load-cookies=${cookiefile} \
      "https://query1.finance.yahoo.com/v7/finance/download/UVXY?period1=1&period2=${period2}&interval=1d&events=history&crumb=${crumb}"
    
    status=$?
    rm -f "${cookiefile}"
    
     
    #146     Jul 30, 2017
    Simples likes this.
  7. inCom

    inCom

    In the end I gave up chasing Yahoo and their mess and started using Quandl API only. Their data seems to be quite stable and reliable, and it's true that they email back within 24h to any question. Also the API is very linear and easy to use.

    They don't offer ETFs such as SPY or QQQ in their free plan, so I will probably subscribe to some of their paid versions.

    Case closed (for now).
     
    #147     Aug 2, 2017
    d08 likes this.
  8. d08

    d08

    Just an update. Yahoo data as of the moment is still full of abnormalities. There are too many to list and some appear to be deliberate so that the data is unusable if downloaded.
     
    #148     Aug 19, 2017
  9. d08

    d08

    Anyone tried them? Looks great but there's no way to properly evaluate their data quality without subscribing.
     
    #149     Aug 19, 2017
  10. mokwit

    mokwit