I've been using yahoo historical quotes and bringing it into excel using vba for years. Over the last couple of days, the historical quotes stop loading after about 30 quotes. To get it working again, I have to wait 10 minutes or so or change my VPN to another proxy. Anyone else experiencing this lately?
No, I meant it stopped working before VPN. After about 30 calls, It only works now when I use a VPN proxy. It's like they are now tracking IP address and limiting it.
It would require updating/adding code and the knowledge to do so, but consider making your flow more free-data-supplier-friendly. Doing so may actually solve your problem, in this case (and help to keep free suppliers ... free). More "friendly" as in the historical data is saved to the harddrive, etc. (csv, xlxs, etc.); and only new data is downloaded daily. Until then, and if you are automating the downloading of multiple instruments; adding an appropriate delay between instruments might result in the 10 min. lockout no longer appearing. You'd have to experiment to find such a delay, if it even exists. Coding this is simpler than adding code to automate dealing with the 10 min. delay. (I assume you currently manually restart the process.)
Thank you for above tips. I will try putting in a delay to help cure problem. I like your suggestion and would like to be a good scraping citizen but for someone with my limited programming experience, it would be a massive undertaking for me to store all that data, figure out how to link it all back in(I currently just use excel) in or update my excel calculations and automate the real time error checking in manipulating the data. Has anyone else recently started experiencing this from Yahoo? I'll like to know if they started ISP tracking or changed something in their system necessitating me to change how I handle cookies and crumbs?
I understand. (btw, it can all be done via Excel and Excel VBA) www.tiingo.com also has a free tier and api.
I don't think this has changed for about four years, but one thing that you could do is refresh the cookie crumb every 10-20 minutes. For example do the equivalent of, https://www.mapleprimes.com/posts/208409-Downloading-Historical-Stock-Quotes Another thing that might help is have your requests alternate between the different IP addresses for yahoo finance. Code: $ nslookup finance.yahoo.com Server: 1.1.1.1 Address: 1.1.1.1#53 Non-authoritative answer: finance.yahoo.com canonical name = edge.gycpi.b.yahoodns.net. Name: edge.gycpi.b.yahoodns.net Address: 69.147.86.12 Name: edge.gycpi.b.yahoodns.net Address: 69.147.86.11 Name: edge.gycpi.b.yahoodns.net Address: 2001:4998:20:800::1000 Name: edge.gycpi.b.yahoodns.net Address: 2001:4998:20:800::1001 And to be friendly, your code should sleep a little between requests (e.g., 100 milliseconds -- more before a retry to recover from a failed request).