I am looking to scrape a list of stocks from a Yahoo screener. I was wondering if anyone knows how to get the list without having to go from page to page. For example this link https://finance.yahoo.com/screener/unsaved/53f27388-eb67-4c89-92a1-132d754a6fb1 (if it is still active), returns a list of 540 stocks but yahoo only allows up to 100 to be viewed at a time. I would like to scrape/download the list in one step and not repeat it 6 times (as in this case).
It is a post to: https://query1.finance.yahoo.com/v1...S&formatted=true&corsDomain=finance.yahoo.com With the following post data: {"offset":0,"size":25,"sortType":"DESC","sortField":"eodvolume","quoteType":"EQUITY","query":{"operator":"and","operands":[{"operator":"or","operands":[{"operator":"EQ","operands":["region","us"]},{"operator":"EQ","operands":["region","ca"]}]},{"operator":"gt","operands":["percentchange",3]},{"operator":"lt","operands":["lastclose52weeklow.lasttwelvemonths",10]}]},"userId":"","userIdType":"guid"} It returns json with the data for all 540 names. You may have to obtain a cookie and include it in your request header. How to do that is covered in a previous thread about yahoo api problems. You should be able to find that thread easily using the ET search facillity.