I wrote some software myself to data scrape intraday data from barchart.com. On November 1st data will no longer be contained in the html. There is a totally new build of the site coming soon. I wrote a program to scrape the new site but it is slow as hell and crude to say the least. I guess I should just be happy that I have a solution. I was wondering if anyone could tell me a better way to scrape https://beta.barchart.com ???? All of my methods were proprietary, I have no knowledge of the more mainstream ways to data scrape with Beautiful Soup, Selenium,Mapit.PY ect..ect.. I am just curious if any of you wizards out there can pull data from it? Thanks.
Barchart data is available on www.quandl.com but you have to pay for it. There is a lot of free data on www.quandl.com what are you scraping?
You could try scraping data from a different site like www.freestockcharts.com or www.tradingview.com Tradingview has realtime data from BATS for free. What about using the interactive brokers API?
The best for this is Nadaq... I think. But, yes I think is better if you pay for it. Interactive Brokers data is not so expensive. You can use this method: Reqmktdata method: https://www.interactivebrokers.com/en/software/api/apiguide/java/reqmktdata.htm
Thanks for the advise everyone. Buying data is the best solution but I am so used to getting data for free I need to update my tech skills. I should be using APIs instead of building everything from scratch. I was basically just treating the website as one big "string" and parsing the unstructured data from there. BTW...getting EOD data for free is no big deal. I was scraping intraday one minute OHLC data. I think the one minute data is harder to get for free.
Yeah you are right. I was not hitting the site with HTTP requests non-stop. I was grabbing everything in a single pass. I don't feel like it was "abuse" but I understand that there is a fine line between high usage and abuse. I guess any form of automation is a form of abuse. They intend for you to view ads ect...