Where and how do you acquire historical prices

Discussion in 'Automated Trading' started by pmuvb, Apr 30, 2021.

  1. pmuvb


    I thought to ask for an advice before I start coding into acquiring the historical data to avoid getting into a dead-end solutions. I am planning to use one year worth of daily bars and 1-min bars of all US stocks for back testing my programs. I've found lists of the historical data providers from Reuters to Quandl and yahoo. I am willing to pay for the data, say, up to couple of hundred bucks a month, hence Reuters is out of equation.

    These are the steps and processes that involve acquiring historical prices (at least in my mind).

    1. Downloading the initial snap. I'd prefer for a provider to have an option for bulk download, that is to download a single zip file that contains all the daily bars and a zip file that contains all the 1-min bars. I realize the 1-min bar file is going to be a large one - around few GB by my estimate. Versus, for instance, yahoo, where you can only download one symbol at a time so I would have to go into trouble of writing some sort of script to download the data for all symbols, if it's possible at all with yahoo - I hear the have been going against allowing to use scripts on their website. Also, yahoo has daily bars but not intra-day bars.

    2. Updating historical data every evening. I see the tasks that can be broken down into:
    a) downloading today's prices for all symbols and then merging them into the main database.

    b) symbols that have been deleted, then delete then from my database (I do not want those).

    c) new symbols that have been added, then add them to the database.

    d) symbols which prices have bee back adjusted due to splits - those have to be downloaded the full one year history.

    Having said that, it looks to me that it would be easier to keep the historical data in a single file per each symbol (or a table per each symbol if I use a database in the future). The main point is I want an easy (if possible) download of the initial snap. Then I want to download the update every evening. I'd like to have a list of the symbols that have been added, deleted, and back-adjusted so I can update those accordingly, that's to avoid to download the whole snap every night.

    How do you handle all of that and what provider would you recommend for me to have an easy life.

  2. Metamega


    I used IQFeed and AMibroker for awhile. Didn’t need the real-time/intraday so ditched it as it was overkill.

    Amibroker handled all the getting the data but theirs a popular product for IQFeed called QCollector. Believe it’ll get you around the 500 limit that IQFeed has for symbol subscription.

    IQFeed doesn’t advertise as a historical data provider and outside of trading hours they’re or we’re lenient with rotating symbol subscriptions.

    QCollector I believe pulls data through IQFeeds API and puts into csv files. Might be worth looking into. Don’t think it’s expensive, maybe 100$ for QCollector, IQFeed is maybe 100$ a month unless you start adding level 2 and exchange packages.

    EOD theirs lots of options. I’d check out Norgate Data. Their tool for downloading will handle a lot of the adjustments and such. But their data gets released a few hours after market close so may be later then you expect.

    CSI believe has a download tool but they only do EOD.

    Yahoo will throttle you if your not timing your requests and if you pull too much. See it pop up here and there on Amibroker forums where users use Amiquote which will pull from yahoo. Generally if your symbol count is less then 2k and your requests are every second or so it’ll work but some modify request time or try and pull 10k stocks multiple times a night and they get blocked.

    Intraday leaves you a lot less options or at least affordable optionsbb
    Last edited: Apr 30, 2021
    pmuvb likes this.
  3. ZBZB


  4. pmuvb


  5. pmuvb



    Thanks for sharing, I am looking into IQFeed and Norgate.

    I have TD Ameritrade that let you only download per symbol, but then I have to do all the legwork of daily updating.
    murray t turtle likes this.
  6. %%
    I get mine[price+ volume] out of my notebooks;
    + off paper candle charts....................................................................... Wisdom is profitable to direct