historical data question

Discussion in 'Data Sets and Feeds' started by caementarius, Dec 4, 2008.

  1. I'm trying to put together the following historical data at a reasonable price:

    1-minute bar data for all SP500 components for 60 trading days randomly selected from the past year

    To buy DVDs of the data would seem to cost a fortune. I started putting it together myself from IB data but the throttling is making it a painful task. Anyone have any ideas?
  2. Tums


    sign up for a 30 day trial at one of the data providers.
  3. heech


    I don't pretend to be an expert in these issues, but I think my Ninjatrader (even with the free account) combined with an account from TDAmeritrade/Izone could give me what you're looking for.
  4. TIMMY57


  5. This may be what you are looking for: Historical Intraday Database, Stocks Edition

    Should be everything you are looking for: 1 min, has the full S&P 500, and is ok in price ($150).

    I use the market edition for back-testing, but the stocks edition has more symbols.
  6. This looks very promising - thanks!

    I've spent a lot of hours trying to scrub data from a few providers. Does anyone have any feedback on the quality of data from pitrading.com?
  7. Here is some data on the quality of the 1-min data from pitrading.com. Overall I am pleased because I think it will be sufficient for system development purposes. However, there are some gaps.

    There should be 390 bars every day. It looks like the most popular stocks have 390 bars consistantly.

    I loaded all the 2008 data for the S&P500 and calculated the following statistics, if anyone is curious.
  8. Hi all:

    I want to test a few ideas and concepts, for which
    I need SPY historical 5-minute intraday OPTIONS data,
    just for about 5 strikes above and below the underlying.

    Please recommend vendors or sources.

  9. I think this is normal for ascii data. Not every minute of the day has active trades and only active minutes are time stamped. Most prevalent on low volume stocks, ETFs, and overnight sessions.

    I am using the market version so I can only comment on the symbols that overlap.

    Your charting package should have a feature that allows you to view empty (NULL) bars so you get 390 bars per day. Some will of course be blank when nothing happens.
  10. optimum


    Caementarius (and zqtrader),

    I found Caementarius' spreadsheet very interesting, since I am contemplating buying their next DVD release.

    I sent an email with the link to Caementarius' posting to PiTrading (support@pitrading.com), also asking when the next DVD would be cut and got this reply, posted with their permission:

    > When is your next scheduled release and when it is released thru what
    > date >will it be current?

    Thanks for your inquiry.

    The historical data CDs are updated on a quarterly basis. The current version is updated through October 3rd, 2008. The next update is scheduled for January 12th, 2009; updated through January 9th, 2009.

    > Can you comment on the gaps in data?

    From a quick review of the forwarded spreadsheet, it appears the author of the analysis is not familiar with the ASCII file format.

    In the ASCII specification, a bar is only recorded if the volume within the time interval is not zero. If no transactions occurred during a 1 minute time frame, the bar is not recorded to the ASCII file as it would be redundant.

    The author assumes each market day is is composed of a constant 390 bars based on a 1 minute interval (6 1/2 hours * 60 min/hr). This is not a correct assumption for various reasons. The simple example of the error in analysis is July 3rd, 2008 where approximately 210 bars are recorded. On this day, the market closed early for the holiday on the 4th, therefore assuming 390 bars would not be correct based on the
    3 1/2 hour trading day.

    All ASCII feeds will show similiar missing bars where no trading volume occurs during the underlying interval of the data.

    You are free to quote the above reply in your forum if you feel the answer will be helpful to your other members.

    Any other questions or comments, please feel free to ask.

    Best Regards...

    -- Michael Schevis
    -- Pi Trading, Inc.

    #10     Dec 22, 2008