EOD Futures Data - Databases Only

Discussion in 'Data Sets and Feeds' started by krraymond, Feb 23, 2011.

  1. krraymond

    krraymond

    Greetings all.... I have a bit of a dilema I'm hoping someone can assist me with...

    I'm building a futures trading application, Order and Trade Management, MTM, Scenarios, Simulations, VaR, etc. I am currently utilizing a database for the daily EOD data, but realize that 30 markets, an average of 6 contracts per year, 10 years of data and 300 plus daily data points will grow the database too large, too quickly. However, a series of flat files makes programming a bit ugly and cumbersome. Ultimately, I'm also looking to automate the entire process of downloading the data into a database.

    So my question is - what are others doing? It seems many vendors out there utilize the flat file format for data storage - i.e. Pinnacle, Reuters DataLink, etc. Are there vendors/applications that you utilize that fit the bill, deploying a database for data storage?

    Thanks in advance for your guidance.

    Best,

    Keith