Time series DB?

Discussion in 'Programming' started by sle, Dec 24, 2017.

  1. sle

    sle

    Any pointers?
    - mostly for futures and stock tick data, but I might move my options there too
    - Linux is the desired platform, but I can live with Windows if that's required
    - Easy integration with Python and C++
    - Free would be nice (especially if it's something that has commercial support level)
    - Is there a reason to go with No-SQL vs SQL etc?
    - I have never used anything besides kdb, but I hear there are better options now.
     
  2. traider

    traider

    You can try to look at what Quantopian is using, they have gone through a few iterations doing the same thing.
     
    Gambit likes this.
  3. sle

    sle

    Hmm. They are using some form of python interface, right, even for tick data? Have you used it?

    Ps. They don’t seem to publicly mention what they use to store the data, unfortunately
     
    Last edited: Dec 24, 2017
  4. Gambit

    Gambit

    If Quantopian is an option, then consider Quantconnect as well. They have 5+ years of options data through Algoseek. Python interface.
    But these are obviously not institutional platforms. The only other option I can think of is KdB which he's already familiar with.
     
  5. Gambit

    Gambit

  6. sle

    sle

    He means I should find out what engine they are using and piggyback on their due diligence. Smart idea, since they have probably put more time into researching the platform components than I ever could.

    Yeah, I looked at them a while ago. Very expensive for what they are trying to sell. Plus, it's a true enterprise product - as a small team, I can't afford to support something like that
     
    Last edited: Dec 25, 2017
  7. Gambit

    Gambit

    Then I guess Algoseek might work. It's what Quantconnect is using.
     
  8. traider

    traider


    Thomas Wiecki

    Jun 6, 2015
    André: We have since switched our storage of financial data as mongo was indeed prohibitively slow for this. We are now using bcolz: https://github.com/Blosc/bcolz which is very fast

    Grant: We certainly encourage all contributions and if such a change were to get merged it would propagate to research within a week or two. However, we'll probably also want these improvements for the quantopian backtester so we should try to solve this problem once the right way. I know that Eddie is starting to work on this so he might correct me on the plans. I'll ask him to chime in here on what the best way forward is and how we can best collaborate with contributors.
     
  9. sle

    sle

    Oh, no, that was not the question - I got the data already (it's a mess). I need the database engine to store it all, the text files are starting to feel a bit heavy :)
     
  10. sle

    sle

    Now we are talking value added! Thanks a bunch! I've heard of bcolz but never used it
     
    #10     Dec 25, 2017