Why use a database?

”Knowing what you intend to build”?? Then we must be masters of all markets, with our crystal balls able to foresee the design of a profitable system before any testing or simulation is actually done.

Discipline balanced with flexibility and adaptability. Markets change. Systems must adapt.

Agreed!

Same too you and Sparohok. Have a great weekend and successful trading in the future!

 

I agree completely.

 

hi everybody!

I'm working on a similar problem-creating a tick by tick database for market analysis.

In your experience, and what is the best data feed for this?

MY SELECTION CRITERIA:
1. I am price-sensitive.
2. I would be happy to start with limited range of stocks - i.e. 300 active issues form NYSE.
3. I prefer Internet delivery rather than dedicated line.

Please tell me what the data feeds you use and what pros and cons you've experienced.

How do you get data form the network into your DB/ Flat file?


Thank you all for replying.

 

Oops!

Forgot to mention:
I need streaming real-time stuff, not end-of-day or1 minute or snapshots.

 

I'm in the same spot, I'm just starting to capture tick data. So I hardly have all the answers. But I've heard multiple people recommend IQFeed for reliability and data quality. However they have a $300 fee to get the API which is really irritating. For now I am just downloading data through QuoteTracker's HTTP server using a Python script and storing it in a PostgreSQL database.

In my experience, broker quote APIs do not provide sufficiently reliable data, few symbols, and no backfill for gaps.

Martin

 

There are exceptions on some of those points. IB's real time data is excellent in terms of accuracy, reliability, speed and cost. You'll have to provide your own backfill.

 

I have actually done some interesting tests on some of this. It turns out that a RAIDed SATA solution worked best on "BLOBs" like movies. RAIDed SCSI still rules on the row type operations (or column operations done now by some dbs) of data stored in relational tables.

nitro

 

I prefer a small db, the entire db can be read into memory.

 

I'm building "KDB for the rest of us" and the core will be open source. Visit http://wagerlabs.com/uptick if you are interested and click on "forum" in the blog headline to join the discussion group. I'm looking to process, say, 1000 ticks per second but my main emphasis will be on similarity search in time series data, i.e. being able to select a portion of a chart and find stocks that match the pattern.

 

Outstanding site. Thanks for the link and good luck with your project. If it works I would be interested...

nitro