Why use a database?

Discussion in 'Data Sets and Feeds' started by onelot, Oct 9, 2004.

  1. The technology has changed considerably.

    Your stats may be accurate for single individual drives but using a program like fancycache can yield a 1000x performance boost.
    http://www.romexsoftware.com/en-us/fancy-cache/index.html

    Infiniband / Striping / hybrid arrays of SSD + disks / clustering etc are all consumer cost reachable technologies today.

    1 GB per second sustained read/writes is doable today and with clustering over infiniband 5gb/sec is attainable while remaining affordable.

    Even some of the journaled posts on index settings and speed may not accurately reflect the hardware impact.

    Try these simple changes for a 1000x boost

    For DB Use covering indexes\ 64k page size (cuts read i/o in half and better cache hit rates)

    For Disk use Fancycache setup with write only cache (1gb 512k blocks write deferred 2 sec) this buffers writes to achieve near SATA 3 spec speeds. Deferring the writes for 2 secs increases cache read hits and reduces hammering of i/o write operations... less fragmented sequentially writes of random data... which in turn optimizes read i/o.

    CAUTION: Write deferred caching can cause serious data corruption on power failure/system lock ups.









     
    #131     Dec 19, 2012
  2. Creating a single point of failure/error...
    At a complex, 3rd party, NAS device...
    Is very poor design for a trading operation.

    You are either spending $5,000,000/year and completing with banks...
    Or you are not playing the latency/data warehousing game...
    There is no real middle ground.
     
    #132     Dec 19, 2012
  3. nitro

    nitro

    #133     Dec 19, 2012
  4. Of course a simple binary file is very fast but there are many issues that need to be handled very carefully. You end up recreating a lot of your own infrastructure to handle these real-cases that you won't even realize unless you do it. It takes time and resources to get this capability and for small timers arguably better spent generating profits rather than trying to catch up the rapidly increasing technology curve.

     
    #134     Dec 30, 2012
  5. lemmer

    lemmer

    And its open-source too, dozens of intelligent people looking at its source code daily and checking for vulnerabilities.
     
    #135     Dec 31, 2012
  6. The basic Art of Trading has not changed much...
    People who try to substitute off-the-shelf technology...
    For hard earned trading EXPERTISE...
    Are wasting their time.

    Actual professional EXPERTISE in any field = 10,000 hours...
    And cannot be circumvented with bells and whistles.
     
    #136     Jan 3, 2013