personal back-testing setup

Discussion in 'Hardware' started by sle, Jan 1, 2018.

  1. Simples


    More like $2300 for prebuilt. I think it's an older version of Storm Trooper, aka. something like this but non-transparent:

    Though bought through local (non US) webshop several years ago. Has been robust enough and no plans to replace it yet. There are fans, but adjustable. Decent size to expand, but haven't had need for RAID or much beyond SSD and external USB HDDs.

    A decent place should provide a good custom build and warranty. I just upped the memory some and the GPU/CPU at the time.
    Last edited: Jan 1, 2018
    #11     Jan 1, 2018
  2. hmcp


    Check out microcenter sometimes they have good deals in the store . Its on 3rd ave in bklyn so not to bad of a trip from manhattan
    #12     Jan 1, 2018
  3. srinir


    #13     Jan 1, 2018
  4. IAS_LLC


    1TB ram? That must be for the whole cluster not for a single processor? Or, are you referring to a PCI solid state hd?

    What are you using for back testing software? You are right that an entire day of ES tick is around a gb, but why do you need to keep all of that data in ram memory?
    #14     Jan 1, 2018
  5. sle


    Nope, it's a single box :)

    NinjaTrader, of course.. just kidding. I got a bunch of different backtesting engines, some are built for higher frequency stuff and deal with tick data, some are more geared toward minutely data for stuff that's less sensitive to latency. In reality, it's the cross-sectional vol stuff that is RAM-hungry, once you start dealing with hyper-cubes of minutely option prices across a few hundred names it gets big very quickly.
    #15     Jan 1, 2018
  6. IAS_LLC


    Damn, I didn't know that even know that existed. What kind of chip is it ? What do y'all use it for ?

    The curse of dimensionality rules us all, unfortunately :( .
    #16     Jan 1, 2018
  7. sle


    I think it's 4x4 Xeon (Broadwell?), but I don't remember. It's blindingly fast for everything, including silly things like 1K x 1K SVD.

    PS. It's not mine, obviously - too grownup of a toy
    #17     Jan 1, 2018
    IAS_LLC likes this.
  8. Hmmmm. 1k x 1k of doubles is 8mb. SVD (assuming that means singular value decomposition) on a 1000x1000 matrix is trivial on almost any system. Did you mean 100k x 100k? That would be ~80gb but would still run in seconds on almost any system, and run much faster if, as is likely the case, you are interested in far fewer than 100k singular values/vectors (see Lanzos algo and other online methods).

    I'm impressed by the 1tb ram though. That's a lot of ram! On my database server (and its clone) I have 384gb, but it's mostly used for Oracle's inmemory column store, I can hardly imagine what it would be like to have 1tb of free memory to play with.
    #18     Jan 1, 2018
  9. sle


    @Kevin Schmit Lol, I meant one million a side - that’s what you get by typing from the cell phone
    #19     Jan 1, 2018
  10. truetype


    OK, I'll bite, how do you manipulate 1m x 1m (=1t) matrix of doubles in 1t of RAM (assuming not sparse or other special case)?
    #20     Jan 2, 2018