Faster Processing For Backtesting?

Discussion in 'Automated Trading' started by Norm, Dec 19, 2005.

  1. A RAM drive is not a good idea. The OS does filesystem buffer caching in RAM, that's more or less a RAM disk for the data you use frequently.

    You're on the right track though. With the CPU usage you posted you are clearly IO bound.

    What will help? Assuming you can't do anything to optimize the code or simplify the dataset:

    1) More system RAM. With enough buffer cache you will no longer be IO bound. Also helps to shut down programs that use a lot of RAM.

    2) If that doesn't help enough, faster IO. A RAID0 with a few 10k drives would probably help.

    Martin
     
    #11     Dec 20, 2005
  2. I still doubt a ramdrive will help....but it's worth a shot. I used to use one from here:

    http://www.ramdisk.tk/

    SSB
     
    #12     Dec 21, 2005
  3. fletch2

    fletch2

    You will get far more dramatic speedup by rewriting your strategy in a compiled language like C++, than by trying to wring more performance out of your hardware.

    It is a lot of work, but if you are serious about accelerating your calculations, it's the only way to go. By tweaking hardware, you might, best case, get 50% speedup. By re-writing, you could probably get a factor of 5 immediately and 10 or 20 with some work.

    Cheers,
    Fletch
     
    #13     Dec 21, 2005
  4. Optimax has solved this problem. Why spend hundreds on hardware or weeks on code to get something that will never even approach what this <$100 service does?
     
    #14     Dec 21, 2005


  5. http://www.tomshardware.com/2005/12/05/hyperos_dram_hard_drive_on_the_block/
     
    #15     Dec 21, 2005
  6. fletch2

    fletch2

    Optimax appears to be just a genetic alg optimizer built on top of TS.

    (BTW, their advertising is disingenuous, comparing exhaustive search to the genetic alg. That's apples to oranges.)

    It might be a reasonable solution for the OP if his complexity is in the expanse of the parameter set, not in the strategy itself. If the complexity is in the strategy, then a genetic alg won't help him. If it's in the parameter space, it probably will.

    I bet you could write a swarm optimizer in less than day that would compare favorably to the genetic alg, and have the combo still test out factors of 10-20x the TS based solution + Optimax, due to the speedup in the strategy eval phase. But, all of that requires skills that may not be worth acquiring for OP.

    Cheers,
    Fletch
     
    #16     Dec 21, 2005
  7. newbunch

    newbunch

    I tried some genetic algorithm optimizer for Excel (don't remember which ones) and found them to be much slower than Excel's solver. But it depends on the type of problem you are trying to solve.
     
    #17     Dec 21, 2005
  8. try amibroker and use io.zip to optimize multiple parameters. shouldnt take long.
     
    #18     Dec 23, 2005
  9. Basically you are right.
    On rewriting: it is often not that much work if you know what you are doing. Mostly, speed has to do with only a few segments of your code. Rewrite these, don't bother with the rest.
    It is difficult to give any definite figures, but typically this may concern less than 5-10 percent of your code. Profiling is a good idea.
     
    #19     Dec 25, 2005