Tick-level backtesting tools?

Discussion in 'Automated Trading' started by foible, Nov 5, 2008.

  1. foible


    I'm trying to investigate a few scalping strategy ideas and if they work, automate them. I've spent some time with NinjaTrader, OpenQuant and RightEdge to see which could meet my goals. Here's what I've found.

    General Requirements/Criteria:
    - support tick-based trading and backtesting
    - would be nice to have custom graphical indicators (useful during development & debugging)
    - eventually nice to support multiple strategies
    - support running strategy on multiple symbols simultaneously

    - free for development, well priced for going live
    - fully supports custom indicators and strategies can easily add graphics to charts
    - fairly mature: bug-free, stable, used by many existing customers
    - excellent support

    - tick-level backtesting is extremely limited, available only as a "Market Replay". Tests cannot run at the same time as a market record and there's no way to use historical tick data. The backtest only runs at the speed of the replay and whenever a change is made to the strategy you need to remove the strategies and re-add them. Stats aren't integrated and there's no option for optimizing.
    - development environment feels piecemeal, dated, awkward and poorly designed especially in comparison to its competitors.
    - running strategies requires adding a new line for each instrument every day, a time-consuming, tedious and error-prone task (may develop strategies which have hard-coded lists of instruments, but this has its own problems)

    - nice development environment which supports debugging in VisualStudio
    - good support
    - can add custom indicators and graphical elements to charts for debugging and development
    - supports command-line options for running strategies making going live a breeze, the best of the lot

    - plans to add tick-level backtesting but this feature is not available yet
    - rapid changes means documentation and samples are out-of-date
    - in my tests, RE has been extremely CPU intensive and a real memory hog
    - problems importing data and limited feedback. Feels unpolished and immature. May be solid in 6mo or a year but is rough right now

    - nice development environment with a friendly IDE
    - smooth and polished, mature feature set
    - claims to support tick-based backtesting
    - claims to support multiple strategies and multiple instruments (I recall this was alright but could be better)

    - lease only available for minimum of 6 months (some song & dance about blaming their credit card company clearing fees. Ever hear of PayPal?) and my trial membership expired over a year ago when I first played with it so have to rely on memory
    - no custom indicators, no means of making annotations on the chart making debugging and development difficult especially when trying to implement pattern-recognition
    - quirky/unreliable/slow/unresponsive support
    - has development stopped? Last version was released in January and nothing has happened since (releases used to be monthly, now nothing). Because of OpenQuant's status as a scaled-down version of SmartQuant, this makes me uncomfortable about its future - will it become abandonware?

    RightEdge sounds good but the high CPU & RAM usage and its unwillingness to recognize imported data make me shy away from it has potential to becomes something good when they can put in a few months to clean up the loose ends they've ignored in the push to release new features.

    NinjaTrader has many good things going for it but the backtesting is clumsy enough for bar-based strategies. To go through the hoops to test with the Market Replay is just painful. NT is reliable and has many good features but they really need some good UI folk to come in and smooth out the workflow and improve the usability.

    OpenQuant seems to be my choice but because I haven't been able to spend any time kicking the tires and testing its tick-level backtesting and other features (like IQFeed support, something that looks very dodgy) I'm reluctant to pay $300 to try, especially when both RE and NT _sounded_ good only to let me down.

    Does anyone have any recommendations for a tick-level backtester? What do people think of OpenQuant or the other choices?
  2. ATLien


    Honestly, most of the retail solutions are very lacking in one way or another. They all want to offer a package to do everything while doing nothing well. You can peek around at some of the open source offerings, like TradeLink from the prior post, and then modify them to your liking. Otherwise, try rolling your own if you have the skill/time.
  3. foible



    I certainly have the technical capability of coding my own backtester and auto trader but I don't think my requirements are so wild that is a project which will take months not days and won't get me much closer to my goal of having automated systems. I don't think it would be a effective use of my time.


    I looked at it briefly but it doesn't have a coherent story, looks like a worse jumble than NinjaTrader and looks like it will cost me more time and effort than any commercial product. At only $50/mo (or $300/6mo for OpenQuant), any open source alternative would have to be very competitive or it would be a waste of time.

    Has anyone tried OpenQuant to do tick-level backtesting before? How is it?
  4. foible


    Neat, but the lack of prices and details about brokers or data feeds makes me think this is targeted at institutions not individuals. Probably out of my league right now.
  5. Tums


  6. foible


    Thanks Tums. Not sure why I missed that one on my first pass but I'm downloading the demo and giving it a try. It's a little more expensive than the alternatives but if it does the trick, it'll be worth it.

    Thanks for the info!
  7. ATLien


    I completely agree. I think the institutional offerings are honestly the best, but also require customization time. I've rolled my own suite, and it's taken tremendous development time, but couldn't be happier now. So that may be a long term solution if you're not satisfied with what's available.

  8. foible


    For the short-term it looks like OpenQuant is the winner. I tried Neoticker but I couldn't grok it and didn't see any compelling reason to justify the increased price over OQ.

    As for home-brewed solutions, the one ATS that I have running right now is entirely custom-built and is nicely profitable though it isn't very flexible. Maybe when I have a few more strategies bringing in an income I'll take the time to develop another one :)
    #10     Nov 6, 2008