AI, Were you running multi-symbol backtests on tick data or time bars? If I could run portfolio/multi-symbol backtests on time bars w/RightEdge and it wasn't unbearably slow I could live with that. Would that work with RightEdge? If it wasn't for that issue would you recommend the product, say, vs. Ninja? I agree, those stats are dangerous. Besides, what conclusion am I supposed to draw from a strategy run on a single realization of a time series? LOL.
it comes down to the same thing. For one to feel comfortable with a back test one has to understand the law of large numbers and its convergence properties, thoroughly understand the underlying statistics and probability theory when analyzing results. But above all I would say you need to operate over sufficiently long data sets which cover different dynamics, environments, market cycles. It always costs more to build bars than simply passing on loaded tick data. Running RightEdge over millions of tick data is slow. Running RightEdge over 1 or 2 years of hourly bars or even a year of minute bars is doable. Are they the same? No, think about how many observations hourly bars over a year get you vs. a year worth of tick data...I guess you get my point. RightEdge is just not really fast iterating over data. That the package is not really optimized for professional users shows that it offers a Jet and MSSQL adapter to load data from such databases.
AI, Using RightEdge do you think I'd be able to test time-based bars on a portfolio? In this case, a portfolio would be running the same strategy but with different parameter sets for multiple instruments. Or, even if it's possible do you think it would be unbearably slow? Yes, I was aware that the database was SQL-based. Same for Ninja I think. The only product I've seen that was truly very fast with tick data was Deltix. Everyone probably has a somewhat different take on what methods are appropriate for testing for a given strategy. For me precision/recall is most relevant. For others, the emphasis might be rigorous cross-validation, randomized trials, bias/variance decomp, etc.
Interesting thread. I tried to use RightEdge about 12 months back and ended up being dissatisfied with (a) the architecture for multi-symbol testing/trading (very clumsy to refer to symbolB when making a trading decision on symbolA, difficult to make decisions based on current portfolio state) and (b) with the support. The staff was sometimes responsive (sometimes not) and rarely helpful. Took many back-and-forth emails to get a straightforward answer. Also the user base seems small, and there doesn't seem to be anything coming out of development (still on 2010r44). I had low confidence that it was a platform to grow with. I'm currently trying a hybrid approach of roll-my-own and Tradelink. Using some TL primitives (basically broker connectors, tick processing engine/archiver, etc...) to focus development on the most important (to me) elements of the system. I'm keeping the interface to TradeLink libraries modular enough so that in the future I will have an option to rip-and-replace. I really like the transparency (at least it's open source, and very permissive licensing for reuse of code) and think the TL developer(s) have done some really solid work. I have a very hard time complaining about something that is being provided for free, even if I'd wish for more documentation etc... I'm quite appreciative that the project exists and am grateful for the developers and community that support it. Another option to consider is WealthLab (see my review in the software reviews section). I really like this platform for backtesting only, and definitely not tick level backtesting. Execution is only available thru Fidelity (they own it) unless you're outside of the US then you could connect to other brokers. Very responsive developers with a very active roadmap of enhancements. Great portfolio level visualizers and statistics. Maybe not satisfying to @AI or others that are more sophisticated about tick level simulations etc... but great for coding up ideas in C# quickly. Free for 30 days, then you need an active trader account with Fidelity (120 trades per year) to continue to use (no charge). @Amazing, I appreciate your commentary on TradeLink - very helpful. I'd be interested in similar opinions from others
For completeness, also check out OpenQuant (smartquant). I put them in the same camp as RightEdge (very similar actually) but slightly better in several ways, slightly worse in a few. Bottom line, their weaknesses are, like rightedge, dealbreakers for me but an option worthy of consideration.
Thx for mentioning it but I'm pretty much done w/the OQ guys. I bot the product a few yrs ago and they really weren't supporting it at all. I posted my dealings with them here a few yrs ago. Maybe things have changed but I'd feel pretty stupid to get burned by the same guy 2x.
Doubtful anything has changed. My biggest annoyance with OQ was the non-response to straightforward questions on the forum, especially bugs etc... I really wish Ninja had a more capable framework. Support on their forums is really fantastic, just can't make some core limitations go away. WealthLab has similar responsiveness from the devs, just isn't a credible option for anything tick related
Yes, non-responsiveness and minimal documentation - that was a problem for me, too. I actually like NT for basic systematic trading and testing. It doesn't crash much at all for me and auto-reconnect is dependable. It's only $1500 or so so I wasn't expecting top-of-the-line. Of course, I'm only using time-based bars right now. My problems with NT: 1) VERY BASIC: The trade blotter gets messed up when there's a reconnect (or, for unknown reasons) and a user can't correct this manually. So, a decent analysis of actual trades can't really be done with the NT data. 2) VERY BASIC: You can't test a portfolio except in one very limited instance. Who the hell trades a single instrument systematically? 3) INTERMEDIATE?: I'd like to test some ideas that are very portfolio-oriented and I'd like to work with limit-order book feature sets but this can't be done in NT.
You might want to look at Deltix's lower cost web-based application. It's the only product that I've seen run portfolios of tick-based strategies very rapidly. I think it's a few hundred $/month.
I can only tell you that I sense something serious happened at RightEdge, either the core developer walked out the door or something else. They were very responsive 2-3 years ago and suddenly they dropped the ball, no feedback, very infrequent updates on their message board, and the software update cycle lengthened by a huge margin...