How to judge a system ?

Discussion in 'Trading' started by cvds16, Nov 20, 2009.

  1. cvds16


    Just a bit about me first: I used to trade options (DIA) untill about 5 years ago through IB, really fun game semi-marketmaking from home, although the hours were odd living in Europe. Arbing things till they were delta-neutral and keeping track of inventory as I was sitting on either bid or ask in about 20 to 30 series of the first two months. Since IB (or better said: exchanges) introduced cancellation costs that game was over. Nowadays it wouldn't be possible anyway because of tighter spreads and penny quotations so no use crying over spilled milk. I have been trying since then doing some intraday-stuff but found out it doesn't mix well with me as a lot of patience is needed for that waiting for the right setup and I have a tendency getting too emotional taking subpar trades.
    So what I tried is finding out why I didn't have those problems when I traded options intraday and since then have been trying to develop some simple systems with hard and fast rules. There is some discretionary aspect to these too, but if explained carefully anyone should be able to trade my 'system' almost exactly the same way. So all data come from papertrading at the moment. I hope to go live in six months or so. My problem is however I now got quite a lot of data for one system (two other systems or in the proces of being developed) but what do I look for to know if these numbers are being significant, what other things should I be looking at and how do I finetune my money-management (I know this will have a big impact on my return). The idea is to improve what I got by looking at significant things.
    For the moment I have a rather agressive version of money-management that probably needs some adjusting. The system takes short-only trades of the Russel3000 stocks (it refreins from trading under certain conditions) and has been doing quite well despite the bullmarket. I can't however give away the keys to the kingdom as I think this is only scaleable up to 150k (200 k at most) with the current money-management without affecting the markets as it often trades not so liquid stocks.
    For the moment I am looking at % return each month, number of trades, max dd, win% and profit factor ...
    Any feedback would be appreciated.

    Edit: this system uses no stoplosses but only time-exit: MOC.
  2. cvds16


  3. I would look at some measure of numerical stability - how dependent is your system on just a few results during testing?

    One metric I use is to see how many of my best trades from the (back/forward)test would have to be replaced with my worst trade from the test before the results become negative.

    If that number is only a handful, then IMO the test means very little regardless of the total number of trades or time period covered.
  4. from the first 100 trades it looks clean.

    books are filled covering performance analysis. compare the results against other systems and strategies.
  5. Like any system someone would want to consider buying - get it audited.

    Depending in what it trades, you can put it on Timertrac, FuturesTruth, collective2, theta research, or other places, free or pay, and you can see how it does with r4eal data. Timertrac is free, C2 is about $100 every 6 months but gives trading stats.

    I see Tradestation stats everywhere also, but never used this.

    Even on a free broker simulator, like OpenECry or others.

    You need an adequate number of trades (INTO THE FUTURE), like 100+, at least 6 months, a profit factor AFTER TRADING COSTS of probably 1.3 or 1.4+, good sharpe/sortino, reasonable drawdowns, etc.

    In other words, treat your own system like you were gong to BUY/LEASE it from someone else.
  6. cvds16


    Thanks for the replies thus far, overall my results don't come from just a few trades, my profit factor is after costs, my biggest loser is -2.07% of the portfolio, my second biggest loser is -1.87% with the current money-management, my biggest winner is 5.85%, my second biggest winner is 3.94% and my third biggest is 3.59% (today :) ) . The other results lie somewhere in between.
    I will have a in depth look at those sites.
    I added some more calculations, let's see if they make me any wiser ...
  7. I would not consider trading a system with a PF less than 3. IMO a PF < 1.5 is noise trading and results from luck, hard to sustain. As of now my real PF is close to 4 and I am still not happy about it. My simulated PF was around 7.5 when I started trading this particular group of systems. It is reasonable to expect something like a 30% - 40% drop in simulated PF after you account for everything in real life.

    I agree about the rest...
  8. 4EXJOE


    What would you (or anyone else for that matter) consider a good sharpe/sortino?
  9. Alexis


    Ideally you'd need at least 1000 trades in backtest to make result statistically valid.
  10. Alexis


    Also, use the drawdown as a measure of risk.

    #10     Nov 23, 2009