System Performance Score

Discussion in 'Automated Trading' started by kut2k2, Feb 28, 2013.

Thread Status:
Not open for further replies.
  1. kut2k2

    kut2k2

    You've completely misinterpreted the formula. There is no absolute P&L. The NOBF is based on percent returns, not absolute returns.

    No need for me to "experiment" further. The formula you posted looks pretty much the same as the SQN with all the same flaws.
     
    #11     Mar 18, 2013
  2. Ok, let's try your latest version of the formula, with the percent returns.

    System A: [-8, -8, -8, +25, +25]
    System B: [-1, +1, +1, +21, +50]

    SPS(A) = 0.450763
    SPS(B) = 0.446332

    So, according to your formula, system A is better than system B, while everyone with common sense would agree that system B is way better.

    My own formula rates system B at about 2.3 times better than system A. Additionally, Sharpe's, Sortino, SQN, Kelly, Profit Factor, Max DD, Net profit, and pretty much every other performance measure would also rate system B higher.
     
    #12     Mar 18, 2013
  3. kut2k2

    kut2k2

    All you've proven is that the NOBF can't handle ridiculous extremes like your system B, which never occur IRL. Calculating the true Kelly fractions gives the correct order for the SPS relationships.

    Kelly(A) = .026
    NOBF(A) = .018 = about 70% of the true Kelly
    NOBF(B) = .0245 = about 3% of the true Kelly
    Kelly(B) = .734

    SPS(A) = 0.65
    SPS(B) = 13.4

    That's quite a lot more than your 2.3 ratio.
     
    #13     Mar 19, 2013
  4. kut2k2

    kut2k2

    In fact, system B is so out of phase with reality that higher-order polynomial approximations of the Kelly fraction get farther away from the exact value rather than the norm of getting closer.

    Linear approximation = 0.0245
    Cubic approximation = 0.0225
    Quintic approximation = 0.0217

    :eek:
     
    #14     Mar 21, 2013
  5. Bottomline: SPS is a "bust"...correct ?

    It's just some sort of statistic, but does not correlate to true performance....right ?
     
    #15     Mar 21, 2013
  6. kut2k2

    kut2k2

    So what performance metric do you prefer?
     
    #16     Mar 21, 2013
  7. Sharpe or Sortino Ratio.
     
    #17     Mar 21, 2013
  8. kut2k2

    kut2k2

    Pick one.
     
    #18     Mar 21, 2013
  9. I'd say Sortino then.
     
    #19     Mar 21, 2013
  10. kut2k2

    kut2k2

    Fail.

    One key characteristic of a good performance measure should be its universality.

    For example, horsepower, a widely accepted measure of engine performance, is universal: its measurement does not vary from user to user or from year to year. But the Sortino ratio, the Sharpe ratio, and similar ratios all depend on something called "risk-free return", which is wildly open to interpretation. This risk-free return varies from year to year and probably from user to user as well, making it very unreliable, and by extension, making the Sortino ratio unreliable as well.

    We want a measure of performance that leaves no value open to misinterpretation because of different benchmarks among users, just as horsepower is not open to misinterpretation by those who know its definition.
     
    #20     Mar 21, 2013
Thread Status:
Not open for further replies.