1. do i understand what market mechanism makes the approach viable? 2. do i understand exactly what a profit-producing market has to look like (relative to model)? does that historically exist? 3. do i understand exactly what a loss-producing market has to look like (relative to model)? does that historically exist? 4. is the expectancy - not probability, expectancy - of 2 higher than that of 3.
Your score will only increase if your average stays the same and your standard deviation stays the same. That said, you are correct, if the average and standard deviation remain unchanged, as you add more and more data points into the picture, these data points rightfully further solidify your expectation that this IS a profitable system that you are assessing. => In other words, yes, the confidence level will rise with increasing supporting data. Often, though, we don't have available to us 1000's of valid data points with which to analyze, and we must make our best determination of the profitability of a system based upon a smaller data set. In this situation, this analysis method can be helpful, IMO. That is close, but I think I should clarify. Think of a system as having 10,000 P&L data points available, but you can only look at 50 of them (and perhaps the other 9950 data points are only known in the future). By looking at ONLY the 50 datapoints, our goal is to estimate the likelihood that the AVERAGE of all 10,000 datapoints is above zero (i.e. profitable system). So, if the results of our calculations say the system is profitable at the 98% confidence level, then that is saying that it is 98% likely that the entire population of 10,000+ trades will have an average P&L above zero, based upon our analysis of the 50 trades we saw. There is still a 2% chance that we just got a lucky 'sample' of data, and that the entire population of trades will have a negative average P&L. (Note as always that this analysis is subject to some errors due to not having a P&L distribution that is Gaussian, etc). I think once you have used this technique for a while, you will start to get a sense on what numbers are needed to give you sufficient confidence in your system in order to trade it. I mentioned the values that I use, but that will not be appropriate for everyone. Recall from Acrary's thread that he didn't want to trade a system unless he was assured that 99%+ of all trading months were profitable. For me, I don't need that level of confidence in order to trade a system, but each trader must answer that question for himself/herself. -Eric
EricP, thanks for a very interesting thread. I can't seem to replicate your values of $2,652 and $1,875 for STDEV and STDEVP. I get $1,760 and $1,607 respectively.
it's worse than that. systems using tight stops (as one example) have arcsine-like P/L distributions and using this approach will not just yield less than perfect results, it will yield flat out wrong results. using variable trading amounts invalidates the numbers even further.
You are 100% correct. Your numbers are correct. I quickly input the six numbers into an existing Excel spreadsheet, and apparently the first four numbers were formated as text, or labels, so when I did the Std Dev calculation, it only based it upon the last two trade values. Thank you for catching that. Perhaps a moderator could correct my earlier post and replace the $2,652 and $1,875 values with your correct values of $1760 and $1607. If so, they can also delete these last two posts, if they would like, to prevent confusion. Thanks, -Eric
Very good point. The more non-Gaussian (non-normal or non-'bell-curve') your P&L data is, the less accurate the results of this analysis will be. I find that it can still be useful data, while perhaps being 'flat-out wrong' in a rigorous statistical sense. Don't forget that the goal is to find a useful technique to analyze our P&L data in which to make rational judgments for activation/deactivation. Despite it's limitations, I have found this to be a very useful method for this purpose and have been using it for 18 months in my trading. I have not tried using this technique for a system such as you are suggesting. For example, a system that makes occassional profits of $400 to $1000, at the expense of having 80% of the trades hit a fixed $75 stop loss. Would this unusual data distribution lead to the results being worthless? I don't know. Thinking off the top of my head, I believe that I would still find the results to be useful and valuable. However, people using any sort of P&L distribution should evaluate how the results look after doing the analysis on various systems and decide for themselves whether it appears to have a use for them. I suspect that it would, regardless of the distribution, although the statistical 'accuracy' would be reduced. -Eric
kestner uses the following when monitoring stategy degradation. he calculates a linear regression of the equity curve with bands plotted two standard errors above and below the fit. he halts trading of the strategy when the curve breaks below the lower channel, and a decision is made to modify or toss the system.
Guys aren't we just talking about the Sharpe Ratio here? A question was raised about variable dollar profit amounts caused by rollover of profits screwing things up. Might this problem be mitigated by analyzing what percentage profit was taken from the underlying (or better yet what portion of profit in ATR terms)? EricP I very much agree that a robust solution now is better than a perfect solution later so I appreciate your methods. I must also commend damir for keeping us on our toes with the specifics. Keep posting guys. This is a good thread.
Another idea is to apply technical analysis to your equity curve. The equity curve in and of itself has pullbacks (drawdowns), it has a moving average, etc. You can devise a position sizing algorithm that adjusts trade size based on your confidence level, i.e., your confidence level and therefore position size may increase as the drawdown percentage approaches historical levels, assuming the system isn't deteriorating. It gets a bit tricky, kind of like a decoherent qubit.