Probably once luckily finding some real edges, you then would know quite well and sure it's not merely luck any more, even without knowing how to validate it.
Here is also a link related to confidence interval: http://www.elitetrader.com/vb/showthread.php?s=&postid=710438&highlight=confidence+level#post710438
Yes, I do use a short term confidence test to determine when to activate/deactive a system, and not necessarily the overall history. Typically, I will arbitrarily use the most recent 120 trades, which might take me back 2-4 months into history. In this way, I will have my system automatically turn off when it begins to get out of synch with the market, hopefully preventing a downward spiraling equity curve. I have not tested the 'performance' of this activate/deactivate strategy. In other words, I have not looked at the oveall equity curve results associated with changing the 120 trade confidence test to 90 days or 200 days, etc. Nor have I tested the results of simply leaving all systems active, all of the time. The overall equity curve looks very stable and smooth to me. My use of the activate/deactive logic is more of a risk control measure to ensure that I downsize my trading when things turn sour, and not necessarily an optimized on/off logic to maximize profits. Also, I have never tried 'trading' my equity curve before, by deactivating my trading when the equity curve looked weak and due for a selloff, etc. -Eric
Let me try to address what the confidence level calculations from the linked thread actually measure (more detailed information may already be on that thread, not sure). The confidence level attempts to mathematically determine the likelihood that a system has an expected average profit above zero, based upon the date you give to it. So, for example, the following three series of trade results will have exactly the same confidence level: a) +$50, -$20, +200, +$10, - $20 b) +$500, -$200, +2000, +$100, - $200 c) +$5000, -$2000, +20000, +$1000, - $2000 In this case, examples b and c are the same P&L results, but multiplied by a factor of 10 and 100, respectively. All three examples will have the exact same statistical confidence, since the statistical confidence does not measure dollar profit or loss, but the likelihood that the average profit is above zero in the entire population (versus this small sample) of trades using this system. For your systems tested, therefore, you can conclude that the implementation of the stop-loss and profit objective makes it more likely that the system will be profitable over the long run. Note that I am assuming that you have roughly the same number of data points for both tests (with stops/targets, versus without), because simply the increase in the number of datapoints will also increase your confidence level. This makes sense, right? You become more confident about a system's profitability after steadily making net profits in your first 30 trades, versus only your first 10 trades. So, to be fair comparing your systems with and without stops/targets, you should be sure that there is not a huge difference in the number of trades being using for each method in the confidence calculation. I agree. Using the statistical confidence calculations is merely a tool, and does have shortcomings. I find it to be very valuable as well, and much better that what I used previously (manually making the on/off decision, without any structured methodology). Another good point. Systems that work across many markets are very nice when you find them. Another thing I look for is a system with very few optimized parameters, or better yet, a system that works well before any optimization is even done. Best of luck, -Eric P.S. I am copying this post to the activate/deactive thread (Journal section) to help consolidate this confidence discussion in a single thread for better reference. http://www.elitetrader.com/vb/showthread.php?s=&postid=746183#post746183
+================ Wyrang; a]Does it work or work around at least 1 bull/1 bear market; not a prediction, but can give you a hint. b]Since 80% of small businesses fail, dont believe in luck; but make sure we are 20% that win. Savings account may help. z]Make sure it fits your gifts/personality, like longhorn bulls ,or ; Teddy Roosevelt wouldnt hunt small bears, but something like polar bear type hunting got in his heart.
I have a great way to figure it out. Look at your account balance 3 months after using system. That will tell you exactly what is happening.
Actually some people also try the reverse logic for the mean-reverting effect... For some trend following systems, the profitability relies mainly on a few big trades. Skipping them might be problematic.
What if what you think you are exploiting is as real as Santa? You need some sort of subjective measure to assure yourself, and do this at low cost.