When you are comparing two distributions, the more different they are, the less samples you need to find out they are different. So the answer to "how many trades" questions is "it depends". You can compare at any time, with any number of trades, and find a % chance that the two distributions are different. The book where I got this from is by Box, Box, and Hunter, called "Statistics for Experimenters". If you have a really strong edge, it won't take long to find out when it dissapears. If the trade methodology isn't much better than random to begin with, it will take a large number of trades. Anyway, it's a suprisingly simple process once you write/obtain that monte carlo program. And the program is not too hard to write either. As for my personal situation, I only have 18 trades so far over the course of 3 months, and everything looks peachy so far. In all my previous 2 yrs trading I didn't have a rock solid methodology like I do now, so in that way i'm a complete newbie. I think the best thing I learned is this: you don't have to lose very much at all to find out when that edge starts to deteriorate. In some cases you might still be making a little money.
The backtesting critereon is this: the distribution of entries has to kick random's ass, not just beat random. I construct a scatter plot of each entry's chance to make money; in this sample plot I put random entries. Visually, it's easy to see if there's a big difference or not.
A proposal for Eric and anyone else wishing to participate: I think it would be VERY interesting and instructive to compare when the Gaussian model and a Monte Carlo study would tell us when to activate/deactivate a system. The nice thing about this is we don't have to talk about the trading methodology itself, so it seems like we could expect a decent amount of participation. Since I do all my testing by hand, and i've just been using this technology relatively recently, I don't have a heck of a lot of examples. The two things I am trading now are working, which make for very boring examples. But I will post an example just to get the ball rolling. I was thinking maybe Eric has an example of something from the past he has deactivated. I would like to run the backtested distribution of results through my monte carlo, and then we can compare what the Gaussian and MC models tell us (and more importantly, WHEN they tell us).
Sounds like a great idea to me. Volatility BO systems worked well a few years ago and quit working. I'll come up with a nice curve system and we can try to find the best method for deactivating it. fan27
Peter, I am interested in continuing the discussion, but I'm not sure if examples will tell us much. With any set of data, optimization of the activate/deactive logic can curve fit the results to different answers. For example, assume that you have a bad trade that was 79 trades ago in your trade history. Your results (Gaussian or Monte Carlo) will depend on how many trades you use in your analysis. If you use the most recent 75 trades, then that bad trade will not be a part of the analysis, and the system will be quite different than if you consider 90 trades in your analysis. Getting back to the initial question. If I understand correctly, you use the Monte Carlo to determine when to DEACTIVATE a system (i.e. the system is no longer performing as well as the history expection). However, when do you decide when a system should be activated? For me, my activation and deactivation rules are also exactly the same (and documented earlier in the thread, if I'm not mistaken). If the paper trade history is better than "X", then the system is activated for live trading. While live trading is being conducted, I simultaneously continue the paper trading of the same system to enable comparison of the real versus paper results. Any real vs paper deviations should result in my reconsidering my paper trading order execution assumptions (mainly slippage). For active systems, the deactivation decision is based upon the results of the most recent "X" number of PAPER trades (typically, 120). It is assumed that the most recent paper trades pretty accurately matches the lilve trades, as this is continually monitored. By only considering the most recent 120 trades, my systems are able to deactivate before a long extended period of poor performance, which is certainly a goal. Typically, 120 trades might take ~2-3 months to generate, so that a bad week or two will not automatically deactivate a system. But, for example, a one month period of breakeven results (30-70 trades) will often result in the deactivation of the system. Similarly, a non-active system that has had breakeven or worse results for several months might begin to trade very well. When this happens, after about a month of very good results, the last 120 trades start to show a statistically high confidence level of profitability, and the system is activated. Anyway, I didn't intend to rehash the way I do my activation/deactivation. But, as I was asking you for more information on how you activate your systems, I figured it was only appropriate to better clarify how I do the same on my end. (i.e. seems like Monte Carlo is more geared towards deactivation for you, as I understant it) -Eric
Suppose you backtest something for the NQ and the distribution of results is: -10 NQ points / 5 times -5 / 1 -2 / 15 -0.5 / 15 +1 / 5 4 / 6 10 / 6 20 / 2 I input these numbers into the program along with a $5 commish per trade. Now, what the program does is pick 20 random trades (or how ever many you want) distributed according to the relative probabilities from the backtest. It does this 100,000 times (or however many times you want). The program outputs the distribution of results (how much $$ you end up with after twenty trades), and equally important info, the lowest account drawdown for each run of 20 trades. Basically, what we've done is give our trading plan to 100,000 trades and told them to make 20 trades and report back. I've attached a picture of the results. Conclusion: if any of our traders report back and their $10k account is below $9k, we should fire them. We should also fire anybody who dips below $9k at any time. Another way to look at it: it costs up to $1k per ctc to find out if this methodology is working. Also suppose we're making money for some time, and then we find ourselves down $1k/ctc in a stretch of 20 trades or less. That means something has changed and our results aren't in accordance with our backtested results. Back to the drawing board. A neat thing to notice: even though the distribution of individual trades is not nearly gaussian, the distribution of trade results is. Hey! That's the central limit theorem in action! When the MC trades, the trades are statistically independent of one another. This is one point I am pondering right now. Without serial dependence of certain price/volume events, I could not enjoy the edges that I do. The trades make a very non-gaussian distribution of results that makes me money. But looking at the individual results themselves, is there a serial dependence there as well? In other words, is a winner more likely after a winner? The answer is: maybe, maybe not-- it's something you have to look at for each methodology. Like Box and Hunter say in their book-- a mere "declaration of independence" is not good enough. Then you have to realize, if you are using MC, that the program assumes consecutive trades are independent.
Our time scales are different. For the edges I can find I get something like 5-10 day trades per month in a market. So I backtest the previous 6 months and the MC says I should make money. So I trade it. I now have three such systems. The first I started in Jan., the second in Feb, and the third I started yesterday. So I am really new at this. Actually, one system I worked out I didn't trade because the entries appeared to give only a slight edge over random entries. I judge entries as a separate issue according to some simple metrics I came up with, basically measuring the money making potential and comparing it to random. I have attached a scatter plot of what a good entry should look like. That's a quantifiable edge. After that, it's up to the exit to extract the cash in a reasonable manner.
Also a message to newbs: I hope you can now see some of the things you need to do to make money. You can't just use the latest gizmo from your software vendor like some of the testimonials I have read on various sites: 'Just a quick note as the trading day progresses. I have to say I was a little worried when you brought out the new ------. As usual you were right. Brilliant is the word that comes to mind. I am using the ------- this morning. I am up 18 points in the Russell using these ---- alone. See attached chart. I am ignoring the other indicators generally. I look for the ------ to be on the bottom for a long and then look at the ----- for a confirmation of entry. If there is no ----- on the ------ I refuse the entry. So far today six trades, four winners for the aforementioned 18 points on two contracts. I can go play golf now! I added an arrow and the word âChopâ to the chart. When there are no -------------- it seems to indicate chop. Thatâs a wonderful chop indicator if it works out. Thank you for your continued outstanding work in our behalf.'