Forums (http://www.elitetrader.com/vb/index.php)
- Journals (http://www.elitetrader.com/vb/forumdisplay.php?forumid=29)
-- Activate/Deactivate System? (http://www.elitetrader.com/vb/showthread.php?threadid=36083)
Activate/Deactivate System?
When reseaching a trading system, at some point you must make a decision about whether your new pet system is 'worthy' of activation. Similarly, once activated most trading systems will, at some point, go through one or more periods of poor performance and you must determine whether this is just an inevitable minor drawdown, or whether this system is now 'broken' and must be deactivated to preserve your capital.
Over the years, I have developed some methods that have worked very well for me, addressing these issues. In the spirit of education, and in memory of acrary's useful posts on this board, I will share my methods for managing system activation/deactivation.
As with acrary, I will no reply to every post if I have nothing to say. In addition, I trade fulltime every day (although automated) and may or may not have time to respond to questions. Please avoid petty bickering on this thread, as it just ruins the thread for everyone.
I'll post some of my initial thoughts on this topic as the day goes on, and after I've posted 3-5 posts would welcome questions if there are any.
-Eric
Optimization
First, before I forget, let me address one of the biggest mistakes people make when developing a system: optimization.
Typical traders will take a program like TradeStation or WealthLab or the equivalent, and create a spectacular trading system with clever and creative ideas, which has a near-linear P&L curve, and with very few minor drawdowns... They are excited and can't wait to start making the 'easy money' with live trading.
Will they actually make money with their new methods? Actually, we don't have a clue, as they've never tested whether their new method will make money in the future on not. All they have actually done is prove that their new system does any excellent job of being optimized to fit the past data, which in and of itself, is worthless.
How do we improve upon this? Test your system in the future! Hmmm... the future, you ask, how would we do that?
Let's assume that you are doing your backtesting on 5 years of data, and the system generates 150 trades per year, for a total of 750 trades. In order to tell how well the system performs in the future, you must withhold all of your data from the optimization tests. Instead, for example, only optimize on the first two years of data (300 trades, pretty decent number). Then, use your optimized system to now test on the 'future' => i.e. test year 3 of data with the system optimized on years 1 and 2.
Next, reoptimize your system on the data from years 2 and 3, and give it a future test with the data from year 4. Similarly, optimize your system with the data from year 3 and 4, giving it a 'real-time' test against the data from year 5. In this way, you can see how your system will perform in the 'future' based upon whatever optimization methods you are using.
This sort of method should give you increased confidence when you go to live activation that it should work well in the 'future'.
-Eric
Excellent reminder, Eric, and not repeated often enough.
activating/Deactivating any system is a system on its own. Example:
Switching between a trending system (MA, LinReg, MACD, etc) and counter trent system (RSI, Bol. bends, Chalkin etc.) is a Nobel Prize challenge.
__________________
MAESTRO
__________________
Create Chaos and then take credit for solving it!
Activation of a System
For me, the question of system activation really depends on answering the question of "Is this system profitable?"
Obviously, once activated, money management can be used to determine the best trade size in order to minimize drawdown on your overall account.
So, how do we know if our system is 'profitable'? Obviously, the average P&L of the trades must be above zero. But, that alone should not give us confidence. For example, the following sequence of trades has an average above zero:
+$2000
+$350
($300)
+$1400
($3000)
+$750
Avg = +$200
=> But I certainly would have little confidence that the system was net profitable. The next trade could be ($2200) and then we would be seeing a net losing system.
We would get better confidence that we have a profitable system if the trades were more consistent (i.e. lower standard deviation). For example, the following six trades also average +$200, but provide greater confidence that the system is likely profitable over a long term:
+$500
($300)
+$400
($50)
+$950
($100)
Avg = +$200
While not ideal, the smaller spread of the P&L results increases our confidence that this system MAY be profitable over the long haul.
How else might we generate increased confidence of a systems long term performance? One key is certainly the number of trades that are in your sample being assessed. Assuming you have got 100 trades that show an average profit of $200 per trade, versus on 6 trades showing an average profit of $200 per trade => you will have much greater confidence in your system based upon the 100 trade analysis.
Fortunately, statistics offers us some tools that we can use to scientifically answer these questions a little more accurately.
What is the likelihood that my system is profitable? What is my confidence level?
(continued)
-Eric
Quote from abogdan:
activating/Deactivating any system is a system on its own. Example:
Switching between a trending system (MA, LinReg, MACD, etc) and counter trent system (RSI, Bol. bends, Chalkin etc.) is a Nobel Prize challenge.
Profitability Confidence Level
Is my system profitable? At what level of confidence?
Assuming we are using 'future' data (whether this is backtested future data or actual forward tested data doesn't matter), then we can use statistical analysis to our data to determine the likelihood that our system is a profitable system over the long run as follows:
x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits)
Once "x" is calculated, you can look up the confidence level on the chart below to see the likelihood that this system will be profitable over the long haul:
x ------------------ Confidence Level
0 ------------------ 50%
0.075 ------------ 53%
0.126 ------------ 55%
0.25 -------------- 60%
0.36 -------------- 64%
0.52 -------------- 70%
0.67 -------------- 75%
0.84 -------------- 80%
1.04 -------------- 85%
1.28 -------------- 90%
1.4 ---------------- 92%
1.645 ------------- 95%
2.05 --------------- 98%
3.0 ----------------- 99%
4.0 ----------------- 99.5%
5.0 ----------------- 99.7%
Note that if the average profit is negative, then this will give you the confidence level that this is a losing system, over the long run.
One quick example to illustrate this equation. Assume that we have 50 trades, yielding an average profit of $40 per trade, with a standard deviation of $150 per trade. Using the equation and table:
x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
x = abs(40)*(50)^0.5/150
x = 1.88, from table, Confidence Level ~97%
-Eric
Limitations
Let me point out that these calculations were designed for series of data that have a Gaussian distribution (i.e. 'bell curve' distribution).
Is this true for our typical stock trade P&L's? Heck no, not even close!! 'Fat tails' from the occasional large stop loss, for example, will make our P&L distribution non-Gaussian.
Therefore, recognize that these equations and methods, while providing a good structured way to analyze your data, will not yield perfect results. The actual system may NOT be 97% likely to be profitable in the future, maybe it's only 93%... So what, big deal. The point is that a value of 97% is better than getting a value of 84%, and worse than getting a value of 99.6%. Greater 'confidence level' results should increase your confidence that you have a good system, while not necessarily being a statistically perfect answer for the true probability of future results. But, we are traders here, not statisticians. I don't mind if my predictions are off a few points, so long as I can make consistent money.
-Eric
This is very good! I like this, good thinking!
__________________
MAESTRO
__________________
Create Chaos and then take credit for solving it!
How to Incorporate Confidence Levels?
As abogdan pointed out, determining when to activate and deactivate a trading system really becomes a system within itself.
My choice has been to develop a very simple method that will shutdown a trading system when I 'lose confidence' in it (statistically), and activate a trading system when it proves itself to be worthy (statistically). I exclusively use the equation and table shown in the above posts for making these decisions.
Anyone can arbitrarily decide their activation/deactivation levels, as well as the number of trades to be used for calculating their confidence levels for a system. For me, I have arbitrarily decided upon using the last 120 trades and I deactivate a system if my confidence level drops below 90%, and I activate a system if my confidence level rises above 93%. Note that I will use as few as 50 trades for these calculation if that is all the data that I have.
Somewhat arbitrary? Yes. Statistically sort of invalid (non-Gaussian)? Sure. But, it provides a solid structure for making the activation/deactivation decisions in a consistent and reasonable manner. And, most importantly, it works for me.
I hope that this has been helpful for a few folks, and this is all that comes to mind on the topic at this time. Questions or comments are welcome, and I'll post additional thoughts if they come to me.
-Eric
Marvelous! Now that I had a chance to think about it, it makes a lot of sense! Thank you for your contribution, very nice!
__________________
MAESTRO
__________________
Create Chaos and then take credit for solving it!
Re: Profitability Confidence Level
Quote from EricP:
Is my system profitable? At what level of confidence?
Assuming we are using 'future' data (whether this is backtested future data or actual forward tested data doesn't matter), then we can use statistical analysis to our data to determine the likelihood that our system is a profitable system over the long run as follows:
x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits)
Once "x" is calculated, you can look up the confidence level on the chart below to see the likelihood that this system will be profitable over the long haul:
x ------------------ Confidence Level
0 ------------------ 50%
0.075 ------------ 53%
0.126 ------------ 55%
0.25 -------------- 60%
0.36 -------------- 64%
0.52 -------------- 70%
0.67 -------------- 75%
0.84 -------------- 80%
1.04 -------------- 85%
1.28 -------------- 90%
1.4 ---------------- 92%
1.645 ------------- 95%
2.05 --------------- 98%
3.0 ----------------- 99%
4.0 ----------------- 99.5%
5.0 ----------------- 99.7%
Note that if the average profit is negative, then this will give you the confidence level that this is a losing system, over the long run.
One quick example to illustrate this equation. Assume that we have 50 trades, yielding an average profit of $40 per trade, with a standard deviation of $150 per trade. Using the equation and table:
x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
x = abs(40)*(50)^0.5/150
x = 1.88, from table, Confidence Level ~97%
-Eric
Re: Re: Profitability Confidence Level
Quote from shorty_mcshort:
Just a quick question. For example, if you were trading stocks and pyramid your profits. Would this test be valid or would you have to test it on the same # of shares or same dollar amounts on each trade?
Re: Re: Re: Profitability Confidence Level
Quote from EricP:
I consider each trade to be an independant event, regardless of whether it is 200 shares or 1500 shares. I think it would still be worthwhile and useful for you, although I would avoid the word 'valid,' as this might imply a sense of statistical accuracy that we do not have or require for our need (in light of distribution not being bell curve). Being less than perfect can be very profitable, while taking the effort to have a statistically 100% supported analysis might be academically interesting (not!) but is a major waste of time for a trader, IMO. We are seeking to get 90% of the results with 10% of the effort, which I think this method achieves nicely for me.
-Eric
Standard Deviation
How does one come up with a value for standard deviation per trade (such as your $150 in the example)?
Re: Standard Deviation
Quote from Hofferino:
How does one come up with a value for standard deviation per trade (such as your $150 in the example)?
Re: Re: Re: Re: Profitability Confidence Level
Quote from shorty_mcshort:
Very interesting. Thanks for sharing. If someone is interested in learning more statistics that you could apply to trading where could I find that information?
Re: Standard Deviation
Quote from Hofferino:
How does one come up with a value for standard deviation per trade (such as your $150 in the example)?
Re: Re: Standard Deviation
Quote from shorty_mcshort:
Also note that the original post talks about standard deviation of profits not per trade.
Fantastic
EricP,
Thanks for your contributions - this is very nicely done.
Question :
you mentioned testing for profit confidence as follows
[ p * n ^0.5 ] / sigma
where
p = net profit per trade
n = # of trades
sigma = std dev per trade
we can factor out n so, it's basically
[ n^0.5] * [p / sigma]
It is clear that assuming p > 0, as n increases, your score is going to increase, and that you can theoretically seek for any score you want by increasing n.
I guess to bring this a step further, would you say that for any given n, the corresponding score will tell you the confidence of being profitable for a period of length n?
Thus : if n = 100 and the score corresponds to 98%, then we can say that the system has a 98% confidence in being profitable every 100 trades. If the same system has a score of 80% at n = 50, then we can say that at 50 trades, the confidence of being profitable drops to 80%.
Thus, the question becomes - say if we are tracking the last 100 trades of a system and working out the corresponding score based on n = 100, and then dropping the system if the score falls below 80% or whatever benchmark we set.
How do we know 100 is the appropriate length to use? How do we know 80% is the appropriate cut-off to use?
Should we just be arbitrary based on our "judgement" or is there a rigourous way of approaching this problem?
Re: Activate/Deactivate System?
Quote from EricP:
...at some point you must make a decision about whether your new pet system is 'worthy' of activation....
__________________
Damir Smitlener
"The misunderstanding here is due to a faulty translation of a German text, in particular the use of the word "mystical."" -- A. Einstein, 1954
Re: Fantastic
Quote from TheStudent:
It is clear that assuming p > 0, as n increases, your score is going to increase, and that you can theoretically seek for any score you want by increasing n.
Quote from TheStudent:
I guess to bring this a step further, would you say that for any given n, the corresponding score will tell you the confidence of being profitable for a period of length n?
Thus : if n = 100 and the score corresponds to 98%, then we can say that the system has a 98% confidence in being profitable every 100 trades. If the same system has a score of 80% at n = 50, then we can say that at 50 trades, the confidence of being profitable drops to 80%.
Quote from TheStudent:
Thus, the question becomes - say if we are tracking the last 100 trades of a system and working out the corresponding score based on n = 100, and then dropping the system if the score falls below 80% or whatever benchmark we set.
How do we know 100 is the appropriate length to use? How do we know 80% is the appropriate cut-off to use?
Should we just be arbitrary based on our "judgement" or is there a rigourous way of approaching this problem?
Re: Re: Re: Standard Deviation
EricP, thanks for a very interesting thread. I can't seem to replicate your values of $2,652 and $1,875 for STDEV and STDEVP. I get $1,760 and $1,607 respectively.
Quote from EricP:
I think you both have the right idea. I am talking about the standard deviation of the profits from the trades. Using the first example posted:
+$2000
+$350
($300)
+$1400
($3000)
+$750
And plugging into Excel for STDEV, I get a standard deviation of $2652. Also, you could use the STDEVP function, which estimates the Standard Deviation over an entire population of data, and I get $1875 as the Standard Deviation.
Re: Limitations
Quote from EricP:
...recognize that these equations and methods, while providing a good structured way to analyze your data, will not yield perfect results.
__________________
Damir Smitlener
"The misunderstanding here is due to a faulty translation of a German text, in particular the use of the word "mystical."" -- A. Einstein, 1954
Re: Re: Limitations
Quote from damir00:
it's worse than that. systems using tight stops (as one example) have arcsine-like P/L distributions and using this approach will not just yield less than perfect results, it will yield flat out wrong results. using variable trading amounts invalidates the numbers even further.
Re: Re: Re: Re: Standard Deviation
Quote from Mr Subliminal:
EricP, thanks for a very interesting thread. I can't seem to replicate your values of $2,652 and $1,875 for STDEV and STDEVP. I get $1,760 and $1,607 respectively.
Re: Re: Limitations
Quote from damir00:
it's worse than that. systems using tight stops (as one example) have arcsine-like P/L distributions and using this approach will not just yield less than perfect results, it will yield flat out wrong results. using variable trading amounts invalidates the numbers even further.
strategy deterioration
kestner uses the following when monitoring stategy degradation.
he calculates a linear regression of the equity curve with bands plotted two standard errors above and below the fit. he halts trading of the strategy when the curve breaks below the lower channel, and a decision is made to modify or toss the system.
Sharpe
Guys aren't we just talking about the Sharpe Ratio here?
A question was raised about variable dollar profit amounts caused by rollover of profits screwing things up. Might this problem be mitigated by analyzing what percentage profit was taken from the underlying (or better yet what portion of profit in ATR terms)?
EricP I very much agree that a robust solution now is better than a perfect solution later so I appreciate your methods. I must also commend damir for keeping us on our toes with the specifics. Keep posting guys. This is a good thread.
Re: How to Incorporate Confidence Levels?
Quote from EricP:
As abogdan pointed out, determining when to activate and deactivate a trading system really becomes a system within itself.
My choice has been to develop a very simple method that will shutdown a trading system when I 'lose confidence' in it (statistically), and activate a trading system when it proves itself to be worthy (statistically). I exclusively use the equation and table shown in the above posts for making these decisions.
...
-Eric
Re:gaussian distribution
my first real ET post...be gentle, please.
(great thread by the way)
regarding non-gaussian p/l distributions:
This is just a random idea -
I remember reading ehlers talking about using a fisher transform to normalize price data so that std. dev. based oscillators would give sharper signals.
Couldn't a similar concept be applied to a p/l that looks like -2, -3, -2.5,, +10?
Maybe you code it up as W,W,W,L? I know there are statistical tests for this kind of categorized data but i don't know them off the top of my head and leave that for people smarter than me.
Paper Trading Feedback
I thought I'd add some additional thoughts to this thread, specifically with respect to paper trading.
Paper trading has a bad reputation among many in the industry as being a worthless exercise, and having no value for the successful trader. I completely disagree with this view, and feel that when done properly, paper trading can be an extremely valuable tool for the successful trader.
First of all, paper trading must be done properly, to ensure that 'impressive' results are not returned for an otherwise worthless system simply due to the over-optimization of testing. There is more on this topic in the posts above.
Let's assume, that a trader uses valid methods to back test his/her system, and then decides that the system is 'worthy' of activation for live trading. Furthermore, let's assume that the system produces an average profit of $100 per trade for this testing with a standard deviation of $200 over 300 trades in the test data.
Once activated, the trader is disappointed to see that Murphy's Law is in effect and the system loses money on it's first 5 consecutive trades, giving a quick loss of $1000. Should this system be immediately deactivated?
On it's own, five trades would certainly not seem to be statistically significant. In addition, adding these five trades to the 300 trades in the prior test results is unlikely to change the overall P&L of the system sufficiently to warrant deactivating the system.
However, the first thing I would do in this situation is to update my paper trading results... Typically, I will simultaneously paper trade all systems while I am also trading the system live, so I already have the information I need. The question here, is:
How did my paper trading perform over the past five trades?
Assuming that my paper trading results match my live trading results (and of course, my back testing was valid in the first place), then I don't have any problem other than bad initial luck with this system and my trading will continue.
However, what if my paper trading shows breakeven, or possibly a small profit for these past five simultaneous trades? => This would indicate that my paper trading execution assumptions are invalid and my entire backtesting needs to be reviewed. The system should be deactivated immediately, in this case, since the activation of the system was based upon incorrect assumptions. We know these assumptions are invalid, since we cannot duplicate our paper trading 'results' with our real performance when 'traded' simultaneously.
This is the way that we 'close the loop' to solidify our trading results. We use 'correctly conducted' paper trading to validate our trading system in order to determine what to activate. Then, once activated, we compare of real results with our simultaneous paper results to validate the execution assumptions of our paper trading.
Assuming that our system passes both of these tests, then we continue to assume that we have a 'good' system, and give the system more time to perform after some initial bad trades.
-Eric
Re: Profitability Confidence Level
I just received a PM regarding this post, and thought I would post the PM (minus name) and my response, to help others that might have the same questions. The PM was:
"My question to you is related to confidence levels you showed in your post. My statistics foundation is very poor and I could not managed to get the whole table of confidence levels with "x" for all % levels. Could you please advice me where I can find table with all confidence levels for a test you suggested for activation/deactivation of a trading system."
My reply was:
You should be able to use the table shown for all %levels as follows:
For any value of x less than zero, you can use the same table. Just make the x value positive, and the table will show you the confidence level that the system is a LOSING system.
For example, if x = -1.04, then we will look up +1.04 in the table.
=> By doing this, we see that the system is a LOSING system at a 85% confidence level... Put another way, the system is a WINNING system at only a 15% confidence level.
I hope this helps. Best of luck,
-Eric
Quote from EricP:
Is my system profitable? At what level of confidence?
Assuming we are using 'future' data (whether this is backtested future data or actual forward tested data doesn't matter), then we can use statistical analysis to our data to determine the likelihood that our system is a profitable system over the long run as follows:
x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits)
Once "x" is calculated, you can look up the confidence level on the chart below to see the likelihood that this system will be profitable over the long haul:
x ------------------ Confidence Level
0 ------------------ 50%
0.075 ------------ 53%
0.126 ------------ 55%
0.25 -------------- 60%
0.36 -------------- 64%
0.52 -------------- 70%
0.67 -------------- 75%
0.84 -------------- 80%
1.04 -------------- 85%
1.28 -------------- 90%
1.4 ---------------- 92%
1.645 ------------- 95%
2.05 --------------- 98%
3.0 ----------------- 99%
4.0 ----------------- 99.5%
5.0 ----------------- 99.7%
Note that if the average profit is negative, then this will give you the confidence level that this is a losing system, over the long run.
One quick example to illustrate this equation. Assume that we have 50 trades, yielding an average profit of $40 per trade, with a standard deviation of $150 per trade. Using the equation and table:
x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
x = abs(40)*(50)^0.5/150
x = 1.88, from table, Confidence Level ~97%
-Eric
Eric, sorry for being so blunt but how smooth does all of that make your combined multi-system equity curve? A picture (or maybe even pictures of several system equity curves + cumulative to illustrate benefits of activation/deactivation??!!?) would be worth a thousand words.
Quote from sunnyskies:
Eric, sorry for being so blunt but how smooth does all of that make your combined multi-system equity curve? A picture (or maybe even pictures of several system equity curves + cumulative to illustrate benefits of activation/deactivation??!!?) would be worth a thousand words.
Thank you. Looks like about a 45 degree angle.
Are all of acrary's posts preserved on this forum and will be preserved in the future? I believe they're of very high value and want to save them for the future generations so to speak. I believe if someone made an archive of all acrary's posts and posted them on a web page somewhere that would be a tremendous contribution.
distribution-free method
Eric, what I do is monte-carlo the trade distribution that I get from testing. If I do as poorly as the bottom 10% then I start to question the methodology. The advantage is you don't have to assume anything about the distribution of results this way.
Re: distribution-free method
Quote from peterfigliozzi:
Eric, what I do is monte-carlo the trade distribution that I get from testing. If I do as poorly as the bottom 10% then I start to question the methodology. The advantage is you don't have to assume anything about the distribution of results this way.
Re: distribution-free method
Quote from peterfigliozzi:
Eric, what I do is monte-carlo the trade distribution that I get from testing. If I do as poorly as the bottom 10% then I start to question the methodology. The advantage is you don't have to assume anything about the distribution of results this way.
__________________
"The test of success is not what you do when your on top. Success is how high you bounce when you hit bottom."
- General George S. Patton, Jr
Re: Re: distribution-free method
Quote from EricP:
A couple of questions for you: How many real time data points (i.e. trades) do you require from your system before you make the comparison with the 10 percentile Monte Carlo results? Also, assuming you have been trading the system live for a very long time (500+ trades), then how many real trades do you use for your comparison? In other words, do you always use your full data set of real trades, or only the most recent x trades?
Finally, with your method, activation/deactivation seems to be a two step process. First, the system must pass your backtesting results in order to be activated in the first place, and then it must continue to perform above the 10 percentile Monte Carlo to continue live trading. Assuming I have stated this correctly, how do you judge that a system has passed your backtest criteria?
Thanks,
-Eric
backtesting
The backtesting critereon is this: the distribution of entries has to kick random's ass, not just beat random. I construct a scatter plot of each entry's chance to make money; in this sample plot I put random entries. Visually, it's easy to see if there's a big difference or not.
peter, are you still trading that trendline break stuff from your journal?
Gaussian vs. Monte Carlo
A proposal for Eric and anyone else wishing to participate:
I think it would be VERY interesting and instructive to compare when the Gaussian model and a Monte Carlo study would tell us when to activate/deactivate a system. The nice thing about this is we don't have to talk about the trading methodology itself, so it seems like we could expect a decent amount of participation.
Since I do all my testing by hand, and i've just been using this technology relatively recently, I don't have a heck of a lot of examples. The two things I am trading now are working, which make for very boring examples. But I will post an example just to get the ball rolling.
I was thinking maybe Eric has an example of something from the past he has deactivated. I would like to run the backtested distribution of results through my monte carlo, and then we can compare what the Gaussian and MC models tell us (and more importantly, WHEN they tell us).
Re: Gaussian vs. Monte Carlo
Quote from peterfigliozzi:
A proposal for Eric and anyone else wishing to participate:
I think it would be VERY interesting and instructive to compare when the Gaussian model and a Monte Carlo study would tell us when to activate/deactivate a system. The nice thing about this is we don't have to talk about the trading methodology itself, so it seems like we could expect a decent amount of participation.
Since I do all my testing by hand, and i've just been using this technology relatively recently, I don't have a heck of a lot of examples. The two things I am trading now are working, which make for very boring examples. But I will post an example just to get the ball rolling.
I was thinking maybe Eric has an example of something from the past he has deactivated. I would like to run the backtested distribution of results through my monte carlo, and then we can compare what the Gaussian and MC models tell us (and more importantly, WHEN they tell us).
__________________
"The test of success is not what you do when your on top. Success is how high you bounce when you hit bottom."
- General George S. Patton, Jr
Re: Gaussian vs. Monte Carlo
Peter,
I am interested in continuing the discussion, but I'm not sure if examples will tell us much. With any set of data, optimization of the activate/deactive logic can curve fit the results to different answers. For example, assume that you have a bad trade that was 79 trades ago in your trade history. Your results (Gaussian or Monte Carlo) will depend on how many trades you use in your analysis. If you use the most recent 75 trades, then that bad trade will not be a part of the analysis, and the system will be quite different than if you consider 90 trades in your analysis.
Getting back to the initial question. If I understand correctly, you use the Monte Carlo to determine when to DEACTIVATE a system (i.e. the system is no longer performing as well as the history expection). However, when do you decide when a system should be activated?
For me, my activation and deactivation rules are also exactly the same (and documented earlier in the thread, if I'm not mistaken). If the paper trade history is better than "X", then the system is activated for live trading. While live trading is being conducted, I simultaneously continue the paper trading of the same system to enable comparison of the real versus paper results. Any real vs paper deviations should result in my reconsidering my paper trading order execution assumptions (mainly slippage).
For active systems, the deactivation decision is based upon the results of the most recent "X" number of PAPER trades (typically, 120). It is assumed that the most recent paper trades pretty accurately matches the lilve trades, as this is continually monitored.
By only considering the most recent 120 trades, my systems are able to deactivate before a long extended period of poor performance, which is certainly a goal. Typically, 120 trades might take ~2-3 months to generate, so that a bad week or two will not automatically deactivate a system. But, for example, a one month period of breakeven results (30-70 trades) will often result in the deactivation of the system.
Similarly, a non-active system that has had breakeven or worse results for several months might begin to trade very well. When this happens, after about a month of very good results, the last 120 trades start to show a statistically high confidence level of profitability, and the system is activated.
Anyway, I didn't intend to rehash the way I do my activation/deactivation. But, as I was asking you for more information on how you activate your systems, I figured it was only appropriate to better clarify how I do the same on my end. (i.e. seems like Monte Carlo is more geared towards deactivation for you, as I understant it)
-Eric
Quote from peterfigliozzi:
A proposal for Eric and anyone else wishing to participate:
I think it would be VERY interesting and instructive to compare when the Gaussian model and a Monte Carlo study would tell us when to activate/deactivate a system. The nice thing about this is we don't have to talk about the trading methodology itself, so it seems like we could expect a decent amount of participation.
Since I do all my testing by hand, and i've just been using this technology relatively recently, I don't have a heck of a lot of examples. The two things I am trading now are working, which make for very boring examples. But I will post an example just to get the ball rolling.
I was thinking maybe Eric has an example of something from the past he has deactivated. I would like to run the backtested distribution of results through my monte carlo, and then we can compare what the Gaussian and MC models tell us (and more importantly, WHEN they tell us).
Basic example
Suppose you backtest something for the NQ and the distribution of results is:
-10 NQ points / 5 times
-5 / 1
-2 / 15
-0.5 / 15
+1 / 5
4 / 6
10 / 6
20 / 2
I input these numbers into the program along with a $5 commish per trade. Now, what the program does is pick 20 random trades (or how ever many you want) distributed according to the relative probabilities from the backtest. It does this 100,000 times (or however many times you want).
The program outputs the distribution of results (how much $$ you end up with after twenty trades), and equally important info, the lowest account drawdown for each run of 20 trades.
Basically, what we've done is give our trading plan to 100,000 trades and told them to make 20 trades and report back.
I've attached a picture of the results.
Conclusion: if any of our traders report back and their $10k account is below $9k, we should fire them. We should also fire anybody who dips below $9k at any time.
Another way to look at it: it costs up to $1k per ctc to find out if this methodology is working.
Also suppose we're making money for some time, and then we find ourselves down $1k/ctc in a stretch of 20 trades or less. That means something has changed and our results aren't in accordance with our backtested results. Back to the drawing board.
A neat thing to notice: even though the distribution of individual trades is not nearly gaussian, the distribution of trade results is. Hey! That's the central limit theorem in action! When the MC trades, the trades are statistically independent of one another.
This is one point I am pondering right now. Without serial dependence of certain price/volume events, I could not enjoy the edges that I do. The trades make a very non-gaussian distribution of results that makes me money. But looking at the individual results themselves, is there a serial dependence there as well? In other words, is a winner more likely after a winner? The answer is: maybe, maybe not-- it's something you have to look at for each methodology. Like Box and Hunter say in their book-- a mere "declaration of independence" is not good enough. Then you have to realize, if you are using MC, that the program assumes consecutive trades are independent.
Re: Re: Gaussian vs. Monte Carlo
Quote from EricP:
Anyway, I didn't intend to rehash the way I do my activation/deactivation. But, as I was asking you for more information on how you activate your systems, I figured it was only appropriate to better clarify how I do the same on my end. (i.e. seems like Monte Carlo is more geared towards deactivation for you, as I understant it)
-Eric
Re: Re: Re: Gaussian vs. Monte Carlo
here's the picture
Also a message to newbs: I hope you can now see some of the things you need to do to make money. You can't just use the latest gizmo from your software vendor like some of the testimonials I have read on various sites:
'Just a quick note as the trading day progresses. I have to say I was a little worried when you brought out the new ------. As usual you were right. Brilliant is the word that comes to mind. I am using the ------- this morning. I am up 18 points in the Russell using these ---- alone. See attached chart. I am ignoring the other indicators generally. I look for the ------ to be on the bottom for a long and then look at the ----- for a confirmation of entry. If there is no ----- on the ------ I refuse the entry. So far today six trades, four winners for the aforementioned 18 points on two contracts. I can go play golf now! I added an arrow and the word “Chop” to the chart. When there are no -------------- it seems to indicate chop. That’s a wonderful chop indicator if it works out. Thank you for your continued outstanding work in our behalf.'
Re: Re: Gaussian vs. Monte Carlo
Quote from EricP:
For active systems, the deactivation decision is based upon the results of the most recent "X" number of PAPER trades (typically, 120). It is assumed that the most recent paper trades pretty accurately matches the lilve trades, as this is continually monitored.
a-ha moment:
Any probability distribution of trades (gaussian or not) will yield a gaussian distribution of results after N trades, as long as N is high enough and the trades are independent (central limit theorem). So if you want to compare the mean and variance of of your live trading results with that of your backtested results, you have two problems:
(1) You need a large number of trades N to generate your gaussian distribution of results.
(2) If the trades are not independent in the first place, you have to be careful with how you interpret "mean" and "variance". Your distribution of results is not gaussian.
Using a monte carlo, you get rid of both of these problems at once.
(1) Many less trades are required, since you can now compare distributions directly. By using a set of 100,000 traders, you can see how likely it would be to achieve a certain result with only 10-20 trades.
(2) Since you are not using "mean" and "variance" to compare your results anymore, it is a moot point whether your results are independent or not. You are comparing distributions directly.
This all makes sense to me now... but i'm open to re-education if neccesary.
1. What software are you using to generate MC analysis?
2. Peter, I don't know if you answered EricP's question but how do you activate a system once it has been deactivated using your MC analysis?
Thanks
another Eric
The program is freely available in the files section of the AustinTraders Yahoo group. You have to download all nine data files (1MB each) plus the setup program.
Max drawdown not gaussian
Interesting point: the max drawdown distribution in that example I posted is NOT gaussian. Looks like poisson. If any stat gods out there could explain why I would appreciate it.
Quote from shorty_mcshort:
how do you activate a system once it has been deactivated using your MC analysis?
Thanks
another Eric
So is it advisable to use Eric's activate/deactivate decision model for optimal bet size as well?
For example if x = 2.0 then use 2.0x leverage or something of that nature, or even risk 2% per trade?
-Taric
Quote from ptunic:
So is it advisable to use Eric's activate/deactivate decision model for optimal bet size as well?
For example if x = 2.0 then use 2.0x leverage or something of that nature, or even risk 2% per trade?
-Taric
That makes sense.
Thanks!
-Taric
Re: Profitability Confidence Level
Quote from EricP:
....x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits).....
__________________
"When the facts change, I change my mind. What do you do, sir?" J.M. Keynes
"I guess I should warn you, if I turn out to be particularly clear, you've probably misunderstood what I've said. " A. Greenspan
Re: Re: Profitability Confidence Level
Quote from hanseng1:
Edit: I suppose what I am saying is the system has "fat tailed" distribution. Therefore this test gives a somewhat poor result.
Eric,
Given that the movement of the market is random by nature, and consequently most of the trading systems out there, will you get better results when you deactivate the system IF it has a confidence level above 99%?
Thanks for sharing your results with us.
Jim
Quote from EricP:
I do.
..............My goal is to ensure that 100% of the system is mechanical in nature. With this sort of system, the systems that are active, as well as the size being traded in any system, are all handled 100% mechanically. Less day-to-day decisions for me equals less stress, fewer worries, and less overall work, IMO.
-Eric
I want to point out that both methods of activation/deactivation in this thread (gaussian/MC) work just as well for mechanical or discretionary trading (and any mix in between).
I have tested many systems for twenty years now. In that time, every time I thought I should turn off a system intraday based on either discretion or some clever analysis I had done, it invariably more often than not the wrong decision. I agree that this may be more a statement about my thinking skills than about the feasibility of being able to come up with a good answer, but I doubt it.
You can embed any question you like about any trading system in the question - Should I turn off this system now?
The markets can and will do anything at any time. The more efficient the market the less memory it has.
IMO the only thing that might work is if you think about it in terms of in/de creasing bet size, but I think that may be as hard a problem in anything but the Options market.
nitro
__________________
"You have to fix your roof when it's sunny outside" - JFK
Quote from nitro:
I have tested many systems for twenty years now. In that time, every time I thought I should turn off a system intraday based on either discretion or some clever analysis I had done, it invariably more often than not the wrong decision. I agree that this may be more a statement about my thinking skills than about the feasibility of being able to come up with a good answer, but I doubt it.
You can embed any question you like about any trading system in the question - Should I turn off this system now?
The markets can and will do anything at any time. The more efficient the market the less memory it has.
The only thing that might work is if you think about it in terms of in/de creasing bet size, but I think that may be as hard a problem in anything but the Options market.
nitro
Quote from metooxx:
The Holy Grail of questions ...
__________________
"You have to fix your roof when it's sunny outside" - JFK
Quote from nitro:
metoo!!!
Where ya been?
nitro
Solving the question you are asking for over a year...
Quote from mc107:
Eric,
Given that the movement of the market is random by nature, and consequently most of the trading systems out there, will you get better results when you deactivate the system IF it has a confidence level above 99%?
Thanks for sharing your results with us.
Jim
Quote from mhashe:
You've been developing automated systems for quite a while now. Curious to know if any of your old systems from few years back still profitable in this market . Do you develop your own indicators or use what's currently in the public domain?
Quote from nitro:
I have tested many systems for twenty years now. In that time, every time I thought I should turn off a system intraday based on either discretion or some clever analysis I had done, it invariably more often than not the wrong decision. I agree that this may be more a statement about my thinking skills than about the feasibility of being able to come up with a good answer, but I doubt it.
Quote from metooxx:
Solving the question you are asking for over a year...
__________________
"You have to fix your roof when it's sunny outside" - JFK
In 2004, I changed systems 4 times. My results were good enough, but subsequent testing revealed that if I had used any one of those systems all year, I would have done considerably better!
To further look into this, I added a feature to my testing software that takes a list of systems and a time interval as inputs, and produces a "switch list" as the output. The switch list can be either "switch to what worked best in the prior period" or "switch to what worked worst". The switch list then becomes the input for a new test.
I then did lots of testing of this sort, using different periods and different systems. The results showed that, on average, it is best to change systems as infrequently as possible, and if you are going to change, it is better to switch to a system that was not one of the best in the most recent period.
I have done similar studies, but the problem is that I did not have voluminous data to do a "real" test.
How long was your testing period? Was it just 2004?
nitro
Quote from akawesson:
In 2004, I changed systems 4 times. My results were good enough, but subsequent testing revealed that if I had used any one of those systems all year, I would have done considerably better!
To further look into this, I added a feature to my testing software that takes a list of systems and a time interval as inputs, and produces a "switch list" as the output. The switch list can be either "switch to what worked best in the prior period" or "switch to what worked worst". The switch list then becomes the input for a new test.
I then did lots of testing of this sort, using different periods and different systems. The results showed that, on average, it is best to change systems as infrequently as possible, and if you are going to change, it is better to switch to a system that was not one of the best in the most recent period.
__________________
"You have to fix your roof when it's sunny outside" - JFK
What I'm using is daily bars for a large universe of stocks. My systems tend to generate about 1000 trades/year. I do most of my testing using the most recent 3 or 4 years, but sometimes go back 8 or even 12 years.
If I run a really long term system comparison test (say 10 years, 10000 trades), all the systems do about the same, even though in a given year, one might do twice as well as another.
All this points to just picking a system and stiking with it, but I still find this difficult when testing shows dramatically better recent results for a different system. So, I continue to search for a system switch algorithm that beats the long-term average.
Ah,
You are using daily data - you are in a totally different place than I am then.
nitro
Quote from akawesson:
What I'm using is daily bars for a large universe of stocks. My systems tend to generate about 1000 trades/year. I do most of my testing using the most recent 3 or 4 years, but sometimes go back 8 or even 12 years.
If I run a really long term system comparison test (say 10 years, 10000 trades), all the systems do about the same, even though in a given year, one might do twice as well as another.
All this points to just picking a system and stiking with it, but I still find this difficult when testing shows dramatically better recent results for a different system. So, I continue to search for a system switch algorithm that beats the long-term average.
__________________
"You have to fix your roof when it's sunny outside" - JFK
Quote from EricP:
I'm not sure I understand your question. First of all, if the movement of the market were fully random, then trying to make money by trading the market would be fruitless. Therefore, in order to be trading at all, you must believe that the movement of the market is predictable to some small extent.
Would it be best to deactivate a system if it's confidence level of profitability falls below 99%? This might be nice for some traders. I suspect most traders are like me in that it is difficult for a system to achieve a 99% confidence level. I do have some systems that meet this criteria, but many do not. By restricting my real trading to only systems with 99%+ confidence levels, I might get a smoother equity curve, but would definitely make less money. Instead, I choose to also trade systems with a 90-99% confidence level and smooth out the equity curve by greater diversification among different systems.
On a side point, factors affecting confidence level include the number of trades in the sample (the more the better), the average profit per trade in the sample (the higher the better) and the standard deviation of the profits in the sample (the lower the better). Typically, most systems that I trade do not have sufficient sample size (I limit to only looking at the last 120 trades) or profitability/variability to reach 99%+.
Best of luck,
-Eric
Quote from mc107:
You misunderstanded my question. I mean that you deactivate your system if its confidence level of profitability is ABOVE 99%.
Thanks,
Jim
Quote from nitro:
Are you saying you are trying to solve it, or that you have solved it ?
Either way, welcome back.
nitro
Quote from metooxx:
almost ...
__________________
"You have to fix your roof when it's sunny outside" - JFK
Quote from EricP:
I'm not sure I understand what you are trying to say. Assuming that their is a statistical confidence level of 99%+ that as system is profitable... => Why in the world would I want to deactivate it from real trading? I wish every single system that I trade had that high of a confidence level. Obviously, I'm using the assumption that my testing was valid and that the results are not "too good to be true" due to an error on my part.
Assuming that there are no significant errors in my testing (which better be the case, or else reviewing the statistics is worthless in the first place), then I would certainly NOT deactivate such a system.
I guess I'm not sure of your point. Under what circumstances would you consider deactivating the systems that appear to be the best ones you have?
-Eric
Eric;
Thank you; brilliant idea if implemented properly. I worked on something similar last year which we did not test properly and discarded; it took your posts to revive the concept.
Michael
Quote from metooxx:
Eric;
Thank you; brilliant idea if implemented properly. I worked on something similar last year which we did not test properly and discarded; it took your posts to revive the concept.
Michael
Quote from mc107:
This system only trades big cap stocks. Your activation/deactivation strategy can turn a money-loss system into a winning system. That is great. However, when x>=2.5, it does not do well. Do you see any results like that in your testing? Or it is system dependent.
Thanks,
Jim
Anyone looking at autocorrelation of returns versus:
1) time scale of returns
2) design space parameters
??
Quote from prophet:
Anyone looking at autocorrelation of returns versus:
1) time scale of returns
2) design space parameters
??
http://en.wikipedia.org/wiki/Autocorrelation
Autocorrelation of returns is correlation(return[1...t-1],return[2...t]), or in other words the correlation between a series of returns and the same series of returns shifted one period forward or backward. Returns can be per-minute, per-day, or per-trade. It is a simple test to answer the questions:
Do returns come in streaks?
Do they oscillate?
Is there no pattern at all (random walk)?
It is particularly interesting to calculate autocorrelation across different time scales, and across the design space of a system. Why? Strong autocorrelating systems on short time scales naturally perform better with stops. So instead of brute force or real money testing different forms of stop losses, profit targets, trailing stops, or MA smoothing rules, look for the root cause of stop loss success… AUTOCORRELATION!
Even if a system’s long-term net profitability is zero or negative, a strong positive (trending returns) or negative (oscillatory returns) autocorrelation on certain time scales can be taken advantage of in a system-of-systems design. Methods include activation/deactivation rules, regression, ranking/screens, or trend following rules.
A system with autocorrelation close to zero, even if nicely profitable will have volatile returns and will be more dangerous and psychologically painful to trade. You won’t know the system is dead until you’ve taken on significant losses… when it effectively becomes strongly autocorrelating. Better to trade a strong autocorrelating system-of-system design that will automatically kill a dying system for you before losses accumulate.
autocorr - trading the equity curve
If there is no autocorrelation of returns then "trading the equity curve" by following the trend of the recent past will fare just as poorly as trend following in a market with minimal autocorrelation.
Well put prophet!
Can I do the autocorrelation test in Excel? What would the formula be?
Thanks,
ges
Quote from prophet:
http://en.wikipedia.org/wiki/Autocorrelation
Autocorrelation of returns is correlation(return[1...t-1],return[2...t]), or in other words the correlation between a series of returns and the same series of returns shifted one period forward or backward. Returns can be per-minute, per-day, or per-trade. It is a simple test to answer the questions:
Do returns come in streaks?
Do they oscillate?
Is there no pattern at all (random walk)?
It is particularly interesting to calculate autocorrelation across different time scales, and across the design space of a system. Why? Strong autocorrelating systems on short time scales naturally perform better with stops. So instead of brute force or real money testing different forms of stop losses, profit targets, trailing stops, or MA smoothing rules, look for the root cause of stop loss success… AUTOCORRELATION!
Even if a system’s long-term net profitability is zero or negative, a strong positive (trending returns) or negative (oscillatory returns) autocorrelation on certain time scales can be taken advantage of in a system-of-systems design. Methods include activation/deactivation rules, regression, ranking/screens, or trend following rules.
A system with autocorrelation close to zero, even if nicely profitable will have volatile returns and will be more dangerous and psychologically painful to trade. You won’t know the system is dead until you’ve taken on significant losses… when it effectively becomes strongly autocorrelating. Better to trade a strong autocorrelating system-of-system design that will automatically kill a dying system for you before losses accumulate.
I'm no doubt asking a dumb question...
Is it this simple:
=CORREL(H2:H2039,H3:H2040)
Quote from ges:
Can I do the autocorrelation test in Excel? What would the formula be?
Thanks,
ges
Doesn't look like it. I compared it to some stats software. The mean needs to be constant for the whole equation and denominator should be constant as well, ie. it should be the variance for the whole series.
Just to confuse you, you could look at the partial autocorrelation function as well. You're looking at the system P&L? This could show if there is some sort of similar pattern, say every five days. It removes the muck of the days in between. I tried to include a picture of what it should look like, but my software isn't cooperating.
The top attachment is a PACF of the S&P 500. The bottom is a non-stationary time series of something.
Hmm, let me try and put up the right attachment. I will add that just because the PACF shows something doesn't mean that it is significant. That needs to be tested as well.
A few loose thoughts on "Activate/Deactivate System?"
(1) I myself would never call "System" something to be activated and deactivated. If I 'deactivate', I would throw it in the garbage can.
Of course, I might do things like turning on and off algorithms or strategies, but I always would treat such procedures as an INTEGRAL PART of the SYSTEM. If you believe that you can make things work by activating/deactivating the whole, you simply don't have a system!
(2) As to evaluation and testing. I don't think that there is any fixed 'optimum' mathematical path to do this. You simply don't have (m)any rigorous assumptions to start from. I used to work on computer control of industrial processes. In this environment, your models being derived from physical processes, are often way better understood, although still stochastic. Nevertheless, I learned from this experience that a successful project always depended on combining theoretical (mathematical) know how with common sense. In fact, this common sense part kind of guides you all the way along (I hope that I don't start sounding like Jack on this 
)
(3) From the above, I adopted a thoroughly 'common sense' approach to the market system development. I use sufficient mathematics, but never too much! Recently, I came across the following by Fredrico Zeri, the titan fine arts historian and astounding expert: "I don't believe in improvising. In truth, great art is always the result of extraordinary technical ability." This might appear to be contradicting 'common sense' but you can't have 'extraordinary technical ability' without applying some common sense in manipulating mathematical tools in the marketplace.
(4) Coming back to turning on and off systems. I myself have only ONE system. If I would turn it off, I would quit for good. I have tried many things that I dumped along the way. The system I kept coming back to is the only one for me. This is very important in testing. I often use lengthy spread out trials in many dissimilar markets before adopting some new idea. I found out though that developing something new can often be done on the basis of very limited data sets - at least in the case of my habitual system. The lengthy tests simply confirm later what you thought to be valid. I came only to this after having injected sufficient common sense in the whole thing: i.e. "System design is testing and testing is system design.© 
"
Be good,
nononsense
nononesense,
It sounds like we come from similar backgrounds. I come from a background of automated control of industrial processes, as well (chemical engineer at a manufacturing plant).
Your point is well taken about simplicity. Too many traders are trying to find the 'perfect' system, or the 'perfect' analysis technique to assist them in their trading. Seems like someone in Market Wizards said something to the effect of "you came become rich, by not being perfect." The point is that having a statistically exact answer is impossible in trading, and almost useless if you could get it. The future is not easily predictable, and we need coarse analysis techniques to show us the way, IMO. Simple is good.
As related to your comments on activation/deactivation, you can look at my methods as a structure for when to activate or deactivate a subsystem, and view all trading being done as the complete 'system'. For me, I trade 4+ systems, with some trading 200+ stocks. So, I use the analysis discussed in this thread to determine when to turn OFF live trading of "System 3" for security INTC, for example.
Thanks for the informative post.
-Eric
Quote from nononsense:
A few loose thoughts on "Activate/Deactivate System?"
(1) I myself would never call "System" something to be activated and deactivated. If I 'deactivate', I would throw it in the garbage can.
Of course, I might do things like turning on and off algorithms or strategies, but I always would treat such procedures as an INTEGRAL PART of the SYSTEM. If you believe that you can make things work by activating/deactivating the whole, you simply don't have a system!
(2) As to evaluation and testing. I don't think that there is any fixed 'optimum' mathematical path to do this. You simply don't have (m)any rigorous assumptions to start from. I used to work on computer control of industrial processes. In this environment, your models being derived from physical processes, are often way better understood, although still stochastic. Nevertheless, I learned from this experience that a successful project always depended on combining theoretical (mathematical) know how with common sense. In fact, this common sense part kind of guides you all the way along (I hope that I don't start sounding like Jack on this
(3) From the above, I adopted a thoroughly 'common sense' approach to the market system development. I use sufficient mathematics, but never too much! Recently, I came across the following by Fredrico Zeri, the titan fine arts historian and astounding expert: "I don't believe in improvising. In truth, great art is always the result of extraordinary technical ability." This might appear to be contradicting 'common sense' but you can't have 'extraordinary technical ability' without applying some common sense in manipulating mathematical tools in the marketplace.
(4) Coming back to turning on and off systems. I myself have only ONE system. If I would turn it off, I would quit for good. I have tried many things that I dumped along the way. The system I kept coming back to is the only one for me. This is very important in testing. I often use lengthy spread out trials in many dissimilar markets before adopting some new idea. I found out though that developing something new can often be done on the basis of very limited data sets - at least in the case of my habitual system. The lengthy tests simply confirm later what you thought to be valid. I came only to this after having injected sufficient common sense in the whole thing: i.e. "System design is testing and testing is system design.©
Be good,
nononsense
prophet,
Excellent post. Thanks for the information and the link. I can certainly see how autocorrelation could be helpful in trading system activation/deactivation.
It seems like an advanced, but possibly very useful tool. Now you've got me also wondering whether other traders use this analysis technique in their system development and activation procedures. If so, hopefully, they will come forward to explain how they use this technique and whether it's been useful for them.
Thanks again for the explanation,
-Eric
Quote from prophet:
http://en.wikipedia.org/wiki/Autocorrelation
Autocorrelation of returns is correlation(return[1...t-1],return[2...t]), or in other words the correlation between a series of returns and the same series of returns shifted one period forward or backward. Returns can be per-minute, per-day, or per-trade. It is a simple test to answer the questions:
Do returns come in streaks?
Do they oscillate?
Is there no pattern at all (random walk)?
It is particularly interesting to calculate autocorrelation across different time scales, and across the design space of a system. Why? Strong autocorrelating systems on short time scales naturally perform better with stops. So instead of brute force or real money testing different forms of stop losses, profit targets, trailing stops, or MA smoothing rules, look for the root cause of stop loss success… AUTOCORRELATION!
Even if a system’s long-term net profitability is zero or negative, a strong positive (trending returns) or negative (oscillatory returns) autocorrelation on certain time scales can be taken advantage of in a system-of-systems design. Methods include activation/deactivation rules, regression, ranking/screens, or trend following rules.
A system with autocorrelation close to zero, even if nicely profitable will have volatile returns and will be more dangerous and psychologically painful to trade. You won’t know the system is dead until you’ve taken on significant losses… when it effectively becomes strongly autocorrelating. Better to trade a strong autocorrelating system-of-system design that will automatically kill a dying system for you before losses accumulate.
The following posts were taken from another thread (http://www.elitetrader.com/vb/showt...6181#post746181), but added here to help consolidate this discussion for reference:
Quote from TriPack:
This is the basic assumption that this type of analysis makes: that a smoother P&L curve will lead to future profits with more certainty than a bumpy P&L curve. One thing I stumbled upon was that I took an always in system, and tested it. It got like 60-70% confidence depending on the particular market it was applied to. Then I took the same system and only added a very tight profit target and stop loss (turning it into a scalping system) and did the tests again. Overall profitability went down significantly but the confidence level shot way up above 90% on virtually every market tested. So the conclusion I came to is that the confidence interval really measures the smoothness of returns, rather than whether that particular system may or may not be profitable over the long haul. This is from forward testing and observing these and other systems side by side. And specifically the confidence interval measures the short term smoothness of returns, which can be affected by a string of wins or losses in the short term, which may or may not continue. There is no guarantee that the system will stay hot or will stay cold, and I haven't found a way to test whether these hot/cold streaks tend to persist or not so I'm not certain that filtering out systems based on a low % is necessarily the best course.
Quote from TriPack:
Don't get me wrong, I appreciate your work and continue to seek out systems that as a general rule have smoother equity curves. I think having a systematic measurable approach is far superior to what I was doing previously. It is a good tool and I use it regularly.
Quote from TriPack:
But the thing that gives me the most confidence in finding an edge is if I apply the same system to multiple markets without changing the system at all, and see if it achieves similar results in many different markets. If it does, my confidence goes up that there might be something there. If the system is a one market anomaly, then it is probably curve fit to the data.
Good post, a couple of thoughts about it. I come from an options perspective so this is not totally applicable. Markets can trade a little differently from my perspective. This is true when comparing equity and commodities, but for a system this may be irrelevent abd signals may work the same. I have no idea on this point as I've never traded a system in either.
The main point I do have some idea, that being the applicability of stats. The confidence interval only tells you something about what you put in. As I use these methods more and more, I realized that you need to go far beyond simple critical values. You need to beat the hell out of the data a little(more like a lot) to get at the underlying process. The first thing I would do is remove the outliers and look at the data again. A system that relies on them for profitability is going to give a higher probability of never getting off the ground.
Quote from Trajan:
The main point I do have some idea, that being the applicability of stats. The confidence interval only tells you something about what you put in. As I use these methods more and more, I realized that you need to go far beyond simple critical values. You need to beat the hell out of the data a little(more like a lot) to get at the underlying process. The first thing I would do is remove the outliers and look at the data again. A system that relies on them for profitability is going to give a higher probability of never getting off the ground.
Definitely don't throw them out. Rather, analyze the outliers seperately. In a lot data analysis you would just toss em and not worry about it, but to us they are important. You could look at the P&L of the system as a whole, the P&L of the outliers and the P&L of those removed.
Quote from EricP:
Good point. To some extent, taking the confidence level will weed out systems that 'look good' based upon only a very few excellent trades. However, if you are trading systems with only moderately high confidence levels, then you should be especially cautious of that sort of system performance (as they might fall through the cracks and pass your confidence level test).
I'm not certain whether I would do the same thing on the outlier bad trades, though. I always like to error on the conservative side, and would tend to include all of the really bad trades without question. For example, I would imagine that a trader writing options would have mostly profitable trades, with the occasional 'outlier' bad trade. These bad trades are part of the risk of writing options and should be included in the system analysis, IMO. These sorts of systems can be especially hard to determine 'real' confidence levels, since it is hard to know if you have got a 'fair' number of outlier bad trades in your data set. I guess this is sort of the Black Swan issue.
-Eric
QUOTE]Quote from EricP:
Sorry, just saw your questions.
1) That's a fortunate problem to have. I would trade them both! If I could only trade one of the two, I would trade the first one and consider increasing trade size by 10% if desired to increase profitability.
2) I'm not sure I understand the need to make further "adjustments" for the sample size. Obviously, the larger the sample size, the greater the degree of confidence you can have in the profitability of the system being reviewed. But, that is precisely what the equation already accounts for. Personally, I like to put an upper limit on the sample size (for me, upper limit ~ 150), to ensure that a huge sample size doesn't make any negligible profitable system have a confidence level of 99%+. Otherwise, you could have a system with a very small profit and a very large standard deviation (an otherwise unattractive system) APPEAR to be extremely attractive based upon a sample size of 20,000 trades, for example.
Other way to handle this would be to require a minimum profit per trade of some fixed amount, in order to trade the system. In this way, a system with an average profit of $1 per trade, with a standard deviation of $1500 could be eliminated, regardless of a potentially high confidence level that would be generated based upon 20,000+ trades.
P.S. Post any other questions in the other thread, to keep the discussion more easy to follow. [/QUOTE]
thx for the feedback - i suppose another approach could be to calculate the "implied" minimum sample size for a given level of confidence, e.g. i have avg profit of 1 and stdev of 5; i want a 98% confidence level, t-value of 2.0 - to get to this level, my sample needs to be at least 100 (i.e. 1/5*sqrt(100)=2)
i suppose what you are saying is that you are comfortable when the implied / necessary sample size is 150 or less (i am assuming that actual sample size will always be reasonable, i.e. hundreds, at least).
i guess the question is - what level is one "comfortable" with in terms of the number of trades which will take your average profit to the level of 2 standard deviations above zero?
I feel that one danger in paper designing systems is to rely too strongly on assumptions of normal distribution. This is fine in most cases, but I feel that ideally you would want to preserve all characteristics of the underlying distribution, something best achieved by random sampling.
In the case of a system with a historical performance of 1 and std dev of 5 based on sample, I would also consider the following simulation : use the sample to construct randomly sampled simulated results, complete with friction cost and whatever position sizing regime you use, and then take measurements from each simulated sample.
You'll want to measure the outcome of course, but also the max drawdown and the time spent in drawdown and all sorts of other metrics that interest you.
If you make thousands of simulations and collect thousands of these metrics, you can build confidence intervals for all those metrics - eg what is the 90th percentile drawdown, what is the 10th percentile outcome after 100 days .... etc, all of which will give you a much better sense of what kind of experience you will have trading the system.
Mean Variance tests - a previous poster had written about a modification of adding tight stops to a system that lowered the expectation but increased the z-stat.
All else being equal, what this result means is that the modification is superior, not inferior.
The fact that it has a lower expectation is more than made up for the fact that it can take on more leverage to produce greater return at the same level of risk.
This is the theoretical argument of course and is subject to close examination of the skewness of the returns.
Increased leverage makes one much more sensitive to outliers.
Quote from TheStudent:
I feel that one danger in paper designing systems is to rely too strongly on assumptions of normal distribution. This is fine in most cases, but I feel that ideally you would want to preserve all characteristics of the underlying distribution, something best achieved by random sampling.
In the case of a system with a historical performance of 1 and std dev of 5 based on sample, I would also consider the following simulation : use the sample to construct randomly sampled simulated results, complete with friction cost and whatever position sizing regime you use, and then take measurements from each simulated sample.
You'll want to measure the outcome of course, but also the max drawdown and the time spent in drawdown and all sorts of other metrics that interest you.
If you make thousands of simulations and collect thousands of these metrics, you can build confidence intervals for all those metrics - eg what is the 90th percentile drawdown, what is the 10th percentile outcome after 100 days .... etc, all of which will give you a much better sense of what kind of experience you will have trading the system.
Re: Profitability Confidence Level
Quote from EricP:
Is my system profitable? At what level of confidence?
Assuming we are using 'future' data (whether this is backtested future data or actual forward tested data doesn't matter), then we can use statistical analysis to our data to determine the likelihood that our system is a profitable system over the long run as follows:
x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits)
Once "x" is calculated, you can look up the confidence level on the chart below to see the likelihood that this system will be profitable over the long haul:
x ------------------ Confidence Level
0 ------------------ 50%
0.075 ------------ 53%
0.126 ------------ 55%
0.25 -------------- 60%
0.36 -------------- 64%
0.52 -------------- 70%
0.67 -------------- 75%
0.84 -------------- 80%
1.04 -------------- 85%
1.28 -------------- 90%
1.4 ---------------- 92%
1.645 ------------- 95%
2.05 --------------- 98%
3.0 ----------------- 99%
4.0 ----------------- 99.5%
5.0 ----------------- 99.7%
Note that if the average profit is negative, then this will give you the confidence level that this is a losing system, over the long run.
One quick example to illustrate this equation. Assume that we have 50 trades, yielding an average profit of $40 per trade, with a standard deviation of $150 per trade. Using the equation and table:
x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
x = abs(40)*(50)^0.5/150
x = 1.88, from table, Confidence Level ~97%
-Eric
Re: Re: Profitability Confidence Level
Quote from nonlinear5:
This amounts to $79.28 profit per trade, with the standard deviation of $156.
Plugging these into your formula gives me 2.16 which maps to confidence interval of between 98% and 99%. It feels like the formula defies the intuition. I mean, would you really feel that the strategy has greater than 98% chance to succeed over the long run based on the results of 18 trades over a 3-month test period?
Hi,
Someone can help to code this formula
x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits) in EL tradestation ?
Thanks.
Best Regards.
Ludo.
http://hquotes.com/tradehard/simulator.html
Eric, the equity curve varys when I input the same data
but when w/l is greater than 2 and w prob >0.4
the equity changes little
why
Quote from robinxing:
http://hquotes.com/tradehard/simulator.html
Eric, the equity curve varys when I input the same data
but when w/l is greater than 2 and w prob >0.4
the equity changes little
why
I use mean-variance optimization to determine the weights of the systems in my portfolio. Because I reoptimize regularly, systems whose performance starts to deteriorate are "gradually" stopped out, i.e. their weight will diminish over time, and may reach zero eventually in case they haven't offered anything in terms of reducing the variance of the returns or improving the profitability of the portfolio.
I also like to plot things like Sharpe and Sortino ratios (together with bootstrapped confidence intervals) over time and see if and how they converge as the sample gets larger. Ideally, they settle around some average after 100-200 obs, while the confidence interval narrows over time.
thank you Eric
another question
in your formula
x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
my question is what is the best number of trades for an intraday
trading system whose frequency is about 100 trades in one month
Re: Optimization
Quote from EricP:
First, before I forget, let me address one of the biggest mistakes people make when developing a system: optimization.
Typical traders will take a program like TradeStation or WealthLab or the equivalent, and create a spectacular trading system with clever and creative ideas, which has a near-linear P&L curve, and with very few minor drawdowns... They are excited and can't wait to start making the 'easy money' with live trading.
Will they actually make money with their new methods? Actually, we don't have a clue, as they've never tested whether their new method will make money in the future on not. All they have actually done is prove that their new system does any excellent job of being optimized to fit the past data, which in and of itself, is worthless.
How do we improve upon this? Test your system in the future! Hmmm... the future, you ask, how would we do that?
Let's assume that you are doing your backtesting on 5 years of data, and the system generates 150 trades per year, for a total of 750 trades. In order to tell how well the system performs in the future, you must withhold all of your data from the optimization tests. Instead, for example, only optimize on the first two years of data (300 trades, pretty decent number). Then, use your optimized system to now test on the 'future' => i.e. test year 3 of data with the system optimized on years 1 and 2.
Next, reoptimize your system on the data from years 2 and 3, and give it a future test with the data from year 4. Similarly, optimize your system with the data from year 3 and 4, giving it a 'real-time' test against the data from year 5. In this way, you can see how your system will perform in the 'future' based upon whatever optimization methods you are using.
This sort of method should give you increased confidence when you go to live activation that it should work well in the 'future'.
-Eric
good job, the only way to know is to put real money behind it.
Quote from NoWorries:
I use mean-variance optimization to determine the weights of the systems in my portfolio. Because I reoptimize regularly, systems whose performance starts to deteriorate are "gradually" stopped out, i.e. their weight will diminish over time, and may reach zero eventually in case they haven't offered anything in terms of reducing the variance of the returns or improving the profitability of the portfolio.
Re: Re: Re: Profitability Confidence Level
Quote from EricP:
As I probably mentioned earlier in the thread, I prefer to have at least 30 trades before using the formula for statistical significance. However, to answer your question => Yes, based upon your data, I would certainly be happy to activate the system you describe. Obviously, going forward, the results might improve or deteriorate, but my activation decision must be made with the information we currently have at hand.
Just to be clear what the formula is saying. Let's assume that a certain trading strategy makes 5000 trades over the course of two weeks (we'll call this the entire 'population' of trades that we have). Now, let's write the results of each of the 5000 trades on a piece of paper and put them in a bucket. Next, we will draw out 18 random pieces of paper from the bucket and do the calculation above, based on the results of those 18 trades. The formula tells you that there is a ~98% chance that the entire population of 5000 trades has an average profit above zero (i.e. net profitable). That's all it tells you. But, I find this to be extremely useful in providing an unbiased foundation for making activation/deactivation decisions.
Best of luck,
-Eric
P.S. I've now 'subscribed' to this thread, so that I will be able to see when future posts are made and will hopefully do a better job of replying to questions in the future. (this is the first thread I've subscribed to, so hopefully it will work). Also, if I fail to address anyone's that posts to me on this thread in the future, please feel free to send me a PM and point out the post. Thanks.
Quote from bolter:
No Worries,
You are using a standard Markowtiz approach (Efficient Frontier)? And what frequency or events trigger a re-opt?
Thanks,
bolter
NoWorries,
Thanks for your response. Out of curiosity what platform are you using, Matlab or similar?
cheers
Quote from bolter:
NoWorries,
Thanks for your response. Out of curiosity what platform are you using, Matlab or similar?
cheers
Quote from robinxing:
thank you Eric
another question
in your formula
x = abs(Avg-Profit)*(Number of Trades)^0.5 / (Std Dev of Profits)
my question is what is the best number of trades for an intraday
trading system whose frequency is about 100 trades in one month
Re: Re: Optimization
Quote from NoWorries:
But then the question becomes: what to do when the out-of-sample results look bad?
Many people will feel tempted to redesign the trading system, and repeat the procedure, until they discover a system that worked well in both the in-sample and out-of-sample period. Needless to say, such a system is over-optimized as well and will likely disappoint when it goes live.
Re: Re: Re: Re: Profitability Confidence Level
Quote from metooxx:
Eric,
How many systems in how many markets are you now running?
Michael
Eric
thank you for your method
but maybe the method is not senstive
look at the trade number
from 350 -500,the equity curve becomes a horizon line
with your method the system should be deactived
but now the system begins to make money
Quote from robinxing:
Eric
thank you for your method
but maybe the method is not senstive
look at the trade number
from 350 -500,the equity curve becomes a horizon line
with your method the system should be deactived
but now the system begins to make money
Quote from Surfeur:
Hi,
Someone can help to code this formula
x = abs(Avg-Profit) * (Number of Trades)^0.5 / (Std Dev of Profits) in EL tradestation ?
Thanks.
Best Regards.
Ludo.
Quote from Surfeur:
Someone for help me ? Please.
__________________
"You have to fix your roof when it's sunny outside" - JFK
Hey Eric,
Excellent thread. In essence you have quantified a way to say when the system that you are trading live is not that profitable compared to the swings it is causing in your portfolio and when to shut it down. Excellent. I currently just do it based on the way it feels on my gut; but good to see it quantified.
Not sure if you have answered this already; but how do you solve the following problem:
Say you deactivate a system based on this technique; essentially you are stopping to trade it after essentially it has caused somewhat of a drawdown (that when the confidence number goes down). Now assume this system stays profitable and it comes back into your green zone (ok to trade), the only way it can do that is, it went from a bad spell to a good spell. Now assuming that you know that the system will stay profitable (albeit with lower profitability) arent you better off with sticking with it all the while?
yes; i know i think thats a trade off you make to stop trading a system that goes from being profitable to not being profitable anymore; but is there a better solution?
-gariki
Quote from kcgoogler:
Hey Eric,
Excellent thread. In essence you have quantified a way to say when the system that you are trading live is not that profitable compared to the swings it is causing in your portfolio and when to shut it down. Excellent. I currently just do it based on the way it feels on my gut; but good to see it quantified.
Not sure if you have answered this already; but how do you solve the following problem:
Say you deactivate a system based on this technique; essentially you are stopping to trade it after essentially it has caused somewhat of a drawdown (that when the confidence number goes down). Now assume this system stays profitable and it comes back into your green zone (ok to trade), the only way it can do that is, it went from a bad spell to a good spell. Now assuming that you know that the system will stay profitable (albeit with lower profitability) arent you better off with sticking with it all the while?
yes; i know i think thats a trade off you make to stop trading a system that goes from being profitable to not being profitable anymore; but is there a better solution?
-gariki
Agree 100% with what you said.
Thanks!
-gariki
All times are GMT. The time now is 07:08 AM.