
Forums (http://www.elitetrader.com/vb/index.php)
- Strategy (System) Design (http://www.elitetrader.com/vb/forumdisplay.php?forumid=34)
-- What is the correct way to think analyse this? (http://www.elitetrader.com/vb/showthread.php?threadid=212151)
What is the correct way to think analyse this?
I am investigating a systematic strategy (stock swing trades).
Over 10 years, the most the strategy trades is 100 times for any given NASDAQ 100 stock, and for some NASDAQ 100 stocks it trades as little as 25 times (over 10 years). With some NASDAQ 100 stocks, the strategy backtests as profitable over 10 years, over others it does not.
If I were analyzing a strategy’s performance over a single instrument, I’m comfortable about how I would do it. In this instance, I would simply say that 100 trades are not enough (in my opinion) for statistically significant testing of the edge, and I would go no further.
However, over all NASDAQ 100 stocks the strategy trades 9000 times (a statistically significant number) in 10-years, and netting all wins against all losses, the strategy was profitable over NASDAQ 100 stocks.
How do I proceed with analyzing the performance and profile of this strategy traded across all NASDAQ 100 stocks?
Do I just proceed the same way as I would for a single instrument (i.e. determine total cumulative return, total Profit Factor, total Sharpe Ratio, total % winners, total average trade, etc for all trades, regardless of the instrument)?
Is this the right way to do it?
Re: What is the correct way to think analyse this?
Quote from abattia:
I am investigating a systematic strategy (stock swing trades).
However, over all NASDAQ 100 stocks the strategy trades 9000 times (a statistically significant number) in 10-years, and netting all wins against all losses, the strategy was profitable over NASDAQ 100 stocks.
How do I proceed with analyzing the performance and profile of this strategy traded across all NASDAQ 100 stocks?
Do I just proceed the same way as I would for a single instrument (i.e. determine total cumulative return, total Profit Factor, total Sharpe Ratio, total % winners, total average trade, etc for all trades, regardless of the instrument)?
Is this the right way to do it?
Re: What is the correct way to think analyse this?
Do I just proceed the same way as I would for a single instrument (i.e. determine total cumulative return, total Profit Factor, total Sharpe Ratio, total % winners, total average trade, etc for all trades, regardless of the instrument)?
Yes, and if you like what you see, start forward testing with real money.
Quote from abattia:
I am investigating a systematic strategy (stock swing trades).
Over 10 years, the most the strategy trades is 100 times for any given NASDAQ 100 stock, and for some NASDAQ 100 stocks it trades as little as 25 times (over 10 years). With some NASDAQ 100 stocks, the strategy backtests as profitable over 10 years, over others it does not.
If I were analyzing a strategy’s performance over a single instrument, I’m comfortable about how I would do it. In this instance, I would simply say that 100 trades are not enough (in my opinion) for statistically significant testing of the edge, and I would go no further.
However, over all NASDAQ 100 stocks the strategy trades 9000 times (a statistically significant number) in 10-years, and netting all wins against all losses, the strategy was profitable over NASDAQ 100 stocks.
How do I proceed with analyzing the performance and profile of this strategy traded across all NASDAQ 100 stocks?
Do I just proceed the same way as I would for a single instrument (i.e. determine total cumulative return, total Profit Factor, total Sharpe Ratio, total % winners, total average trade, etc for all trades, regardless of the instrument)?
Is this the right way to do it?
Re: Re: What is the correct way to think analyse this?
Thanks for the excellent response Stoxtrader
Quote from Stoxtrader:
How many variables go into determining the buy/sell signal(s)? For many variables, 9000 trades may not be enough.
Quote from Stoxtrader:
Has the system been tested out-of-sample? In other words, backtest for 10 years looking back, test for 1 year blind..
Quote from Stoxtrader:
Figure out return, profit factor, Sharpe ratio, etc. per year. If the system does not outperform the index for all 10 years taken individually, it might not be worth trading. What I'm saying is, the Nasdaq 100 becomes your benchmark to beat, consistently, every year.
Re: Re: What is the correct way to think analyse this?
Quote from oraclewizard77:
Yes, and if you like what you see, start forward testing with real money.
I would think about testing it as a portfolio. You would like to know if your signals are clustered in time or evenly distributed, and also how the signal performance varies over time. Also if you want to be statistically anal, your tests are not independent when treat all the individual trades as a big sample and ignore whether they happened at the same time or not, and running your test on the full portfolio is one way to handle this.
I would also consider testing it by sub-portfolios - does it work in stocks of all market cap, trading volume, industry groups, etc?
__________________
black diamond
Quote from black diamond:
I would think about testing it as a portfolio. You would like to know if your signals are clustered in time or evenly distributed, and also how the signal performance varies over time. Also if you want to be statistically anal, your tests are not independent when treat all the individual trades as a big sample and ignore whether they happened at the same time or not, and running your test on the full portfolio is one way to handle this.
I would also consider testing it by sub-portfolios - does it work in stocks of all market cap, trading volume, industry groups, etc?
Also, please keep in mind that the index composition has changed over the years and comparing performance to its returns by trading current composition may not reflect realistic conditions.
I think the greatest challenge is the allocation. It makes no sense to allocate the same amount of money to stocks with different alpha and betas.
I have faced all these problems before and I never found a good solution neither for backtesting, nor for allocation. For the backtesting I chose just a subgroup of stocks that mostly replicated the index but were always in it for the testing period and for allocation I used ATR.
Quote from intradaybill:
Also, please keep in mind that the index composition has changed over the years and comparing performance to its returns by trading current composition may not reflect realistic conditions.
I think the greatest challenge is the allocation. It makes no sense to allocate the same amount of money to stocks with different alpha and betas.
I have faced all these problems before and I never found a good solution neither for backtesting, nor for allocation. For the backtesting I chose just a subgroup of stocks that mostly replicated the index but were always in it for the testing period and for allocation I used ATR.
... for allocation I used ATR.
I ‘m wondering whether there’s a way to save myself unnecessary work... but my maths and statistics aren’t good enough to figure this out for myself!
I rely on Sharpe Ratio as a go/no-go metric for further detailed investigation (and eventually trading hopefully!) of any system.
If the range of Sharpe Ratios for the strategy applied to individual NASDAQ 100 stocks (with no more than 100 trades over 10 years in each case) is between X – Y, can I guestimate that the “TOTAL Sharpe Ratio” for the strategy applied across ALL NASDAQ 100 stocks together will ALSO fall in the X – Y range (in which case it won’t reach my threshold)?
Or could TOTAL Sharpe Ratio end up being above the X – Y range?
[... as I write the above I have the sneaking suspicion that the next book I read should be about Portfolio Theory ...
]
Quote from abattia:
... suspicion that the next book I read should be about Portfolio Theory
Many thanks!
Quote from Steven.Davis:
Your intuition is good. If you are trading completely uncorrelated markets with uncorrelated signals, then one still gets a diversification benefit which bumps the portfolio Sharpe. In real life, the markets are correlated and your signals are correlated. This can help or hurt the portfolio depending upon whether you consider this in your trading.
R is a column vector of returns
w is a column vector of your portfolio weighting (w>=0) (sum w = 1)
V = covariance matrix.
R0 is Risk-free rate = 0% these days
Sharpe_port = Expected excess return / expected volatility
= R * w -R0/ sqrt( w * V * w)
If no correlation, then V is diagonal. If all N markets have equal volatility v, and an equal weighting
Sharpe_port = R/N / sqrt( v * v * 1/N) = Sharpe_each /sqrt(N)
If all of your systems on all of your markets perform equally, then the portfolio Sharpe is 1/sqrt(Number of traded market system pairs)
Adding one degree of realism, your system-market pairs are not expected to perform identically so one needs to weight intelligently. If almost all of your money is in one stock, then the portfolio is really just the one stock, and the Sharpe Ratio will follow accordingly.
Adding another degree of realism, your system-market pairs are not uncorrelated. This is especially true when major adverse affect occur. If you want to buy stocks that all fit the same mold, then bad news affects several trades so w * V * w is closer to N*N*v*v/N/N = v*v so you expect the average return with no diversification benefit.
One can keep at this, and do some really sophisticated money-management/portfolio allocation math to look at your current positions and your current opportunities to find your smartest moves. For example, if you already have a long Euro trade, and have a signal for a short Swiss Franc trade, should you resize your Euro trade? Good traders know how their markets are related, and take this kind of thing into account. Objectively doing this has a bad name. Real-life rebalancing costs are substantial, and too often these models are unrealistic. I have my own way of doing this.
One easy way to get some diversification benefit without going huge on analysis is to use pair-trading/spread trading. If you want to bet on the general market, then one can just bet an index. If you have an opinion on a particular stock or future, then find a correlated/cointegrated reference point. For example, maybe you like a particular gold mining stock relative to an index of gold mining stocks. If you are right, then buying one and selling the other should net a positive return, but if balanced, a low volatility. This makes for a good Sharpe Ratio.
By making a portfolio of pair-trades, one can lower volatility through diversification. (How correlated is a gold stock vs a gold index trade when compared with an Internet stock vs a technology index?)
By minimizing the volatility, one can trade larger for the same risk, and make more money off of the same trading opportunities.
Quote from intradaybill:
... for allocation I used ATR.
Curve fitting? Yes or No?
To the various thread contributors, thanks for your helpful and insightful suggestions!
Quote from black diamond:
... I would also consider testing it by sub-portfolios - does it work in stocks of all market cap, trading volume, industry groups, etc? ...
Quote from intradaybill:
Also, please keep in mind that the index composition has changed over the years and comparing performance to its returns by trading current composition may not reflect realistic conditions.
I think the greatest challenge is the allocation. It makes no sense to allocate the same amount of money to stocks with different alpha and betas.
I have faced all these problems before and I never found a good solution neither for backtesting, nor for allocation. For the backtesting I chose just a subgroup of stocks that mostly replicated the index but were always in it for the testing period and for allocation I used ATR.
Re: Curve fitting? Yes or No?
Quote from abattia:
As a result of the above, I've determined that the NASDAQ 100 stocks less 24 of them (Biotech, 10; Healthcare, 8; Media, 4; and Transportation, 2) would have performed better than trading the strategy over all 100 of the NASDAQ 100.
But if I decide to trade just the 76 "good" stocks (rather than the full 100) aren't I just "curve fitting"?
_________
(*) The sector assignments are my own
Re: Re: Curve fitting? Yes or No?
Quote from Stoxtrader:
Curve fitting, IMHO. Especially since the sector assignments are your own. One of the "gurus", Zweig I'm thinking but I can't find it in a quick google search, excludes the real estate sector. Ridiculous, I say. The only "sector" to exclude might be ADR's since foreign companies might be operating under different financial regulations.
OTOH, you might convince me that you're right if in backtesting the performance improves significantly *and* in blind forward testing for 1-2 years the performance difference persists.
Quote from Stoxtrader:
Can't the data just be considered censored?
http://en.wikipedia.org/wiki/Censoring_%28statistics%29
Re: Re: Curve fitting? Yes or No?
Stoxtrader and intradaybill, many thanks.
So it would be "curve fitting". Fair enough ...
Quote from Stoxtrader:
... OTOH, you might convince me that you're right if in backtesting the performance improves significantly ...
Quote from intradaybill:
Good question...I'm not sure about that. Can you elaborate how to apply this idea? I think it is more of selection bias issue. Most people do not realize that the performance of stock indices in the long term is chosen purposly to be positive by constantly eliminating worse performers and inluding best ones.
Thank you to the "ET brain trust" members still with me on this one ...
To re-cap:
Quote from black diamond:
...You would like to know if your signals are clustered in time or evenly distributed...your tests are not independent when treat all the individual trades as a big sample and ignore whether they happened at the same time or not...
| Year | Total # of unique trades | Avg # of "statistically dependent" trades occuring each time strategy trades | Std Dev of # of "statistically dependent" trades occuring each time strategy trades |
|---|---|---|---|
| 2010 | 521 | 2.8 | 3.7 |
| 2009 | 435 | 2.5 | 3.6 |
| 2008 | 666 | 2.9 | 3.6 |
| 2007 | 577 | 2.4 | 2.3 |
I'm sure any semi-competent 19 yo math type with a PC...
Could use what used to be called "data mining"...
And today is called "data dredging"...
To come up with endless similar results in 100 hours.
The problem is Detachment From Reality...
The market is not a bunch of data points...
It's 10,000 or 50,000 experts that impose very high costs on you...
That take no fucking prisoners...
And chisel you for $0.01... over and over up and down the Food Chain...
Something like this might have been marginally profitable in 90s before decimalization.
The Correct Way To Analyze This
(1) Very specifically, why does this approach work?
(2) Why does this TAKE MONEY AWAY from the Top 10,000 traders?
(3) What is your Competitive Advantage...
That allows you to overcome transaction costs + spread + cheating...
And take money way from Market Makers, Fund Managers, and Insiders?
Unless you can answer these questions SPECIFICALLY... you have nothing.
Eventually you will have to give up on this...
And develop some actual expertise...
And an actual Competitive Advantage.
Quote from DeeDeeTwo:
... Unless you can answer these questions SPECIFICALLY... you have nothing.
Eventually you will have to give up on this...
And develop some actual expertise...
And an actual Competitive Advantage.
Quote from Stoxtrader:
Selection bias is the same as censored/truncated data. Search for Heckman algorithm, Heckman solution, Heckman selection model, Heckman two-step procedure, Heckman correction. One implementation is in the R package sampleSelection.
http://cran.r-project.org/web/packa...s/selection.pdf
Quote from goodgoing:
Censoring involves partially known data and it is the beyond control of the researcher.
Selection bias is related to the method of collecting samples by the researcher.
I cannot see why you equate the two although I don't claim to be a statistics expert. Maybe someone with a formal graduate degree in the subject could help in resolving this.
Quote from abattia:
Following black diamond’s suggestion, I have been analysing how individual trades are distributed in time.
[b]METHOD
Over 4 years of backtest data, I analysed whether multiple trades of different NASDAQ 100 stocks were occurring together.
I assumed any two trades were not statistically independent (i.e. were “statistically dependent”?) if :
a) BOTH entered on day “A”,
b) BOTH exited on the day “B”, and
c) BOTH were winners (or both were losers).
Otherwise I assumed they were statistically independent.
__________________
black diamond
Re: Curve fitting? Yes or No?
Quote from abattia:
But if I decide to trade just the 76 "good" stocks (rather than the full 100) aren't I just "curve fitting"?
Or does the fact that I would be taking out whole sectors (*) (rather than individual companies) redeem me - at least to some extent - from this cardinal sin?
__________________
black diamond
Re: Re: Curve fitting? Yes or No?
Quote from black diamond:
This is a tougher question. Some purists would consider most of this stuff data mining. ... so I would run tests like these without worrying about data mining.
Thank you, Black Diamond
Quote from black diamond:
...The usual way to handle this is to create an actual portfolio and measure the results in calendar time, not across individual trades. So assuming you are working on daily signals, each day you would average the returns over all your trades, weighted by the position in each. Then calculate your stats on the daily return series...
All times are GMT. The time now is 05:27 PM.