 |
abattia
Registered: Dec 2008
Posts: 983 |
12-30-10 10:26 AM
Stoxtrader and intradaybill, many thanks.
So it would be "curve fitting". Fair enough ...
Quote from Stoxtrader:
... OTOH, you might convince me that you're right if in backtesting the performance improves significantly ...
No, the improvement in backtest performance is not earth-shattering ...
|
| |
|
Edit/Delete • Quote • Complain |
Stoxtrader
Registered: Jul 2006
Posts: 228 |
12-30-10 09:23 PM
Quote from intradaybill:
Good question...I'm not sure about that. Can you elaborate how to apply this idea? I think it is more of selection bias issue. Most people do not realize that the performance of stock indices in the long term is chosen purposly to be positive by constantly eliminating worse performers and inluding best ones.
Selection bias is the same as censored/truncated data. Search for Heckman algorithm, Heckman solution, Heckman selection model, Heckman two-step procedure, Heckman correction. One implementation is in the R package sampleSelection.
http://cran.r-project.org/web/packa...s/selection.pdf
|
| |
|
Edit/Delete • Quote • Complain |
abattia
Registered: Dec 2008
Posts: 983 |
12-30-10 10:08 PM
Thank you to the "ET brain trust" members still with me on this one ...
To re-cap:
- I am investigating a systematic strategy (stock swing trades, daily bars).
- When backtested against individual NASDAQ 100 stocks, the strategy sometimes trades only a low (i.e. single digit) number of times each year, not enough for any performance analysis to be statistically significant.
- However, backtested against the current NASDAQ 100 stocks as a group, the strategy trades on average 500+ times in a year (over the last 4 years).
- I am trying to determine whether I have a statistically significant number of backtest trades to analyze the strategy viz-a-viz trading against all NASDAQ 100 stocks as a group (can I avoid being “fooled by randomness”?).
Quote from black diamond:
...You would like to know if your signals are clustered in time or evenly distributed...your tests are not independent when treat all the individual trades as a big sample and ignore whether they happened at the same time or not...
Following black diamond’s suggestion, I have been analysing how individual trades are distributed in time.
METHOD
Over 4 years of backtest data, I analysed whether multiple trades of different NASDAQ 100 stocks were occurring together.
I assumed any two trades were not statistically independent (i.e. were “statistically dependent”?) if :
a) BOTH entered on day “A”,
b) BOTH exited on the day “B”, and
c) BOTH were winners (or both were losers).
Otherwise I assumed they were statistically independent.
Then I averaged the number of “statistically dependent” trades that occurred each time the system traded.
RESULTS
The results are below (and I have no idea why they are so much further down the page LOL!).
CONCLUSION
From the results, is it fair to conclude that a good guestimate of the number of statistically independent "tests" of the strategy will be the total number of unique trades divided by 3 (i.e. approx 170+/yr)?
Year |
Total # of
unique trades |
Avg # of
"statistically dependent" trades occuring each time strategy trades |
Std Dev of # of "statistically dependent" trades occuring each time strategy trades |
|---|
| 2010 |
521 |
2.8 |
3.7 |
| 2009 |
435 |
2.5 |
3.6 |
| 2008 |
666 |
2.9 |
3.6 |
| 2007 |
577 |
2.4 |
2.3 |
|
| |
|
Edit/Delete • Quote • Complain |
DeeDeeTwo
Registered: Dec 2007
Posts: 623 |
01-01-11 04:52 PM
I'm sure any semi-competent 19 yo math type with a PC...
Could use what used to be called "data mining"...
And today is called "data dredging"...
To come up with endless similar results in 100 hours.
The problem is Detachment From Reality...
The market is not a bunch of data points...
It's 10,000 or 50,000 experts that impose very high costs on you...
That take no fucking prisoners...
And chisel you for $0.01... over and over up and down the Food Chain...
Something like this might have been marginally profitable in 90s before decimalization.
The Correct Way To Analyze This
(1) Very specifically, why does this approach work?
(2) Why does this TAKE MONEY AWAY from the Top 10,000 traders?
(3) What is your Competitive Advantage...
That allows you to overcome transaction costs + spread + cheating...
And take money way from Market Makers, Fund Managers, and Insiders?
Unless you can answer these questions SPECIFICALLY... you have nothing.
Eventually you will have to give up on this...
And develop some actual expertise...
And an actual Competitive Advantage.
|
| |
|
Edit/Delete • Quote • Complain |
abattia
Registered: Dec 2008
Posts: 983 |
01-01-11 06:03 PM
Quote from DeeDeeTwo:
... Unless you can answer these questions SPECIFICALLY... you have nothing.
Eventually you will have to give up on this...
And develop some actual expertise...
And an actual Competitive Advantage.
Many thanks!
Don't mean to seem as though I am ignoring you; I'm not!
If you’d like to start a separate thread with the title of "Backtesting is Pointless", or whatever most closely represents the opinions you express above, I'll be happy to contribute. I’d be an interesting thread, I’m sure ...
My own opinion is that it’s not pointless, and I’m trying to solicit help here from others with a similar view ...
|
| |
|
Edit/Delete • Quote • Complain |
goodgoing
Registered: Jun 2009
Posts: 340 |
01-02-11 10:48 AM
Quote from Stoxtrader:
Selection bias is the same as censored/truncated data. Search for Heckman algorithm, Heckman solution, Heckman selection model, Heckman two-step procedure, Heckman correction. One implementation is in the R package sampleSelection.
http://cran.r-project.org/web/packa...s/selection.pdf
Censoring involves partially known data and it is the beyond control of the researcher.
Selection bias is related to the method of collecting samples by the researcher.
I cannot see why you equate the two although I don't claim to be a statistics expert. Maybe someone with a formal graduate degree in the subject could help in resolving this.
|
| |
|
Edit/Delete • Quote • Complain |
| Receive
an email whenever a new post is added to this thread by subscribing
to it. |
|
|
|
|
