Your Account  •  Become a Member  •  Help  •  Search
 Forums ›› Technically Speaking ›› Strategy (System) Design ›› What is the correct way to think analyse this?

 Page 4 of 5:   1  2  3   4  5
 abattia   Registered: Dec 2008 Posts: 983 12-30-10 10:26 AM Stoxtrader and intradaybill, many thanks. So it would be "curve fitting". Fair enough ... Quote from Stoxtrader: ... OTOH, you might convince me that you're right if in backtesting the performance improves significantly ... No, the improvement in backtest performance is not earth-shattering ... Edit/Delete • Quote • Complain
 Stoxtrader   Registered: Jul 2006 Posts: 228 12-30-10 09:23 PM Quote from intradaybill: Good question...I'm not sure about that. Can you elaborate how to apply this idea? I think it is more of selection bias issue. Most people do not realize that the performance of stock indices in the long term is chosen purposly to be positive by constantly eliminating worse performers and inluding best ones. Selection bias is the same as censored/truncated data. Search for Heckman algorithm, Heckman solution, Heckman selection model, Heckman two-step procedure, Heckman correction. One implementation is in the R package sampleSelection. http://cran.r-project.org/web/packa...s/selection.pdf Edit/Delete • Quote • Complain
abattia

Registered: Dec 2008
Posts: 983

12-30-10 10:08 PM

Thank you to the "ET brain trust" members still with me on this one ...

To re-cap:

• I am investigating a systematic strategy (stock swing trades, daily bars).
• When backtested against individual NASDAQ 100 stocks, the strategy sometimes trades only a low (i.e. single digit) number of times each year, not enough for any performance analysis to be statistically significant.
• However, backtested against the current NASDAQ 100 stocks as a group, the strategy trades on average 500+ times in a year (over the last 4 years).
• I am trying to determine whether I have a statistically significant number of backtest trades to analyze the strategy viz-a-viz trading against all NASDAQ 100 stocks as a group (can I avoid being “fooled by randomness”?).

Quote from black diamond:
...You would like to know if your signals are clustered in time or evenly distributed...your tests are not independent when treat all the individual trades as a big sample and ignore whether they happened at the same time or not...

Following black diamond’s suggestion, I have been analysing how individual trades are distributed in time.

METHOD
Over 4 years of backtest data, I analysed whether multiple trades of different NASDAQ 100 stocks were occurring together.

I assumed any two trades were not statistically independent (i.e. were “statistically dependent”?) if :
a) BOTH entered on day “A”,
b) BOTH exited on the day “B”, and
c) BOTH were winners (or both were losers).

Otherwise I assumed they were statistically independent.

Then I averaged the number of “statistically dependent” trades that occurred each time the system traded.

RESULTS
The results are below (and I have no idea why they are so much further down the page LOL!).

CONCLUSION
From the results, is it fair to conclude that a good guestimate of the number of statistically independent "tests" of the strategy will be the total number of unique trades divided by 3 (i.e. approx 170+/yr)?

Year Total # of
Avg # of
Std Dev of # of "statistically dependent" trades occuring each time strategy trades
2010 521 2.8 3.7
2009 435 2.5 3.6
2008 666 2.9 3.6
2007 577 2.4 2.3

Edit/Delete Quote Complain
 DeeDeeTwo   Registered: Dec 2007 Posts: 623 01-01-11 04:52 PM I'm sure any semi-competent 19 yo math type with a PC... Could use what used to be called "data mining"... And today is called "data dredging"... To come up with endless similar results in 100 hours. The problem is Detachment From Reality... The market is not a bunch of data points... It's 10,000 or 50,000 experts that impose very high costs on you... That take no fucking prisoners... And chisel you for \$0.01... over and over up and down the Food Chain... Something like this might have been marginally profitable in 90s before decimalization. The Correct Way To Analyze This (1) Very specifically, why does this approach work? (2) Why does this TAKE MONEY AWAY from the Top 10,000 traders? (3) What is your Competitive Advantage... That allows you to overcome transaction costs + spread + cheating... And take money way from Market Makers, Fund Managers, and Insiders? Unless you can answer these questions SPECIFICALLY... you have nothing. Eventually you will have to give up on this... And develop some actual expertise... And an actual Competitive Advantage. Edit/Delete • Quote • Complain
 abattia   Registered: Dec 2008 Posts: 983 01-01-11 06:03 PM Quote from DeeDeeTwo: ... Unless you can answer these questions SPECIFICALLY... you have nothing. Eventually you will have to give up on this... And develop some actual expertise... And an actual Competitive Advantage. Many thanks! Don't mean to seem as though I am ignoring you; I'm not! If you’d like to start a separate thread with the title of "Backtesting is Pointless", or whatever most closely represents the opinions you express above, I'll be happy to contribute. I’d be an interesting thread, I’m sure ... My own opinion is that it’s not pointless, and I’m trying to solicit help here from others with a similar view ... Edit/Delete • Quote • Complain
 goodgoing   Registered: Jun 2009 Posts: 340 01-02-11 10:48 AM Quote from Stoxtrader: Selection bias is the same as censored/truncated data. Search for Heckman algorithm, Heckman solution, Heckman selection model, Heckman two-step procedure, Heckman correction. One implementation is in the R package sampleSelection. http://cran.r-project.org/web/packa...s/selection.pdf Censoring involves partially known data and it is the beyond control of the researcher. Selection bias is related to the method of collecting samples by the researcher. I cannot see why you equate the two although I don't claim to be a statistics expert. Maybe someone with a formal graduate degree in the subject could help in resolving this. Edit/Delete • Quote • Complain
 Page 4 of 5:   1  2  3   4  5