Why it's so hard to beat the market (Wisdom of Crowds)

Discussion in 'Trading' started by Steve Ladd, Nov 10, 2020.

  1. It is common knowledge that most hedge funds, mutual funds, and advisors under-perform the general market. Even Warren Buffet has fallen behind! It seems that the vast majority of market participants do worse than average. I’ve always wondered, how can this be so? Why is it so hard to beat the market given that just as many people must do better than an average as do worse than the average? Well I think I have found an explanation. Part of it comes from the concept of the Wisdom of Crowds and part comes from clearer statistical thinking.

    A favorite example of the Wisdom of Crowds is a guessing contest at a county fair where everybody guesses the weight of an ox. Each person writes their estimate on a piece of paper. When the guesses are averaged it is found that said average is closer to the ox's true weight than the estimates of most individuals, even the experts. Note that the crowd is not a mob. Rather, they are a diverse collection of independent individuals making a certain type of prediction.

    Now an analogy. For the guessing contest substitute the market. For the guessers substitute the market participants. For the average of all guesses substitute the portfolio of the S&P 500 or whatever benchmark you follow. The analogy tells us that no individual is likely to beat the market because their wisdom is inferior to the collective wisdom.

    But our analogy remains incomplete. If the average of all guesses represents the holdings of the S&P 500, what does the true weight of the ox represent? Note that the Wisdom of Crowds theory doesn’t require that the average of the guesses be exactly right. The average guess is surprisingly good but there will be some discrepancy between it and the true weight.

    Perhaps the true weight of the ox should represent that system which, of all possible systems, would give the highest return. After all, we aren’t just trying to match a benchmark, we’re trying to get as rich as possible. In other words for the true weight of the ox substitute the Holy Grail, the best imaginable trading system.

    This requires us to rethink the significance of the benchmark. Perhaps it’s incorrect to think of it as the average of all the returns experienced in that market in that time frame. The S&P 500 isn’t the average return of everyone who dabbled in that market. It’s simply what those exact stocks actually achieved.

    If this is so, consider the ramification. The gap between the average ox-weight guess and the actual ox weight becomes analogous to the gap between what the benchmark achieves and what the Holy Grail would achieve. Here the analogy presents a problem. The ox weight is an actual number, and the average guess comes pretty close to it. But the Holy Grail is hypothetical, and would seemingly be far superior to holding the S&P 500.

    Or must it? Maybe all those Holy Grails are more constrained than we think. Maybe the Holy Grail’s realistic return, over time, isn’t so much higher than that of the benchmark. If this is so, the little gap between the average ox-weight guess and the actual ox weight represents the difference between the benchmark and a portfolio or strategy that does only a little better on a consistent basis. Now it’s not so hard to imagine a sufficient number of benchmark-beaters.

    To summarize, my thesis is that it’s hard to beat the market because 1) the market benefits from the Wisdom of Crowds whereas we, as individuals, do not, and 2) the realistic return of the best possible system or portfolio isn’t as great as we imagine due to real-world constraints. Just as very few guessers will guess more accurately than the average guess, very few market participants will get returns superior to that of the benchmark. And the average of all participant returns won’t equal that of the benchmark. Rather, it will be well below.
     
    .sigma, christielowe and Axon like this.
  2. maxinger

    maxinger

    Don't attempt to beat the market.

    The market does not make our lives miserable.
    The market does not create trouble for us.
    The market is innocent.
    In fact, it provides tons of opportunities.
    It is up to us to seize the opportunities.

    Always attempt to beat our old lousy bad self.

    Word of Wisdom: Blame ourselves and not the market.
     
    .sigma, christielowe, orbit23 and 7 others like this.
  3. CharlesS

    CharlesS

    bc few participants have the requisite
    • Knowledge,
    • Determination,
    • Wits,
    • Self-awareness, &
    • Self-control
    to successfully play the winner-take-all games of short and intermediate-term trading

    & bc those that do have scant motivation to share their knowledge
     
    persistence and Nobert like this.
  4. Occam

    Occam

    I thin you are right in that the market benefits from crowd wisdom. But keep in mind that the market price of any given stock represents expected future returns, so the cow guessing game is interesting, but a different problem; with the market, we don't know the answer to the "cow weight" until the company dissolves, and even then you have to take into account opportunity cost.

    Countless academic papers seem to indicate that mutual fund managers underperform, on average, by almost exactly their average fee.

    Net of transaction costs/fees, the average fully invested market participant will exactly match their benchmark, if it is the perfect benchmark for that investor (an impossible ideal but close enough to reality IMO), IF by average you mean "arithmetic mean". If however you mean "median", it can deviate quite a bit.
     
  5. Overnight

    Overnight

    Exactly. There is nothing to "beat". Why is that considered important in the grand scheme of things? Who cares about "beating it"? Just go with the flow, man.
     
    formikatrading and maxinger like this.
  6. Peter8519

    Peter8519

    Try the best to be on the market side but it's not easy. The market is irrational but it is what it is. A promise of good vaccine against COVID-19 is enough to spike those industries affected by the virus.
     
    Last edited: Nov 10, 2020
    murray t turtle likes this.
  7. Dazz

    Dazz

    apparently traders fail for 5 reasons, some of which are not fixable.....
     
    beginner66 likes this.
  8.  
    formikatrading likes this.
  9. I googled Dr. Dean Handley scam. Not the most credible source for trading advice IMHO
     
    christielowe likes this.
  10. Overnight

    Overnight

    Ouch @KCalhoun Dr. Handley called you out in that vid! How does it feel, to be shit upon by the world's worst reviewer ever?
     
    #10     Nov 10, 2020
    christielowe likes this.