Why do I see "Trends" in Randomly Generated Data?

Discussion in 'Data Sets and Feeds' started by Rahula, Feb 21, 2008.

  1. I don't know which order to give you some pages to read.

    It is one of the funniest books a person will ever read.

    Read pp 273 to see how niave these guys were by nearly the end of the book.

    For kicks also read pp 197 from the paragraph beginning: " We should have focused on this earlier...

    These guys have real money and real people are fundung them to do what they do.

    There is a theme in the book about how one after another people leave as a consequence of their personal realizations.... These people only fight the system for so long before they quit.... The reasons they quit are kinda compelling in some ways.

    Bass was handicapped and just doi9ng what people like him do. Think of Covell, Derman, Mehrling, and Lowenstein as similar birds.

    As the read got funnier and funnier I got out little 3M tabs and clocked thirty such things as those mentioned above.

    People read this stuff and, apparently, are taken in by the fact that the writers don't really catch on to what is going on. How can Bass write up stuff the way he did if he knew anything?

    The TA cutesy is on page 112.
     
    #531     Mar 31, 2008
  2. Jack,
    As much as I hate Tucson, I have to make the effort to come out there for a few days. You're amazing and I mean that in a positive way.
     
    #532     Mar 31, 2008
  3. Please do. Trading starts at 6:30 am local. One more would be fun. The pool hit 92 again today; we are going to reverse the solar.....LOL.

    I sign EKG's weekly....
     
    #533     Mar 31, 2008
  4. kut2k2

    kut2k2

    I'll go slowly.

    If a trend occurs in a random sequence, you won't know it until after the fact, so you can't "catch it" unless you are a compulsive gambler who always bets with the trend, no matter the source. "Expert" casino craps players bet with the trend and sometimes they win big. But that's nothing compared to the amount of money they've probably lost overall by playing a negative expectation game.

    But if the trend is from a nonrandom source, it can be depended on to last for a while or to reoccur if it dissipates. That's the key difference: being able to detect and exploit a nonrandom trend.

    As for your claim that entry into any trend has to be random, that's what makes no sense whatsoever.
     
    #534     Mar 31, 2008
  5. I get Vitamin D overload out there.
    I need clouds and rain occasionally.
    I will suffer later this year and force myself, promise.
    You need to share the EKG though . . . and put ice cubes in the pool.
     
    #535     Mar 31, 2008
  6. Just curious what do you mean by distribution being more or less stable?:) :confused:
     
    #536     Apr 9, 2008
  7. MAESTRO

    MAESTRO

    #537     Apr 9, 2008
  8. So basically there are two camps: those who believe in patterns and those who don't.

    There are known successful (and not so successful) examples of both sides:

    Pattern believers:

    Victor Niederhoffer (blew up?)
    "Niederhoffer, like Buffett and Soros, was a brilliant man. He had a Ph.D. in economics from the University of Chicago. He had pioneered the idea that through proper statistical analysis of patterns in the market an investor could identify profitable anomalies." (THE NEW YORKER, APRIL 22 & 29, 2002)

    Toby Crabel (manages $3.2 billion, Niederhoffer's apprentice)
    Wrote a book "Day Trading With Short Term Price Patterns and Opening Range Breakout." (1990)

    Monroe Trout (retired, networth ~$100 million, Niederhoffer's apprentice)
    "I found that prices were not independent. That is, there were some statistically significant patterns." (The New Market Wizards, interview with Monroe Trout)

    Alan Crary (aka acrary)
    Says he was influenced by Toby Crabel (http://www.elitetrader.com/vb/showthread.php?postid=137766#post137766) and Monroe Trout (http://www.elitetrader.com/vb/showthread.php?postid=858115#post858115)
    Here he talks about non-randomness of the markets (http://www.elitetrader.com/vb/showthread.php?postid=140656#post140656)



    Pattern non-believers:

    Nassim Taleb
    "Taleb, by contrast [to Victor Niederhoffer], has constructed a trading philosophy predicated entirely on the existence of black swans—on the possibility of some random, unexpected event sweeping the markets. He never sells options, then. He only buys them. He’s never the one who can lose a great deal of money if G.M. stock suddenly plunges. Nor does he ever bet on the market’s moving in one direction or another." (THE NEW YORKER, APRIL 22 & 29, 2002)

    MAESTRO (I only know that his name is Alex): his posts on this thread show his views on the markets.

    Also, acrary said he wasn't pattern oriented trader early in his career (here: http://www.elitetrader.com/vb/showthread.php?postid=150348#post150348)



    My own research on trying to find meaningful patterns were fruitless. I've coded the whole pattern mining system. I also took care to remove data mining bias. The input data was daily OHLC data (comparing various data points and ranges, aka "binary identifiers"). The system does find significant patterns, but the main point is that they don't pass out-of-sample tests. I've found a few that persisted, but their occurrence frequency were too low to satisfy my needs. Of course this does not mean there are none patterns, because my search space was limited and I've probably missed some important concept that people mentioned above know.

    On the other hand, fat-tails in the markets are obvious. It's easy to prove with a simple excel sheet.
     
    #538     Oct 26, 2008
  9. Jerry030

    Jerry030

    Indrionas,

    As any good Quantum physicist would tell you, what the observer sees is dependant on their point of view and the instruments they use in observation. Look at a rock in you hand and it appear solid. View it at a magnification approaching the Plank length and it is 99.999999% empty space and the rest is constantly moving strings of energy. Which is true? They both are as the "truth" is totally dependant on the location of the observer. Your question on the markets is the same. Each side in this question is true as the truth is dependant on the point of view and methods of measurement of the different traders.

    I've found that certain markets, such as EUR/JPY can be predicted is a very high level of accuracy: PF > 6 and a total account doubling time < 30 days.

    The tools needed to "observe" the patterns that enable this kind of trading are advanced Neural Networks. NNs don't lend themselves to the kind of statistical analysis that most participants use in this kind of debate. They want to define a tidy simple relationship "how often does the market move higher when the close breaks the trend line after a key reversal on a Doji Star, etc, etc". These measures are simple things the human mind can understand since it dreamed them up in the first place. They are far too simplistic to find dependable patterns in the market.

    In contrast a NN powerful enough to generate consistently profitable trades may use 30 or 40 variables and each one is interrelated to each of the others by a weight to create a network. The complexity of this structure is indicated by the term "black box"....it can't be understood by the human mind. You can print it out but you get 100 pages of numbers that define these interrelationships and weights but it's not something you can understand.

    However a well trained NN can trade with a high level of accuracy which is the essence of successful prediction. So the answer to your question on market predictability depends totally on the skill of the observer in selecting the method of observation and the tools used to measure and organize the elements that make it predictable. Methods and tools that are too simplistic give the correct conclusion that it is not predictable. Those sophisticated enough to capture its complex patterns give the also true answer that that it is predictable. I'm among the latter. But is pointless to ague with folks in the former group as their answer is just a true and mine is.

    I’d suggest you forget this theoretical debate and focus on actual trading instead. Contact me privately if you’d like some links to resources on repositioning yourself as the Observer.

    Jerry030
     
    #539     Oct 26, 2008

  10. Hi,

    I completely understand what you mean. My input was 200-400 binary variables, derived from OHLC data. And output is a single binary variable (1 indicates probable target behavior tomorrow, 0 indicates "stay out"). I tried both approaches: the "naive" brute-force data mining, and neural network. My experience with neural network is that it either does not converge to the specified accuracy level or overfits to the data and fails miserably in out of sample test.

    Another thing to note here is that my inputs and output are binary variables. So it actually does not matter what method of mining you use: brute-force that covers the whole search space, or some "intelligent" heuristic method like neural network or genetic algorithm, that cover only some part of the whole search space.
     
    #540     Oct 26, 2008