Why do I see "Trends" in Randomly Generated Data?

Discussion in 'Data Sets and Feeds' started by Rahula, Feb 21, 2008.

  1. Jerry030

    Jerry030

    Based on the above I'd ask the following questions:

    1) What NN package were you using? The stuff marketed to the trading community is junk for the most part (NeuroShell, Bioscopy, NuroDimensions, etc). What you want to use is a general purpose corporate modeling applications. These are used successfully every day on very complex tasks: credit car authorizations, oil explorations, climate modeling, national security surveillance.

    2) What is you formal education in Computer Science? A BS is good, an MS is better.

    3) What is your experience using models to predict other things?
    There are lots of formal datasets out there that people use to calibrate and compare modeling strategies or approaches. If you can't get them to generate the published results then likely your usage/method/approach is defective as opposed to the software.

    Jerry030
     
    #541     Oct 26, 2008
  2. 1) BrainMaker Pro

    2) BS in software engineering. I work full-time as a software developer. I disagree with your statement that MS is better than BS. In real-life application experience is much more important. A BS with 5 years of real work experience is better than MS with zero experience in most cases. (but that's very offtopic)

    3) I haven't tried to create models for anything else.

    I will reiterate: I view NNs and similar approaches as subset of brute-force data mining approach that goes over all possible combinations of inputs. Why? Simply because if you take a trained neural network and run all possible inputs (I'm referring to binary data) and register output values, you will get the same patterns that you would by using straightforward mining approach. The difference between "dumb" and "AI" methods is that "AI" uses some heuristic rule that reduces search space. In other words, it's based on hope that the heuristic helps to converge to the global optima with a trade-off of getting stuck on the local optima. But the main point is that both approaches are susceptible to overfiting. You may as well run NN on random data and find spurious patterns that mean nothing.
     
    #542     Oct 26, 2008
  3. monti1a

    monti1a

    #543     Oct 26, 2008
  4. Jerry030

    Jerry030

    To your points:

    1) While I'm not familiar with it BrainMaker appears to be a low end product at a few hundred dollars. Expect to spend $10,000 to $50,000 for a corporate grade industrial strength application.

    2) Agreed, 5 years of experience gives one the ability to know what to do. Many people with no background in CS buy a package for a few hundred dollars, throw a few years of prices at it and then conclude NN don't work when it fail to make money.

    3) Test if your methodology is correct by getting something like the Austrian Credit Problem from CME, UCLA or other sites and see if you get the published results. IF you can't then either the software is junk or you did it wrong. IF either of these is true you' never get a trading system that works.

    Any data mining approach could over fit if you fail to do correct pre processing and partitioning of training and validation sets. All of these things are solvable if you know data mining. Higher end packages prune Independent Variables with high cross correlation. This eliminates much of the issue with models that train well and fall apart in real life.

    What was the nature of you Dependant Variable (what you were trying to predict?

    My feeling is that brute force can’t deliver the same results as well engineered net as brute force is either black or white. The hidden layer gives shades or grey and inflection that discreet logic can’t. And in my trading they work very well…so go figure.


    Jerry030
     
    #544     Oct 26, 2008
  5. jtnet

    jtnet

    where is the trend asshole?

    [​IMG]
     
    #545     Oct 26, 2008
  6. Jerry030

    Jerry030

    Doubt you could tell the difference between one and your penis. Go back to junior high.
     
    #546     Oct 26, 2008


  7. its right there ^ in the center......
     
    #547     Nov 1, 2008
  8. whats really random in life... nothing really is everything is a pattern and thats why you see it in everything... might as well be asking the question does the universe define mathematics or does mathematics define the universe... in there a natural structure in the universe because it needs to follow rules to exhist? and those rules are defined by mathematics?

    why are humans top of the food chain really? cause we can see and read the rules of the universe/ laws of nature and create use those rules to make things or understand things with formulas? if everything is defined by mathematics like a intrauniversal web of laws then nothing can be random.... because its defined by rules and if something is within a set of rules how can it be random?

    to random point creation in excel..... honestly?? computer run off code the code is rules how can the numbers be truly random.... if the creation of those "random numbers" come from code...?

    price is not random...because nothing is random

    so my guess is you see trends in random data because its not random data and through the law of large numbers compress the data and zoom out (such as price points of 5min then move out to daily your gonna see a trend) your gonna get closer to finding the formula that "derives" the "random" points..


    i feel bad for code breakers who listen to static and pick out repeating patterns.
     
    #548     Nov 1, 2008
  9. It's a good thing price doesn't trade or plot like this or we all would be in a world of poo.

    Price can be read by a computer for direction and strength and there has to be a logical environment there for this to take place.

    Price is randon and chaotic but structured into a relatively smooth flow of supply/demand exchanges.

    Trends do not exist in markets but they DO exist and can be objectively defined on actual and real non-variable price charts.
     
    #549     Nov 1, 2008
  10. danielc1

    danielc1

    This guy or girl will be the richest person on ET, if he or she isn't allready if it keeps going in that direction...

    And I'm not kidding...
     
    #550     Nov 1, 2008