My back-testing is too good to be true. What am I missing?

Discussion in 'Commodity Futures' started by TradeSparrow, Dec 5, 2010.

  1. Most main stream indicators when subjected to actual systematic, objective testing fail to have any significant predictive value in short term trading. Even setting that aside, based on his usage of indicators, it is essentially like finding a correlation between two different data formats of the same data set. In other words, there is only one variable, which violates the requirement that both an independent and a dependent variable be present.

    However, it is possible to partition one data set into multiple ones for forward-looking prediction (IE: Pattern mining), but this still requires that testing be done for correlation strength (if anything, it would require this more). As I said, as soon as I get some more time, I'm going to make an extensive post on the matter. Keep an eye out for the post if you're interested. I always look forward to getting different opinions to learn more. I just jumped into this thread to save a guy that reminded me of me from losing money.
     
    #31     Dec 7, 2010
  2. LMAO
     
    #32     Dec 7, 2010
  3. :D
     
    #33     Dec 7, 2010
  4. Dogfish

    Dogfish

    http://ensign.editme.com/t57heikin Issues with Heikin Ashi

    "In fact, haUp candles will ALWAYS have a high wick, and haDown candles will ALWAYS have a low wick. This is a built in behavior that may surprise most Heikin-Ashi candle readers. It is one of the primary areas I feel is misleading."
     
    #34     Dec 8, 2010
  5. Define:

    Actual
    Systematic
    Objective
    Testing
    Significant
    Predictive
    Short term trading

    I can bet, those 7 things mean different things to different people. It is possible that if you give an idea to 10 traders to test it you will get 10 different results.

    Before you start a long post please keep in mind that extensive experience (mine and of others) has shown that it is not the concepts but their use that makes a difference.

    It is too, way too, dogmatic to say that certain methods do not work.

    I agree that naive testing does not provide an edge. But those 7 things you mentioned and I listed above offer a very broad spectrum of operation from very naive to extremely sophisticated.
     
    #35     Dec 8, 2010

  6. They can mean different things. However, present them to most statisticians and they will tell you the following:


    Actual - utilizing testable statistical methods rather than just visual confirmation.

    Systematic - the testing method can be repeated over and over again in the same manner using a clearly defined set of steps. It is preferable that the rigorousness of the system be such that others utilizing the same steps should get the same results for the test for the same data.

    Objective - (Actual and systematic combined). Furthermore, conclusions are based on test results rather than what outcome the tester was expecting.

    Testing - The statistical methods were actually used rather than just assuming the data passes the tests through haphazard testing (in the case of trading, usually just visual confirmation)

    Significant - A null and alternate hypothesis were clearly defined before the testing was done for what the result should be. The level of significance would the alpha value chosen (the level of significance) at which the test would be run. I prefer to run tests at an alpha level no less than 0.10. In other words, it is reasonable to expect the outcome of the test on the sample data to be the outcome at least 90% of the time for the overall population. However, many people might feel comfortable with lower alpha levels which is reasonable. Either way, most mainstream indicators would not pass profitability testing at alpha levels < 0.3 which is very unreliable.

    Predictive - The testing strongly suggests that the edge is something that should exist in the future. This is perhaps the hardest thing to test for. The most common way is to create a system that yields a certain type of results for one set of data, then yields similar results for a forward looking set of data. Predictive value must be subjected to its own tests and can have its own alpha value as well.

    Short term trading - This is the only term that I think can really be very, very vague and can reasonably have "wildly" different definitions. For the sake of this conversation, lets refer to it as any position opened and closed within the same market session (no swing trades). I've never trader forex so I cannot confidently provide a definition for the forex market. If I had to pick, I'd say any position opened and closed within a 12 hour window, but like I said, I've never been a forex guy.

    You are certainly right that usage can alter the results. However, I would be right to say that greater than 90% of the time main stream TA fails to give a profitable edge to traders irrespective of usage. The people that are profitable and still use main stream TA (such people are so rare that they can literally be considered statistical anomalies). It is far more likely that other methods are being used for an edge and the concept of an indicator is something completely different for these traders.

    The majority of short term traders (as defined above) that say they are profitable only think they are profitable, but over the long run will end up losing money. Hence, the wide belief that mainstream TA is profitable is a bunch of traders succeeding for a short while and propagating the notion that main stream TA is working, then failing with only relatively few people coming back to propagate the reality that their methods never really worked in the first place.

    There are certain points I do want to address that I think are needed to properly convey the message I want to get across, but I don't have the time to run the analyses to present that information. When school and work settle down, I will come back to the site for a more in-depth friendly debate.
     
    #36     Dec 8, 2010
  7. dave4532

    dave4532

    http://www.investopedia.com/articles/trading/09/short-term-trading.asp

    You confuse short-term trading and intraday trading. Can I safely assume then that you are confused with other issues as well? Probably it is so.

    Do you think that an MA crossover is some type of an anomaly? Why do you people repeat terminology you have read some place without understanding its meaning? Do you know there are funds that manage billions with no losing year with a system based on MA crossovers?

    Why do all losers want to convince everyone that TA doesn't work?

    If it doesn't work for you you are responsible for that. Please, do not try to convince other traders that your failures are normal. Try to imrpove yourself. I think you are spending too much of your time trying to convince others that your failures are justifiable. They are not. You didn't do your homework.
     
    #37     Dec 8, 2010
  8. First of all, please do not turn this into an arena for personal insults. I clearly stated that I am open to healthy debate. I do not mind having my ideas questions, but stick to attacking the ideas and not me. With that being said:

    The link that you posted defines short term trading as, word for word, "It can last for as little as a few minutes to as long as several days.". That is a definition which I acknowledged. I chose intraday positions since it seemed that I was being asked for concrete definitions. Just like all squares are a rhombus but not all rhombi are squares, the relationship between intraday trading and short-term trading are the same, respectively. I more than clearly stated that short term positions can be defined differently.

    As for those multi-billion dollar funds running there systems on simple TA, check their job postings. They are all hiring quants that do anything but simple TA. Additionally, they rarely, if ever hold short term positions - even by the broad range definition given by the link you provided. Heck, if you compare how most people regard moving averages to how moving averages are regarded in time series analysis, you will see a huge difference.
     
    #38     Dec 8, 2010
  9. Another question:


    I know what the bid/ask/last prices represent, but in Think or Swim, these prices are completely different from the prices show on my candles. What gives?
     
    #39     Dec 8, 2010
  10. Yet another question:

    Does anyone think an 8 cent stop would get hit often?

    I've been thinking of trading one contract per 10,000 instead of one per 5,000. I've also been thinking a 4% stop loss would be fine, as this would mean an 8 cent stop.

    Of course, my profits would be more than half of what they are now, which is still quite good. It makes sense to go more slowly, especially since I am inexperienced.

    The past three days, I've been forward testing using a 2 cent stop and futures do seem to wiggle around more than I thought. An 8 cent stop ought to cut out the noise much better. Any opinions?
     
    #40     Dec 8, 2010