Another limitation is testing for candle stick formations. TS for example require one to specify the parameters for a doji, also hammers. Visual pattern trading is also very difficult to backtest with confidence.
...that is ontologically and epistemologically very distressing to me. If it is true that a profitable system may not backtest well, then one must question whether or not a system which backtests well is profitable. Horrors! But I think you are on the right track to BT only time, price, and volume (although I ain't so sure about the volume, never found anything there). IMO it is very fruitful (i.e., full of fruit) to kick back with a bottle (or two) of cheap champagne and BT outrageous ideas that you get looking at pure time-price patterns (one must ignore the psycho-sexual ideas one gets from these ink blots as mere distractions). FWIW, I have found that that the patterns themselves found thereby are reliable, but the trade parameters are quite evanescent, requiring weekly or semi-monthly reoptimization to keep sharp. Trading randomness is such an art! It is this shifting tectonics of the market that leads me to the conclusion that all intuitive traders are either geniuses or full of shit.
...re your other limitation, the obverse is also true. Backtesting may find patterns that you cannot see. I have one system that constantly surprises me when it triggers, even though I am looking for it to occur. However, I hold the supercilious opinion that there is no visual pattern for which an algorithm cannot be written. Alas, I have found such cases where the code was so complex that E-Signal bogged down on it, making them impractical to trade mechanically.
Mark Douglas (Trading in The Zone) specifies there are three stages - Mechanical, Subjective, and Intuitive, "If you asked me to distill trading down to its simplest form, I would say that it is a pattern recognition numbers game. ... In the process, I'll take you through the three stages of development of a trader." http://www.elitetrader.com/vb/showthread.php?s=&threadid=21167&highlight=douglas
I often remember the following teaching (from one of my very respectful professor): To foresee is to manage!
...yes I remember that quite well from the book. Hopeless for me, though. My intuition is usually wrong, but not even consistently enough that I could trade against it.
You are right, in backtesting you can't be sure how your order would have gone, especially on the shorter timeframes. It's even true for realtime testing with a broker simulator. That is what drives me away from the shortest timeframe systems. I assume that I can trade 10% of the volume at the price where testing gives me the entry signal, if all those other traders got those prices I should have been able to as well. On the other hand, if my order actually causes the price to move in the direction of my trade and shortens my time in the trade then I will not have the money tied up in the trade as long and will be able to use it elsewhere!!
...Absolutely! The only thing that keeps me from automating is that I learn a lot from the thoughts that go through my head and from the emotions I experience when the order has to be placed, especially when it is clear for many seconds that the trade will trigger. Holding until the exit is called is even more enlightening. It is quite clear that there are several people in my head, each with his or her own reasons for not taking the calls, or for making exits early. Also I find that being in a trade, especially an uncomfortable one, is highly conducive to attending to evolving price patterns for the purpose of wild-assed scheming about system ideas.
Belief! In trading, so many things would be just our belief (Although the outcomes could be quite different)!