Just to clarify my first post: I never meant to imply that lag is a good thing or even an inconsequential thing. Too much lag is a very bad thing. But there is an attitude prevalent among the indicator haters that even the slightest lag is detrimental. This is nonsense. If you are a trend trader and your indicator (say, a moving average) is strongly lagging during heavy chop, this would be a good thing. The lag will keep you from being whipsawed by the chop. On the other hand, during a strong trend, the lag will get you late signals. Not good. There is a middle ground where some lag is at least tolerable because it does mitigate some chop while still allowing you to catch most of a trend. My point is that bar-by-bar PA is difficult and some reference to the recent past is necessary to validate most trading signals. The main problem with most public indicators isn't lag, it's poor design. They just don't do very well (or at all) what their creators claim they do. The best indicators with very few exceptions are proprietary. You don't need 20 different hammers to drive a nail. But there are at least 20 different technical indicators all claiming to do the same thing: find a trend. Clearly most of them are doing a pisspoor job, which is why new ones are always being invented. For the public. Most of the proprietary indicators are already getting the job done.
How are you going to look at order flow in the first place? Have you think about it? Order flow is known to the broker or someone who acts like one before it is posted or executed internally or on the exchange. They are not going to let you look at it. Trust me. You can assume that anything posted is either a bait, a head fake or order that they have no expectation of making money on. Another words your idea of looking at order flow is similar to asking poker players to reveal their hands before you make your move. How realistic is that?
Moving average is low pass filter and as a result of applying this filter to signal there is a phase shift. Phase response of the filter characterises such filter. What you call lag in TA is called phase shift in DSP. How you want to chart it or interpret it does not change fundamental characteristics of low pass filtering.
I'm feeling adventurous so I may as well jump into the fray since I haven't been banned or attacked yet. I think the OP's question can be refined and rephrased to ask "Can historical data have predictive value?". I think that's unlikely for the most popular technical patterns and indicators, but there is potential for predictive value using 1) custom patterns/indicators you develop yourself, or 2) using the popular ones in a different and unique way. Also consider sentiment indicators, like the put/call ratio, which are claimed by some to have predictive value when they reach extreme values. As well as high levels of short interest where a setup for a short squeeze may be in the making. These are examples where past data may have some forward-looking value. Addressing the OP's original question about why traders use lagging indicators, the most obvious answer is because that's what is taught by vendors and is found in virtually every software charting program. This is a business that has a lot of churn and thrives on a continuous supply of new students who believe what they see and are told. I remember going through that phase myself.
Would you design a filter for high frequency noise and apply it when there is very little noise. Maybe switch to bar by bar analysis when noise diminishes. Implies adaptable filters and proper application. In electrical terms , use hysterisis (schmitt trigger) in noise and dont otherwise. Easier said than done. I should add that Kut2k2 has mentioned adaptable filters many times here. One or two more compliments and he may give me some proprietary gems.
i personally don't trade with indicators because i consider them to be useless on a 5 minute chart.most of the indicators can be interpreted in just looking at the charts so their is no use for me.a trend is a trend and a trading range is a trading range.you can see it for yourself. lagging indicators are more useful on daily charts as to see price confirmation on their trade and possibly divergence between indicators and price which gives them a warning for a reversal. as for my believe why people still use indicators is that they want something else to give them a confirmation.people do not like to take responsibility for their trading they want an indicator,advice from other people,or some computer program to take responsible for their trades (suppose you can say the fear of failure).it also takes away a lot of emotion in trading,it's much easier to trade when two lines of an indicator crossover and it gives them a buy signal rather than the way i do it by slapping my face,breathing like a Buddhist monk and acting like anthony robbins at the same time just to put one trade on.
The biggest introduction of technical indicators are the following order with #1 as the biggest. 1. Data vendors (e.g. TradeStation, eSignal, Multicharts, NinjaTrader) 2. Brokers 3. Trade Forum Discussions (e.g. EliteTrader, BigMikes, ForexFactory) 4. Indicator Vendors...most of which I see involving FX markets 5. Exchange (e.g. CME and other exchanges with those educators they have contracts with for education). The above 1 and 2 will sometimes share your email list with vendors on their approval list that have paid for it...reason why you should never use your real ISP email address especially if you hate junk mail. Best to use an alternate email address like hotmail, yahoo or gmail. Technical indicators is a big promotion tool for many. Its also a big promotion tool by forum owners to bring in new members. Many forums now have "indicator" sections where people can upload their indicator codes for anyone to download. Almost all forums now have "technical analysis" discussion threads. Heck, even some algorithms are exploiting indicators by the quants. I still remember my first online broker. Within about 2 weeks of opening that account...I got junk emails from Omega Research (now called TradeStation), Qcharts (now owned by eSignal) and Realtick. All the emails key promotion was the advertising of their "award winning technical analysis indicators". Heck, I even remember how I heard about Elitetrader.com It was on a Yahoo message forum like year 2000 and ET was heavily being spammed by "someone" from this forum that could only be from management. I even met Joe (someone from ET management) on IRC about the same time span. Guess what was talked about as the highlight of this forum...the technical analysis talks. Here's the ironic thing I don't get. The bashers or haters themselves use TA and some use technical indicators...three of the most well known haters at this forum even have/had blogs that promoted it...its like some twisted psychotic game being played out at this forum.
So you are saying that TA for markets is what card counting is for casino: to attract people who would hope that applying numbers and math will help them to beat the house.
i agree with a lot of things that you posted. my journey = 4 very hard years of understanding the australian futures market. and then i had to battle with my emotions for another 6 months.ive been trying to get fluency in my trading and get rid of hesitation in other words trying to be in the zone for trading. now putting this altogether i am currently starting to get much much more profitable trades than losses. because i am a discretionary trader and my mind is very open about the market, such as a simple example, i could have price moving to a support zone but i personally believe it will not reach it ,because their is buying pressure so i jump in., but the computer program will tell me to wait until it reaches the support zone,it doesn't so i miss on the trade. in other words how do you program a discretionary traders mind,where he chops and changes his mind every second or minute to adapt to the market conditions.
I can't debate about any correlations like such with card counting. I can only give facts about whom are the biggest promoters of TA and its big business for those data vendors and brokers.