Thanks for your answers. Could you please clarify what you mean by "event profiler"? I am not familiar with this term. IMO the decay of the signal is simply market efficiency at work! Every trading system makes money from some kind of regularity in the data (a deviation from randomness). We can sometimes see short temporal deviations from the EMH, but this does not last forever. As soon as a market inefficiency becomes too "visible", some sophisticated market participants identify it and trade it away. All my trading strategies suffer from performance decay and stop working at a point in time. I think the competition for "trading patterns discovery" is strong. In order to make money in this business, we need to be able to identify quickly market anomalies when they occur and trade them profitably before they get noticed by the competition and disappear.
I do not thing there is something wrong in training many models or testing many hypotheses on the same data set. But by trying many models or testing many hypotheses, there will always be something which works nicely on this particular data set, just by pure coincidence/chance. What is wrong is to assume that the trading system found in the training data set has some explanatory power outside of this data set. Performance measures and statistical significance in the training data set are strongly positively biased and must not be used. A simple solution to this problem is to use cross-validation : use a data set which was not used at all during the training/development to quantify the expected performance / statistical significance of the trading system. Good explanation on wikipedia : http://en.wikipedia.org/wiki/Data_dredging http://en.wikipedia.org/wiki/Testing_hypotheses_suggested_by_the_data
the event profiler is a tool that performs an event study over a particular pattern in the market data. http://en.wikipedia.org/wiki/Event_study it was designed by Tuckler Balch from GA Tech http://www.youtube.com/watch?v=pNLIB88XQoY yeah, I expect the signals to decay, or at least to fluctuate over time. As the market's sentiment switches from one state to the other, or the attention of participants becomes more fragmented or less fragmented.
Thanks for the links, I did not know about this particular tool. I have developed something similar to analyze the entries of my trading systems but I am not sure how to actually make use of it at this point. As the x-axis is time, I believe the idea is to use some kind of time-based exits (=exit N bars after the entry)? I am not sure time is necessarily the best x-axis for some strategies. In fact, some strategies can be inconsistent in time (they do not respond consistently the same way as function of time) but can be consistent in amplitude (i.e. we know the price trajectory is going to move up by a certain amount but we do not know how much time it will take).
Coming back to the strategy development framework... Some open questions and comments... * Can we design the entry conditions of a trading system without defining the exit conditions? Or the entry alone has no statistical edge by itself and requires the exit condition to be defined? Splitting the problem in 2 would make things easier but I am not sure it is feasible. * Assuming patterns have a short lifetime. The discovery process must therefore constantly watch for the occurrence of new patterns or the resurgence of old ones. This requires a kind of moving window. Does it make sense at all to look for patterns over very long time periods? And does it make sense at all to look at the shape of the equity curve? * Again assuming patterns have a short lifetime, the trading systems themselves have little value because they will die. All that matters is the system development process. In order to beat the competition, we need a very short development cycle and enough confidence in the process to bring quickly a trading system to live trading once discovered and to trade it profitably before it dies...
I've been playing with the idea of moving window systems where the system is learning from the data for a period of time, but then the data "expires" and goes out of scope... The system development process... and the tools to efficiently develop new systems, and try out/discard ideas that come up... that's what this thread is all about
very nice read. thank you. i specially liked the last line. """ the methods for analyzing systems for the presence of bias are in many cases more important that the methods used to generate the systems in the first place and are considered an integral part of a trading edge. """