Alright, I'll bite -- why would spectra generated from MESA -- a Butterworth filter for detrending (which is a fancy type of moving average) be superior to spectra generated from an FFT?
Butterworth has nothing to do with MEM. MEM means that the spectra is choosen which maximizes entropie function (under given constraints). If properly done it can be slighly superior to FFT since FFT has the problem to choose a window function (http://en.wikipedia.org/wiki/Window_function) .
Butterworth has to do with MESA, as I understand it. Regarding Butterworth, detrending and MESA it would seem to me to be problematic to detrend where the true trend is unknown. Therefore this appears superficially to be a problem with MESA. Regarding your MEM comments, it seems to me that "choosing" spectra in a non-stationary distribution is fraught with selection bias. Would it not be better to have an FFT with leakage, no window function and just put up with any leakage and noise? But I am far from an expert and I appreciate your feedback and comments.
Hi bdd, There is a limited resolution in the spectral bin width. You will only get harmonic bin resolution in 2^n tones. I.e. 512 days, 256days, etc... I just recently did a writeup on the FFT topic (with other problems noted)... http://intelligenttradingtech.blogspot.com/
Detrending (with Butterworth at MESA, but it could be any reasonable detrending method) is just pre-processing. As we want to examine cyclic behaviour we try to flatten out a linear component that could disturb the spectra. Of course this is highly subjective, as well as there is no strict stationarity as you mentioned. I don´t understand what you mean by leakage. There is always a window function involved (e.g. a rectangle) unless you have infinite amount of data (in which case you wouldn´t need FFT anymore).
Hi there, and thanks for your post. Regarding harmonic bin resolution, can you explain how this would be a factor? An FFT can extract 1/32 wavelengths from a 64 point dataset, for example. It seems to me that those objections (such as Nyquist) are primarily concerned with total inversion and reconstruction of the waveform, when with regard to trading, lower frequency filters are primarily the concern. Endpoint errors are less of a concern in a low frequency or medium frequency bandpass filter.
Yes -- and it's been an ongoing concern of mine to eliminate measures of standard deviation as they assume a stationary distribution. Same with detrending. It simply seems to be a false way of grafting a (momentarily) normal distribution on to a non-stationary one. Leakage refers to the spreading around individual frequencies produced from the frequency spectrum resulting from the FFT. This happens when the "frequency component of the original signal is not an integer multiple of the sampling period." Since we cannot know what the true frequency components are in market data, and we cannot know which optimum window to use, my thought is to leave the leakage as is and focus on a data subset which is an appropriate size in relation to the time periods being forecast. Too large introduces irrelevant data, too small results in incomplete data. The only window being a bandpass filter of appropriate length for the prediction period. Agree/disagree?
It's not so much the bin resolution, but translating the bin wavelengths back to daily cycles is not uniform due to sampling. It's not easy to explain in a few sentences, but you won't get 1 day, 2day, 3day... etc. granularity in the transformed spectrum. Those are problems with FFT construction/alteration/reconstruction, yes. But if you only want to use a LPF, then there's no need for a FFT. However, discrete LPFs have endpoint problems as well. Nothing is simple, I'm afraid.
I spent the last few years off and on looking for a practical application of spectral analysis to improve my trading and so far no real luck (despite degrees in math and numerical modelling, career involving time series analysis). Perhaps the closest I've come is using some combination of dominant frequency and weighted rate of change of frequency to extend a centered Hurst envelope on a daily chart, but aside from generating a couple of curves that are comforting to look at have found no real evidence it predicts what prices are likely to do more reliably than what one might predict by eyeballing the chart. I'm rapidly coming to the conclusion my time is better spent practicing trading using the few indicators I've found that work for me.