A spread/pair-trade is still a directional bet on whatever is causing the anomalous spread. For example, new management/new technology. Is the share price being hyped or has the game changed. This is especially relevant to the timing of the trade. "Deep understanding" is what is needed, but how does one get it? I can easily automate filtering for consistency of a hundred different correlations. I can use cointegration methods to find relative pricing situations, but that only gets one in the neighborhood.
Today looks to be a little boring so I will take some time and try to relate my own experiences. My experience is in grains. I worked for a couple of grain companies before trading as a local. Working within the industry and on the floor gave me the opportunity to serve on committees that designed the futures contracts I traded. Serving on these committees was enlightning, they were typically dominated by a bloc of commercials and ultimately any changes were designed to enhance their positions. The experience drawn from those early years has served me well since trading as a local. I will give an example, one that I related on another thread. Several years ago, Jim Rogers starts trading a contract that I had traded for probably 20 years. He establishes a large long position, probably 60 % of the open interest. Now I know that he is going to roll his position prior to first delivery day so I take the opposite side by shorting the nearby spread and waiting for him to roll. At times I am short 20% of the nearby. Works out very well for about 5 years until he decides to liquidate after losing a substantial amount. I cite this experience as an example of how someone with a "deep understanding" has an obvious edge over someone trying to trade off of simply recognizing a trend. Profiting from recognizing trends is possible in this environment of increased volalility, but over the long run, it becomes more challenging, Ask Mr. Rogers. Regards, local
I appreciate your sharing. I have known someone with such deep knowledge of the international monetary system who is able to consistently front-run based on an exceptionally deep understanding of public information and practical implementation details. About all that I can really conclude is that there will always be a need for subject matter experts in the markets.
I hoped that providing a concrete example of some inversely related futures markets might spur this thread along. I forgot to mention in the original post that the tables is a correlation after converting to whole contract value in historical US dollars. I will change the code to exclude Forex series.
Yeah, the exclusion would be a good idea, as there's some very mixed metaphors (multicollinearity) going on there. For example, Danish Krone features a lot on the list, but we all know why that is and why it's a bit of a red herring.
I went through and removed most of the Forex markets from my analysis. (The trading profit/loss on SGX/JPY converted to historical USD isn't very intuitive.) Before giving an updated table, I want to talk about time period. I find correlation and cointegration to be highly interesting. They both provide some fairly easy tools to get a quick understanding of the relationships between the markets. On the other hand, I have read some academic work by someone I respect that said that correlation in the markets is a characteristic of the particular sample and not of the underlying markets. I have certainly seen many bogus pairs which had some coincident correlation, but I have also seen plenty where upon research, a real reason could be uncovered. My first line of defense is to consistency in different time periods. For example, consistency in nested time frames: Code: Correlation Time Span Linear Rank 0.125 Year 98.5_% 97.4_% 0.25 Year 98.3_% 96.6_% 0.5 Year 98.3_% 97.1_% 1 Year 98.6_% 97.7_% 2 Year 97.9_% 97.3_% 2.5 Year 95.0_% 97.3_% 5 Year 95.7_% 97.9_% 7.5 Year 96.3_% 98.5_% 10 Year 97.0_% 98.9_% 12.5 Year 97.6_% 99.1_% 15 Year 97.8_% 99.2_% and consistency using different metrics in disjoint time frames: Code: Year Price Correlation Log Correlation Return Correlation Linear Rank Linear Rank Linear Rank Pearson Spearman Pearson Spearman Pearson Spearman 2010 100.0_% 99.9_% 100.0_% 99.9_% 98.5_% 97.6_% 2009 99.9_% 99.9_% 99.9_% 99.9_% 97.9_% 97.2_% 2008 99.5_% 99.1_% 99.5_% 99.1_% 93.0_% 96.8_% 2007 100.0_% 99.9_% 100.0_% 99.9_% 99.6_% 99.5_% Bone's comment that 10 years is too long, made me wonder, what is a good initial time frame for searching. Is there a time frame for which a strong reading implies that the other time frames are also likely to have strong readings. To get an idea, I looked at 10 nested time frames (40 years down to 1/8 of a year), and measured the conditional expectation that if a market pair had a positive linear correlation of daily return at 90% or higher, then the same pair would also have a 90% of higher correlation in the second time period. Code: Year40 Year30 Year20 Year10 Year5 Year2 Year1 Year0.5 Year0.25 Year0.125 Year40 100% 100% 100% 60% 60% 60% 60% 100% 100% 60% Year30 100% 100% 100% 60% 60% 60% 60% 100% 100% 60% Year20 26% 26% 100% 74% 79% 68% 79% 89% 84% 37% Year10 7% 7% 30% 100% 78% 89% 57% 93% 57% 63% Year5 3% 3% 17% 42% 100% 90% 84% 88% 62% 40% Year2 1% 1% 3% 9% 16% 100% 78% 72% 47% 41% Year1 1% 1% 3% 5% 14% 71% 100% 71% 51% 33% Year0.5 1% 1% 3% 8% 14% 65% 71% 100% 58% 43% Year0.25 1% 1% 5% 8% 16% 67% 80% 91% 100% 55% Year0.125 1% 1% 2% 9% 11% 64% 57% 74% 60% 100% I know that this is not a well controlled test, but that is the nature of working with real data. The lower-left values should be small since there is an explosion of markets in recent years. It is unlikely, by chance, that a pair with 1/8 year of history consists of two markets both of which have existed for 40 years. The fact that the first two rows are duplicate and that the first two columns are duplicate reflects the small sample size of markets which have been existent and highly correlated for such long time frames. The 100%'s along the diagonal is just a truism. One would expect that the first cells above the diagonal to be large since knowing that for the last 20 years a pair was highly correlated does imply that for each of the first and second 10 years in the sample, the pair was fairly highly correlated. I have some ideas about how to compute a null-hypothesis, but let's proceed anyhow. With these basic biases pointed out, the columns for 1/8th of a year has quite small values 30-50%. Even the off-diagonal 55% is quite small. (Knowing that a pair was highly correlated for the last 1/4 of a year suggests only a 55% chance that it will be highly correlated the the last 1/8th of a year.) This suggests to me at the 1/8th of a year time frame is not very predictable from longer time frames, and I would even go further to say that it is probably not very characteristics of the market pair itself but rather just on the current circumstances. A similar pattern is demonstrated in the bottom of the 1/4 of a year column. Another feature of the numbers in this table is that for both 1/2 and 1/4 of a year, the conditional probabilities increase as one goes up from the diagonal. Thus knowing that a pair has been correlated for 20 years is a stronger implication that it will be correlated in the last 1/2 year then only saying that it has been correlated for 10 years. This tells me that there is something legitimate in correlation. That having a longer period correlation is a function of the market and the underlying economics. Back to the original question of whether there were any time frames for which high correlation would suggest high correlation in other time frames, the rows from 5 years and up satisfies this property. As the underlying economics change over long spans and as there aren't many markets with 30-40 years of history, I can't recommend using these. The 5-20 time frames have similar conditional probabilities within the assumed sampling error. Myself, I usually use the 10 year time frame due to the inconvenience in sifting through recently introduced markets.
Here is an updates list of futures markets with 10 year rank correlation on daily return (log difference) at 90% or higher (market price converted to historical USD value first.) Code: Name Name Linear Rank Pair CorrelationCorrelationStability Euroyen-3Mth-SGX (SIMEX) Euroyen LIBOR 3 Mth-(SIMEX)-SGX 100.0_% 100.0_% 3.8 Euro Swiss Franc-EURONEXT(LIFFE) EURIBOR-3 Mth-EURONEXT(LIFFE) 99.9_% 99.8_% 1.3 Australian Govt Bond 6%(3Yr)-(Flo Australian Govt Bond 6%(10Yr)-(Flo 99.9_% 99.8_% 1.3 Euro Swiss Franc-EURONEXT(LIFFE) Euroyen-3Mth-(LIFFE)-EURONEXT 99.8_% 99.8_% 1.4 New Zealand Bank Bills(90 Day)-(N Australian Bank Bills(90 Day)-(Flo 99.8_% 99.8_% 1.4 Australian Bank Bills(90 Day)-(Fl Australian Govt Bond 6%(3Yr)-(Floo 99.8_% 99.8_% 1.4 Euroyen-3Mth-(LIFFE)-EURONEXT EURIBOR-3 Mth-EURONEXT(LIFFE) 99.8_% 99.7_% 1.4 Sterling Rate-3Mth-EURONEXT(LIFFE EURIBOR-3 Mth-EURONEXT(LIFFE) 99.8_% 99.7_% 1.4 Australian Bank Bills(90 Day)-(Fl Australian Govt Bond 6%(10Yr)-(Flo 99.8_% 99.7_% 1.3 Sterling Rate-3Mth-EURONEXT(LIFFE Euro Swiss Franc-EURONEXT(LIFFE) 99.7_% 99.7_% 1.4 New Zealand Bank Bills(90 Day)-(N Australian Govt Bond 6%(10Yr)-(Flo 99.7_% 99.6_% 1.3 Sterling Rate-3Mth-EURONEXT(LIFFE Euroyen-3Mth-(LIFFE)-EURONEXT 99.7_% 99.6_% 1.4 New Zealand Bank Bills(90 Day)-(N Australian Govt Bond 6%(3Yr)-(Floo 99.7_% 99.6_% 1.3 New Zealand Bank Bills(90 Day)-(N New Zealand Govt Stock(3 Yr)-(NZFE 99.7_% 99.6_% 1.3 New Zealand Govt Stock(3 Yr)-(NZF Australian Bank Bills(90 Day)-(Flo 99.7_% 99.5_% 1.3 New Zealand Govt Stock(3 Yr)-(NZF Australian Govt Bond 6%(10Yr)-(Flo 99.7_% 99.4_% 1.3 New Zealand Govt Stock(3 Yr)-(NZF Australian Govt Bond 6%(3Yr)-(Floo 99.6_% 99.4_% 1.2 EURIBOR-3 Mth-EUREX Euro German Schatz-EUREX 99.4_% 99.4_% 1.3 Euroyen-3Mth-SGX (SIMEX) SIBOR-3 Mth-(SIMEX)-SGX 99.4_% 99.1_% 1.5 Euroyen LIBOR 3 Mth-(SIMEX)-SGX SIBOR-3 Mth-(SIMEX)-SGX 99.4_% 99.1_% 1.5 Russell 2000 Index at CME Russell 2000 (Floor+Electronic Com 97.0_% 98.9_% 4.8 Eurodollar(3Mth)-SGX(SIMEX) Euroyen-3Mth-SGX (SIMEX) 98.7_% 98.1_% 1.4 Eurodollar(3Mth)-SGX(SIMEX) Euroyen LIBOR 3 Mth-(SIMEX)-SGX 98.7_% 98.1_% 1.4 Euro German Bobl-EUREX Euro German Schatz-EUREX 98.1_% 97.8_% 1.2 Eurodollar(3Mth)-SGX(SIMEX) SIBOR-3 Mth-(SIMEX)-SGX 98.3_% 97.8_% 1.4 Dow Jones Industrial Index at CME Dow Jones Industrial Avg Index 92.8_% 97.2_% 1.5 10 Yr Treasury Note at CME-CBOT 5 Yr Treasury Note at CME-CBOT 96.3_% 96.4_% 1.3 Major Market Index-(Floor Trading Dow Jones Industrial Avg Index 92.5_% 95.8_% 1.2 EURIBOR-3 Mth-EUREX Euro German Bobl-EUREX 96.0_% 95.5_% 1.3 Swiss Govt Bond(10Yr)-(SOFFEX)-EU Euro German Bobl-EUREX 95.4_% 94.7_% 1.3 3 Month Eurodollar at CME LIBOR(1Mth)-(Floor+Electronic Comb 95.6_% 94.6_% 1.5 US Treasury Bond at CME-CBOT 10 Yr Treasury Note at CME-CBOT 94.5_% 94.5_% 1.3 S&P MidCap 400 Index at CME Russell 2000 (Floor+Electronic Com 93.4_% 94.3_% 1.2 Russell 2000 Index at CME S&P MidCap 400 Index at CME 94.6_% 93.8_% 1.2 Wheat-Kansas City-KCBT Wheat at CME at CBOT 94.7_% 93.5_% 1.3 Major Market Index-(Floor Trading Dow Jones Industrial Index at CME- 96.1_% 93.3_% 1.3 Swiss Govt Bond(10Yr)-(SOFFEX)-EU Euro German Schatz-EUREX 94.0_% 93.0_% 1.3 Japanese Govt Bond-EURONEXT(LIFFE Euro Swiss Franc-EURONEXT(LIFFE) 93.8_% 91.9_% 1.2 Japanese Govt Bond-EURONEXT(LIFFE Euroyen-3Mth-(LIFFE)-EURONEXT 93.6_% 91.8_% 1.3 Japanese Govt Bond-EURONEXT(LIFFE EURIBOR-3 Mth-EURONEXT(LIFFE) 93.7_% 91.8_% 1.3 Sterling Rate-3Mth-EURONEXT(LIFFE Japanese Govt Bond-EURONEXT(LIFFE) 93.7_% 91.7_% 1.3 Wheat-Kansas City-KCBT Spring Wheat at MGE 92.4_% 91.2_% 1.2 Swiss Govt Bond(10Yr)-(SOFFEX)-EU EURIBOR-3 Mth-EUREX 92.4_% 91.2_% 1.2 NYSE Composite Index at US ICE Dow Jones Industrial Avg Index 89.1_% 90.6_% 1.2 5 Yr Treasury Note at CME-CBOT 2 Yr Treasury Note at CME-CBOT 86.5_% 90.5_% 1.4 Dow Jones Industrial Index at CME NYSE Composite Index at US ICE 93.5_% 90.5_% 1.2 Had to manually remove a few dozen redundant markets. Hope this helps.