Pairs Trading = Numbers Game One of the first things you quickly come to understand in equity pairs trading is how important it is to spread your risk. The reason is obvious: stocks are subject to a multitude of risk factors – amongst them earning shocks and corporate actions -that can blow up an otherwise profitable pairs trade. Instead of the pair re-converging, they continue to diverge until you are stopped out of the position. There is not much you can do about this, because equities are inherently risky. Some arbitrageurs prefer trading ETF pairs for precisely this reason. But risk and reward are two sides of the same coin: risks tend to be lower in ETF pairs trades, but so, too, are the rewards. Another factor to consider is that there are many more opportunities to be found amongst the vast number of stock combinations than in the much smaller universe of ETFs. So equities remain the preferred asset class of choice for the great majority of arbitrageurs. So, because of the risk in trading equities, it is vitally important to spread the risk amongst a large number of pairs. That way, when one of your pairs trades inevitably blows up for one reason or another, the capital allocation is low enough not to cause irreparable damage to the overall portfolio. Nor are you over-reliant on one or two star performers that may cease to contribute if, for example, one of the stock pairs is subject to a merger or takeover. Does that mean that pairs trading is accessible only to managers with deep enough pockets to allocate broadly in the investment universe? Yes and no. On the one hand, of course, you need sufficient capital to allocate a meaningful sum to each of your pairs. But pairs trading is highly efficient in its use of capital: margin requirements are greatly reduced by the much lower risk of a dollar-neutral portfolio. So your capital goes further than in would in a long-only strategy, for example. How many pair combinations would you need to research to build an investment portfolio of the required size? The answer might shock you: millions. Or even tens of millions. In the case of the Gemini Pairs strategy, for example, the universe comprises around 10m stock pairs and 200,000 ETF combinations. It turns out to be much more challenging to find reliable stock pairs to trade than one might imagine, for reasons I am about to discuss. So what tends to discourage investors from exploring pairs trading as an investment strategy is not because the strategy is inherently hard to understand; nor because the methods are unknown; nor because it requires vast amounts of investment capital to be viable. It is that the research effort required to build a successful statistical arbitrage strategy is beyond the capability of the great majority of investors. Before you become two discouraged, I will just say that there are at least two solutions to this challenge I can offer, which I will discuss later. Methodology Isn’t a Decider I have traded pairs successfully using all of the techniques described in the first part of the post (i.e. Ratio, Regression, Kalman and Copula methods). Equally, I have seen a great many failed pairs strategies produced by using every available technique. There is no silver bullet. One often finds that a pair that perform poorly using the ratio method produces decent returns when a regression or Kalman Filter model is applied. From experience, there is no pattern that allows you to discern which technique, if any, is gong to work. You have to be prepared to try all of them, at least in back-test. Correlation is Not the Answer In a typical description of pairs trading the first order of business is often to look for a highly correlated pairs to trade. While this makes sense as a starting point, it can never provide a complete answer. The reason is well known: correlations are unstable, and can often arise from random chance rather than as a result of a real connection between two stock processes. The concept of spurious correlation is most easily grasped with an example, for instance: Of course, no rational person believes that there is a causal connection between cheese consumption and death by bedsheet entanglement – it is a spurious correlation that has arisen due to the random fluctuations in the two time series. And because the correlation is spurious, the apparent relationship is likely to break down in future. We can provide a slightly more realistic illustration as follows. Let us suppose we have two correlated stocks, one with annual drift (i.e trend of 5% and annual volatility of 25%, the other with annual drift of 20% and annual volatility of 50%. We assume that returns from the two processes follow a Normal distribution, with true correlation of 0.3. Let’s assume that we sample the returns for the two stocks over 90 days to estimate the correlation, simulating the real-world situation in which the true correlation is unknown. Unlike in the real-world scenario, we can sample the 90-day returns many times (100,000 in this experiment) and look at the range of correlation estimates we observe: We find that, over the 100,000 repeated experiments the average correlation estimate is very close indeed to the true correlation. However, in the real-world situation we only have a single observation, based on the returns from the two stock processes over the prior 90 days. If we are very lucky, we might happen to pick a period in which the processes correlate at a level close to the true value of 0.3. But as the experiment shows, we might be unlucky enough to see an estimate as high as 0.64, or as low as zero! So when we look at historical data and use estimates of the correlation coefficient to gauge the strength of the relationship between two stocks, we are at the mercy of random variation in the sampling process, one that could suggest a much stronger (or weaker) connection than is actually the case. One is on firmer ground in selecting pairs of stocks in the same sector, for example oil or gold-mining stocks, because we are able to identify causal factors that should provide a basis for a reliable correlation, such as the price of oil or gold. This is indeed one of the “screens” that statistical arbitrageurs often use to select pairs for analysis. But there are many examples of stocks that “ought” to be correlated but which nonetheless break down and drift apart. This can happen for many reasons: changes in the capital structure of one of the companies; a major product launch; regulatory action; or corporate actions such as mergers and takeovers. The bottom line is that correlation, while important, is not by itself a sufficiently reliable measure to provide a basis for pair selection. Cointegration: the Drunk and His Dog Suppose you see two drunks (i.e., two random walks) wandering around. The drunks don’t know each other (they’re independent), so there’s no meaningful relationship between their paths. But suppose instead you have a drunk walking with his dog. This time there is a connection. What’s the nature of this connection? Notice that although each path individually is still an unpredictable random walk, given the location of one of the drunk or dog, we have a pretty good idea of where the other is; that is, the distance between the two is fairly predictable. (For example, if the dog wanders too far away from his owner, he’ll tend to move in his direction to avoid losing him, so the two stay close together despite a tendency to wander around on their own.) We describe this relationship by saying that the drunk and her dog form a cointegrating pair. In more technical terms, if we have two non-stationary time series X and Y that become stationary when differenced (these are called integrated of order one series, or I(1) series; random walks are one example) such that some linear combination of X and Y is stationary (aka, I(0)), then we say that X and Y are cointegrated. In other words, while neither X nor Y alone hovers around a constant value, some combination of them does, so we can think of cointegration as describing a particular kind of long-run equilibrium relationship. (The definition of cointegration can be extended to multiple time series, with higher orders of integration.) Other examples of cointegrated pairs: Income and consumption: as income increases/decreases, so too does consumption. Size of police force and amount of criminal activity A book and its movie adaptation: while the book and the movie may differ in small details, the overall plot will remain the same. Number of patients entering or leaving a hospital So why do we care about cointegration? Someone else can probably give more econometric applications, but in quantitative finance, cointegration forms the basis of the pairs trading strategy: suppose we have two cointegrated stocks X and Y, with the particular (for concreteness) cointegrating relationship X – 2Y = Z, where Z is a stationary series of zero mean. For example, X could be McDonald’s, Y could be Burger King, and the cointegration relationship would mean that X tends to be priced twice as high as Y, so that when X is more than twice the price of Y, we expect X to move down or Y to move up in the near future (and analogously, if X is less than twice the price of Y, we expect X to move up or Y to move down). This suggests the following trading strategy: if X – 2Y > d, for some positive threshold d, then we should sell X and buy Y (since we expect X to decrease in price and Y to increase), and similarly, if X – 2Y < -d, then we should buy X and sell Y. So, just to summarize a bit, cointegration is an equilibrium relationship between time series that individually aren’t in equilibrium (you can kind of contrast this with (Pearson) correlation, which describes a linear relationship), and it’s useful because it allows us to incorporate both short-term dynamics (deviations from equilibrium) and long-run expectations , i.e. corrections to equilibrium. (My thanks to Edwin Chen for this entertaining explanation) Cointegration is Not the Answer So a typical workflow for researching possible pairs trade might be to examine a large number of pairs in a sector of interest, select those that meet some correlation threshold (e.e. 90%), test those pairs for cointegration and select those that appear to be cointegrated. The problem is: it doesn’t work! The pairs thrown up by this process are likely to work for a while, but many (even the majority) will break down at some point, typically soon after you begin live trading. `The reason is that all of the major statistical tests for cointegration have relatively low power and pairs that are apparently cointegrated break down suddenly, with consequential losses for the trader. The following posts delves into the subject in some detail: Other Practical “Gotchas” Apart from correlations/cointegration breakdowns there is a long list of things that can go wrong with a pairs trade that the practitioner needs to take account of, for instance: A stock may become difficult or expensive to short The overall backtest performance stats for a pair may look great, but the P&L per share is too small to overcome trading costs and other frictions. Corporate actions (mergers, takeovers) and earnings can blow up one side of an otherwise profitable pair. It is possible to trade passively, crossing the spread to trade the other leg when the first leg trades. But this trade expression is challenging to test. If paying the spread on both legs is going to jeopardize the profitability of the strategy, it is probably better to reject the pair. What Works From my experience, the testing phase of the process of building a statistical arbitrage strategy is absolutely critical. By this I mean that, after screening for correlation and cointegration, and back-testing all of the possible types of model, it is essential to conduct an extensive simulation test over a period of several weeks before adding a new pair to the production system. Testing is important for any algorithmic strategy, of course, but it is an integral part of the selection process where pairs trading is concerned. You should expect 60% to 80% of your candidates to fail in simulated trading, even after they have been carefully selected and thoroughly back-tested. The good good news is that those pairs that pass the final stage of testing usually are successful in a production setting. Implementation Putting all of this information together, it should be apparent that the major challenge in pairs trading lies not so much in understanding and implementing methodologies and techniques, but in implementing the research process on an industrial scale, sufficient to collate and analyze tens of millions of pairs. This is beyond the reach of most retail investors, and indeed, many small trading firms: I once worked with a trading firm for over a year on a similar research project, but in the end it proved to be capabilities of even their highly competent development team. So does this mean that for the average quantitative strategist investors statistical arbitrage must remain an investment concept of purely theoretical interest? Actually, no. Firstly, for the investor, there are plenty of investment products available that they can access via hedge fund structures (or even our algotrading platform, as I have previously mentioned). For those interested in building stat arb strategies there is an excellent resource that collates all of the data and analysis on tens of millions of stock pairs that enables the researcher to identify promising pairs, test their level of cointegration, backtest strategies using different methodologies and even put selected pars strategies into production (see example below). Those interested should contact me for more information.
Please consider posting your application of the particle swarm method to determine trade entries. Thanks in advance.
I used to trade pairs. One thing I learned is you better know your stocks and the industries their in inside and out. I saw a lot of guys blow up adding to a pair trade when the fundamentals had changed or they just didn't understand the stocks or the market we were in. A good example was a baby bell trade (can't remember the stocks). Everyone was long the large cap and short the small cap. Well guess what happens in a bull market in a deregulating market. The large cap bought the small gap. Everything gapped. One guy came into the office down $20k on that trade alone, enough to wash him out. I'm leery of any strategy that is constantly in the market.
You make a good point. And, indeed, one approach is to implement the strategy in a limited number of pairs in stocks you know inside out, as you suggest. My preference, on the other hand, is to implement the strategy in a larger number of pairs so that the capital allocated to any particular pair is sufficiently small not to blow up the entire strategy if the trade goes wrong.
I don't think the problem in this case is that the strategy is constantly in the market. I believe the problem is the underlying premise of successful pair trading is flawed. In modern times where indexing of stocks is a major part of the trading volume, pair trading no longer works. Certainly, you can find periods of time where pairs are cointegrated, but by the time you establish this relationship (whether spurious or genuine), it's over. With all the technical search capabilities available to the masses, any long term cointegration relationship would be discovered by every retail trader and their brother-in-law and would be quickly arbed out. And pair trading on a large number of pairs to somehow diversify away this risk is like the old adage, "we lose money on every sale but make it up in volume". The problem isn't in the statistical techniques applied or the number of pairs. Pair trading is one of yesterday's strategies in a different market than we have today. There may still be opportunity in n-way spread trading where you are trading relationships between 3 or more stocks or baskets of stocks. These are always customized or bespoke spreads that are typically unique to each trader and not getting hammered by 2-way pair scanners.
I have to disagree. There are plenty of examples of pairs that continue to work. However, I do agree with you that triplets offer fresh opportunities. We also trade some of these.
The problem with triplets is the curve fitting. There's some commonly known/related ones like the relationship between GDX/GLD/USO but those are very rare. All the other ones seem to be force fitted into an arbitrary relationship that may or may not hold. I found all the ones that take interest rates into account get messed up heavily by fed policy changes.
Excellent article. I also think that correlation and cointegration is key to be tested before adding a new pair in arbitrage model. I also think that industry diversification might be helpful for better correlation performance. What are your thoughts on this?