After spending another day with this, I figured out how to get it working properly. In addition to a number of minor improvements, the major problem I had was the way I was estimating trading costs relative to tracking error. I'm sure my current estimates are far from perfect, but they are good enough for now. Thanks again Rob (and Doug) for bringing this method to our attention.
It probably doesn't calculate individual correlation but infers the final value from exponentially weighted covariance and variance figures GAT
yeah, checked pandas does and it does seem that way: "return a.cov(b, **kwargs) / (a.std(**kwargs) * b.std(**kwargs))". This seems like a lot of work to re-implement in C#, so I compared visually several options on some random prices converted to returns: 1. raw Excel correlations with constant lookback(25), 2. exp-weighted #1 (with span=25, alpha=0.0769 as expCorr_t=alpha*RawCorr_t+(1-alpha)*expCorr_t-1) 3. Pandas' ewm(span=75).corr() 4. Pandas' ewm(span=25).corr() The #2 (red line) looks the least jiggly, it's even smoother than Pandas' with span=75(orange) and I already have this implementation, so I'm going to use that one
A cautionary tale TLDR: I've found an issue in the dynamic optimisation method. It's probably not a massive problem unless you're really pushing it to extremes like me (over 100 instruments with $500K of capital). I plan to update the blog post today with my updated findings (which I'm still working on), but I thought I'd let you guys know first in case anyone is about to implement this thing with real money. Long form: Well I'm a flipping idiot. I rushed this thing into production, violating every rule in the book, and it started so well. I put the shadow cost up very high (200 rather than the calibrated value of 10); There were quite a few adjustment trades, more than I expected by hey ho, but then when I relaxed it the thing was trading like crazy. I reckon I spent 0.3% of my capital on costs in a week which may not sound like much, but bear in mind I plan to spend about 1.5% in a year: I've spent over two months budget on trading costs in a few days. (I might also have gained or lost versus keeping the old static system just in terms of raw returns, but I don't plan to check by comparing backtest results since I've already screwed up) Of course I should have run it in paper trading for at least a couple of weeks, but I was too bloody impatient. Anyway to cut a long story short I put the system on 'pause' about a day and a half ago (positions held, but no trading), and really started to dig into it. Initially I assumed there were some kind of difference between the sim and production implementations. I haven't been as careful as I should have been out ensuring these were identical; mainly because in my previous system the only part of code that was outside the backtest in production was the buffering, which is fairly trivial. In this new system I had the optimisation code outside of the backtest, so it was a lot more complicated. Here's the stuff that should be the same: The optimal positions are passed from sim to production, so these will be the same Both version use the core class objectiveFunctionForGreedy to do the optimisation So here's a list of possible differences between the two codebases. First the trivial stuff: There is a lot of diagnostic information in the production version that isn't in the sim, but that's fine The production version gets all the additional constraints information (maximum weights, don't trade, reduce only); in the sim version these are usually not passed unless we're testing out that functionality. In the sim version the previous positions are passed in as an argument, since we run this lots of times in a loop, in production we use the actual positions we hold so we have to get that from the database. The functions that go from weights to contracts (and vice versa) is vectorised in the sim version as it operates on data frames not sets of portfolioWeights. However it's sufficiently simple (just a division) that's probably okay. The shadow cost can be configured in the production version; in the sim version we use the default value. It would be better to allow the shadow cost to be configured in the sim. Now some stuff that isn't so minor: The per_contract values could be slightly different since these are calculated as dataframes in sim and as single instances in production using current values, but to an extent this is self correcting since in both code bases we use the same per contract values before and after the optimisation (at the margin it may make slight differences to the optimisation results). Still it's worth checking that the current values of these are as we'd expect. The costs could be slightly different. Mainly this is because they have to be divided by the current capital to get a percentage version. Still it's worth checking that the current values of these are as we'd expect. Now the stuff that might make more of a difference: The covariance matrix Some of those differences are justified because the vectorisation in sim speeds things up a lot, but I also found plenty of examples of different functions and different parameters being used in particular to calculate the correlation and standard deviations. So my original thought was that the production version used a much faster correlation update and that was causing a lot of trading. Here's a list of the notes I made on the subject: Both use daily data for standard deviation calculations For the production code the denominator of the % calculation is the current adjusted price, whereas for sim it's the current priced contract. Unless we've rolled recently they should be the same, and in any case shouldn't make much difference The robust_vol_calc in sim has no parameters passed to it from the config, all arguments are hard coded Production uses a simple 30 day rolling standard deviation, whereas sim uses the robust_vol_calc which is exponentially weighted and does other fancy stuff. This shouldn't make a huge difference, but amongst other things will lead to vol moving around more. Sim correlation uses a span =75 exponential calculation on weekly data (should be equivalent to the daily span, but...). The correlation matrix is calculated monthly. Some extra parameters are passed in to override default arguments. Production correlation uses a span=375 day exponential calculation on daily data. The correlation matrix is recalculated every day. It's not obvious from the code, but under the hood this is the same calculation function as used in sim. But no extra parameters are passed in we just use the defaults Basically what happened was I had all these carefully written estimation functions in the backtest, but then I'd only needed these types of things for the risk report so I'd just written some additional functions which were much quicker and dirtier, with hard coded parameters. Then when I introduced dynamic optimisation, I just used the risk reporting functions. I know it sounds stupid when I write it down... Anyway I fixed all this up so at least I knew that whatever I had in the sim would be reflected in production, using common code and keying of the same configuration file. Mostly this didn't change the sim behaviour, except that for the thorny issue of correlations. I can think of the following options, bearing in mind the levers we have to tweak are the frequency of data (daily, weekly) and the frequency of estimation (daily, ... monthly): Make production exactly like sim; every month the correlation matrix would change and we'd get a rippled of trades every month. Make sim exactly like production; we'd calculate a new correlation matrix every day based on daily data. Correlations would be biased downwards, and my computer would melt down. Use weekly data for both, with weekly estimations. I'd have to check if this was feasible from a sim resources point of view: it ought to be since under the hood the pandas function actually calculates weekly correlation estimates, I just throw away 3 out of every 4 of them. Use weekly data with weekly estimations for sim (again need to check it's okay) do the same for production but make the weeks 'rolling weeks'. In other words on monday we'd use a whole series of monday closing prices to form a weekly series, from which we calculate a correlation. Then on tuesday we do the same. This should result in the correlations being exactly the same as sim once a week, but then gradually changing throughout the week to reduce trading costs. I like this option. I went for the final option, which is trivial to implement (a config change in sim, and a code change so that the weekly dates are generated finishing with today) with only a small time penalty in the backtest. I also rechecked that a six month halflife was roughly okay to predict future correlations (by this stage I was getting paranoid). I then decided to recheck the shadow cost. I decided the fairest thing to do was compare an unoptimised system with 25 instruments (about what I could run with my old capital) against an optimised strategy with over 100 instruments (what I plan to run), and try and match the turnovers (averaged across instruments). I had a gut feeling that the calibration work I'd done with a smaller dynamic system and some notional capital might give me the wrong results (I was right, but for the wrong reasons). That's when I had my face palm moment. My calculation of turnover is based on the idea of an 'average position'. Basically you say oh look I've traded 100 contracts a year, with an average position of 10, so my turnover is 10. But I don't measure the average based on, well the average, I base it on the typical position I'd have with a constant forecast of 10 (with some smooothing). This makes complete sense in my original static system. Basically the turnover numbers for the dynamic system were going to be weird, since the average positions would mostly be very small numbers (since each instrument has on average 1% of capital allocated to it). I needed an alternative measure of turnover; and it turns out there is a good way of measuring turnover across the entire system, which is to measure turnover for each instrument using an average position calculated as if it was consuming your entire capital. This will obviously be a small number. Then if you add up all these small numbers you effectively have the total turnover for the system. Interestingly one consequence of this is that your turnover will be higher if your IDM is higher; which seems fair. Of course, and some of you might be ahead of me here, the way I calculate costs is to do the following calculation: Sharpe Ratio cost * turnover. And I use Sharpe Ratio, not 'cash' costs, as a rule since the latter aren't vol adjusted. For example if I use the current cost of trading S&P 500 in cash back in 1981 when my data starts, it costs something like 10% of the notional to trade it. SR costs is a nice neat shortcut that I've always liked, since you dont' have to calculate the cost of trading every single contract individually. And again it makes complete sense for my old system where most instruments usually had a position on. But if my turnover is flawed, then so is my SR cost calculation. I decided I would use cash costs instead, but vol adjust them in the past. So basically back in 1981 I'm literally saying, okay in vol equivalent terms it would have cost me $x to trade, and every time I trade one contract I methodically subtract $x from the top line. (Note I'd still keep SR costs for forecasts, since it's very hard to attribute numbers of contracts to a trading rule signal, but I'd switch to this new method for subsystem and portfolio costs) I set up a benchmark 25 instrument old school system and the figures under my new methodology came in at 1.21 gross SR, 1.14 net SR for 0.07 SR costs. That's pretty much in line with what I had when I was using SR costs, at least since I included the costs of rolling my positions (they weren't in my backtest until a few months ago). Oh and the total turnover was 25.9. That might seem high, but it's still a holding period of a couple of weeks, and bear in mind the IDM is 2.5 it equates to an average of 10 if any single instrument had all the capital. Of course you all know the punchline by now, but here goes. Using the 'optimal' shadow cost of 10, the dynamic system with 103 instruments and the same capital came in at a gross SR of 1.5 (nice!) but the net was 1.27. The costs are a substantial 0.23 SR points; much higher than my normal 'speed limit' of about 0.13 SR points, and more than 3 times the existing system. The turnover was a heart stopping 110 times a year. Even though there was still a theoretical improvement in net SR there was no way I was going to run with this. Now my first thought was I can still save this by increasing the shadow cost. With a shadow cost of 100 the gross SR is unchanged, the net is 1.3, so we're down to 0.20 SR points of costs, with a turnover of 71. For shadow cost 200, gross SR is starting to come down to 1.48, net 1.29, so we've only knocked off another 0.01 SR in costs which are down to 0.19 SR. However the turnover is lower, 55. We're clearly going to need a much higher shadow cost. Anyway I'm currently trying to see if a very high shadow cost will get the turnover and costs down to some more reasonable level; slow enough that I can turn the system back on at least for next week. However it's possible this will be at a level where the net SR doesn't really justify using the dynamic system (the higher the shadow cost the less closely we can match the raw system as we saw going from costs of 100 to 200). It's also possible that it's impossible to get the costs of the dynamic system down to below 0.10 SR points in costs. Why? There is one basic problem here: the greedy algo. In particular when the sign of the underlying forecast flips it always close any trade it has on in that instrument. Let's assume that each instrument has a forecast turnover of 10. With over 100 instruments each will be flipping it's forecast every 35 days or so; which means on any given day about 3 instruments will be flipping their forecast and closing their positions. The shadow cost can't really help us there. This isn't a flaw of the idea; it's just because I'm pushing it to an extreme level with a much larger number of instruments than I have any right to hold. [To put it another way, the optimal shadow cost for the toy example in my blog post is likely to be lower than the one for my extreme system, although probably not equal to 10] So when I come back though I plan to look more seriously at CVXPY. I still like the idea of the tracking error minimisation as the objective function, but I want a more general integer solution. Basically the opitimisation needs to be able to hold a long position on in an instrument even when the forecast has flipped, if it's too expensive to do the closing trade. Most probably it would then go long(er) a cheap hedging instrument, to compensate. This isn't as nice, since it makes the optimiser feel less robust and puts more weight on our correlations, but I have high hopes... GAT PS What's annoying about this story is that something very similar happened in my professional career, albeit to someone else. They introduced a fancy optimisation algo that looked great in paper, but traded far too much, and our backtest didn't deal properly with the costs of trading that quickly. The thing was pushed into live trading (yes it was paper traded but with much smaller capital so the additional market impact wasn't apparent), and when it started racking up too much in trading costs we had trouble matching live and sim code to find out what was going on. I won't tell you how much that thing cost in trading costs by the time it was eventually shut down but it was a hell of a lot.
I've gone for a shadow cost of 2000 which puts the stats at gross 1.35 / net 1.23 / costs 0.12 / turnover 25.5 A net SR of 1.23 isn't the 1.43 or so I thought I'd get, but it's still okay. Costs are slightly higher, but turnover is matched. I think my production costs will be lower anyway, since I've excluded really expensive instruments from trading which are allowed in sim. Anyway the system will be okay for a week or so. Spent a bit of time diddling around with CVXPY but the problem as specified isn't convex so that didn't work; also tried tweaking the greedy method without success. Going to take a week off anyway so plenty of time to think.... GAT
Hi Rob, Have you tried this implementation with cvxpy? https://github.com/golforado/tracking-portfolio/blob/main/tracking_portfolio.py I didn't have any problem due to convexity Thanks
I needed to address the problem in a slight different way so I could solve it with cvxpy. Actually the interpretation of the shadow cost changes. I'm using around 50, but I'm not in production yet. I will review it after your recent experience. I tried to explain the way I addressed the problem here: https://github.com/golforado/tracking-portfolio Please let me know if I can help. Thanks
Hey GAT...If this was the TLDR version, then the verbose version is going to break Baron's internet data-amount costs. Holy smokes!