Fully automated futures trading

HobbyTrading · May 21, 2021

That's quite some progress. It gives me the impression that you are in some sort of lockdown and have nothing else to work on....

newbunch · May 21, 2021

globalarbtrader said:
Progress has been quite swift over the last couple of weeks.

I'm continuing to add markets but clearly this is going to take a while. I've put all the markets for both the first and second batches into my configuration file and there is a sobering total of 227 rows; of which 40 or so were my legacy markets, and where I've added another 16 of the remaining 187 so far (including the new micro and mini markets). This is going to be a long haul!

For some markets where there isn't Barchart data I created some code that will 'seed' the database with as much IB data as it can get (again thanks to @HobbyTrading for the heads up about expired contracts).

I decided to add the micro/mini instruments as additional markets rather than replacing existing ones. That means things like risk and position reports in the past will still make sense, and overall just seemed a less risky approach. That also means for the time being I'm still sampling the prices of the original markets; given the number I'm adding to my system another half dozen is neither here or there.

I did some profiling work and was able to get some quick wins quite easily. Broadly speaking I did this by removing use of pandas apply (on rows) using my own functions, and instead did the relevant operation on the entire dataframe in situ.

A particular time sink was the calculation of the carry signal; in particular working out what the distance between two expiries is in a fraction of a year. Now I could assume the distance is always what it is specified in the roll configuration, but to be safe I'd rather not do that.

I was doing this with apply and also working with the string "20060300" converting it into a datetime, and working out the exact number of days. This is crazy since it assumes the expiry is on the first of the month, so we're doing a massive approximation, so it doesn't really matter if (for example) we account for the fact that February is 28 days long. Instead I changed the contract series to floats so the operation could be done on the entire dataframe, assuming each month was exactly 1/12 of a year.

There were also little things like "".join(["x","y"]) being faster than "x"+"y", introducing a new 'simple' contractDate object where both inputs are strings (since the full object is very flexible and therefore slow).

Then there is this little beauty, run every time a contractDate is initialised (since I specify spread contracts as eg '20200100_20200200':
Code:
MISSING_STR = -1

def list_of_items_separated_by_underscores(this_str, result = ()):
    find_underscore = this_str.find("_")
    if find_underscore is MISSING_STR:
        result_as_list = list(result)
        result_as_list.append(this_str)
        return result_as_list

    partial_str = this_str[:find_underscore]
    result = result + tuple([partial_str])

    remaining_str = this_str[find_underscore+1:]

    return list_of_items_separated_by_underscores(remaining_str, result=result)
Which I replaced with the surprisingly quick:
Code:
date_str_as_list = date_str.split("_")
WTF was I thinking?! Anyway, moving on, nothing to see here...

In a change from the plan above, I decided to circle back to potential trading signals and changes to the core model before I went ahead with anything else.

When I sat down and went through the list, there weren't actually that many things that I intended to add in terms of new signals (which wouldn't be more suited to a newer seperate strategy, such as spread trading). I added some 'fast' cross sectional momentum (I already had slow cross sectional mean reversion within asset classes, so this was just a case of reversing the sign and shortening the window). I added the skew and kurtosis rules. This also means I can take out the 'short vol bias' rule. That's basically it. A 'feature' that isn't strictly a new signal is the idea of attenuating momentum and carry style signals.

The next step will be to 'build' the new system as a traditional model. So that means generating forecast scalars, and then fitting forecast weights. I have a method for fitting forecast weights that I will probably blog about. It involves fitting on each instrument individually, then clustering and fitting clusters; finally fitting across the whole set of instruments. The final forecast weights will be a blend of the weights from each of the three methods, the blend depending on the amount of data an instrument or cluster. Quite slow to do, I'll probably do it as a single in sample fit once I've tested it, and quite heuristic, but I would like to introduce some more instrument specific weightings where that is justified. I expect this will be in my next blog post.

Then it's down to calculate the FDM, and fit instrument weights (something I'll have to do regularly as I add new instruments). One consequence of the new methodology, by the way, is that no IDM is needed (but I'll probably set it to the maximum 2.5 just for old times sake).

Once the system is working in the traditional model, it's a case of adding the additional optimisation stage that accounts for discrete positions. As part of the optimisation I will be including my idea for dynamically targeting risk according to average forecast strength (this is why an IDM is no longer needed).

At this stage I also want to include my new blended vol estimate that incorporates mean reverting vol. Another thing I will have is a 'don't trade' list of instruments that is updated automatically according to my criteria for adding new markets (in practice we can trade, but it won't allow new positions to be opened); in particular volume and cost constraints (the 'too large to trade' will be dealt with by the opimisation).

I will make some simplifications. I can remove the hacky forecast mapping that I use right now for larger contract size. And the exogenous risk overlay will also be redundant. Finally buffering will no longer be required, since the optimisation will include a cost penalty.

GAT
More...
Wow! That sure is complicated. I'm sure there must be something between "caveman trend following" and whatever it is you're doing.

Kernfusion · May 23, 2021

globalarbtrader said:
Progress has been quite swift over the last couple of weeks.

I'm continuing to add markets but clearly this is going to take a while. I've put all the markets for both the first and second batches into my configuration file and there is a sobering total of 227 rows; of which 40 or so were my legacy markets, and where I've added another 16 of the remaining 187 so far (including the new micro and mini markets). This is going to be a long haul!

For some markets where there isn't Barchart data I created some code that will 'seed' the database with as much IB data as it can get (again thanks to @HobbyTrading for the heads up about expired contracts).

I decided to add the micro/mini instruments as additional markets rather than replacing existing ones. That means things like risk and position reports in the past will still make sense, and overall just seemed a less risky approach. That also means for the time being I'm still sampling the prices of the original markets; given the number I'm adding to my system another half dozen is neither here or there.

I did some profiling work and was able to get some quick wins quite easily. Broadly speaking I did this by removing use of pandas apply (on rows) using my own functions, and instead did the relevant operation on the entire dataframe in situ.

A particular time sink was the calculation of the carry signal; in particular working out what the distance between two expiries is in a fraction of a year. Now I could assume the distance is always what it is specified in the roll configuration, but to be safe I'd rather not do that.

I was doing this with apply and also working with the string "20060300" converting it into a datetime, and working out the exact number of days. This is crazy since it assumes the expiry is on the first of the month, so we're doing a massive approximation, so it doesn't really matter if (for example) we account for the fact that February is 28 days long. Instead I changed the contract series to floats so the operation could be done on the entire dataframe, assuming each month was exactly 1/12 of a year.

There were also little things like "".join(["x","y"]) being faster than "x"+"y", introducing a new 'simple' contractDate object where both inputs are strings (since the full object is very flexible and therefore slow).

Then there is this little beauty, run every time a contractDate is initialised (since I specify spread contracts as eg '20200100_20200200':
Code:
MISSING_STR = -1

def list_of_items_separated_by_underscores(this_str, result = ()):
    find_underscore = this_str.find("_")
    if find_underscore is MISSING_STR:
        result_as_list = list(result)
        result_as_list.append(this_str)
        return result_as_list

    partial_str = this_str[:find_underscore]
    result = result + tuple([partial_str])

    remaining_str = this_str[find_underscore+1:]

    return list_of_items_separated_by_underscores(remaining_str, result=result)
Which I replaced with the surprisingly quick:
Code:
date_str_as_list = date_str.split("_")
WTF was I thinking?! Anyway, moving on, nothing to see here...

In a change from the plan above, I decided to circle back to potential trading signals and changes to the core model before I went ahead with anything else.

When I sat down and went through the list, there weren't actually that many things that I intended to add in terms of new signals (which wouldn't be more suited to a newer seperate strategy, such as spread trading). I added some 'fast' cross sectional momentum (I already had slow cross sectional mean reversion within asset classes, so this was just a case of reversing the sign and shortening the window). I added the skew and kurtosis rules. This also means I can take out the 'short vol bias' rule. That's basically it. A 'feature' that isn't strictly a new signal is the idea of attenuating momentum and carry style signals.

The next step will be to 'build' the new system as a traditional model. So that means generating forecast scalars, and then fitting forecast weights. I have a method for fitting forecast weights that I will probably blog about. It involves fitting on each instrument individually, then clustering and fitting clusters; finally fitting across the whole set of instruments. The final forecast weights will be a blend of the weights from each of the three methods, the blend depending on the amount of data an instrument or cluster. Quite slow to do, I'll probably do it as a single in sample fit once I've tested it, and quite heuristic, but I would like to introduce some more instrument specific weightings where that is justified. I expect this will be in my next blog post.

Then it's down to calculate the FDM, and fit instrument weights (something I'll have to do regularly as I add new instruments). One consequence of the new methodology, by the way, is that no IDM is needed (but I'll probably set it to the maximum 2.5 just for old times sake).

Once the system is working in the traditional model, it's a case of adding the additional optimisation stage that accounts for discrete positions. As part of the optimisation I will be including my idea for dynamically targeting risk according to average forecast strength (this is why an IDM is no longer needed).

At this stage I also want to include my new blended vol estimate that incorporates mean reverting vol. Another thing I will have is a 'don't trade' list of instruments that is updated automatically according to my criteria for adding new markets (in practice we can trade, but it won't allow new positions to be opened); in particular volume and cost constraints (the 'too large to trade' will be dealt with by the opimisation).

I will make some simplifications. I can remove the hacky forecast mapping that I use right now for larger contract size. And the exogenous risk overlay will also be redundant. Finally buffering will no longer be required, since the optimisation will include a cost penalty.

GAT
More...
Oh, looks like I will need to upgrade my trading server (finally there's a good reason to do it )., will buy some mean crazy-fast pci-e ssd-drive, lots of ram.., it's gonna be fun!

HobbyTrading · May 23, 2021

Kernfusion said:
Oh, looks like I will need to upgrade my trading server (finally there's a good excuse reason to do it ).
More...

FTFY

wopr · May 23, 2021

globalarbtrader said:
Progress has been quite swift over the last couple of weeks.

I'm continuing to add markets but clearly this is going to take a while. I've put all the markets for both the first and second batches into my configuration file and there is a sobering total of 227 rows; of which 40 or so were my legacy markets, and where I've added another 16 of the remaining 187 so far (including the new micro and mini markets). This is going to be a long haul!

For some markets where there isn't Barchart data I created some code that will 'seed' the database with as much IB data as it can get (again thanks to @HobbyTrading for the heads up about expired contracts).

I decided to add the micro/mini instruments as additional markets rather than replacing existing ones. That means things like risk and position reports in the past will still make sense, and overall just seemed a less risky approach. That also means for the time being I'm still sampling the prices of the original markets; given the number I'm adding to my system another half dozen is neither here or there.

I did some profiling work and was able to get some quick wins quite easily. Broadly speaking I did this by removing use of pandas apply (on rows) using my own functions, and instead did the relevant operation on the entire dataframe in situ.

A particular time sink was the calculation of the carry signal; in particular working out what the distance between two expiries is in a fraction of a year. Now I could assume the distance is always what it is specified in the roll configuration, but to be safe I'd rather not do that.

I was doing this with apply and also working with the string "20060300" converting it into a datetime, and working out the exact number of days. This is crazy since it assumes the expiry is on the first of the month, so we're doing a massive approximation, so it doesn't really matter if (for example) we account for the fact that February is 28 days long. Instead I changed the contract series to floats so the operation could be done on the entire dataframe, assuming each month was exactly 1/12 of a year.

There were also little things like "".join(["x","y"]) being faster than "x"+"y", introducing a new 'simple' contractDate object where both inputs are strings (since the full object is very flexible and therefore slow).

Then there is this little beauty, run every time a contractDate is initialised (since I specify spread contracts as eg '20200100_20200200':
Code:
MISSING_STR = -1

def list_of_items_separated_by_underscores(this_str, result = ()):
    find_underscore = this_str.find("_")
    if find_underscore is MISSING_STR:
        result_as_list = list(result)
        result_as_list.append(this_str)
        return result_as_list

    partial_str = this_str[:find_underscore]
    result = result + tuple([partial_str])

    remaining_str = this_str[find_underscore+1:]

    return list_of_items_separated_by_underscores(remaining_str, result=result)
Which I replaced with the surprisingly quick:
Code:
date_str_as_list = date_str.split("_")
WTF was I thinking?! Anyway, moving on, nothing to see here...

In a change from the plan above, I decided to circle back to potential trading signals and changes to the core model before I went ahead with anything else.

When I sat down and went through the list, there weren't actually that many things that I intended to add in terms of new signals (which wouldn't be more suited to a newer seperate strategy, such as spread trading). I added some 'fast' cross sectional momentum (I already had slow cross sectional mean reversion within asset classes, so this was just a case of reversing the sign and shortening the window). I added the skew and kurtosis rules. This also means I can take out the 'short vol bias' rule. That's basically it. A 'feature' that isn't strictly a new signal is the idea of attenuating momentum and carry style signals.

The next step will be to 'build' the new system as a traditional model. So that means generating forecast scalars, and then fitting forecast weights. I have a method for fitting forecast weights that I will probably blog about. It involves fitting on each instrument individually, then clustering and fitting clusters; finally fitting across the whole set of instruments. The final forecast weights will be a blend of the weights from each of the three methods, the blend depending on the amount of data an instrument or cluster. Quite slow to do, I'll probably do it as a single in sample fit once I've tested it, and quite heuristic, but I would like to introduce some more instrument specific weightings where that is justified. I expect this will be in my next blog post.

Then it's down to calculate the FDM, and fit instrument weights (something I'll have to do regularly as I add new instruments). One consequence of the new methodology, by the way, is that no IDM is needed (but I'll probably set it to the maximum 2.5 just for old times sake).

Once the system is working in the traditional model, it's a case of adding the additional optimisation stage that accounts for discrete positions. As part of the optimisation I will be including my idea for dynamically targeting risk according to average forecast strength (this is why an IDM is no longer needed).

At this stage I also want to include my new blended vol estimate that incorporates mean reverting vol. Another thing I will have is a 'don't trade' list of instruments that is updated automatically according to my criteria for adding new markets (in practice we can trade, but it won't allow new positions to be opened); in particular volume and cost constraints (the 'too large to trade' will be dealt with by the opimisation).

I will make some simplifications. I can remove the hacky forecast mapping that I use right now for larger contract size. And the exogenous risk overlay will also be redundant. Finally buffering will no longer be required, since the optimisation will include a cost penalty.

GAT
More...
Impressive work and a great writeup, thanks Rob!
I'm really looking forward to the next post in the series, on how exactly you trade all that. I'm currently tracking 55 markets and trading 22 (capital is the limit), so am interested in how you approach this.
You mentioned you're interested in doing something like this, and provided a little bit of detail on the TTU podcast sometime last year, and when you said that, I tried prototyping an implementation, but it quickly became quite complex with some edge cases (eg. your "ranking" says long Corn is a great position, you go long, then that position falls down in the ranking, and what do you do then. Backtesting this was also a bit of a pain as it's path-dependant and no way to parallelize). Second thing I got stuck on was accounting for instrument diversification, I really wanted to make it dynamic, so that it's computed based on the instruments I currently have positions in, but that was changing quite a bit, surely more than the fixed IDM that you (and therefore I ) use right now.

So thanks again for documenting all this, it is immensely useful.

Kernfusion · May 23, 2021

wopr said:
Impressive work and a great writeup, thanks Rob!
I'm really looking forward to the next post in the series, on how exactly you trade all that. I'm currently tracking 55 markets and trading 22 (capital is the limit), so am interested in how you approach this.
You mentioned you're interested in doing something like this, and provided a little bit of detail on the TTU podcast sometime last year, and when you said that, I tried prototyping an implementation, but it quickly became quite complex with some edge cases (eg. your "ranking" says long Corn is a great position, you go long, then that position falls down in the ranking, and what do you do then. Backtesting this was also a bit of a pain as it's path-dependant and no way to parallelize). Second thing I got stuck on was accounting for instrument diversification, I really wanted to make it dynamic, so that it's computed based on the instruments I currently have positions in, but that was changing quite a bit, surely more than the fixed IDM that you (and therefore I ) use right now.

So thanks again for documenting all this, it is immensely useful.
More...

Regarding parallelization I'm now thinking to maybe do it NOT between instruments (i.e. process each instrument in a separate thread), which clearly doesn't, work because in many places we look at all instruments at the same time to make decisions, but instead do certain stages of the workflow of a single instrument price in parallel. So essentially, there will still be only 1 main processing thread, but it will delegate certain operations to other threads, wait for all results and continue..
E.g. an easy win would be to calculate forecasts, of which I already have 12 counting permutations, in parallel. Because individual forecasts don't care about other forecasts, they can be computed simultaneously in separate threads and then results returned back to the main thread, which will continue only after receiving all the results. Also, for every new day my system stitches prices for each instrument from scratch, which I bet takes a lot of time, that should also be fine to do in parallel for different instruments, but it's a bit harder to implement in my code..

globalarbtrader · May 24, 2021

wopr said:
Impressive work and a great writeup, thanks Rob!
I'm really looking forward to the next post in the series, on how exactly you trade all that. I'm currently tracking 55 markets and trading 22 (capital is the limit), so am interested in how you approach this.
You mentioned you're interested in doing something like this, and provided a little bit of detail on the TTU podcast sometime last year, and when you said that, I tried prototyping an implementation, but it quickly became quite complex with some edge cases (eg. your "ranking" says long Corn is a great position, you go long, then that position falls down in the ranking, and what do you do then. Backtesting this was also a bit of a pain as it's path-dependant and no way to parallelize). Second thing I got stuck on was accounting for instrument diversification, I really wanted to make it dynamic, so that it's computed based on the instruments I currently have positions in, but that was changing quite a bit, surely more than the fixed IDM that you (and therefore I ) use right now.

So thanks again for documenting all this, it is immensely useful.
More...

Very briefly in the Corn case, if -once allowing for costs - the expected return is higher on another instrument you'd close the Corn and open something else instead.

You're right about backtesting, but this is something I'd probably demonstrate as a proof of concept on a small number of markets. And you can still as @Kernfusion says parralize (a word I can never spell!) almost all the calculations (basically everything that's identical to the current system).

Instrument diversification is an interesting one, and one I am still toying with. Effectively though I think the system I have in mind (I still haven't written a single line of code!) will have a 'long run' concept of instrument diversification (identical to what I have now, based on long run correlations of subsystem returns), but the implicit instrument weights can be modified using 'short run' correlation between market returns.

GAT

wopr · May 24, 2021

Kernfusion said:
E.g. an easy win would be to calculate forecasts, of which I already have 12 counting permutations, in parallel. Because individual forecasts don't care about other forecasts, they can be computed simultaneously in separate threads and then results returned back to the main thread, which will continue only after receiving all the results. Also, for every new day my system stitches prices for each instrument from scratch, which I bet takes a lot of time, that should also be fine to do in parallel for different instruments, but it's a bit harder to implement in my code..
More...

Good point. I've started to extract my backtesting plumbing away from my system and my long term goal is to have it completely separate (and have it support backtesting stocks systems as well).

However, from my experience so far, all the path-independant stuff can usually be calculated in a vectorized fashion and is already quite fast. For example, my backtesting system computes all moving averages, carry and all forecasts before starting the main loop, using Pandas. For a 20-year backtest of 22 markets that takes 35 min, that whole forecast computation takes 30 seconds. So parallelizing that would be almost no gain, the real benefits (at least in my case) would be gained from parallelizing the path-dependant calculations which is much harder.

Kernfusion · May 24, 2021

globalarbtrader said:
Instrument diversification is an interesting one, and one I am still toying with. Effectively though I think the system I have in mind (I still haven't written a single line of code!) will have a 'long run' concept of instrument diversification (identical to what I have now, based on long run correlations of subsystem returns), but the implicit instrument weights can be modified using 'short run' correlation between market returns.

GAT
More...

- Sort of like the new fed's inflation tracking policy - 2% "on average" ?
But also, as I understand, you will modify the short-term weights based on the current forecast strength as well, not only based on the short-term correlations? Or maybe these things will be independent of each-other.. Anyway, we probably have wait for the next full blog-posts to find out

wopr said:
However, from my experience so far, all the path-independant stuff can usually be calculated in a vectorized fashion and is already quite fast. For example, my backtesting system computes all moving averages, carry and all forecasts before starting the main loop, using Pandas. For a 20-year backtest of 22 markets that takes 35 min, that whole forecast computation takes 30 seconds. So parallelizing that would be almost no gain, the real benefits (at least in my case) would be gained from parallelizing the path-dependant calculations which is much harder.
More...

Yeah, for sure vectorized approach will be faster, and you can probably precompute all these stitched series, averages, etc. in parallel before starting..
It's just I'm running backtests on essentially execution engine, which is not vectorized, it treats every historical price I pump into it as if it was a new real-time price.. The advantage here is that I'm testing exactly what I'll be trading, and also can check that way the backtest before and after every production change to make sure I didn't introduce bugs.. But yes, it's slow, currently with 40 instruments it takes ~5 hours, and with 200 instruments it'll probably go into weeks but I believe there's a room for improvement there..

globalarbtrader · May 27, 2021

" I have a method for fitting forecast weights that I will probably blog about. It involves fitting on each instrument individually, then clustering and fitting clusters; finally fitting across the whole set of instruments. The final forecast weights will be a blend of the weights from each of the three methods, the blend depending on the amount of data an instrument or cluster. Quite slow to do, I'll probably do it as a single in sample fit once I've tested it, and quite heuristic, but I would like to introduce some more instrument specific weightings where that is justified. I expect this will be in my next blog post."

I expect I was right
https://qoppac.blogspot.com/2021/05/fit-forecast-weights-by-instrument-by.html

GAT