Home > Technical Topics > Programming > Oh no, not another python backtester...

Oh no, not another python backtester...

  1. There are already many python packages where you can back test trading strategies. Some of them also include a framework for automatic execution and complete position management.

    I can't give an exhaustive list but I'll pick out:

    - Quantopian's zipline
    - BT
    - pythalesians
    - pyalgotrade
    There is a longer list here.

    Does the world really need another one? Well, probably not, but today I've released one anyway (this explains at least partly why it's been a month since my last post, since I've spent the last month writing it).

    You can find pysystemtrade on github, here. This is version 0.0.1. It provides a basic backtesting enviroment that implements the 'staunch systems trader' example in my book. And it's GPL 3 open source.

    However it's my intention that this will become a fully featured back tester that will include all the optimisation and calibration techniques that I've discussed here on the blog, and in my book. It will also, eventually, include a complete end to end system that can be used for fully automated futures trading with interactive brokers.

    More on my blog, in the usual place.

  2. this is a stupid question, but do these use python or do I need to know python to use them?
  3. They use python, and you wouldn't get very far using them unless you knew python
  4. Which is the most mature with the largest user base ?
    Why is Python becoming a new standard in algo trading ?
  5. Assume this is directed at a certain user I'm ignoring, rather than me...

  6. Looks interesting. I was able to clone it from GitHub and to recreate the graph and data for the demo system from your book. I think that equity curve was really missing in your book, when I read it.
    One question, though:

    Your price data for the instruments in the system start at different dates:

    EDOLLAR: 1983-09-26
    US10: 1982-05-04
    EUROSTX: 2013-03-19
    V2X: 2013-03-26
    MXP: 1995-04-26
    CORN: 1981-09-25

    How is that handled in your backtester? Are the instrument weights adjusted, so that they add up to one at each day?
    Most of the testing time, only EDOLLAR, US10 and CORN are available, after 1995 MXP.
    EUROSTX and V2X are basically not used at all, since their history starts so late.

    Sorry, I could not find it in your code, yet.

    I printed the stats() for the system:

    [[('min', '-0.764'), 
     ('max', '0.1589'), 
     ('median', '0'), 
     ('mean', '0.000671'), 
     ('std', '0.02224'), 
     ('skew', '-5.727'), 
     ('ann_daily_mean', '0.1718'), 
     ('ann_daily_std', '0.3559'), 
     ('sharpe', '0.4827'), 
     ('sortino', '0.4677'), 
     ('avg_drawdown', '-0.339'), 
     ('time_in_drawdown', '0.9864'), 
     ('calmar', '0.1109'), 
     ('avg_return_to_drawdown', '0.5067'), 
     ('avg_loss', '-0.01581'), 
     ('avg_gain', '0.01534'), 
     ('gaintolossratio', '0.9702'), 
     ('profitfactor', '1.113'), 
     ('hitrate', '0.5343')], 
    ('You can also plot:', ['rolling_ann_std', 'drawdown', 'curve']), 
    ('You can also print:', ['weekly', 'monthly', 'annual'])]
    Why is the skew so negative? Shouldn't it be positive from the trend-following part?
    What are min/max/median/mean? Return in %?
  7. The instrument weights are adjusted to add up to 1.0

    However the instrument div. multiplier should also be adjusted - it should be lower in the past. Future releases will calculate the idm on an out of sample basis (and also optimise the instr. weights).

    I didn't choose these instruments because they had long history (or god forbid, better back test performance), but because they had smaller risk, so better for the account size in the book.

    Skew would probably be positive on a monthly basis - the positive skew will only appear when measured at a frequency analagous to the holding period. Anything with a holding period of more than a few days will inherit the daily skew of the underlying assets rather than the trading strategy.

    Again a future release will calculate skew on different time periods.

    min/max/median/mean refer to daily returns.


  8. why do we keep seeing all those frameworks when any good Python programmers could make one of its own with more flexibility and more accountability.

    seems like a lot of bored developers are putting good effort into something that few needs really
  9. Perhaps. But having already 'made one of my own', and since plenty of people asked me to release it into the wild, it seemed like no extra work to do so

  10. ^ is that really necessary or even useful ?
  11. if you have a novel rule you might not know what the scale should be (it's not always possible to work it out without using data)

  12. apologies for the abrupt time boxing of the chat from my side.
    I found that useful, thanks for the input to portfolio construction, will be looking at the impact of 'smoothed binary' and binary for trade-able lot sizes of 1 over the next 2 weeks. cheers all
  13. Think you meant to put this on private thread!
  14. It is unclear where you get your data from.

    quandl.py contains only:
    get data from quandl

    Presumably you are using legacy csv data, but where did this come from and how long are the time frames?
  15. Hi

    Yes this is a work in progress...

    csvdata is using data from my existing database, which is basically daily quandl data, then for the last couple of years I've been sampling intraday mid prices from IB tick stream every hour or so, plus closing prices (which are always timestamped 23:00). All times are local UK system time by the way.

  16. Hi GAT,

    I’ve been using pysystemtrade to backtest several strategies, that I would now like to try to track in real time.

    I am not sure exactly how I should proceed here in terms of pysytemtrade. Please correct me if I’m wrong or if there is a better way to do it, but the steps I am thinking of are as follows:

    1. Run the following code with my config:

    from systems.provided.futures_chapter15.basesystem import *
    from syscore.pdutils import align_to_joint, uniquets, divide_df_single_column
    from syscore.dateutils import generate_fitting_dates
    from syscore.algos import robust_vol_calc
    from systems.portfolio import Portfolios
    config = Config("private.my_system.futuresestimateconfig.yaml")
    system = System([Account(), Portfolios(), PositionSizing(), FuturesRawData(), ForecastCombine(), ForecastScaleCap(), Rules()], csvFuturesData(), config)

    2. I then obtain each position by using:

    data = csvFuturesData()
    for instrument in data.get_instrument_list():
    system.accounts.get_buffered_position(instrument, roundpositions=True).tail()

    3. Assume then nothing changes except that I get new price data coming in for each instrument. How should I proceed in order to obtain new updated positions based on new data without having to run the whole system again?
  17. Read the section on the userguide for caching specifically this bit.

    Basically you'd:

    - run the entire system
    - when you get a new price for a particular instrument, remove the relevant contents of the cache
    - get the buffered position

  18. I see, thanks GAT. With respect to the 'delayfill=True' function, does this only apply to accounting (e.g., 'system.accounts.portfolio(delayfill=False)' ) and not position generation with the 'system.accounts.get_buffered_position(instrument, roundpositions=True)' method, or does the 'delayfill' feature not affect the get_buffered_position feature?
  19. No positions aren't delayed, only inside the accounting function (otherwise it would be confusing with a risk of delaying things twice)

  20. Hi GAT, in pysystemtrade, does the system account/are trades generated for rolls when the contracts change in the carrydata files?
  21. Not yet.

  22. Hi GAT,
    How does pysystemtrade treat the carry system when there is no recent price for the carry contract? I see in the historical data that sometimes there is gaps in the data for carry contracts (e.g., when you only trade particular annual contracts and the next carry contract is too far how to have liquidity)?
  23. Hi AG

    Good question. Basically carry is calculated on each day when there is both a current and a carry contract price. The rest of the time it will be NAN. All the other forecasting calculations are then done, which again will propogate NANs. Finally just before combining with other forecasts I forward fill the forecast values. It's important not to do this too early or it will throw off things like calculating the standard deviation of forecasts for scaling purposes.

    Hope that makes sense

  24. Hi GAT, Why do the returns and metrics in pysystemtrade change so markedly if you change the base currency? Shouldn't the returns be similar for various currencies given they are mostly USD instruments and most of the account stays in cash? The vol is often much lower than target vol when using non-USD currency balances. How do you grapple with this?
  25. I'm surprised to be honest. I've opened a ticket on this, and I'll look at it when I get a chance https://github.com/robcarver17/pysystemtrade/issues/37

  26. Hi GAT, just wondering if there is potential for adding stops into pysystemtrade? I am considering adding a mean reversion system but want to add some conditions to exit if a position moves too far against me.
  27. Yes I'll add them at some point, for exactly that (a mean reversion system).

    I need to think about the most elegant way of doing it plus I am full speed on book writing right now so it will be a few months before I get a chance to do any python again

  28. Hi GAT, thanks for releasing pysystemtrade to the great unwashed. finding it
    very educational.

    I'm trying to recreate the included data. The file EDOLLAR_price.csv currently
    has a last entry of "2016-07-08,98.9725" I understand that we are looking at
    around 40 months out on the curve so i would expect the contract to be one of
    201906, 201909, 2019012

    Looking at Quandl CME data 201906 gives last price on our date of 2016-07-08
    201906 98.98. But where does the price actually come from? is it your last
    sampled price from IB?

    I'm begining to create my own Panama adjusted price series and am using the
    included price csv's as reference.

    Any tips and pointers for this process? Would it be possible to give the
    contracts which forms each of the price series files?

    When reconstructing the carry files should I be rolling them on volume for the
    carry contract or price contract? This sould be easier to check as I have the
    source contracts in the file by construction.

    Kind regards,
  29. Don't roll ED contracts on volume, roll by date. Volume shifts back and forth, often on a daily basis.

  30. The carry data file gives the current contracts being used for prices

    So for example in here:


    The final row (#8646) has a price contract of 201909 and a price of 98.1175. The carry contract is 201906.

    Then in the price file there is the same price (at the end of these files the current adjusted price will always ). That price will be the closing price from IB on that day, which should be the same as quandl although obviously no guarantee.

    The rolling is done on a constant maturity basis, adjusted in earlier years if there isn't liquidity far enough out on the curve.

    I'm hoping to publish an extension to pysysystemtrade that creates this data, after pulling individual contracts from quandl, plus explanatory blogpost.

  31. I've looked into this and I can't repeat the problem, nor can I see any obvious bugs that might be the cause. For example with USD and GBP I get sharpe ratios within 0.02 of each other. There are still some small differences as you'd expect for the following reasons:

    The main difference is with euros as the base currency you only have the FX history since 1999 so a lot of the earlier back test vanishes.

    The concept of "cash" is irrelevant since the backtest doesn't know about margin. Obviously if you have £1,000,000 that will give you different positions to having $1,000,000; and the difference will vary over time as those amounts are different. Plus profits will be daily marked to market into the base currency and so will be different.

    The differences are smaller if you use roundpositions=False when doing accounting or turn off buffering (config.buffer_size=0.0)

    Small amounts of capital and rounded positions combined with buffering may lead to weird non linearities where you have a position in one currency but not in another which could be a cause of lower vol.

  32. I look forward to the extension (and blogpost). Would be reassuringly complete to have the stitching code for the example data files. Thanks
  33. Hi GAT,

    I notice a bug(?) in pysystemtrade.

    >>> system.data.daily_prices("GBP").tail(10)
    Loading csv data for GBP
    2017-01-05    1.2427
    2017-01-06    1.2293
    2017-01-09    1.2178
    2017-01-10    1.2179
    2017-01-11    1.2223
    2017-01-12    1.2178
    2017-01-13    1.2200
    2017-01-16       NaN <-------- Holiday
    2017-01-17    1.2411
    2017-01-18    1.2352
    >>> system.accounts.get_notional_position("GBP").tail(10)
    2017-01-05   -84.629566
    2017-01-06   -77.148345
    2017-01-09   -75.307283
    2017-01-10   -78.993038
    2017-01-11   -81.460877
    2017-01-12   -85.617533
    2017-01-13   -88.694337
    2017-01-16   -88.694339
    2017-01-17   -86.485507
    2017-01-18   -88.339476
    >>> system.accounts.pandl_for_instrument("GBP").tail(10)
    2017-01-05    -7555.350031
    2017-01-06    10596.153621
    2017-01-09     8760.748599
    2017-01-10      -72.328423
    2017-01-11    -2942.568750
    2017-01-12     3170.171272
    2017-01-13    -1588.764330
    2017-01-16        0.000000
    2017-01-17        0.000000 <-------- not supposed to be zero
    2017-01-18     4656.453583
  34. Probably a subtle date / time alignment; I've raised an issue


  35. Thanks for the reply (in Jan), I'm starting with recreating the carry data as this seems simpler.

    If I remember correctly, you said that you are not rolling with fixed rules, that there is some discretion applied as to when contracts are rolled.

    Does this mean that the carry data files provided in pysystemtrade show the
    dates of when each of your active instruments were rolled? Meaning you are recording the date when the roll is initiated for use in constructing backtest data.

    For data which is outside of the date range your system has been live and rolling, how did you come up with the roll dates? Are these roll dates volume based or bid ask spread based?

    Regarding this reply:

    Do the change in carry pair dates indicate points in the price file where panama adjustment is applied?
  36. Yes since around January 2014 these are actual roll dates.

    They are based on a fixed time before the expiry date, with adjustments to those dates when there isn't a sychronous price between the current and new contracts.


  37. Hi GAT,
    I was wondering if there is a feature in pysystemtrade to be able to conduct portfolio attributions?
    Also, have you ever experimented with 'value' rules in futures markets?
  38. Depends what you mean by 'attributions'. There is a lot of diagnostics in the accounting stage but if you mean something distinct perhaps you could give an example?

    'Value' as distinct from carry I'm not sure what that means in futures. If you mean the definition used by Anti Illmanen, which is a long run mean reversion to an equilibrium, then:

    a) for absolute value (reversion to the univariate equilibrium for that instrument) I haven't looked at it because the time scales involved mean it isn't statistically significant
    b) for relative value (say within asset class) then yes, I actually use this myself

  39. By attributions I mean: the contribution of each market to this month's return. To decompose the monthly return into its individual contributors...
    Value I meant some form of measurement of value such as e.g., P/E with equities. But it sounds like you've not looked at this, which is fine.
    Relative value sounds interesting...
  40. easy:


    Value: oh sure, yes I looked at aggregated PE for equity indices and other metrics like yields, quality etc. It sort of works okay but it's quite highly correlated with carry. I also looked at using bond yields, but again they're very similar to carry. In other futures the notion of value is rather abstract - what is value in Corn? The main problem with these strategies is they are big consumers of data for not very much extra return.

  41. Hi GAT,

    I noticed in your instrumentconfig.csv file your Github account that the contract multiplier for KOSPI is off vs Interactive Brokers contract specifications, by an order of x10 (you can click the links). Is this intentional due to some sort of indirect currency multiplier, a mistake or is there something I am missing?


  42. I think you've linked to the mini contract there.

  43. I note that the larger contract has also a different multiplier (250,000), or have I gone wrong somewhere?
  44. Yes, whereas pysystemtrade's multiplier is 500,000
  45. You're correct. For the contract I trade (large size) it should be 250K so out by a factor of two. For interest I verified this by dividing the current contract value on IB page (KRW 76,916,748.05) by the price (KRW 307.666992).

    Github should now reflect this.

    Good spot. An excellent example of the benefits of open sourcing your code (many eyes make all bugs shallow, and all that).

  46. It's a pain whenever the exchange changes the contract specs. Ideally IB should notify its clients trading this.

    Pysystemtrade is great and it's extremely generous that you share this with the world. I learnt a great deal, thank you for that. Out of curiosity: on what aspects would think it falls short of what professional hedge funds are running? Also: being a few years into it, are there design choices you regret?

    Thanks for your insights and generosity again.
  47. It might not be clear but psystemtrade is 'version 2' - Version 1 of my code I wrote a few years ago, and I'm actually still running since pysystemtrade isn't yet a full featured end to end trading system. Most of the design choices I regretted in version 1 I've subsequently fixed (or will fix going forwards as I develop it further).

    I've given a lot of thought as to how to put together the various components of a backtest in a natural way, although to be fair I've seen systems that have 'magic glue' that works better (i.e. if you create a new method of position sizing you don't have to override the class methods of an existing object to make it work); however I think the internals of these systems are more difficult to understand and I prefer the transparency.

    In terms of how it compares to a professional system, well firstly it's incomplete as I've already said. However in its complete form it will fall short in two main areas - tests and robustness (the main difference between amateur and professional software!), and speed (it's all python, whereas a professional system would probably have critical parts rewritten in C++).

  48. Got it. What do you mean by robustness? It's probably possible to speed things up using Cython, but I reckon it's not worth it if you trade a system with long holding periods.
  49. Purely that code written by amateurs is more likely to be buggy than code written by teams of professional software engineers.

  50. How and when did you pick up Python actually? Any recommended courses or books to get from beginner to your level?
  51. When I started in hedge funds I used R, the same language I used in university for my masters dissertation so it was fresh in my mind. We also used S+, Matlab, C++ and a proprietary language elsewhere in the firm. Then the firm decided to consolidate and rewrite everything in python. So at that point I had to learn how to use it. The firm ran a single 2 hour course, and then mostly it was from osmosis (having three software engineers in the team that could already use the language) and the normal way you learn to use any language (googling anything you don't know). Plus at this point I'd already been programming in various languages for nearly 30 years.

    I've only got two python books on my shelf. The 'beginner' book is 'learning python' by Lutz. The intermediate book is the 'python cookbook'. I've never taken a publicly available programming course in my life so I can't recommend any. But I've heard people rave about https://learnpythonthehardway.org/

  52. Do you conduct research using pysystemtraded? I remember you mentioned that research and production should be seperate systems. If not, what system do you use for research? Is there anything that you would recommend for quick prototyping?

  53. Currently I'm using my older system for live trading, but I do plan to replace it with pysystemtrade for both live and production trading.

    I keep changing my mind on the 'separate systems' issue. For the kind of amateur low frequency trading I'm doing I think it's perfectly okay to have a single system (and indeed optimal).

  54. Hi GAT,
    How hard would it be to add alternate portfolio construction methods such as e.g., minimum variance?
  55. Relatively easy, and I'll add it to the 'to do' list.

    However I note in passing that if you equalise asset sharpe ratios, as I prefer to do, minimum variance and maximum sharpe ratio portfolios will be identical.

  56. Also, is it possible to run systems in pysystemtrade with the continuous scaling and without? I am thinking particularly of faster, short-term mean reversion systems which are likely to be somewhat more binary in nature (or at least at a minimum, you don't want to be adding to them if they move against you...)?
  57. All things are possible, although this would require some more re-engineering. This is already on the list - I'm particularly interested in this type of system myself.

  58. Hello GAT,
    I'm roadtesting my recently hacked together code to construct Panama curves, but having trouble recreating your results.

    Taking V2TX as an example I have the following rules defined:

    tenor: m
    # 1 month behind
    carry: -1
    distance: 2
    day_of_month: 16

    these rules are used to calculate the Panama adjusted continuous price curve.

    Comparing the contract date - the index date of my algo and the data in your package:


    Which results in a price curve:

    Your curve with MAs and carry rules give a Sharpe ~ -0.1 whilst my version ~ -1


    From reading your blog I understand that for real trading you are applying discretion in the rolling process - fair enough that my sledge hammer algo. and your data diverge. But how are you determining the roll dates before you started real trading in this instrument? I get similar results for other instruments.

    Many thanks for all the education.
  59. @djames I fear that you have taken an instrument (V2TX) which is not really suited for Panama stitching. Other instruments, such as metals and grains, are much better suited for Panama stitching.
  60. I'm not sure what you mean by that - could you be more specific? I guess in some ways the point is moot as I'm trying to recreate GATs provided Panama data. In the data he provided is he using an algorithm to determine roll dates for contracts before live trading or is he hand picking them?

    If it's an algorithm, I wonder what it is doing as it surely is more sophisticated than mine (which isn't hard to do)
  61. Panama stitching is a method which is best suited to stitch two (or more) price lists which have a fixed offset between the two of them. I have an example for corn (ZC): three contracts with different expiry dates. You can stitch the older line to the newer line by "lifting it up". For example: to stitch the blue line to the red line you would add approximately 7 points to the blue line. Then, to stitch the red line to the yellow line you add approximately 10 points to the red line.
    The second example is for V2TX. Here you see that the price files do not show a more or less stable price gap between the contracts. In fact: the prices are diverging and the gap increases when you get closer to the expiry date. This makes that you would need to do some additional "hand-holding" to stitch these contracts together.
    Although I prefer to use Panama stitching for most instruments as it is an easy solution, for instruments such as V2TX am I doing a manual way of stitching the prices together. The process is then more based on what I think is right (or visually looks right) for that particular instrument.

    @globalarbtrader wrote more about stitching on his own blog at https://qoppac.blogspot.my/2015/05/systems-building-futures-rolling.html
  62. Thanks @HobbyTrading, makes sense. I'm still keen to find out from @globalarbtrader how the data for v2tx (and others) included in pysystemtrade was constructed. Perhaps he is using open interest or traded daily volumes as triggers for rolling in this dataset. I'd prefer to code up as much of this logic as possible for the data in backtest .
    I appreciate that in trading the roll dates become flexible with discretion and passive / active rolling.
  63. I remember him saying that he did it manually.
  64. That is also my impression. But I can't remember where he wrote that.
  65. I seem to have gotten out of the mire by moving further out on the curve, this gives:

    (contract date) - date:
    Panama stitch:

    I think I can with that discrepancy between GATs data and mine, will see if that's foolhardy or not!
  66. @djames, for what it is worth to you: this is my hand-crafted stitch file for V2TX:
    It looks not that much different from the one you have created.
  67. Sorry for the delay, been on holiday.

    Prior to starting collecting live data (~ January 2014) I used a fixed method to determine roll dates; from memory for V2X this would be something like 'trade the second contract roll then trade out the day before it becomes the first contract' (I didn't use volume or any. After that it's down to discretion - but 99% of the time I roll within a week or so of the second contract ageing to become the first.

    Yes the effect should be smaller as you go out the curve as the roll jumps will be smaller.

  68. OK, stupid question coming up.

    I'm about to place my first trade for Eurodollar (GE), buying 2 lots, in my test portfolio i have GE and V2TX, varget vol 15% and 50k account balance. With these params i should be long 2 GE, short 5 V2TX with longer MAs and carry rules.

    But, when placing the LIMIT order through IB, i get a warning "The estimated order value of 490,800 USD exceeds the value limit of 77,701 GBP."

    1. Do you guys set this limit to very high amounts?
    2. Does this limit apply to the gateway as well when I go fully automatic?
    3. Ok, i under stand that i am buying using leverage, but that level of leverage is more than i would have anticipated for nominal trading amount of 50k GBP - can someone shed light for a futures newbie? I think I have been to wrapped up in the systems programming and am having a shock when actually placing a trade.
  69. Best way to get a sense of the dollars n cents, in your demo account, buy 1 lot, then watch the price fluctuations, also look at the margins used. its all very nicely computed by IB
  70. 1. Yes
    2. No, you won't get that warning.
    3. You have to consider the difference between the total contract value and the risk of value fluctuation (the value volatility).
  71. The notional value of a Eurodollar contract is 250,000 * price / 100 hence the size of the order value (because it's a quarter of a years interest on a million dollars). This is probably the largest notional value you will see, unless you trade Japanese Government bonds (for which you need a massive account size).

  72. Hi GAT,
    Is it possible to assign static weights to instruments for specific dates? I know you can assign static weights for all dates but am sure how to set these for e.g., specific years. The reason being, due to the bootstrapping nature of the estimation techniques used, the weights generated lead to slightly different positions over time, so when running live some trades that were previously generated don't appear when run at a later date, increasing the tracking error between live trading and simulated trading. If I assign static weights, due to the different instrument history lengths the system remains 'underinvested' in the past when there are fewer instruments...
    Also, if I input static weights, how does this impact the Instrument Diversification Multiplier?
    One last question, is the IDM the same for all instruments or does it vary by instrument? I suspect it varies by instrument given that theoretically instruments with a greater diversification benefit should receive a larger weight.
  73. Hacky, but can be done: In /pysystemtrade/systems/portfolio.py you'd override the method get_raw_fixed_instrument_weights so it returned a specific pandas dataframe (assuming config.use_instrument_weight_estimates = False)

    There is no underinvestment problem. If you set config.use_instrument_div_mult_estimates=True then an IDM will be calculated that will compensate for changes in the length of price history in the past. This is equally true for static instruments weights, semi-static weights like you want to use, or fully estimated weights.

    By construction the IDM is something that is the same for all instruments. The IDM of a system with a single instrument would be 1. An instrument which had better diversification properties would get a higher instrument weight.

  74. would it be that crazy to reoptimise weights every day as new data is added? is it not better to use all available data as it's available rather than fixing weights statically?
  75. Hi GAT,
    I'm still struggling with this. I noticed with my config that each time I fix the weights, the IDM always shows up as 1 over the entire sample period. How can I fix the weights but allow the IDM to fluctuate?

    My yaml config is as follows:

    method: bootstrap
    frequency: W
    pool_instruments: True
    date_method: expanding
    monte_runs: 100
    cleaning: True
    bootstrap_length: 50
    floor_at_zero: True
    ewma_span: 0
    use_forecast_scale_estimates: True
    pool_instruments: True
    use_forecast_weight_estimates: False #(I'm using my own)
    rule_variations: ['carry', ... etc ]
    instruments: ['NKY', 'NASDAQ', ...etc]
    pool_instruments: True
    use_instrument_weight_estimates: False
    use_instrument_div_mult_estimates: True
    func: syscore.divmultipliers.diversification_multiplier_from_list
    ewma_span: 125 ## smooth to apply
    floor_at_zero: True ## floor negative correlations
    dm_max: 2.5 ## maximum
    method: bootstrap
    frequency: M
    pool_instruments: True
    date_method: expanding
    monte_runs: 100
    cleaning: True
    bootstrap_length: 50
    floor_at_zero: True
    ewma_span: 0
    NKY: 0.0344
    NASDAQ: 0.0349
  76. Are you using the latest version?
    If so, what happens if you look at system.portfolio.get_instrument_correlation_matrix()
    Does it have different correlation matrices inside it?

  77. Unless your data series is really short the weights should hardly change at all. But do it if you want to.

  78. Hi GAT,

    How can one insert a rule into pysystemtrade that results in a long position if greater than e.g., 5 day simple moving average and short if below? This is of course not what I am directly trying to test, but I found it embarrassing that I cannot seem to code such a simple thing into pysystem trade.

    I've tried:

    def trend(price):
    trend = pd.DataFrame(np.where(pd.rolling_mean(price,5)>0,1,-1),index=price.index,columns=price.columns)
    return trend
  79. This is a python / pandas issue rather than with pysystemtrade, but as I'm a nice guy who likes to show off how pythonic I can be:

    trend=price - price.rolling(5).mean()
    position=trend.apply(lambda x:1 if x>0.0 else -1)

  80. Can I check if risk free rate is used in the calculation of sharpe ratio in pysystemtrade? From reading the code it doesn't seem to be. Is there a reason to include it vs not including?
    Can you elaborate roughly how the mean return for futures investment calculated since they are leveraged products?
  81. No futures returns are already excess returns so you don't subtract rf. Any interest on unused margin would need to be added on.

  82. I'm getting an error
    ImportError: cannot import name 'align_to_joint'

    from syscore.pdutils import align_to_joint, uniquets, divide_df_single_column

    I checked the pdutils file but it doesn't seem to have the align_to_joint.
    Does anyone know how to resolve this?
  83. which script are you trying to run? i f i remember correctly the core of pysystemtrade is solid (i'm using parts in my core system) but periphery scripts require some tweaks due to changes made during refactoring. If you do fix them up, please push them back to upstream
  84. Yes, on the 'to do' list is to move all the examples out to a different library apart from the core ones that belong to the instructions.

  85. In function
    get_cash_costs(self, instrument_code):

    Does anyone know the difference between
    value_of_block_commission vs value_of_pertrade_commission ?

    For example JPY futures, value block is $2.46 which coincides with the comm charged by IB.
    wholesystem.py inside mythbusting folder.
    My current workaround is to just comment it out, realized that this script doesn't use those fn
  86. Per trade is a cost per trade, regardless of size.
    Per block is a cost that scales with the number of contracts traded.

  87. Hi GAT, I have a simple question regarding the forecast position of pysystemtrade.

    I'm using close prices and therefore pysystemtrade spits out a position to take for the day of the close price. Is there a function which gives me a position to take for the next day? Or do you use these positions to put on at the open of next trading day?
  88. Exactly that

  89. Hi GAT, Is there a way to dig out the information on trades in pysystemtrade i.e. to evaluate the distribution of trades (not the daily distribution of returns)?
  90. Not in an easy way right now

  91. @AvantGarde perhaps the simplest way right now is to get_position and then process the dataframe to yield trades

  92. Hi GAT,

    you wrote a blog post about using Native IB API. However I don't see the code in your github, what's available is some code that works with IBSwig framework. Do you plan to release this part on github? If you are, will it be a live trading framework or some tests and examples similar to the IBSwig?
  93. If you actually read the series of blog posts they each reference several gists on git hub. A summary is here:


    This section: gists: python-IB-API

  94. On the topic of IB and python, i've found this library https://github.com/erdewit/ib_insync to be a very useful abstraction of the IB API. It is using the IB Python API under the hood.
  95. from systems.provided.futures_chapter15.estimatedsystem import futures_system

    system = futures_system(log_level="on")



    The above code fails at

    W:\GDrive\19\Code\Python\pysystemtrade\syscore\optimisation.py in decompose_group_pandl(data, identifier, pool_costs, pool_gross, backfillavgcosts)
    371 """
    --> 372 assert identifier in data.keys()

    It seems that initially, the system has 38 instruments but when I try to focus on a single instrument, data.keys() contains only "CORN", "EDOLLAR", "EUROSTX", "MXP", "US10", "V2X"

    Is there any reason why this is happening?
  96. Basically there is data for 38 instruments, but the default example only has the subset of instruments covered in my book.


  97. Very late in the day but this seems to be fixed.

  98. It doesn't seem to be fixed on the latest version.
    C:\pysystemtrade>git log
    commit 95ce276859bfe6cf6c5220e052af708b2667976b
    Author: rob@systematicmoney.org <rob@qoppac.com>
    Date:   Thu Jan 4 11:01:40 2018 +0000
        version 16.1
    The P&L for the next day after exchange holiday is still zero.
    >>> system.data.daily_prices("GBP").tail(10)
    2017-12-25       NaN   <--- exchange holiday
    2017-12-26    1.3431
    2017-12-27    1.3440
    2017-12-28    1.3479
    2017-12-29    1.3557
    2018-01-01       NaN   <--- exchange holiday
    2018-01-02    1.3628
    2018-01-03    1.3547
    2018-01-04    1.3586
    2018-01-05    1.3597
    Freq: B, Name: price, dtype: float64
    >>> system.accounts.get_notional_position("GBP").tail(10)
    2017-12-25    44.949005
    2017-12-26    44.469445
    2017-12-27    46.485309
    2017-12-28    48.276678
    2017-12-29    48.615382
    2018-01-01    48.615389
    2018-01-02    51.620324
    2018-01-03    49.690158
    2018-01-04    52.426681
    2018-01-05    56.820880
    Freq: B, dtype: float64
    >>> system.accounts.pandl_for_instrument("GBP").tail(10)
    Calculating pandl for instrument for GBP
    Calculating buffered positions
    2017-12-25       0.000000   <--- exchange holiday
    2017-12-26       0.000000   <--- still zero
    2017-12-27     308.806873
    2017-12-28    1328.743155
    2017-12-29    2992.167840
    2018-01-01       0.000000   <--- exchange holiday
    2018-01-02       0.000000   <--- still zero
    2018-01-03   -3095.374570
    2018-01-04    1488.347216
    2018-01-05     410.335081
  99. Does anyone know if PRICE column of data in CRUDE_W_carrydata.csv refers to settlement price or last price? I'm trying to extend the files already provided.

  100. Try updating now

  101. Hey GAT,

    trying to extend your csv data files, may I know if PRICE column of data in CRUDE_W_carrydata.csv (also all the futures data file) refers to settlement price or last price? I'm trying to extend the files already provided.

  102. It's the price IB provide with the historical data function set to get daily prices. I think it's settlement, but not sure.

  103. I think that IB only provides OHLC bars, so it would be the close price if that is what you filter out. I'm not sure whether IB even provides settlement prices.
  104. Do you know if the futures all close at the same time?
    Does it depend on exchange ?
    Does it depend on instrument?
  105. I numbered your questions for simplicity:
    1. Not all futures close at the same time.
    2. Yes, it does depend on the exchange.
    3. Yes, it does depend on instrument.

    Also, if IB is your source of data and you use daily OLHC bars, you should be careful what you request: either the full day (which might be almost 24 hours, depending on instrument). Or a time period which IB calls "liquid trading hours" or "regular trading hours". These time periods are defined for certain instruments, and depend per instrument.
  106. On a daily price level my research indicated that it is irrelevant whether we use close or settlement price.

    I recreated the provided csv files using Quandl sourced data (sometimes stitching GATs as he has more history for some instruments than that which Quandl provide) - the backtests were comparable. But don't take my word for it!
  107. My experience is that you have to be careful with data obtained from Quandl. In some cases do they provide open-high-low-close data, whereas in other cases they provide open-high-low-settle data.
    The difference between using settlement prices versus daily close prices might be not large for a system which reviews once per day. But if your system works in a different way (e.g. once per hour) it might have an influence.