Fully automated futures trading

Discussion in 'Journals' started by globalarbtrader, Feb 11, 2015.

  1. Both open and close
     
    #251     Jan 16, 2016
  2. About 4K was Oil, but there was more to it than that.

    Here are yesterdays moves, fifth column is move in standard deviation terms, next column is my signal:

    Code:
    Headline chanages: (norm price change>1 sigma)
           code  last_price datetime_last_price  last_norm_return  rawsignal  rawsigchange  multisignal  multisigchange
    28      MXP       0.055 2016-01-15 19:01:25             -2.46      -1.21         0.063        -12.6          -0.273
    24      AUD       0.684 2016-01-15 18:53:32             -2.30      -1.51        -0.135        -15.8          -0.062
    34  CRUDE_W      36.335 2016-01-15 18:10:44             -2.22      -3.00         0.000        -37.6           0.535
    20      CAC    4179.000 2016-01-15 16:15:39             -2.21      -1.37        -0.145         -9.4          -0.005
    19      AEX     399.450 2016-01-15 16:14:14             -2.17      -1.23        -0.218         -8.5          -0.007
    26      GBP       1.426 2016-01-15 18:57:16             -2.13      -2.52        -0.061        -26.3           0.071
    37  EUROSTX    2934.500 2016-01-15 16:17:01             -2.13      -0.68        -0.034         -0.0           0.000
    22   NASDAQ    4100.375 2016-01-15 17:48:25             -2.10      -1.56        -0.214        -10.6           0.249
    2   LIVECOW     117.375 2016-01-15 18:18:59             -2.01      -0.92        -0.047         -9.6           0.168
    21      SMI    8009.000 2016-01-15 16:18:26             -1.99      -0.23        -0.235         -1.6          -0.004
    23    SP500    1857.875 2016-01-15 17:50:05             -1.94      -1.19        -0.201         -8.1           0.199
    18    KOSPI     228.575 2016-01-15 04:42:30             -1.79      -1.74        -0.214        -11.8          -0.124
    30   COPPER       1.943 2016-01-15 19:04:54             -1.10      -2.53        -0.069        -31.6           0.119
    
    8       BTP     138.445 2016-01-15 16:02:19              1.07       1.69         0.225         15.0          -0.253
    10      OAT     151.690 2016-01-15 16:05:05              1.15       2.09         0.209         18.6           0.037
    7      BOBL     131.515 2016-01-15 16:00:55              1.21       0.75         0.044          3.3          -0.001
    11    SHATZ     111.642 2016-01-15 16:06:27              1.28       1.07        -0.001          0.0          -0.000
    15      US5     119.996 2016-01-15 18:46:53              1.36       1.76         0.139          3.9           0.011
    12     US10     128.445 2016-01-15 18:41:25              1.46       1.37         0.132          3.0           0.005
    14     US20     159.266 2016-01-15 18:45:02              1.48       1.00         0.213          2.2           0.007
    36  EDOLLAR      98.303 2016-01-15 19:14:24              1.52       1.84         0.040         22.9           0.025
    16      V2X      27.750 2016-01-15 15:19:34              1.97       0.38         0.003          3.0          -0.007
    13      US2     109.090 2016-01-15 18:43:09              2.06       1.93         0.003          4.2           0.011
    
    Wrong
    25      EUR       1.096 2016-01-15 18:55:23              1.13      -0.11         0.325         -1.1          -0.434
    17      VIX      23.725 2016-01-15 18:51:46              1.82      -0.53         0.055         -4.2          -0.002
    1   LEANHOG      77.975 2016-01-15 18:17:23             -1.07       0.23         0.011          2.4          -0.041
    
    
    So it was more a case that all the stars were aligned in the right direction. Of course this could equally go the other way...

    GAT
     
    #252     Jan 16, 2016
    taq likes this.
  3. I've now added the ability to estimate forecast scalars using a rolling window to the code for pysystemtrade.

    GAT
     
    #253     Jan 18, 2016
  4. Hi GAT,

    Looking at your above formula, it is not clear to me what you mean by L/4 smooth - can you elaborate? Why 4 and where does the L/4 smooth enter the EWMA? I assume you then volatility standardise this EWMA.


    Also, in order to obtain forecast scalars that are on average equal to the absolute value of 10, you mention that 'you need to backtest the behaviour of the rule with real data to find average forecasts'. Is this backtesting conducted for each individual instrument in isolation of the others or across all instruments? Depending on what it is, the outcome at the instrument level wil be different:

    i)if it is at the instrument level in isolation of other instruments, each instrument will have a forecast scalar equal to the average absolute value of 10, which ultimately leads to all instruments having an average absolute forecast scaler of 10

    ii) if it is at the portfolio level across all instruments, the average across all instruments will equal 10 but the average absolute value for individual instruments will not necessarily equal 10
     
    Last edited: Jan 18, 2016
    #254     Jan 18, 2016
  5. It's probably easier to show you my code.

    Code:
    def breakout(x, ws):
      max_x=pd.rolling_max(x, ws, min_periods=min(len(x),int(ws/2)))
      min_x=pd.rolling_min(x, ws, min_periods=min(len(x), int(ws/2)))
      sig=[breakout_one_row(idx, x, max_x, min_x) for idx in range(len(x.index))]
      sig=pd.TimeSeries(sig, index=x.index)
      sig=pd.ewma(sig, span=int(ws/4.0), min_periods=int(ws/8.0))
      return sig
    
    def breakout_one_row(idx, x, max_x, min_x):
      r_px=x[idx]
      r_min=min_x[idx]
      r_max=max_x[idx]
      return 4*(r_px - mean([r_min, r_max]))/(r_max - r_min)
    
    
    
    
    I'd do it by pooling across instruments (by coincidence just blogged on this http://qoppac.blogspot.co.uk/2016/01/pysystemtrader-estimated-forecast.html)
     
    #255     Jan 18, 2016
  6. nemo4242

    nemo4242

    Hi, you write on your blog about scalar estimation 'The odd one's out are V2X (with a very low scalar) and Eurostoxx (very high) - both have only a year and a half of data - not really enough to be sure of the scalar value.'

    I downloaded data for the Eurex Eurostoxx 50 from Quandl (https://www.quandl.com/collections/futures/eurex-euro-stoxx-50-index-futures) going back to Sep 1998. Are you talking about a different contract?

    This thing shows some peculiarities. Here is a plot of it from my system:
    estx50.png

    Settle is the panama adjusted settle, OSettle the 'original' settle price.
    Carry shows the difference to the next contract, adjusted to annual (In this case multiplied by four, since ESTX50 has quarterly contracts).

    Do you have any idea why the March contract shows those regular spikes?
    Seems to be really consistent over the years.
    Other contracts like DAX, SMI don't show this, the CAC40 also looks strange.
     
    #256     Jan 19, 2016
  7. Over time Quandl have been backfilling their data. When I first tried to get this from quandl, a couple of years ago, they didn't have any Eurostoxx data. So I had to just use the IB data.

    At some point I'll add this extra data.

    The carry 'bumps' are due to the different timing of dividend payments. It affects different contracts to different degrees. This is a well known effect and not a problem with the data.

    GAT
     
    #257     Jan 19, 2016
  8. nemo4242

    nemo4242

    Thank you! Do you have some more information why ESTX50 is so strongly affected by this?
    The DAX is a performance index (includes dividends), is this why it doesn't show that pattern?
    There is probably no easy way to use this for trading?

    Another thing: Your code for the breakout rule above uses Python list comprehensions along the series axis.
    I think this horrendously slow and not really needed.
    You can write it fully vectorized (loops in pandas/numpy C code) like this:
    (stochastic here is scaled to -0.5 .. 0.5)

    Code:
    def stochastic(data, column, length, smooth):
        roll_max = pd.rolling_max(data[column], length, min_periods=length // 2)
        roll_min = pd.rolling_min(data[column], length, min_periods=length // 2)
        stoch = (data[column] - roll_min) / (roll_max - roll_min) - 0.5
        if smooth > 1:
            return pd.ewma(stoch, span=smooth)
        else:
            return stoch
    
     
    #258     Jan 19, 2016
  9. Yes the DAX includes dividends; so the effect doesn't apply. I think the yield on the Eurostoxx is quite high. I guess also there are more firms paying semi-annual dividends (quarterly divis rarer in europe than the US).

    Thanks for the code tip. Speed isn't a concern with a daily system, but it's always nice to have more efficient code even for the sake of it.

    GAT
     
    #259     Jan 19, 2016
  10. nemo4242

    nemo4242

    About forecast scalars:

    I am using artificial random data to estimate the scalars and correlations between rules.
    For example, I created 1000 different random walks, each 100000 bars('days') long and applied trading rules to them:

    Code:
                                    count  mean  std  min  25%  50%  75%  max
    scalar_ewmac256  1000  1.628219  0.070967  1.420118  1.581897  1.626521  1.678534  1.827633
    scalar_ewmac128  1000  2.301430  0.071873  2.092129  2.253581  2.300945  2.349642  2.524300
    scalar_ewmac64   1000  3.262073  0.073323  3.039892  3.211492  3.261055  3.313205  3.511254
    scalar_ewmac32   1000  4.639525  0.075807  4.417496  4.589838  4.636334  4.694308  4.875523
    
    The 50% value is the median along the 1000 tries (data is from the pandas describe() function).
    ewmac256 means the EWMA 256/64 rule and so on.
    The medians are lower than the values that you give in your book (1.87, 2.65, 3.75, 5.3), probably reflecting the fact that the artificial data has no trends.
    But even with 1000 tries of 100000 days, i.e. about 400 years, you can see the error in the values is still significant.
    I wonder if it is really possible to get stable values from real data where we only have maybe 40 futures and 50 years of data.
     
    #260     Jan 19, 2016