The missing feature from every back tesing platform

Discussion in 'Programming' started by fan27, Nov 4, 2018.

  1. fan27


    All back testing platforms or methodologies I have seen involve some sort of testing (AI, ML, if then else, etc) on historical data and perhaps followed by some sort of walk forward and or live trading analysis. The result is one to many strategies to trade. Let's say I have data from 2000 to 2018 and let's also say I tested on several sub data sets and I have 18 years worth of simulated results. The results are flawed because the earlier results are from strategies that were formulated after that slice of data. For example, if we were to go back in time to 2010, the group of strategies that would pass testing and be selected at that time would be different than the strategies that would pass testing and be selected today.

    Here is a more realistic approach:

    Determine the minimum amount of data you need to test with and split it up and test accordingly. Let's say it is 10 years (2000 to 2010) and we arrive at a group of strategies. In simulation, we trade those strategies for the first quarter of 2010. At the end of the first quarter, we now perform the same analysis on the expanded set of data and perhaps wind up with a different set of strategies. Those strategies are traded in simulation for the next quarter (I am arbitrarily picking a quarter. Could be another time frame). To me, this seems like a true simulation. Is anyone else doing anything similar or know of any research platforms that can do this?
    d08 likes this.
  2. tommcginnis


    If you assume stationarity of data, then the simplest of trend decomposition works, and your forecasts can be whittled to where 'yt - y^ ≡ ε' ...("Yayyyy!")

    But as long as you have non-stationarity, you *must* examine errors for systematic content that can be discerned and then accounted for in a new model specification.

    It's not so much whether/how you account for changes of fit, but simply that you recognize that it is going to occur. One person's 'noise' can easily be another person's channeling trend.

    "Love your local maxima."
    fan27 likes this.
  3. fan27


    Yes. This is my point. Financial markets do not involve stationary data. "New model specifications" should be created based on new data. My position is to walk the simulation forward in time and re-calibrate based on the "new" data. Being able to do this is at the top of my list. It would not be that hard with my platform but it would take way to long on my laptop. Currently, it takes about 4 hours to look for edges, execute various order combinations and then take the "best of breed" to construct a portfolio of strategies. This is all automated. Assuming I use 2010 as my starting point, 2010 is 27 quarters of data so the time on my laptop is 27 x 4 = 108 hours. Once I go live with my initial set of strategies, will start running on AWS (64 core instance is only $3.20 per hour).
    tommcginnis likes this.
  4. tommcginnis


    Being a cheap-ass (:)D)), I would suggest you go in the other direction:
    set a minimum-n sample size, of the most current data, and run your analyses.
    (Let's say, 6 months' worth.)
    Then run it at 1 year, and see the effect on PnL and risk (and the whole gamut of system checks, right!?!.
    Then at 2 years, then 3, etc.

    Unsurprising (when you think about it), the persistence of exponential models in time series is going to suggest that the benefit to you of going waythehell back in time, diminishes to where there is no net benefit. Where is that point? I don't know but my instinct would be wherever there was a major regime change -- An election, a FED 180° on policy, March 2009 -- things like that which will have a decreasing impact over time.

    FWIW, I think 2010 is wayyyyyyyyyyy back there, and certainly marks your studies as "complete" as far as being useful for forward-looking money-making. Way big :thumbsup::thumbsup::thumbsup:.
    fan27 likes this.
  5. fan27


    This exactly the question I will be able to answer with my proposed approach of walking forward strategy portfolio construction at set intervals.
  6. Metamega


    I personally don’t see the point except for making a pretty compound equity growth chart.

    The point of backtesting is to find an edge for today’s market. So if an old system doesn’t work anymore, why bother with it’s past results.

    Theirs so much possible variance in past results vs future results plus more possibility of overfitting, I can’t see the value myself.

    Amibroker has a built in walk forward optimization where you can set your start/end dates and frequency to optimize. I’ve never used it myself except to try it but for the stuff I test it hasn’t really shown me any reason for it. Perhaps with the right rules to optimize theird be some benefit.
  7. Handle123


    What I find amazing in my experiences, every ten years I have evolved to think in different way. What I thought was important forty years ago is of no importance today, and what is important today, I didn't even know of it then. I have come to realize it is all luck of being on board of trends, but what I know today, I will eventually be onboard without cause much losses finding them from day trading to very long term trading. What is not luck, is risk management, when to not take signals based on previous charting or TA, to managing ongoing position. So now I only back test concentration of risk management, how close to zero drawdown can I get. Scalping and options for signals, backtesting gives clues whether entries help.

    Duration of time I seek the "mean" whether trending or chop or retracements.

    I don't use commercial platforms any more, too much missing from them imho, like you can't have it identify reports and build systems off of them at least when I had started to get mine developed. I have found electronics have aided in far more than just using charting and moving averages, love the challenges.
    comagnum likes this.
  8. RedDuke


    Not sure what you mean. There is no coding limits in what you can do in NinjaTrader. It fully supports c#. The only limitation is that it does not work with options.
  9. fan27


    Exactly. That is why all new quant hires at Renasainse or Dalio's hedge fund are equipped with the latest version of NinjaTrader.
  10. RedDuke


    This is a dumb post. I can write in C#, Java, R (to some degree) and etc. The question was asked and I answered. C# gives one full flexibility to write what ever they need (apart from HFT).

    Renasainse, btw, developed their own language to write all their code.
    #10     Nov 4, 2018