Proper back and forward testing

Discussion in 'Automated Trading' started by Allistah, May 4, 2013.

  1. Allistah

    Allistah

    Very interesting. Good to know that you like Tradestation. It is going to take me some time to get used to the platform.

    One thing that I'm trying to learn is.. How do you go from idea to live trading? From the little time I've had with Tradestation, it seems like you need to backtest it on a small amount of time like a month or something for intraday bars. Then once you get it to work there, expand it a bit then keep adding until you get to 5 years. What I have found is that when you're working on ideas, you don't want to start with 5 years because it would take forever to optimize and backtest. Start with smaller times at first to see if it even works. Then go to larger times. Does this sound like something you guys do when making a new strategy?
     
    #11     May 4, 2013
  2. ofthomas

    ofthomas

    they can't handle FOB... but they are pretty good though, I find it a bit hard to use which is why I got my money back after I tried it for 30 days...

    my only experience with tools handling FOB (or the whole data stream as you call it) is with in-house custom apps or CEP tools like APAMA or OneTick, I have deployed architecture for both and I have used OT... but now you are talking about a subscription feel per core for PROD $$$$ (DR/UAT/Dev are free) ... but it enables you to do real time analytics and queries, come up with a model, and implement those within "minutes" per say... so TTM for models is reduced significantly...

     
    #12     May 4, 2013
  3. The built-in optimizer is definitely slow, but I don't use it with new ideas. I just manually play with the parameters for about an hour, doing backtests over 5 years (which take about 2 seconds on my laptop). This gives you great insight on the stability of the strategy. If its equity fluctuates all over the place as you tweak the settingz, it's unstable and should be discarded. However, if you can get it to work with many different parameters the likelihood is high that it will survive in the wild.

    And my profit factors are coming in at about 1.5-1.8, just like the strat that dom993 posted. I can hit around 2.5 with manual trading, using "purely subjective" (Quant-repulsive) methodz

    You want to build an old fashioned pushrod V8 engine, not a turbocharged rotary that will blow if the boost controller fails, or Ferrari flat-12 that will lose compression if it ingests a bug :eek:
     
    #13     May 4, 2013
  4. Allistah

    Allistah

    Thanks for the reply, this is helpful. Can you tell me what sort of things you've had the best luck with? I hear such a mixed bag of opinions from people like what works and what doesn't. Can you give me an overall type of opinion on what sort of things you've had the best experience with? If you were starting over right now what would you want to tell yourself to get yourself going in the right direction?
     
    #14     May 4, 2013
  5. Trend following has been my main approach, and some range algos have worked as well. One entry, no averaging in.

    Edit- what would I do differently? I'd get tradestation sooner in my career, trading stocks only (no options) until I really had an edge, but the overall approach would be the same. The Market will always have a signal for you to harness, because there will always be hedgers, suckers, and freaked out traders who create tails.
     
    #15     May 4, 2013
  6. Allistah

    Allistah

    Since you're doing trend following mainly, do you have items that you need to optimize for that? Or is it just using price action and not any indicators to follow the trend?

    I also heard someone else say that your targets and stops need to be dynamic.

    Trying to get something that works for me is pretty overwhelming.
     
    #16     May 4, 2013
  7. Knowing what to use and what to optimize will come with experience. If anyone shows you their secret recipe, it is probably already obsolete or soon to be, depending on how many beginners try to use it in The Real Market.

    :)

    I just finished polishing this one off. This was based on something I thought I saw as I watched the index every day, while trading stox.
     
    #17     May 4, 2013
  8. Allistah

    Allistah

    I've had a number of strategies that have worked for a while and then they stop working. Should you try to make it continue to work so that it keeps going or do you just scrap it and start over with something else?

    I would think that you would want to develop a system that has worked for at least 5 years in the past and also does well with walk forward testing. You'll never know but I would think it would continue to work for a while after that.

    You also touched on an interesting point. If multiple people start using a system will it cause it not work anymore? Is this why when people have a working system that they guard it with their life (so to speak)?
     
    #18     May 4, 2013
  9. Exact thing happened to me when I optimized the "price beetle" on one year of data. It instantly failed the forward test. :eek:

    :cool:

    If too many night clubz open in a small town, what would cause their business model to stop working?
     
    #19     May 5, 2013
  10. AmiBroker is able to do that.
     
    #20     May 5, 2013