Reasons for failur in automatic trading systems

Discussion in 'Automated Trading' started by cohvi, May 10, 2006.

  1. jerryz

    jerryz

    can anyone give an example that shows it's a condition of the market but not an edge?

    for example, let says gap ups are filled 100% of the time within the first 30 minutes. you made 100 trades over the past 2 years to exploit this. is this a condition of the market or is this an edge?

    in this example,
    what is the condition of the market?
    what is the edge?
     
    #31     May 12, 2006
  2. man

    man


    not sure if i completely agree to the part on walk forward testing ...
     
    #32     May 12, 2006
  3. Initial test was done... with data 1 year ago

    You check the market today.

    Does that one year period of no tests matter?

    Edge doesn't matter.

    Conditions do.
     
    #33     May 12, 2006
  4. jerryz

    jerryz

    or is this curve fit?

     
    #34     May 12, 2006
  5. man

    man


    i thought alan meant soemthing different.

    you start in 1990 with a system and optimise it, run it until one month later, reoptimise, run for a month and so forth. that is my understanding of walk forward testing, what you said is what i would call out of sample period.

    my point is that if you do not do the walk forward testing too often with changed (meta-)parameters, it can help you in the bout against miss fit.
     
    #35     May 12, 2006
  6. man

    man

    i think the basic thing starts with the utility function you choose. whenever it is not a return/risk figure you are already flawed.

    random trades produce sharpe ratios above 1.0 in 3% of all cases. thus, if you find a system with a sharpe above 1.0 your chances are about 30:1 that it is not a fit. GIVEN that your underlying systemSearchMethodology is comparable to the random entry test.

    assume that you do 10000 random entry tests, resulting about 300 equity curves with a sharpe above 1.0. now assume you take these 300 and you add a single parameter and run these 300 tests again. will you expect NOW to have 9 tests with a sharpe above 1.0, which corresponds to 3% chance? clearly not, since you already start with nonRandom systems but only those that already show a sharpe above 1.0, so you would expect to jump up to about 50% of all systems yield a sharpe above 1.0. BUT, that is still pure chance, no edge at all around here. nothing tradeable.

    if you think this example is abstract, then ask yourself how often you started working again on something that looked good once? if this test then was already a result of a fit, your derived variation will be too ...

    does it help if you look at the time series itself before you make a system backtest? thus, you are trying to find conditions that predict the market will go up tomorrow and only after you have found them you start a backtest?

    i must admit that i still do not see a difference in principal. why shoudl you not be able to fit this kind of statistic analysis as well by introducing many differnet paramters?

    yet, i am still puzzled with the fact that the kind of systems derived by such test seem to do better. a possibility i am thinking of is that it is more psychological than one might think in the first place. let me elaborate.

    system backtesting is a boring thing. most stuff simply does not work, especially those ideas, that sound super convincing in the first place. so you start playing around, adding something here, tweaking something there and you add up frustration since hardly anything seems to make sense. timeconsuming, frustrating, fitting around. and nothing seems good enough for trading it.

    when you look for edge, things do not need to make trading sense in the first place. just a slight shift in hit ratios is already some inspiring "learning effect". you feel like you start to get what this is all about and you add up effect after effect, still most do not work the way you would wish, but you see slight effects all over the place and you enjoy the search process, thus you do more of it. and after all that edge searching you glue that all together, add some stop here and some pyramding there and ready you are.

    now was that process "per se" a better weapon against fitting? i'd say not in principle, but very much so in terms of psychologically enabling you to dig deeper and deeper.

    sorry ... i think i just wrote that for myself ...
     
    #36     May 12, 2006
  7. man

    man

    what a pathetic long post .... :)
     
    #37     May 12, 2006
  8. cohvi

    cohvi


    1. absolutely not - my system does not have an edge. I go with the flow, wave surfing, jump on the wagon, the more the merrier. As I understand perfectly the success of edge systems, mine is not like that. I recognize traders behavior on the graph and act in accordance with it. With them not against them. I go with the trend so I like the trend to be strong.

    2. As continue to 1. - The thing that makes the system stronger is the fact that it does not take advantage of an edge in the market so as the market volume increases in time, the profits gain. These are the conditions and characters of the market that define the success of the system. Small markets dont work well for me.
     
    #38     May 12, 2006
  9. cohvi

    cohvi


    I'm testing the system in C++ & VB.NET.
    The production environment will be the same.
    The trading system I will use doesn't matter at all, as long as I will get all my inputs in time. From my house it will probably be on top of Strategy Runner, but in bigger scale from the hedging fund I'll interface to what ever system they have. The method is what's important.
     
    #39     May 12, 2006
  10. squeeze

    squeeze

    This post makes me think you are likely to run into trouble sooner or later but good luck with your endeavours.
     
    #40     May 12, 2006