The accuracy of trading performance back testing

Discussion in 'Strategy Building' started by bzinchenko, Feb 16, 2008.

  1. Even beginner trader is well aware how important is testing of trading strategy on historical price data to estimate its efficiency. Everybody who tried to write own strategies in Metastock® or Trade Station® was able to watch performance charts of initial capital growth throughout the period of strategy execution and also everybody observed typical statements: “past performance is not a guarantee or a reliable indication of future results” which typically accompany performance charts. One can wonder why at all to pay so much attention to quantities without any exact reliability. The simple answer tells: because there is nothing better to measure trading prospects.

    People with basic knowledge of mathematics would surely remember to mention probability theory associated with stock market modeling and high degree of error associated with market trend forecasting. This serves as a root to numerous strategies of risk management, which seek to achieve the balance between profits from aggressive trades and associated losses in case when the market direction estimate is incorrect. However, in many cases such estimates appear also wrong.

    There is simple and evident proof for such a high volatility of market forecasts. Market itself is extremely complex system and to date there is no proven theory to explain it in any finite model allowing for easy trend calculation. The most risk estimates suppose that market behaves as some black box system with associated plain probability measures such as moving averages and variances approximating error limits. Nevertheless, there is quite generic evidence that these criteria appear inadequate in most market conditions.

    This phenomenon is due to multiplication effect of market influences. Consider long pipeline transporting raw oil. Suppose every part of pipeline can suffer accident with some known probability. It may seem that the ultimate probability of pipeline fault as a whole will appear as a simple sum of associated probabilities for every fragment of a pipeline. However, it is not the case because the fault in every pipeline segment would cause the fault of the pipeline as a whole. Therefore, cumulative probability would appear not as a sum but as a product of associated probabilities. In mathematical terms, it will mean that there will be severely violated the conditions of so known Central Limit Theorem, which defines the statistical variance as a reliable measure of probability distribution associated with the value. It reveals that the most risk management strategies implemented worldwide are not reliable.

    Only few mathematicians with good background in statistics can understand and rigidly prove this point. However, one should not be mathematician to observe its real world manifestation in abrupt huge losses of trading institutions and world market instabilities, which, by far, are not the results of errors of some evil minded individuals, as often told to wide public, but are due to fundamental stochastic laws which rule the modern market.

    There exist special mathematical techniques devised to diminish the influences of cause multiplication factor on the accuracy of ultimate statistical estimates. Known as nonparametric statistics, they are well described in many statistical textbooks. The simplest of such measures is median point of distribution, which acts as a reliable nonparametric replacement of the average value. Another measures concern special statistical hypothesis testing techniques aimed to estimate reliability in cases where regular variance looses its sense as illustrated above.
     
  2. You may understand mathematics...
    But you have no clue how pros make money in the financial markets.

    Your entire post is predicated on the assumption...
    That pros predict market future movements...
    Make directional bets... and profit from this.

    FALSE.

    Virtually ALL pro trading firms...
    Practice some combination of market making and risk arbitrage...
    Within a very well-hedged, market neutral framework.

    Last year I made 100,000 trades... and not a single directional bet.
    I don't care which way the markets go next week... it does not matter.
     
  3. 400 trades a day?

    LC
     
  4. You think that 400 trades/day is a lot?
    It might be typical of a TEENY, TINY 2-3 person trading firm.

    Or a single highly automated trader can do it...
    Using a lot of custom software...
    Because every trade has to be a quality decision.

    I would guess that within one year...
    As my Automated Systems evolve...
    I will be pushing 1,000 trades/day...
    Perhaps 50% of them Automated.

    But the point is...
    EVERY time I bought stock... I shorted an equal amount of something very similar.
    So there is ZERO effort put into determining market or individual stock direction...
    All effort goes into staying as market neutral as possible...
    Which is not trivial.

    This type of market neutral scalping is very profitable...
    ESPECIALLY is in a high volatility environment like we saw last year.
     
  5. Bob111

    Bob111

    DeeDeeTwo at work-

    [​IMG]


    :) :) :)
     
  6. Hound, Are you agnostic on the volatility of the positions too or is that where your edge lies?
     
  7. I just wanted to draw attention to inherent instability and low reliability of common technical indicators and strategy testing. Surprisingly, it impacts both very complex arbitrage models of market makers who trade huge hashing and also ordinary traders with typical industry charting.

    Multiplication effect impacts everybody on the market just because it is in the ground of market itself and is as strong as nature. Though the most of wide spread technical indicators are just not capable catching these effects. In other words, trading on averages with system which does not have average is very same as just bling guessing.
     
  8. Your statement does not make any sense. Fist of all, cumulative probability is defined as P[X <= x] meaning that it is the probability of a result x less or equal to X.

    Now, you confuse probabilities. There is the probability of a flaw in the pipeline and the probability of failure of pipeline. The latter is conditional and it is always the certain event given the former for certain types of faults.

    The rest you say about money management are conclusions disconnected from your argument and make non sense. If you like you can state again your point in such a way that it makes sense for traders in this forum.

    Bill
     
  9. When you hedge a long position...
    With a "very similar" short position...
    Both sides of the hedge will have similar volatility...
    Or it's not a good hedge.

    More specifically...
    I'm talking long basket vs short basket...
    50 stocks long vs 50 stocks short.

    But within the baskets are 50 pairs...
    Because each long position is specifically matched to a short position.

    If this is done WELL...
    It's VERY market-neutral...
    And volatility-neutral...
    Because both baskets will exhibit similar volatility.

    If you are set up this way...
    You do not care which way the market goes... or how volatile it gets.

    So how do you make money?

    There are only TWO (2) ways to make money trading:

    (1) By exploiting market inefficiency

    In terms of above example...
    You are using quant analysis to set up pairs that are underpriced/overpriced...
    And will revert to mean over time (hours or days).

    (2) By exploiting the bid/ask spread

    In terms of above example...
    You are hi-volume scalping your way into positions and scalping your way out...
    Using automation and algos as much as possible.

    Volatility is your friend.
    A choppy environment increases both #1 and #2...
    Pricing becomes more inefficient...
    And your scalping volume goes up.

    This market-neutral framework is not some big secret...
    But a classic approach...
    And a variation of this is used by virtually every pro trading firm.

    But unlike staring at charts...
    It's VERY HARD to do well enough to be in the Top 5%...
    That can make a living from a Zero Sum Game.
     
  10. As a rule, we cannot accurately deduce the exact probability as a measure over a set of elementary events. It means that we hardly will ever know particular probabilities associated with each particular trade. Useful results appear because we are typically interested in some generic functions defined over a set of elementary events which little depend on the exact probability distribution.

    This more realistic notion of volatility is closely related with the Gaussian normal distribution of probability. Gaussian random value typically appears as the result of many loosely related impacts of approximately equal magnitude as follows from Central Limit theorem. Gaussian distribution has remarkably easy representation with just two characteristic parameters, mean value and variance against this mean value (second moment of distribution). This Gaussian approach is the ground of the vast majority of modern trading and risk management models. However, everybody knows that market events are closely related, which implies that the real distributions will considerably deviate from Gaussian normal law.

    It is well known that a random value can have peaks. In most practical cases, Gaussian random value is estimated to have the maximum magnitude equal to its triple standard deviation. Such approximation is very robust in routine least squares calculation of experimental results in laboratory, but it can have dramatic impact in trading experience. On a relatively short trading session, one can safely use traditional Gaussian risk measures. However, on a long time prospect the probability of a peak deviation will inevitably increase in a nonlinear proportion. Practical result of such a peak deviation will be typical bankruptcy or, at best, heavy losses.

    Probability of abnormal peaks can considerably increase in case of deviations from the Gaussian probability law. For instance, lognormal distribution, which appears in cases when logarithms of random value have Gaussian distribution, will have much more often and adverse peaks. Such distributions with long probability distribution tails must be typical in real market conditions because of close correlation and multiplication effect of market events.
     
    #10     Feb 17, 2008