How much is an ATS worth?

OK how do you overfit 1000 data points with, say, five parameters?

 

For starters, you have to agree a bit about what it means to be overfit.
It doesn't require matching every single point of the data series precisely.

Suppose I had an model that was able to fit 70% of the directions in hindsight;
would that be overfit? What if I had a model that had a terminal wealth that was 80% greater than the underlying series, but a low hit-rate? would that be overfit?
If you agree that both of those scenarios are examples of overfitting, then what I'm saying is that it's very possible to overfit. And it does not necessarily even require 5 parameters, let alone 1000.

 

" Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations." - Wikipedia

 

Unfortunately, you are pasting a 'general' definition of overfitting from Wikipedia without really trying to understand the points I made.

If you think overfitting a 1000 pt data series requires 1000 parameters, I'm sorry to inform you that's wrong (even with the wiki definition you pasted).

Sorry to sidetrack the discussion.

 

My point in part is that there is no "objective evaluation" of an ATS (see my earlier posts). Its value, if any, depends highly on who developed it and how, and that ties into the issue of overfitting.

And as you point out I'm not interested in buying an ATS, and I'm unlikely ever to sell one, for the aforementioned reasons. But I do feel some obligation to post here in case some poor, uninformed fellow comes on ET wanting to buy an ATS; they should know that the vast majority of ATS purchasers simply get fleeced.

 

The point of the thread is to come up with an objective system of evaluation that would benefit buyers and sellers alike. A fiat declaration that it can't be done is unacceptable, at least to the OP.

And I don't know of any OPM market where a system developer would take his system for ready cash infusion.

I've already posted a method of pricing in the winrate. It may be wrong but it's a starting point for others to evaluate.

Looking forward to others contributing ways of pricing other performance factors.

 

Could be done with one.

 

Here's an excerpt from the 2011 interview by Futures magazine of William Eckhardt:

FM: How do you ward off curve-fitting?

BE: What most people use to ward it off is the in-sample/out-of-sample technique where they keep half their data for optimization and half their data for testing. That is an industry standard. We don’t do that; it wastes half of the data. We have our own proprietary techniques for over-fitting that we actually just improved on a year ago. It is important to test for over-fitting; if you don’t have your own test use the in-sample/out-of-sample [technique].

I can talk a little more about over-fitting, if not my personal proprietary techniques. First of all I like the [term] over-fitting rather than curve-fitting because curve-fitting is a term from non-linear regression analysis. It is where you have a lot of data and you are fitting the data points to some curve. Well, you are not doing that with futures. Technically there is no curve-fitting here; the term does not apply. But what you can do is you can over-fit. The reason I like the term over-fit rather than curve-fit is that over-fit shows that you also can under-fit. The people who do not optimize are under-fitting.

Now the two numbers that most determine if you are over-fitting are the number of degrees of freedom in the system. Every time you need a number to define the system, like a certain number of days back, a certain distance in price, a certain threshold, anything like that is a degree of freedom. The more degrees of freedom that you have the more likely that you are to over-fit. Now the other side of it is the number of trades you have. The more trades you have, the less you tend to over-fit, so you can afford slightly more degrees of freedom. We don’t allow more than 12 degrees of freedom in any system. If you put more bells and whistles on your system it is easy to get 40 degrees of freedom but we hold it to 12. On the other side of that, for us to make a trade we have to have a sample of at least 1,800; we won’t make a trade unless we have 1,800 examples. That is our absolute minimum. Typically we would have 15,000 trades of a certain kind before we would make an inference as to whether we want to do it.

The reason you need so many is the heavy tail phenomena. It is not only that heavy tails cause extreme events, which can mess up your life, the real problem with the heavy tails is that they can weaken your ability to make proper inferences. Normal distribution people say that large samples kick in around 35. In other words, if you have a normal distribution and you are trying to estimate a mean, if you have more than 35 you’ve got a good estimate. [In] contrast, with the kind of distributions we have with futures trading you can have hundreds of samples and they could still be inadequate; that is why we go for 1,800 as a minimum. That is strictly a function of the fatness of tails of the distribution. You have to use robust statistical techniques and these robust statistical techniques are blunt instruments. [They] are data hogs, so both seem to be disadvantages but they have the advantages of tending to be correct.

 

Prove it.

 

What does the normal distribution of 1,800 mean?