I was thinking about two types of parameters one can use in a model. Open-ended parameters like SMAs or percentage-based filters, where the value used can be anything from zero to infinite and those like percent, where the value can only be within a certain range, like 0% to 100%. Are either of these two types more susceptible to curve-fitting? It would seem that you could say the first type is because the ability to calibrate the value of the parameter to historical data is nearly infinite, so someone could come to the conclusion that the 53-day SMA with a 13.2% volatility filter were the optimal values for an entry strategy, whereas with the second type, your ability to fit to a curve is limited by the range the variable can take on, meaning you'd be better off basing a model on the second type, to the extent that you can. I'm just thinking out loud a bit, so if this simple comparison and conclusion is flawed, I'm happy to hear why. I realize that you'd kind of have to ignore the potential for infinite subdividing of the range-constrained parameter, so that you don't end up with a value like 15.898798798% as your model input.
Open-ended parameters are contrained by maxbarsback. If you have 4,000 bars in a file it does not make any sense to use a 4001 bars sma. So there goes you otherwise nice try.
OK, so there is a practical difficulty, in some cases. Does that mean that the distinction is invalid and that there really aren't two types of parameter here? I suppose on the most macro level, if a market has been traded for 50,000 days, it can't make sense to use the 50,001 SMA, so not only would there be data limitations on the parameter values, there would be historical limitations as well. It still seems intuitive (which isn't always correct, obviously) that the fewer values a parameter can take on, the less susceptible to curve-fitting the model would be, which would mean that binary parameters would be the least likely to be curve fit, which seems correct.
You have to look at the sensitivities of the changes in parameters values and not at the range of values. There are infinite real numbers between 1 and 2 and between -100 and +100. Ranges mean nothing. Sensitivity is important. You have to look at the partial derivative (where is that moron quant by the way?) of the objective wrt that parameter as a function of time. It is a nasty problem but it can be done numerically using polynomial fitting.
You had me until you mentioned needing to look at it as a function of time. Sounds like you are saying the less sensitive, the better, unless you want to use polynomials. Personally, I do not use them, with one exception.
LogicMan: I know relatively little about advanced statistics and the science of modeling. However, I have worked with Neuroshell for a number of years ( a neural networking program for the market). I have found that range limited items work better for longer modeling periods. If I look at the change in the Close over a period of years, as the price of an instrument climbs, the network sees a $1 change in a $50 stock as different than a $2 change in the same stock when it is at $100. However, if I look at the percent change in the close (2%), the net sees the same value. Thus, range limited info, for longer-term models, seems to get more robust results. Short-term modeling does not suffer to the same extent. At least this is true in the models I can develop. More sophisticated modelers might not find this to be true in their models. This might not apply to instruments, which by their nature are more range-bound, like short-bonds. But I've never tried modeling them. . Jack
No, a curve-fit is a curve-fit. With a 0-100 integer (not floating point) range on one parameter, combined with 3 more you get 100 million possible combinations. That's a lot of room for a curve-fit with only 4 parameters. Even using 3 possible values for each will give you a curve-fit. The idea is to have other statistics that backs up a possible fact that what is found is a fundamental market rule, and not a curve-fit. How that's done is a different matter. If you are not using integer values in your example (like you said 15.something%) then the ranges 0-100 or 0-100000 really don't matter since floating point value is just that, an almost unlimited number of numerical representations in 8 bytes of information. In other words, it doesn't matter if there are constraints. It might just happen that a value of a parameter at 22.22222 % produces a great looking model, but 22.22223% doesn't. That means it's just a "magic number" that's there due to luck, not real substance (same thing alexandermerwe mentioned).
OK, let me phrase the issue slightly differently, if you don't mind. If I have a one parameter model that can take a value from 0% to 100% (e.g., the parameter measures the odds of some other thing happening) and the outcomes when that parameter value is from 0% to 50% are negative (on average) and from 51% to 100% are positive (on average), is that likely to be a "fundamental market rule" or is that curve-fitting and it is likely to be the case in the future that even those scenarios where the parameter value is from 51% to 100% will be negative, thus making the model's overall value negative? I realize that perhaps the rule could be more finely cut and the likely outcome might be negative all the way up to 50.4999999999% and the positive outcomes actually start at 50.5%, but let's just keep this example a bit more simple than that. So, I'm not selecting a single value for the parameter, but I am defining a set of values within the parameter's allowable range which I would know from my historical data would lead to a negative outcome on average, meaning that now I have myself a viable way of filtering out trades as undesirable as well as a way of identifying positive expectancy trades, based on the one parameter. Or, at least, is it less likely to be curve fitting than saying that 50 is the ideal parameter value for a moving average which could range from 1 to almost any number? Because while you can identify a range in the example above, you can't say that any moving average in a range of 0 to 50 will work as your parameter value, it has to be one number to drive the model. It's either 50 or it's something else, whether that be 49, 51, whatever. This is the distinction I am trying to articulate, now that I think about it more.
I think you ignore the fact that any indicator can be forced to be range-constrained between some fixed numbers.