Genetic Optimizers usually work by randomly seeding and selecting populations. You should not expect results to be exactly the same each time. If you truly want to compare apples to apples, you need to brute force all of the possible outcomes. That will of course, truly over-fit the best outcome, but then you can start to try to understand generalizing. If the default optimizer is doing brute force one by one, you'd need to let it complete all possible outcomes to expect the Gen Optimizer results be a subset of those. If they are not included, either 1) you did not run all possible outcomes or 2) there are outcomes or parameters in the Gen Optimizer that are not equivalent to the search space of the brute force method. Either way, trying to match those isn't all that important as trying to understand generalizing and why you need it.
Hi dtrader98, thanks for your reply. It is clearly the second condition you mentioned. Parameters generated by Gen Optimizer is some value in between the scan steps. And the problem is, those slightest changes in parameters, produces huge difference of performance(far more better than the brute force search). So, I guess this in only a sign of over-fit. BTW, what do you mean by "trying to understand generalizing and why you need it" ?
How many parameters are you trying to optimise? I don’t understand the issue here, I get fine useable profitable results from general basic optimisation of a few key parameters. Try metatrader 4, it works for most people really well, but it’s mainly for forex traders
As someone already stated, the default optimizer is not considering the exact same search space that the genetic optimizer is considering. You are 'stepping' over values with the default "brute force" search. Additionally, you may want to consider using the Walk Forward Optimizer instead of the genetic optimizer (unless you really know what you doing); otherwise, you are likely over-fitting.
So, the GA is likely generating random continuous parameters with a granularity less than your step size of the brute force generator. The fact that many of these do much better, shows it is highly sensitive to small parameter changes and thus not likely to perform well out of sample. Not only is that overkill, but shows you that the system you are developing is overly sensitive to its parameters -- generally, not a good thing. Generalizing is finding a way to reduce the complexity of your system and parameter fitting, such that it has a better chance of continuing to perform well out of sample. As mentioned in a prior thread, one way to perform this is using walk forward validation, some other commonly known are validation, and cross-validation.
My focus here is not the system. The system performed badly in walk forward. While I think this test is not a firm evidence against the reliability of the default Genetic Optimizer in NinjaTrader 8. And I will need further tests. It is my first heard of the generalizing, though, much appreciated !!
This thread reminded of this classic article from PAL blog since we just discussed another of their articles in another thread. Please read the article before using genetic and NN algos. When you use anything that optimizes you have to think multiple comparisons. If you optimize anything on the past then chances are it has no good use for the future.
Yeah this is just a weird bug where NT8 is somehow setting the increment on your first parameter to 1e-14 when you select genetic optimizer. I'm not sure if that's better or worse than the genetic optimizer spitting out false results. Probably an easier fix for the NT programmer?