I work in data science, i understand what you mean. One could also use k-fold cross validation for that matter. However, when the (hypothetical) equity curve is smooth this would essentially mean the same thing, imo
You should be focused upon "what's working". If something is not, eliminate it. You need a certain amount of diversification to protect against "single issue adverse event risk", but that's all. You don't need "diversification across all sprectrums". "Relative Strength" and "sector rotation" are big parts of premium performance.
Can you tell your broker that this stock sucks, I want my money back? Or take my orders at the middle of the chart and not at the right edge(a quote from Alex Elders).
The only way you could convince me it's not overfitting is if you had a good reason why those 10 didn't work. Is there something in common between them that would plausibly lead to them performing poorly with your strategy? If so, what's the mechanism? If you can't answer that... probably an overfit.
I solved that problem by always using the same system, but in different timeframes. Faster moving means shorter timeframes; slower moving means longer timeframes. I can use the same system in any commodity and in any market. Don't have to change anything.
Not necessarily. I backtested my system when it was in development without doing any optimization. I searched for situations where my system got in trouble and tried to find a global improvement in the basics of my system. So not in testing variable parameters and optimize the system for that specific set of data. A system should be build on logic. If something does not work it means the logic is not good, so you don't fix that with an optimization of parameters. You improve your logic. To me a good system should perform well in any market within changing any parameter. Within certain limits parameters in a good system can be changed without influencing the performance. That shows that the system is well balanced, and does not get affected by "noise". The basic logic of my system never changed the last 25 years and still works. There is some kind of logic in the market behavior, markets are not random. Question is how to find what the logic is. I tried to find a logical solution, not a mathematical one. And when the logic is OK, the math in general is OK too. As math is about logic.
Why did the 10 of 50 perform poorly? Different sectors, different asset classes, or was there anything that stood out? Did you run the backtests with different starting dates?
It is realy strict as you said and in the fact I also think so, but it is really difficult; Instead I sometime train a strategy by "APPL", and I would believe it most if the trained strategy runs well on "IBM" (at least several tradings) even though it runs not so well another. Is it ok?