Hi guys, quite new here. I am just curious to know; in the process of trading signal development, how do you ultimately decide which signals to include in your repertoire. I am currently working on some models, where I ultimately decide to include the signal where I find that it replicates in 2 separate forward datasets. But the original process of developing the model included multi-fold cross validation on the training data, before the check on 2 data sets. So I have to ask myself why the signals didn't work completely on the first dataset (in spite of multi-fold validation) and in some case why those that did went on to fail on the second dataset. That is; are the successes on test data set 1 and then test data set simply incidental?
Use your charting software to run a screen and not backtest from 2007 to 2021. This should cover the bull and bear market condition. A screen will let you see the entry and exit. Choose a stock that you expect of your model and see what is your model action.
It might not work if you backtest far too much. about a decade ago, there were major currency wars. currencies were moving massively. Nowadays, currencies move just a little. The environment has changed very significantly.