Good info. I am working on a soon to be commercial product for algo discovery and should have it in a deliverable state in less than two months. If you are interested I will send it your way to check out...no financial obligation on your part. Just interested in collecting some feedback.
Interesting, sounds like you have already formed a strategic plan to tackle your observations. Glad to hear it, and perhaps a brief description of method of application. Likewise, happy to reciprocate and share this code base under a no resale NDA. (soley due to the commercial value of the per instrument state-full LSTM processing engine framework). I think ultimately the success in this game is to merge the best of each resource which is out there. If we could be a source for one for eachother . great.
This describes the concept of the product. There are other apps out there that are similar geared toward non programmers but what I plan to offer will be an app and tool solution with source code (the core SDK component will be compiled with no source code). I have also added automation capabilities to intelligently combine strategies which I have not seen in other products. Basically, I deconstructed how I would manually code and backtest a strategy and wrote a platform to automate that process. https://www.elitetrader.com/et/thre...esting-algorithmic-trading-strategies.323189/ I am pushing hard to get live trading in the next several weeks but would be interested in a "show and tell" after that.
Wow, dude I don't think you give yourself enough credit. This is an amazing concept and code discovery articulation/implementation. You know I think you would like re-reinforcement learning very much and I can already see in your code you have the aptitude to actually 'code' at a lower (better) level than I can - and I think what I find most interesting is the concept itself. A lot of what I am seeing is also very relevant to creating/working with agents exploratory order functions. Also, even on my existing platform - I can already see how such factors when dynamically calculated over time, could substantially improve the 'values' or discover better ones for the more traditional indicator analysis derived strategy components. This would be a major value add for any automated trading platform imo. Yes these things can be done with linear regression as you point out, but I think your approach and stated processing times are something to boast about. Look forward to a show and tell.
DQN / Re-Inforcement Learning (aka Q-Learning) reading materials http://www.wildml.com/2018/02/introduction-to-learning-to-trade-with-reinforcement-learning/