There are two different segments of data-mining. One is a discovery phase (while this is continuously in development, it is very useful in it's present state). The other is a phase to increase the probability from results that were generated in back-testing. There will be also be a version inside the back-testing reporting system where the system will show the trader the scenario / pattern that occurred before each winner vs. each loser. This will then enable to developer to modify his strategy / system accordingly. All of this is part of a process: 1. Discovery - mine data to find situations that can present trading opportunities (mainly price patterns using daily data) while this is productive now, it is still currently in a development phase. This is a web-based platform and uses PHP as the language and a MySQL database for historical data. 2. Develop systems around information found in discovery 3. Back-Test the systems 4. Mine the data using a web-based PHP / MySQL platform that looks for patterns that may increase the probabilities of the system. This can also be used to find when not to trade. It's just as important to find a scenario to stay out of as opposed to trade into. 5. Test trading system with Live Data 6. Live Implementation
The SQL idea is a great one -- I'm working now on getting my backtester to insert the indicator values into SQL instead of CSV's and spreadsheets. This will give me better querying ability. Another idea I've tried researching is: instead of "a" system, with one set of rules, increase to 2 or more sets of rules, with a switch that toggles between them. There may be synergies between some such sets. My research on this is slow, because of my platform and I have to build my own tools to work with data I can get from it. I can see the advantage of a platform made from scratch -- one would have more control to implement a feature like this.
You did not answer my question. If you don't want to answer it is fine. However, I find it peculiar that you did that. I gather you understand what GA, GP and NN stand for? Tallking about data mining is one thing. Doing it in ways that make sense is another totally different story. Plus, I believe that MySQL and PHP will be awfully slow for the job.
Take a look at what TradersStudio can do for you. I have posted a few video's in this thread. I also have a course I did a while ago on developing trading systems. If you PM me and give me your real e-mail address I will make sure you get a copy of it.
I thought he is more interested in developing a platform that integrates system backtesting and system discovery based on data mining. Can TradersStudio do that?
Let me first say that : GA - Genetic algorithms is more of a search technique than a data mining algorithm. This would be better suited in back-testing and could be referred to as a form of optimization. GP - Genetic Programming - this can be done in SQL, however it is for more complex problems than I am trying to accomplish. NN - Neural Networks - This method will be used when we complete our back-testing reporting application. It is our goal to accomplish this using PHP. I will be using Decision Tree/ Rule Induction as the method for data-mining. This method is the simplest form and for what I am trying to do here and it is more than efficient.
Thanks. Good luck. I guess most of this stuff is already in the market in bits and pieces. I already use a data mining tool and I am very satisfied with it within its limitations. Data mining provides an edge to those that use it properly this is why you do not hear people talking about it a lot or you even hear people try to discredit it.