Absolutely not ironic... I am really asking for help and help other in this tough journey of trading.
Well, I'm up 8% using linear regression the last two months compared to the market only returning 5%. So, linear regression can beat the market, but in my recent experience, only very modestly. It's making correct predictions 55% percent of the time, not 95%. Is it even worth the bother compared to buy and hold? Perhaps not unless you enjoy the calculations (it's fun for me). I have not tried the Ernie Chan method of incorporating ML but it's on my to-do list.
Currently it is basic student homework to predict stock prices using various AI/NN techniques, likely to demonstrate how pointless this is. There are now over 10,000 related open source projects and the list is growing every day: https://github.com/search?q=stock+price&type=
Anyone who is genuinely interested in getting some experience applying ML to financial data should look into the data-science competition at Numerai. It will quickly fix any questions you have about, does it work and what works. It's not something you can do half way though, it takes a lot of work in the beginning to grind through the processes required to really put together a meaningful pipeline. I was one of the original participants in the comp, and I just pulled out of the main competition this fall after nearly five years of participation. My six month performance average has been in the top twenty many times over the years and I've had dozens of models land in the +90% performance percentile out of nearly 2000 models. Now I'm focused 100% on their new "Signals" comp, which is significantly harder. Only a couple hundred models in that comp right now as we're all just getting our pipelines sorted. I will say this. If you do join the comp and the community, leave any bias you have at the door. Opinions don't matter, results do.
i flew out and hung around with si so if results matter and nothing else would i be able to not use ai to win? is there somewhere i can see the results of actual trading is someone independently auditing the returns? btw i am really put off from the touting of Renaissance Technologies cause i know they are bs.
I've probably said all I will about it. If someone wants to get their hands dirty with some deep ML modeling, it is, imo, a very good place to do it. I'll leave it at that...
It might be possible, but it's very difficult. And it's unlikely that someone who has just read a few papers and used an off the shelf package will be able to find the holy grail. As you've discovered, the most likely outcome is that you will end up massively overfittedd. A more benign outcome is that you'll reinvent the wheel of something much simpler (I remember someone doing a similar exercise, and coming up with a highly complex model that essentially did the same as a pair of moving average crossovers). I'd advise you to spend time learning and understanding more old fashioned techniques first (like someone said nothing wrong with linear regression, on which almost the entire equity market quant fund industry is run), then you'll have a better appreciation of statistical significance and an intuition about fitting that a black box method like NN won't give you. And then read Marcus' book. GAT (I'm using old fashioned statistical backtesting to trade. Not living on a beach, but left my job 7 years ago)
FWIW, I wrote a custom ML implementation and initially had great success trading the models. Once they started the fall apart, I pulled the plug and reassessed to discover there was a flaw in the process. Fortunately, I was still quite ahead from where I started. I have abandoned the ML approach and instead now look for ways to best exploit known tendencies of particular markets.