What Nyquist Didn’t Say I won't say much more in this discussion about this, for fear of other more knowledgeable experts, who may bring their insights on the table. Since I'm no expert, I leave it to the proper authority.
You keep on repeating this "input is more important than the actual algorithm" story. When you were confronted with facts and evidence by some of the top ai researchers in this industry you just conveniently ignore it. Makes me wonder what your posting rationale is...
I agree with JCL....Next Bar objectives are stupid....the objective must be based on these factors: 1) Trades Per Day (frequency) 2) Commisions and Fees Per Trade measured in TICKS 3) Future price vs. Current Price in TICKS So constrain #1, and optimize on #3 minus #2. If you don't constrain on #1 i.e. limit the # of trades per day, the ML algo will do tick scalping....with marginal results. If you force it to trade a longer time-frame, it may discover a pattern that maximizes #3-#2.
IMO most if not all of you are wasting your time. Markets spend 90% to 95% of the time in what I call bid/ask spread relative value efficient. That means that the only way to make money in that market regime at any given time, is to be an HFT/MMer with somewhat deep pockets to hedge and to smooth out bad statistics and "sell the noise" to chartists etc. Of the other 5/10 % of the time that markets are outside their efficiency regime, it is almost always due to a news event. Therefore, strictly inputing price to predict the same instrument's price at a later time into any of these machine learning algorithms, which probably 9 or 10 out of 10 of you are doing, is going to lead you nowhere. You will feed these things data, it will spit out a model (probably a momentum model), you will trade it, but you will be driving Uber or collecting Social Security to actually pay your bills. If it spits out a MMing model, you will either get run over by a news event, or you will not be able to trade it because most retail traders aren't equipped to trade a MMing model. Imo, the only edge left is either HFT/MMing (ruling out 99 out of 100 traders on ET), or understanding how news events affects different symbols. Doesn't mean you can't use machine learning to do news/price impact, but the database is not so easy to attain. News drives markets (the plural is important) out of efficiency for a non-trivial amount of time. Since news is essentially unpredictable, but markets rarely immediately know how to go to the efficiency regime instantly, this edge should "never" die even on say 15 minute time frames. Plenty of time. You can play around with these algorithms if this is your hobby, but trying to make it a profession (actually being able to make a living trading ML models) is going to be nearly impossible in my experience. FWIW.
Well that is your opinion and you are entitled to it. But a huge host of hedge funds, and investment banks, buy side firms, and mutual funds and others seem to disagree with you. All the above are not market makers and are not engaging in hft. They all invest hundreds of millions in Ai research and even if it's just intended to hedge themselves. At the very least Ai properly applied (and I believe 99% of the posts on this website are a testimony of a complete lack or false understanding of the mechanics of deep learning networks) beats the odds of "technical analysis" at any time. We are talking a rigorous and mathematical and statistical discipline vs hocus pocus that can never be validated based on hard numbers. Anyone who understands the least about Ai at least concedes that should TA have the slightest value then even the most simplistic deep learning networks would unearth such value. Most anyone with an IQ of more than 120 and one who is up to date on technological developments understands that Ai will transform the world more than the Internet has, more than mobile phones have, more than any other technology in the past 50 years. The same applies to Ai in financial services and trading.
In fact hedge funds do not yet invest largely in AI research. They began hiring ML developers only recently. Most of their research still goes into conventional algorithms. ML for financial prediction bears two challenges, that's why other methods still prevail. First, whatever you predict, you can safely assume that the price curve has at best 2% signal and 98% noise. Second, you have limited training samples, especially on high time frames. Both is not good for ML. Depending on what signals you use for inputs, for instance the components of a spectral analysis, a simple ML algorithm can be more successful than a deep learning network.
I do not know what proportion of funding is invested in deep learning networks vs more conventional algorithms on the Quant trading side at hedge funds and I would claim neither do you just because you work at one firm. But how important research in AI is at investment firms can be easily gleaned by looking at the public statements many such funds make and the sometimes attached amounts, invested. And I have numerous friends who work in this space who share their experiences and we actively exchange thoughts on some of the latest AI developments. I guess we can both agree that most on the retail side approach this whole area with a very naive mindset and a lack of rigorous training in mathematics and statistics and professional training on the trading side. We also both probably agree that software vendors for retail products jump on every bandwagon to keep pace and offer new "fratures" to their customers. It has been shown that many such features contained at times blatant errors and mistakes. "Portfolio back testing" is a perfect example. Or proper currency conversions for pnl for assets of different base currencies as part of backtests. A few years ago brute force optimizations were the latest hype, then genetic optimizations, now ML, and soon enough I am sure the ninja traders and multicharts of the world will come out with AI features. Means nothing without a proper and deep understanding of the underlying methodologies. By the way a hedge fund I interned at after grad school in 2005 already plowed huge amounts of money into ML research and implementation. It's nothing new to the hedge fund world at all.
The hedge fund industry has been decimated the last six years. Recent examples: Unmanageable Money: Hedge Funds Keep Losing (And Closing) - Why It Matters http://www.zerohedge.com/news/2016-...-funds-keep-losing-and-closing-why-it-matters