I used to think it was useless, but now there are a few things I would like to backtest. Most of it kinda requires more advanced stuff than I am able to do. Few ideas.... Distribution of deviations from VWAP during the cash open for index spreads. Volume distributions of aggregated futures data at significant deviation. Sector correlation rank type analysis for synthetic indexes. (which sectors does my synthetic index track most closely?)Volatility distribution for the basis spread (index basis). Some kind of ATR distributions for rate spreads to help me analyze the treasury fly. A lot of this stuff is highly non trivial to get good analytics on. Many issues here, and there is latency sensitive stuff that could be a hurdle as well. There is much more to do with this type of stuff (maybe more than I even care to do).
Visual backtesting using lagging indicators (most indicators are) is a perfect way to blow your account......
Billions have been made back testing paired with other meaningful things. See: Medallion fund from.Renaissance Technologies. Of course few people reading ET will be.able to successfully make anything meaningful from backresting except maybe being scared away from some really bad ideas
So if I understand it correctly there are 3 types of backtests that you are mentioning 1) backtesting of a strict mathematical/"mechanical" rule 2) manual backtesting of a (semi)discretionary trader 3) backtesting of bar patterns/indicators which has been tested trillions of times and has no edge (the stuff you see on Youtube). What types of tests would encompass the first point ? As you mentioned that that is what the smart people are doing which makes me curious as I never believed in point 2 (too many variables), and point 3. Thanks
How can you be sure that backtesting can save a lot of time and capital? Can you make a control experiment? How?