I thought that if a system cannot be backtested it is probably not used much. Does anyone know any such system? I can suggest a system based on market context. It is too broad to be able to backtest.
Some things are very hard to backtest, like the workings of the queues in the order book... with enough book feeds data and some creative logic you can get a good approximation but it is very difficult to get it right and it'll never be perfect. Other things like the impact of your orders in the market price are simply impossible to simulate. This means that strategies that rely heavily in market microstructure, liquidity constraints and the latency race would be very difficult or even impossible to backtest... but this doesn't stop the HFT crowd from deploying systems that mostly fall on this space.
There are a lot of new data sources coming online that simply didn't exist before. (auto/programatically) Getting market sentiment from seekingalpha.com, or something of that sort. There's a ton of survivorship bias in most backtesting no matter what you do. As mentioned above, the very short time frame (market making) doesn't generally have enough data for back testing, but simulating trading on a go-forward basis can work well. On very large time scales using new data sources (not just time/price/volume), bias and lack of historical data comes into play. I'd rather approach with "testing", and then determine the appropriate testing mechanism rather than just throw around "backtest" all the time like it's a panacea.
It is more difficult as you requiring fastest of equipment and best internet connection bringing it in storage then T-1 connection to put on the trades, the systems themselves are far less difficult. Once you have blown all the bucks, back testing is done, then real work takes place of using it to better advantages of those who don't have skills and moola to do it all. Impact of quantity of volume falls under the "what if" and that is impossible, and what I have witnessed of huge orders in past all at one time is truly foolish in my opinion cause you last getting in. Some areas fall in testing as you are trading to find happy medium of what is acceptable without moving the market. You can watch DOME late at night and sometimes see one's like in Crude Oil and some price have no volume, so if you tossed in 50 lot, CL would scream maybe 25 ticks to get filled and someone else might quickly go other way twice as many to force the loss.