This shows you have not done any serious backtesting work. Speak only for yourself. You have no idea what backtesting is anyway. You guessed right. I am a trader. By the way, do you have a vested interest in Amibroker? I find that program extremely hard to use. Unlike other programs I have used, its design, language and interface are not user-friendly at all. There are many other user-friendly programs traders can use to make money without first getting a degree in CS and having to wrestle with esoteric designs.
No vested interest in Amibroker, or Ninjatrader or any of the other programs that I use. As for user-friendly interface, I find it on par with most packages oriented towards testing - in many ways it is simpler - the code length required for typical systems is much, much shorter in AB than other packages. What packages do you use for testing that you find simpler? As for whether or not Yahoo is reliable data - my point is that if someone wants to test a system, it's not a bad place to start given that the data is free. And while it has problems, if you want to do test on, say, major indices or major ETFs, it generally doesn't have problems. I've compared the data with that from major vendors including CSI, Premium Data, Bloomberg and it matches. As for whether or not I know anything of backtesting - I'm quite comfortable with the fact that I know a good deal about backtesting having been doing for a long, long time, and a really good idea of what the pitfalls (over-optimization/curve fitting, data snooping bias) are and what you have to expect when you bring a system a live. As for your experience - I can't comment because you've given no details on why you think that backtesting is a joke - you've just stated it. I've simply said that, in my experience, backtesting can work. Now if you'd like to give details of why you think backtesting is a joke, please do. And if you don't use backtesting, then what is your methodology? I'm not asking for your trading rules - what is your general approach, and do you trade with any knowledge of historical price history? Or are you a fundamental trader?
Historical speed is, of course, great. How well does Ami perform at replicating historical performance with fully mechanical trading using the same strategy? Specifically, how do you automate a strategy with Ami? Then, how reliably does it run unattended? And if you compare the real time results to the the same data run through a historical test, do they match? If not, how far off is the correlation? To put that another way, how well does it "simulate" real time/live entries and exits when historical testing? There's plenty of platforms to use for decent historical testing some faster than others. But where do you go to perform "beta extraction" via automation short of a software platform with a 6 figure price tag? Only one.
You can currently only automate a strategy with AB using InteractiveBrokers and the IB Controller add-on. As has been pointed out, the focus of AB is really more on analysis than automated trading. It is particularly strong at doing portfolio-level backtesting - I consider it second only to TradingBlox. By "only one" - do you mean Tradestation? It definitely is a more straightforward product for automation.
TS doesn't have automated portfolio trading support. Building strategies on bar data and then trading them live gives low correlation when compared to running the same data through historical simulation.
I don't know what you mean by "low correlation" - do you mean that backtesting doesn't line up with future trading?
That's correct. Many platforms have that problem. It's not necessarily their fault per se. It's just in the nature of using bar data to simulate real time trading. Unfortunately, the highs and lows of each bar require 3 prices rather than only one that you have at the top and bottom of the humble bar. In real life, each high or low has a 3 prices--bid, ask, last trade. So stop or limit orders that happen to occur near the high or low of a bar in historical may never get filled in real life because one or the other of bid or ask prices either never made it that far or went further. What is more, some trades will appear in real time trading near the highs or lows that never occurred in historical. So the fundamental issue is that bar data is forced to pick one of the 3 prices at the high. Ordinarily, the last trade price for highs lows, open and close gets used. But that's only 1/3 of the market picture and bid or ask nearly always go farther than the last trade price--sometimes by a wide margin. So, again, you will have entire trades that appear in historical but not it real time and vice versa. It's really painful if you ever try it. Surprisingly, some traders eventually figure out how to make money even with this appalling situation. They may cease using historical testing altogether and do only forward testing but that's very dangerous way to find programming errors in your strategy rules. You'll hear horror stories that happen on occasion due to typos in their strategies. The other was is to test ideas historically but trade them manually in real life so as to force trades to more closely match historical. Another solution is to use tick data. But Ami, I'm told, only supports bar data. The only way to use tick in Ami is to feed a single tick and treat it as a bar with the ask as the high, bid as the low, open/close as the trade, etc. But that's doesn't come close to solving the real issue. Plus it performs far worse for an equivalent period of time due to the extremely high number of bars. There's another platform, more expensive, mentioned on this thread which solves all these problems in one of several differents ways for the user to select by having all the following very essential features for true mechanical "beta" extraction: 1. It accepts multiple data input types and granularities all based on tick data--even bar data gets treated like tick data as you will see below. 2. Of course, feeding full tick data to the platform gives the most accurate simulation. Then it generates bar data on the fly and gives you extremely accurate entries and exits that match perfectly real time trading going forward. 3. Another option it supports is to use full tick data to generate bar data but with complete bid/ask/last prices for each of the open/high/low/and close. That means that each bar contains a total of 12 prices associated with it instead of only 4 OHLC. This runs faster that full tick data and about the same as Ami for the same amount of bars. Plus, it assures that the number and timing of trades match real time going forward. Still, since it has to simulate the intrabar data so you get some fill prices with more slippage. People who have tick data like this option to operate at bar data speeds with acceptable forward trading simulation. NOTE: The above 2 options require the user to have or purchase tick data whereas the next 2 allow using ordinary bar data with some trade offs to still get accurate forward trading that matches historical. 4. Additionally, the platform allows you to use ordinary OHLC data to convert to tick data as in 3 above but the Bid/Ask/Last prices for each tick hold the same price. A different feature must be enabled to solve the problem because this choice leaves you with only 4 prices per bar which is too little in other platforms for good real time trade simulation. Still, along with the next feature, this solution has been found to allow accurate forward trading that matches historical... 5. This platform offers 2 choices for handling orders in real time trading. Of course, you have the option of sending limit, stop, and such orders to the broker/exchange, but the platform also allows holding them by the platform and treating them synthetically so that the platform internally watches the real time market and fills the limit and stops with market orders. 6. How does the above solve the fundamental problem? Well on more features is needed to complete the picture. The platform allows you to select which price(s) in the real time tick to watch to fill orders. In this case, you must tell it to watch only the last trade price for filling orders. Since that's the only price used to build the bars, you ignore bid & ask during real time trading. That allows the real time trades to produce the same results going forward as historically on bar data. That solution works exceedingly well on very liquid markets without any manual intervention. However you choose to do it, it's absolutely critical that you find a way to make your forward trading match your historical testing even if you must do it manually to save on software licenses. That's the only way to build any confidence that strategies you build and test historically will ever work in real life. This whole issue is, by the way, only one of several reasons why so many people on these ET forums have forsaken historical testing altogether. But technology is bringing a solution. By the way, there are at least 2 other major reasons why historical testing is extremely hard. That platform solves those also making forward testing and trading of historical developed trading closer to a science than a "black art". And all of that is possible only because the platform in question eats tick data for lunch. Those solutions are impossible on platforms that can't handle tick data to begin with. Again, as mentioned earlier there's only one platform that makes it possible to historical and mechanical forward trade reliably for smooth beta extraction with a winning trading idea.
Signalstop - you point to some inherit limitations in the data - the answer, for me, is when using daily bars to not rely on limit orders and only trade on the open or close (preferably the close as the open price has problems), and, finally, to make sure you've got a good amount of slippage incorporated into the testing. Using these guidelines, I've had no problem getting forward testing to match backtested results. In fact, in many cases, my systems have benefited from slippage in a positive way. I don't know if you're right about AB and the tick data - I believe AB can deal with tick data fine and it is more a function of whether you have the infrastructure to support it (both in terms of data feed and in terms of storage). What is the platform you're talking about? What's the name?
Droskill, that's very smart and another technique to add to the list. Just avoid limit and stop orders altogther, trade only market orders on the close for making historical match real time trading. Congratulations. Each trader finds his own way. Hey, I'm going by what several people told me about Ami, never tried it since when I checked it out it didn't have any builtin mechanism for fully automated trading. I hear there are improvements in that area. The platform I mention and prefer not to specify although it's already mentioned in this thread goes a lot further in helping match historical and real time. For example, have you figured out that you can't use money management stops like Profit Target and Stop Loss on Ami and most other popular platforms? Most successful traders leave them out of historical tests because the platforms handle them by "mixing" them with the natural trades. Instead those traders select their money management on a spreadsheet by analyzing the trade history. Then they trade the strategy manually and apply their chosen stop losses. The platform mentioned does that same "manual overlay" automatically by keeping the money management separate. If you're unclear of this problem it's a question of whether to mix the money management exits with the "natural trades" or overlay them. Mixing is terrible and how all the popular platforms do it. So you get weird results when optimizing and more likely to "curve fit". Again, all these problems are solvable in a variety of ways manually.
Droskill, another benefit of you using only market orders is avoiding hassels with partial fills. The other platform I mentiond elegantly handles partial fills with limits and stops but you avoid that entirely with only market orders. Do you do any advanced techniques like variable position size or portfolio trading? Do you do any portfolio trading with either multiple strategies per symbol or a strategy on multiple symbols? If so do you do on-the-fly automated asset allocation position sizing? That's well nigh impossible on most platforms in real time. Just curious if you found another clever work around for those areas also.