ssrrkk, When collaboration fails is when one side cannot give anything to the other side. You must take from each other and excel...you know two heads are better than one. I am not a big fan of backtesting and very creative and sound a lot like your friend. The coder I work with on and off over the years seems to think the same as I and we get along together. But you got me thinking....Do I do enough in our collaboration? ES
With regards to commission and slippage, I basically embarked on an academic exercise: you can categorize algorithms from worst to best as: A. consistently unprofitable without commission and slippage B. consistently profitable without commission and slippage C. profitable within certain regimes with commission and slippage D. consistently profitable with commission and slippage (holy grail) So there are clearly many algorithms of type A and B. That is the first test: does it even work in a frictionless ideal world? If it's A, then it's really really a bad idea. It simply doesn't work no matter what. If it's B, it is a good start. It seems to have the right mechanics in place, or it is latching on to a real market phenomenon that might be exploitable. It's just that the profitability of the algorithm is not enough to overcome the slippage and commission. Most moving average cross type algorithms fall into B. Now if I find several algorithms of type C, I am willing to go live given that I know which regimes it works in, and I know with enough confidence that currently we are in that regime. Regarding D, I am still of the opinion that you can't find one that works for 10 years with no tweaking. But the real question is: do I want to spend years trying to find it? Or do I want to settle with C and start making profits. I agree with the importance of money management and its importance in relation to your PL volatility (i.e., draw downs).
hi ES, Well, believe me, I don't have any problems with creative types -- I work with them every day in my day job and I have no issues. I would venture to say that I come up with some pretty out-there ideas myself but it's just that I don't take them seriously until I subject them to a battery of tests. But what bothers me is this: he would go out of his way to disparage my scientific mindset, saying this is foolish and useless in the trading world. And that especially gets me going because I'm the one who has spent double working hours over the last year implementing absolutely everything to make things work. He not only wants me to develop a working system, he wants me to spend most of my time on things that has a very low probability of working (backed up by statistics and real evidence, not just my opinions), and also he tells me to make my work less rigorous. I just don't get it. If it weren't for that, I would value my partner's input even if it was just bouncing off ideas and having discussions. It's when he starts to interfere and disparage me that I start to think whether it's worth it. Anyway I think I've been ranting on this board too much.
ssrrkk, You are not ranting...look...you got me to think and your thread helped me. I sent an email to the coder I am working with (He is a genius) and asked how I may help. It is important for friends to be aware and sensitive to their conduct to one another...or else you would not be friends right? You thread brings out many postable replies and this is what ET is all about. Discussion! Anyways...I find the fanactical backtesters cannot be changed and besides why would a friend be in the business of requiring you to change? ES
Just curious, for those of you who don't believe in back testing, how do you evaluate a strategy? How do you decide to go live trading using a system? I suppose you could emulate a trader with a successful track record, but that's almost like back testing too. If you rely solely on forward-testing, then how long do you do that before going live? Just wondering...
You did not ask me directly... but I can tell you that I have looked at thousands of EA's in a private forum with backtested graphs and results (2 YEARS TO 10 YEARS) and forward tested them with those settings AND NOT ONE EA TRADED PROFITABLY IN FORWARD MODE! This has taken me two years to do. ES P.S. Lets not turn this thread into a backtesting debate...it will ruin it. I know what I know for myself and there are debatable points for and against.
hi ES, thanks for your reply -- I understand that this might be a controversial topic but I was too curious not to ask. I would say this, not to doubt any of the back tested graphs that you refer to, but at least in my case, I have found that the results of back tests (the PL curve) is an extremely sensitive function of various parameters, obviously the slippage being an important one, but also including the accuracy of the minute bars. By extremely sensitive, I mean that you can make an unprofitable algorithm look profitable and vice versa just with small changes in parameters. I have for several months struggled with this and I think only recently I have gotten a handle on this matter. The way I got things straightened out is that I have been running about 2 months of forward tests using my trading code with IB's paper account. I found that it took a huge amount of tweaking to get the two to match up. I found also that it is absolutely important to check every single trade, the times and the prices that it got, versus the back test. After doing this, I was able to get all the trades to match up, but even then, I saw a difference of up to may be +/- 10-15% of the daily PL. The reason for this difference is not only the slippage difference, but also because the minute bars in my trading code is subtly different from the historical data downloaded from IB. My bars are built in my code based on IB's real time stream, and also my computer clock but I adjust that time to match up with IB's server as best as I can, even estimating the time it takes for a time query to go back and forth between my computer and IB's time server. That difference in the bars can sometimes cause a trigger to happen one minute later or earlier. Also that difference can change the SPX minute bar closing price enough to create a 5-7% difference in entry and exit prices of the ES (with the 50 multiplier). But I have plotted the PL curves of my forward and back tests and they overlay very well now. The individual points are not spot on, but generally when there is a profit in the forward test, I also have a profit in the back test and vice versa and the difference is consistently within 10-15%. So now I can finally trust my back test results.
By the way the reason I started talking about slippage and commission after you (braincell) mentioned the high number of trades is because slippage and commission make a huge difference especially for algorithms with high number of trades -- I think it's pretty obvious but I never explicitly stated it.
I've seen the EAs he's talking about and the platforms and methodologies they were backtested on. Let's not turn this into a backtesting debate, but I have to say, those backtests which give the equity curves for those EAs are pretty useless. Most of them just use the typical price of a bar and don't even simulate ticks between OHLC and volume data (as far as i know), so the highs/lows don't get hit quite as much in those backtests. They also don't have any model for accounting for slippage. This is why any backtests posted by EA creators shouldn't really be trusted. The only way to backtest and EA is convert the code to a format for your own backtesting, and then run it with a credible methodology. Slippage needs to increase when the price is changing fast, but by how much is up to you (the tester) to decide. Regarding the difference between historical and real time bars created with IB feed, that is an interesting observation. I'm doing the same thing when creating bars inside of my own code, and I didn't compare them to see if the differences would be large enough. That's a good hint, and I should probably do that. If they're big differences I suppose I'll better look for a high quality data feed - or - on each new bar download the historical one from IB (one bar per minute) to make sure.
As I said, until you verify each and every trade even for a short period in your forward tests, you really cannot tell what is going on. Sometimes, the back test will miss a signal entirely if you are using some kind of hard threshold in your signalling, just because of a 0.01 price difference which can easily show up in the bar differences. So that's why I said the outcome is extremely sensitive to all parameters, because in my system, if you miss a signal, that could in the extreme case mean a +/- $2000 profit difference for that day -- and you multiply that by the number of times that happens and that can quickly add up. The importance of slippage cannot be overstated. I made the mistake both ways: not using enough slippage, and using too much. At one point early on, I was using the absolute worst slippage thinking that the profit margin was big enough to find algos that worked in that case (my slippage at that point was the worst price in the next bar after the signal, e.g., for buys, use the high of the next bar, for sells, use the low of the next bar). When I was doing this, I could not find anything that worked. The PL curves were basically going straight down. But after my forward testing comparisons, I have found that that is an exceedingly harsh overestimate of the slippage. When I started to use a more realistic and simpler slippage (just add/subtract a fraction of the close as entries exits), my algorithms started to match up. I agree I could go to a more sophisticated slippage model based on the current bar or previous bar's height (scale by the percentage of bar height) but so far, I can get the back test to match with the forward test without going to that extent. Of course the other extreme case of slippage is none -- and that absolutely does not tell you anything about the algorithm. Like I said earlier, this is like a type B algorithm (of course, if you include commission then it's not exactly type B but). when my friend first approached me, he brought with him some PL curves that appeared to go straight up. So I spent the first month trying to reproduce that curve until I realized that he wasn't using slippage. Ever since then my friend has been very skeptical of back testing.