Hey Austin! Good to see you again here! Frost, Austin provides some great advice here, hopefully you will listen.
I am reducing the importance of, and time spent in, backtesting. Currently I'm scrolling through charts and developing rules based on what I see. The next step is take all the rules and find the conflicts between them and to find out what rule set probably works best overall in regard to profitability and my ability to sleep at night. The next step is to forward test on the IB test account and simultaneously trade the system live. After that work, a backtest might be in order but probably not all that necessary of a step. The next step is to change the account to the real money one and watch the thing to see if anything works differently than [or if anything works remotely the same as] expected. Backtesting is so screwed for so many reasons I can't think of them all at this point. The suggestion to get RINA software and do monte carlo shakeups and whatever else you can do to expose the systems weaknesses is a good one probably but even with all those capabilities I don't want to go there that much. Backtesting is just one small step in the overall development cycle, too much emphasis is placed on it.
What is monte-carlo testing? In laymans terms please. Thanks, Osorico EDIT: nevermind found it... http://en.wikipedia.org/wiki/Monte_Carlo_method EDIT: and even more relevant: http://en.wikipedia.org/wiki/Monte_Carlo_methods_in_finance
i think (but can't prove it yet) that the data from the IB's simulated account vs the "real" account is slightly less than identical. i base it on 2 identical systems running side by side on each account. i have not looked closely at it however, so.. you prolly will be able to confirm (or deny) this assertion when you move fwd w/ your testing.. it's most likely insignificant..if at all. just something i've noticed. bb
I have to agree, setting daily profit targets (or really anything) based on backtesting is just curve fitting in the worst way. Been there, done that. Doesn't matter how much data you have. I didn't believe it until we did it. Our older (failed) systems were based on backtesting data and curve fitting (conscious and unconscious). The system has to first fundamentally make sense. It's really hard to explain because it's somewhat of a catch 22. You don't know how profitable a bot is before you backtest, but once you backtest you can't change anything to make it more profitable because that's just curve fitting. So in summation, backtesting will tell you if a bot WOULD have been profitable, but can never tell you if it WILL be profitable. That's why most bots based on MA's fail. You think "well the principle behind them never changes" but then somehow the system fails a few months into running. The only real way (and then you never really know) is to just forward test with new data. Reserve some of your backtesting data for forward-testing and then each day test it with brand new data from today. DO NOT CHANGE ANYTHING DURING THIS TIME. After a month or two you will know whether you have a chance of making money or not.
All these posts about out-sample test and RINA is telling me that none can really program or isn't testing systems enough. Most the posts are based on assumptions and/or just presenting the limitations of how they run a test. Anyone ever thought about and tested whether the out-sample tests setup is significant? How the tests should be conducted? Whether their risk-metrics are really significant? Of course, I haven't seen a 3rd-party backtesting packages giving you the ability to truly test... That why all system traders need to learn how to program. For example: I've programmed my tester to be able to walk-forward automatically when a trigger is sent. 1. Walk-forward a parameter with a specific interval (eg. Walk-forward every 3 months looking back 5 months of data...) 2. Walk-forward when a trigger is hit. Triggers were based on a specific risk metric. (eg. Walk-forward if the Risk/Reward Ratio for the past 30 trades are less than 1.5) 3. Test multiple risk metrics to find the best fitness to use when I pick the parameter value. (eg. Which parameter value should I pick? The system that has the best Sharpe ratio? The best % profitability? The best profit factor? ) Etc. Etc. ======================== When system traders mention, "TEST EVERYTHING". It means you test everything. You test all the decisions, process, and everything quantifiable that you can think of. I'd also like to mention that I didn't have access to a fast computer or some computer cluster when I started running the tests above. I was able to run all my tests at home, using 2 cheap Dell computers.
BigBubba, I believe you may be correct about the data in sim account and live accounts are different. Not sure how different, but I been running the bot simultaneously on both sim and real money accounts, and the results sometimes are slightly different... and in 2 days VERY different... so I do believe there is some truth to the data being different. I"m currently trying to asses how much if any this difference may be causing current issues. Since a lot of the backtesting data tested on was retrieved from being live on a sim account.
BS. If minor data difference affects your trading to your degree where it determines profits and losses... It means you're over over over over (x100) curve-fitting. Stop finding excuses. Take a serious look at what YOU're doing.
not true for all cases. i found one difference between sim and live which effected me was when my system attempted to grab liquidity which faster systems got to first. obviously that had a negative effect on the system and has nothing to do with curve fitting. IB gives you a sim fill for a market order at the next 'last price' after you transmit (not realistic), with limit orders IB gives you a fill only if someone else trades at your limit price (also not realistic).
Let's rephrase that: Does this seem right? Also, there is no such thing as "true edge". They will always go away, just like the trend in a market will change.