When optimising parameters, I noticed something funny: a small tweak to just one value can drastically change the final result. This happens when trading stocks without margin. The main reason is that when the system runs out of cash, it ignores new signals until it closes some older positions. So even a tiny change can shift the moment when funds become available, which affects when the next trade is opened or closed. That shift ripples forward all the way to the end of the backtest. It doesn't necessarily mean one parameter value is "better" than another—it just led the system down a different path through the historical data. The same thing can happen if I just move the start date forward or backward by a day. So I came up with a little workaround, and I'm curious what you think: Instead of taking every trade, I randomly skip half of them. For each buy or sell signal, there's a 50% chance I ignore it. If it's ignored, the system can still act on it the next day (if there is a signal). Then I run the backtest multiple times (with the same parameter values) and take the median result as the representative one. This actually helped—the predictive power for out-of-sample results improved compared to just using all trades. Do you think it makes any sense, at all?
It makes sense it helped but it's because you've contaminated the signal. Separate out signal generation from execution constraints (eg margin) as separate processes; your margin doesn't impact the chance of price turning up or down, so it shouldn't figure in your analysis. as to why it has such a marked impact before you skipped trades, usually it's because volatility clusters so same conditions persist for longer than random chance. so adding time breaks by randomly skipping trades takes you to new clusters faster. I suspect if you delve into hold times for trades, you'll see what's really causing P&L to bounce but the way you're doing it it'll be hard to establish wtf is going on.
Interesting take. I was wondering about adding some quirk to the backtest engine: pretend there is unlimited amount of money so that I never run out. Once the money is added (to satisfy a signal) it's never removed. Unfortunately, it breaks some statistics like drawdown and others. The fact that I can run out of money in the backtest also helps to find a right parameter to exit trades early. My point is: even if there is decent probability of price going up, I may want to exit existing trade to have cash ready for another one. With unlimited money this is not simulated. Another way would be to just increase initial capital and decrease initial position to a value small enough to be able to open all positions required.
Is this some kind of all in one tester that's mixed MM and signal in a most unholy way? It gives me the shudders. Personally, I'd be moving platform but that's me.
I wrote it for my self (I'm a software dev). It simulates real trading with capital constraints, so yeah, you can say that it "mixes" MM and signal. Would you suggest to split test in two: test signals only (entry and exit only, disregarding other opened positions) and then figure out MM?
Absolutely. If not you could end up creating all kinds of wrong uns— eg a blind martingale system that made 2-300% last year but contingent on knowing the volatility and sizing to fit. No actual validity to signal. Separate out the two and treat as separate problems Being a dev will be useful- you know the maths behind the calculations so no black boxes in sight