Well, considering that these people are probably trading something 20ish-10ish delta, it does not take that much data to price these options. A simple spreadsheet doing a historical payoff back-test with ATM vol rescaling would give you a reasonable idea if you are doing well or just deluding yourself. As a matter of fact, there is one BIG investment fund that does a very similar strategy in interest rate options and the guy that runs this book knows his risk pretty well. I am not defending anyone, but there are very few trading strategies out there where you can't quantify your risk in one or another form.
If you know that you have a 90% success rate then you should be able to calculate the expected value fairly easily. I still think you will be able to back test it and come up with some meaningful numbers.
How would you calculate the "expected value fairly easily"? Price movements every day are just outputs from a random variable with a distribution that we can only observe, not actually know. Let's go with you and sle's theory, and run backtests and came up with the conclusion that you would've made $1 in 91 out of the last 100 months... and lost $10 the other 9 months... so what? Does that mean you, as a manager, have positive expectancy in running this strategy? From a statistical point of view, absolutely not. I'll refer to this: http://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fair
Let's make this easy and say we have a 90% success rate(over the course of this fictional test). So we have 90 winners that profit an average of $1 and 10 losers that lose an average of $5. I could say that this strategy is a positive expectation strategy, but I can only say so to a certain confidence level, which is what I think you are saying. Although if I were to say that I ran a back test that found 900 winners of $1 and 100 losers of $5, I could be much more confident in the expectation of the strategy.
Let's say we have God-like powers, and we know some option spread (I'll call it the Copper Beaver) is profitable exactly 90% of the time, and a loser 10% of the time. And of course, the Copper Beaver is usually priced correctly. (This is just going to be a simplistic model... we're not even looking at scale of losses during the down months.) This is basically the scenario you're describing, jb514. Now, along comes manager A... who claims he has special skills, and knows when he should put on the Copper Beaver (and/or can "manage" it), such that he can beat the fairly priced market. How many trading decisions from manager A do we need to see, before we can be confident that he does indeed have special skills... with let's say, 95% confidence? - If manager A makes money for 91 months out of 100 (barely "beating" the odds), we would need to see 10,000 months (800 years+) before we really know for sure. - If manager A makes money 95 months out of 100 (absolutely killing the odds)... we still need to see 196 months of actual results (16 years!) before we are confident. - Even if manager A makes money EVERY SINGLE MONTH... we still should wait for 4 years of trading results before we have confidence he knows what he's doing with the Copper Beaver. (If you prefer to think of it in backtesting terms... how many months of back-testing results before you can determine whether your super-duper strategy A is actually "beating the odds"?) Now, on the other hand... let's talk about manager B who's trading direction (long/short), or something else that we "know" to be roughly 50/50 odds. How many months does it take for us to know they actually are beating the 50/50 odds? If they're profitable 60% of the months, then we can say he's skilled after just 4 years. If they're profitable 70% of the months, then we will know in just one year! And this is why I say... if you don't want to cheat others or yourself about your likelihood of success, stay away from the low probability stuff.
But of course. If you have the distribution of returns, a bootstrap test (for example) will tell you if your strategy is actually statistically sound even for a fairly small sample size. For a 100 years of S&P data where you generate overlapping N-day returns, you would not even need that.
Forget about options spread, lets take much simple example - people who do event-driven arbitrage. They have a fairly small sample to work with yet somehow they are confident that the strategy has a statistically significant edge. E.g. a guy who does month-end-extensions in bonds would probably have to test his strategy on maybe 240 samples and even that is pushing it. Yet he somehow believes that he has a statistical edge... Do you think he is stupid?
No! You're totally missing the point. When the statistical edge is glaring, it doesn't require many samples to validate it. When the statistical edge is out of necessity minimal (and has to be for a low-delta, low-probability spread), then it requires MANY samples. If I'm contemplating investing in a merger-arbitrage fund manager... it doesn't take many samples at all. The probability that their 80% positive hit-rate (let's say) is due to "luck", when they're just taking 50/50 bets on two different positions, is very low. It's easy to prove statistical significance. If I'm contemplating investing in an options-writing manager who's selling stuff 2 std dev out from the underlying? It is *very* difficult to prove that they're skilled... and requires a very large number of samples. This likely means they aren't skilled at all.
70% edge on a sample of 120 months is just as "un"-glaring as 1% on a sample of 1000. This is becoming an increasingly academic discussion on low probability statistics. There are many thing people can do to evaluate statistical qualities of outer tails in distributions and there is a lot of non-financial work being done in that area. In the end, most of your risk is not in evaluation of fair value (which is fairly straight forward), but rather in preventing a ruinous loss. That's where the skill of the portfolio manager comes in.
You guys are way over this amateurs head. Not that it isn´t interesting. But as an hands on guy, I just don´t trust all that statistical stuff for making money. I like to read it and so forth, just don´t trust it. Maybe it comes from building a lot of boats up to 50 ft in wood. If it looks good, it is good. If you followed the plans and it doesn´t look right, then it is probably not right. Simplistic I know. In the meantime i am in a SAD mood. Went from being up 5% last month in cash, to down 9% so far this month, from starting account balance. Made a STUPID mistake. Entered my volatility, premium ballooning trade fine. But in the couple of weeks of doldrums in between, I had been playing with scalping the spy and somehow during my volatility bread and butter trade, I did not take my profit when I should and switched, or rationalized myself into going for a longer couple of day trade and finally had to close out losing. I did not stick to my rules and kicked my butt all over the house for a couple of days. Ahh well! Wait now patiently for the next one. Had been sick the day before to the doctors and missed an excellent trade, so next day I took a weaker version, which worked, except I changed the parameters mid stream. Sheessh! Probably old news to all you smart guys who are so rich from trading options.