Are you getting a kickback from price action lab? You seem to be just as obsessed with data mining bias as that website is... Also, can you really take a website that is telling you not to buy trading systems... yet is selling a trading system seriously? That being said... I agree... don't buy trading systems (including PALs)... If they worked, they wouldn't be for sale.
Not sure if you're asking me or the OP... but I'll answer: I have no idea what to think of his systems... He clearly knows a thing or two about statistics, which I appreciate... but I also see it to be a sales pitch more than a proof of concept for his trading systems. It almost seems like a politician just throwing numbers out there to confuse the masses. The systems may work, they may not... but they certainly aren't working THAT well, otherwise they wouldn't be for sale. His claim that he is only selling limited software licenses so he can cover his development cost is bogus... the profits produced from trading the systems should take care of the dev costs. In my opinion, his systems are targeted towards those who are fascinated with highly technical trading systems... but don't have the slightest idea how they work. Just my two cents.... they probably aren't even worth that.
This article is a perfect example of why you must forward-test any system, either purchased or home-grown. The big issue is: How long must it be tested ? The biggest factor in that determination is "average time in trade" (ATIT)....5 seconds, 5 minutes, 5 hours, 5 days. The longer the ATIT, the longer the forward test must be.
How many observations make a relevant study? Sharpe ratio doesn't seem reliable in the hundreds, imho. Probably worse for Drawdown. I remember elementary statistics saying the magic number for sample size is 30 or something like that.
You need enough to "reject the Null Hypothesis", with a given confidence level. With something as complex as the market, Monte Carlo simulation is the best way to go about it, in my opinion. If you make 15 trades in a day, with an average holding time of 5 minutes and a standard deviation of 1 minute. Set up a a monte carlo study that results in 15 entries uniformly distributed across the day, with holding period statistics that match your trades. Run this simulation several thousand times... if the monte carlos are producing returns that are greater than your backtest... your results may not be statistically significant. If 19/30 monte carlos produce returns that are LESS than your backtest, than your result is significant with 95% confidence, using the Binomial Distribution with a probability of success of 50%. All this means is you are 95% sure what is happening is not random......it DOES NOT mean 95% of your trades will be winners.