This is an interesting thread, and you seem like quite a bright guy. Good luck to you. You've spoken about R a few times, so it might be worth mentioning -- re "chokes on gig+ files" -- Revo's new Win64 product (I assume from context you're using Win) -- http://www.revolution-computing.com/products/whats-new-revolution-enterprise-version2.php http://www.revolution-computing.com/products/windows-64bit.php They're a vendor of a "supported" build of R, and charge a few dollars -- but I wouldn't be surprised if they made you a deal, since you're an individual trader. They're very, very good. One of their team is David Smith, ex-S+, who's the coauthor of the intro to R manual that comes with the base R distribution. And there are also many other ways people solve the big-dataset-in-R problem.
If daily trading stocks/etfs it's certainly very important to make a wise choice of instruments. And actually, as far as i know, the number of instruments which can be profitable in a day trading approach are really very few (and, clearly, also partially dependent on the strategy used). The ability to "capture enough price movement" is certainly one crucial factor. Tommaso
fantastic post, thanks for your refreshing way of writing. There are so many, even in this thread, who come across as being increadibly arrogant and then when reading their material it occurs to me they write in such fuzzy ways they seem to actually not really understand what they themselves are talking about or where they try to get to. You seem to have a pretty clear picture and appear to be on the way you chose. You reminded me on the times back in NYC when I underwent a very similar program as you while working. Good luck to you!!!
I really don't know where this "backtested over 20 years" stuff comes from. There are short term phenomenon that, if you understand, you can exploit. The 20-year requirement really doesn't make sense for market microstructure practitioners because rules change, liquidity changes, and the game changes constantly.
I built a bridge from TradeStation to R via a DLL written in C++ today. The DLL gets loaded by TradeStation, and builds a singleton around a connection to R. The singleton arbitrates access to the R dll [poorly] and returns relevant portions of the resultsets back into TradeStation. For me, the development cycle was somewhat of a nuisance because what would happen is that I would export the data from TradeStation to a text file via the "Data Window" menu. Then I'd go load that as a CSV into R, then run my mathematical ideas in R, then go back to TradeStation and write EL to model what I was doing in R. Then I'd use TradeStation to optimize the parameters and fit. This was just impractical. I cut the manual middle man out, so now there are functions that get called directly out of TradeStation into the DLL. The bonus I get here is the optimization setup will be automated for me and I won't have to write tedious R code to loop over the sets of parameters I want to optimize for anymore. I am happy with my results. Granted, I don't have a super fast system here; however, I don't have the luxury of time to sit around reimplementing time-series or matrix inversion in C++. I don't have the time to build a serious backtesting infrastructure either, and I settled on TradeStation for a reason. Now, if TradeStation gives me my results, I will likely re-use the DLL interface for when I plug Genesis in as the source of the data. Onwards.
Spot on, markets change all the time. This month the nikkei index could start behaving the way zinc or crude oil did in May 1986. No one knows for sure. That's why systems have to work over a very long period of time. Tell me, how will you know when to switch to your 'new' system when markets change? At the point that the old one stops working? And did you understand why intraday trading is not a good idea when you're competing with people with no trading costs? Or, why low variance is important? Believe me, I'm trying to help you here. There are enough brave souls who walk out of well paid jobs thinking they're going to do better on their own and fail for reasons they never tried to understand. Spending a week or two knocking up an algorithm or three is a pretty sure route to disappointment.
As long as he knows what his edge is, there are statistical tests he can perform to see if a <a href="http://en.wikipedia.org/wiki/Structural_break">structural break</a> has occurred... Carry on.
So, all the separate systems/algorithms and the tests that tell you when to switch from one to another taken together form ONE system. I'm suggesting you test THAT system over 20 years' data.
sorry but I agree, you suggest nonesense. While having more data is better than less a lot of empirical evidence shows that testing a high frequency trading strategy over 1 or 2 years of tick data will not perform a whole lot different statistically than testing it over 5 years worth of tick data, regardless of the time frame chosen. Given you scrambled tick data (possibly, depending on the core of your strategy, running a test on some sort of bm discretizations, generated by iid random vars) you can be pretty much certain you are onto something useful if your system makes it through such tests. I see no valid reason in regards to your 20+ year backtest postulation. I could show you a high frequency strategy that performs almost identically regardless of whether I backtest it on 1 year of data during the hight of the "internet bubble" or 2008 data. Surprised?