"THAT" really isn't a system. SEC regulations change, rules change, and where your liquidity will come from also changes. You really aren't going to find microsecond timestamps from 2001 as hard as you try, and even if you did, it probably wouldn't help because even the ECNS that were around then are different. You find an arb and you milk it for as long as you can. Once the arb disappears, find another one and run with it. If that class of arb disappears, find another form of arb.
But what HE is saying is that he is capitalizing on market microstructure inefficiencies, which only last a very short amount of time. In fact, he may take advantage of an aspect that only exists for a month or two before it disappears. If he is lucky, it will work going forward. In this case, backtesting is almost meaningless, because the inefficiency didn't EXIST six months ago, much less twenty years ago. So he isn't trying to develop a single unified strategy, he is trying to develop strategies to take advantage of market inefficiencies he notices.
TradeStation is a fantastic broker. I asked for help with a sort of exotic problem I encountered and they called me on the phone to help me out today. I've never had a broker do that in my life. This is by far the highest quality retail platform for testing and running ideas. It's not what I have on the inside of big banks, but for a retail guy, I don't think you can do better for the price.
Excellent thread! Best of luck to you and congrats for having the guts to take a big risk in your life. I can respect what you're attempting and it sounds like you're in the correct frame of mind to be successful. How much and what method you use for backtesting is ultimately up to you. As with everything in life there's a middle ground between not testing enough and spending too much time on it. You could backtest for 50 years and still end up losing money due to the nature of the markets...at some point you have to pull the trigger. Stay in touch with your instincts and you'll know when you feel comfortable to begin trading. The great thing about being independent is that YOU are the boss, not anybody making suggestions on this board. It sounds like you already know that though... With that said, I'm curious about how you're using R in the setup. My disclaimer is that I have absolutely zero experience with R...but when you mention that it's choking on gigs of data, my thought is that perhaps it's not the best tool to use for large volumes of data? I'm currently working on a similar issue where my front-end (.NET) just wasn't giving me the processing speed I was hoping for. I eventually decided to build a dedicated SQL Server environment to store my data and handle all the heavy lifting of compiling millions of records. The database does a better job of slamming huge datasets together than anything I could hope to code myself and I can feed only the aggregated results that I actually need to the analysis engine. Just wondering if you have any SQL exposure and if you've already considered this as an option? Even if you don't have experience, I think SQL is one of the easiest "languages" to learn.
The reason R is handy is because when you model relationships and want to know what's tradeable, R gives you the summaries and visualizations of the phenomenon quickly. It isn't for speed, so much as it is rapid prototyping and understanding the data. I'm only integrating it into my trading software (TradeStation) because testing ideas manually in R is a pain -- it wasn't designed for traders. When I say large data, ... I had someone give me a 3GB file with ticks. Was never practical to load it all, but I tried it anyway. R took a very long time to load and manage. Will this occur often? Probably not. SQL is a bad idea from what everyone says. Most groups I have worked with just seem to have lots of flat files and/or kdb and Q. Everyone raves about Q, but Q is expensive. I haven't had to deal with this problem yet in my home-brew retail trading, but it's an issue at work. Work cannot be discussed on this blog.
Fair enough. I haven't had the issues with SQL that I hear other people complaining about here. I'd like to be the first to tell you that it doesn't have to be a bad idea. I've worked with companies in the past who put far too much emphasis on Excel for example, asking it to do things that are out of its league. They told me not to waste my time with SQL when Excel has some database functions built in...I no longer consider that to be good advice. My point is there are ways to look at data in a relational database that are not practical in other programs...maybe that's too much of a blanket statement. I should say, I've haven't yet found a way to do some very simple SQL operations in an OO language with the power and speed of a well-designed database. My guess is that many of the people complaining about poor SQL performance are not using it very efficiently. They will say that it's not cut out for working with linear data or tick data...but all I can say is that I've had great results myself whether it's daily, minute or tick data. Like any other tool, it's all about how you use it. Database design is already a big part of my background though... So would it be a wise investment of your time to explore SQL (and I'm assuming you don't have that background)? Maybe, maybe not. Anyway sorry for the rant. Just something I wanted to suggest to you, do with it what you will. Cheers
I concur, I actually tried an SQL implementation myself and it was way slower in terms of processing large amounts of data. It stores the data, yes, but requests are very inefficient and way slower than some of the proprietary ways of storing and requesting data. My previous firm also used Q and it looked to me to be the best solution I have seen so far. But then its not just about the ability to make requests and store data but how to operate on them. Example given, Amibroker processes data in a completely different way (array processing) than, for example, Right Edge or Smart Quant. It makes a huge difference regardless of how you store and request the data. I currently work with a C# front-end and the only bottle neck at the moment is the speed of processing large amounts of data. So, yes I am very curious about suggestions. But I am also curious to hear from others who used SQL to get the job done...
Porche's have nice shiny engines, but they will be disappointing to drive if you only use the first 3 gears. Again, it's all about how you use it. I get great results, but I devote a good bit of time to designing the database for efficiency, and keeping in mind which types of requests will be inefficient and which won't. I can't speak to your previous experience, asiaprop, and I don't mean to imply that you weren't getting to "4th gear" but I am curious about how you were composing those requests. Of course SQL is no more of a magic bullet than any other application. I just feel that it gets a bad rap (unfairly) on this board...and amongst traders in general. Anyway, I don't want to hijack the OP's journal with this, so maybe I'll start a "SQL: love it/hate it" thread in another category.
No question it's worth investigation -- I'm just not there yet with my retail-idiot[1] trading. I should also state that I don't have much of a background in databases. I hack solutions together, but I'm not the definitive voice on this matter. My expertise is more in the area of realtime systems and operating system level things, like device drivers, and I have graduate level training in statistics and time-series analysis. The database stuff will come, but I have much simpler objectives. As I stated before, I'm shooting for $30/day average. For this type of goal, the backend systems given to me by tradestation are probably good enough. [1] The idiot being me and my wild dreams.
Here's the thread I mentioned above, if anyone cares to discuss their feelings about SQL in detail: www.elitetrader.com/vb/showthread.php?threadid=168685