Starting from absolute basics. This is how it works: Incoming market data + other data ----> Black Box[ Analyze the situation and come to any buy/sell decisions] -----> Submit order(s) to broker! Rinse and repeat. That's it. How hard can it be? The same process is applied for backtesting and forward testing. Let's break it down a little bit further: The general theme that I have seen in many systems is that there is an "inner loop" of sorts. This inner loop is what triggers the black box to do it's magic and come up with buy/sell outputs. Interestingly, Wealthlabs has the loop in the strategy logic (pros/cons people???) More specifically, the black box or bit of code that contains your money making algorithm is run or called when specific situations occur. These specific situations may be one of the following: 1) When the latest bar of data for an instrument time series is completed. (Multiple instruments may be monitored) 2) When the latest tick for an instrument is received. 3) On an timer basis e.g. every 1 minute. Regardless of market data coming in. 4) When some other custom situation arises (Ideas please) Thoughts, comments? Additions, errata?
Hi Valdis, Being a vendor of a commercial offering in this product space (QuantDeveloper) and also having in-depth and highly technical experience you bring a fairly unique perspective to this thread! Your contribution is most welcome though I'm not entirely sure what you are implying. I'll give you the benefit of the doubt that your comments are well intentioned. It's interesting that your commerical product was based initially on an open source project unless I'm mistaken. As I alluded to in my inital post, my expectation of people contributing their code etc. is close to zero! My only hope is that we can collaborate on ideas and best practices. If people with experience such as yourself do not wish to contribute, I fully respect that. Your knowledge and experience is no doubt hard-gained. I wish to move forward starting from the absolute basics so that even newbies can follow along and hopefully catch any fundamental flaws in the system before it progresses too far. I will be writing code as we go along. In this respect, I hope it to be an educational thread too. I will be shamelessly stealing ideas, and even API concepts from various commercial products including yours. I hope you take that as a compliment!
Think of it like an operating system that manages state and consider ticks as hardware interrupts. With regard to the loop design, one of the problems is that any computational models that execute inline with the loop will results in a compute-time based latency before the next tick/packet is serviced. A better approach is to dedicate the "loop" to doing one thing and that is pulling in ticks. This isolates you from problems when there are bursts of data. Like a hardware interrupt, you only want to capture relevant data and post it elsewhere for post-processing later. The key is to make sure the data is received and not lost. Past experience dictates that ITCH and ARCA feeds do, in fact, experience large bursts. Those bursts can be deadly for poorly written software. Any dropped connections or TCP flow control problems would result in potential miscalculations of theoretical price if, in a discrete time model, price p(t) is such that it's a function of p(t-1). I have seen poor implementations where calculation time was so high that connections would result in drops or delayed data. Even if the forecasted volatility or theoretical price was good, it didn't matter because of the delays. In the 2.4 Linux kernel, for example, a device driver would possibly block on a wait queue. In the event of an interrupt or I/O completion, the thread processing information would be woken up. They divided the concept into fast handlers and slow handers. This is largely the model I am using, and view ticks as a form of asynchronous external interrupt. Windows NT sees the whole thing as various IRQ-levels and uses a similar concept in the form of what are called delayed procedure calls (DPCs). I'd try to avoid actual blocking where I can. I try to avoid context switches by using spinlocks in cases where there may be high contention for information. For example, certain products and feeds result in an almost constant stream of data. Write queues become busy, so it makes sense to incorporate mechanisms to use test-and-set based spinlocks instead of constructs which result in context switches. Treat models as threads or components of execution with their own mini-contexts. View model execution in the manner an operating system kernel views threads, and avoid starvation for models which require more immediate responses.
Suffice it to say you are way more techie than I ever will be! Perhaps a poor choice of words on my part. I didn't mean to imply an actual loop. Agree. This is where I plan to utilize messaging, more specifically publish/subscribe topics. Asynchronous with multiple subscribers/listeners. In combination with queues or queue features so that data does not get lost. Durable subscriptions etc. Much much much more on this later, we are getting a little ahead of ourselves. Actually, bursts are something I've been able to ignore with the snapshot data feed from Interactive Brokers but I would definitely want to deal with this in a robust fashion. Yes, you are way more techie than I. Agree. The intention is not to block unless necessary. Keep 'em coming.
I have a system which requires user interaction to trade (i.e., pressing a button to confirm trades). Otherwise, you might consider it an ATS. The sole reason I made it was for quicker order entry - I don't have to type anything - I just press a button. I have attempted to start general purpose ATS projects and I always seem to abandon them eventually. I have thought about why I don't pursue the effort further and I think it is because there is no guarantee that I am going to find a successful automatic method for trading the markets even after completing the project. It has been much easier for me to observe or perform real-life trading and then take that experience and expand on it to make money (rather than build an ATS to find a way to make money). I have never found a way to make money in a completely automated fashion - I can make programs to alert me to potential opportunities, but in the end I am always the one making the final decision on whether to enter a position. In my case the only time automation has paid off is when I already figured out a way (through real manual trading) to make money. And in these cases, I didn't design a general purpose automated trading system but instead specifically designed the software for the method. Sure, I have saved some time by reusing code from other projects, but essentially all of these systems have started from the ground up. However, I know that others have succesfully made profitable automated trading systems (with a general foundation) and that gives me hope that I may be able to build one someday too. Sorry for the rambling - but when I read that you are assuming the reader already has a method to make money, then it seems easier just to make custom software for that method. The hardest part is finding the method - making the software is the grunt work.
Feel free to ramble away. You're right, having a trading method that works is definitely the hard part! The software is arguably the easy bit. I would also add, that there are some trading methods where it is just not feasible to be tested without custom software and that too is a valid reason for developing a framework; so that ideas can be tested out without the artificial limitations that are imposed by commercial software. More tomorrow and/or the weekend...
In the words of the wise: "Let's get this party started" I will be drawing up some high level diagrams for the system architecture soonish so that anyone interested can discuss pros/cons ommissions/improvements etc. At the same time, I'd like to drill right down and focus on getting a particular component implemented. The one I have in mind is the data feed component and data feed provider adapters for the various sources that exist. My reasons for choosing this component to start with is that hopefully it will yield something useful for people to actually use that doesn't require the entire ATS framework. Indeed, hopefully most of the ATS framework components can be developed in such a way as to be used individually with minimal dependencies on other components in the framework. Dealing with the data feed interface presents some interesting challenges: 1) Normalization of instruments/symbols 2) Normalization of data feed capabilities etc. The two feed interfaces I have some experience with are: 1) Interactive Brokers 2) OpenTick If anyone else has API documentation for other feeds available and are able to detail it here, that would allow me to consider as wide a range as possible when designing the interfaces. I'd rather not subscribe to or open accounts with any other providers myself. I am in discussions with www.limebrokerage.com with respect to their API so hopefully I'll be able to incorporate that in. I'll detail my ideas on the matter and then people can jump in and critique.
Confidence is a little shaken in Javaforge.com since it breaks when I try to sign up! Ideally I'm looking for somewhere with Subversion support. Integrated documentation, wiki, issue tracking tools etc. would be preferable. I'm even considering a Subversion and Trac solution. If anyone knows somewhere that supports continuous integration tools like Anthill, Cruise Control, Continuum etc. that would be good too. I'm just getting up to speed on Maven so like I said, one big learning curve!
Little tip for newbies: You might find this thread, or indeed all threads easier to read and follow by adjusting the number of posts you can see per page. Click on Your Account at the top, then Edit Options, then adjust Default Posts per Thread. I've set mine to 40.
More admin stuff... I'm not up to date on hosting, domain name deals these days. Anyone with suggestions on super-cheap domain name/simple hosting combos? Primarily I just want to lock down the domain name for Java package naming purposes and have a basic page with some brief details and a link to the code hosting site etc. Curiously, mojotrader.com is already taken I'll make do with mojotrader.org given it is an open source effort. Yes, I've decided to call my development efforts MojoTrader or just Mojo