I am quite envious of your knowledge and experience. High performance/real-time systems are not something I have a lot of experience with so I suspect I will be struggling on those aspects. In fact, I am probably a very poor coder! Hopefully, when I start discussing and releasing code here some of you guys will point out the flaws and inefficiences therein.
Actually, to re-phrase that, if you have developed your own custom ATS software, why did you do it? Was there a specific feature or capability that you needed to support that wasn't available in commercial offerings? If so, what was it? (if it doesn't reveal your edge)
I'd say the biggest reason I'm doing it is because it's very clunky to get information from other frameworks into C++ code to analyze information, and the cost for the better packages for minor features doesn't seem like it's worth it. Take e-Signal for example. Dealing with time in their proprietary language is a royal pain in the ass. It ought to be something as simple as: SELECT PRICE FROM EMINI_FRONTMONTH WHERE TIME=<something> I absolutely hate using their framework for working with time. I looked at other packages, and they all had trial periods that were non-existent or too low for me to develop anything substantial. Yet, all vendors wanted to charge me several hundred dollars. e-Signal's way of dealing with DLLs also angers me. When there's errors or problems, it doesn't report them intelligently. Some of their javascript language has odd quirks. For example, you can't make a comment with the word 'main()' in it, because their parser will accidentally think that is main. I mean, come on, what kind of half-ass javascript parser is that? Also, their API behaves differently during the day than when the market is closed. You don't get access to tick data and exchange data during data-replay. Dealing with time frames is a pain. If I want the high bar for a period of 15 minutes, it should be as simple as: SELECT MAX( PRICE ) FROM CONTRACT_TABLE WHERE <some data range> Finally, I looked at support. Third-party support has always been dismal for developers and is always worse than for users. The only time in the software industry I've ever gotten decent support from a company was when we've had service agreements. There's layers upon layers of bureaucracy in companies that technical support requests travel through before your question is addressed by a programmer. The products that demand recurring fees (like Tradestation) bind you to the platform. The features that I would use from Tradestation I could implement myself without their monthly fee. Third party products just angered me too much.
I think you are being too ambitious. While it is good to keep in mind various design objectives, you (or anybody else) are not likely to get it right the first time around. It might be better to omit some things and resign yourself to some significant redesign/rewrite at some point in the future. IMHO candidates for omission might be: 1. Redundancy/fail over. Zero downtime is hard to achieve. Sometimes the most bizarre issues can crop up. Assuming no software faults, is the MTBF of a server grade box with mirrored disks, ECC memory, dual power supplies etc going to be less than that of the broker/data feed provider/comms service provider or even the exchange ? It might sound good but does hw redundancy actually buy you much ? 2. Grid computing / clustering. With fast quad core CPUs here now, what are you going to be doing that can't be done on an SMP box ? Running on a single box has got to be easier in terms of software design. (As a curiosity, I read somewhere about a research JVM that distributes threads across a network). 3. UI - Browser and Swing. One or the other - not both. I can well imagine that both is a lot more than twice the work of just one. My preference would be Swing for performance reasons and it would probably be easier to code. I think that things like GA and ANN are actually relatively easy to incorporate. Whether they help you make money is another issue. I have had a little play with jgap for optimizing trading systems and I had no real problems integrating it. I don't profess to have any expertise with GA but it's not hard to use the library. I've had a look at joone and not done anything with it, but it looks very usable and I didn't see any real stumbling blocks in integrating it. At one stage I did write a sort of an ATS platform for backtesting/research and live trading including portfolio level stuff but I've put it on the backburner until I learn more about the market. It needs to be rewritten anyway - hence my opening comments. Anyway that's just a few rambling thoughts.
Nice summary. I wanted to puke when I tried doing anything with EFS or whatever it's called. There's a myriad things that can't be done, it's all tried to charts and I'll bet things like EFS are dog slow compared to C/C++ or Java. Things might have changed since I tried, but even to get data out of the damned thing was a real PITA.
Thanks for the input and feedback. I suspect you are not the only one to come to that conclusion based on my high level system requirements! With that in mind, now is perhaps a good time to adjust expectation levels for those that I hope are following along with this thread. For me, the project and hence this thread is a long term effort. I'm not expecting to get things done in the space of a few months. I will be taking things very slowly. With an adjusted perspective on time frame, perhaps things don't look quite as ambitious. Another product may come along in the mean time e.g. Frosty's frostengine that makes this thread redundant. I'm 100% certain I won't get things right first time around. I have already built very poor ATS before and they certainly aren't right! My intention to have collaboration on ideas is to minimize the mistakes made going forward and to leverage lessons learned from other people if they are willing to share. I would like to follow an iterative development process. I would like to produce a solution that is just as comfortable working on a desktop PC with a retail data feed and broker interface as it is working hosted at an exchange or data warehouse on multiple servers with an institutional/exchange feed and execution. The fail over I was referring to was with respect to application server or JMS. Depending on your level of knowledge building this can be handled by the third-party components e.g. J2EE servers. What am I going to be doing that can't be done on an SMP (I assume you mean shared memory multi-proc) box? that would be telling Again, depending on your level of knowledge I don't forsee it being too difficult to grid enable an ATS if it is designed with that in mind to begin with. I may be proven wrong. Actually, I was going to do neither (I hate GUI programming)! My intention of mentioning the client aspect of the ATS was to ensure that the ATS server was built with absolutely no dependency on a GUI thus leaving the door open for any kind of GUI later on as long as a suitable API was exposed. I'm thinking of situations where the ATS is hosted and you want to connect to it without using terminal services/VNC etc. you could have a web client which was accessible via PDA/cellphone/PC or you could have a thick client which connected over the Internet to your server via web services or some other protocol etc. I think Eclipse RCP would make a great base for an ATS system but it would be more along the lines of a Rightedge product or even JSystemTrader where the GUI was somewhat integral to the ATS. After much deliberation (yes, I've been planning this for some time), I decided I did not want to pursue that route and prefer a server-based solution instead. Of course, Eclipse RCP could still be used as a thick rich client. I've also had a play around with Spring Rich which is a neat solution for building Swing-based clients - so that is a possibility later on too. However, it is still a very young project. I certainly don't want to re-invent the charting wheel. There are plenty of charting packages out there. I even don't want to build a charting component based on JFreeChart or even BIRT charts (part of Eclipse BIRT reporting). This is largely because I hate charts and my algorithms don't really rely on traditional TA indicators etc. However, I'm aware that most people do come from a charting background and progress from custom indicators to alerts to finally ATS based on these things so this may need to be accomodated somehow. I know what you're thinking...Waaaay to ambitious! Yes they are, I have used both. Thanks for bringing that up. To take another preemptive strike before others chime in, lets make the assumption on this thread that we already have an edge and successful trading system that we want to automate. Or at least, want to test some ideas out that can't be done otherwise. Your comments and input are greatly appreciated. I would love to hear about your experiences and any insight you gained from developing your ATS platform as this thread unfolds. Perhaps you can take away some ideas from this thread and we can mutually benefit. That's the idea anyway!
Thanks for the input. Sounds like you had plenty of reasons for developing your own ATS! I've never experienced e-signal as I heard too many negative comments about them - largely on price. Guess I dodged that bullet. How does one put a price on software you can control in totality? Perhaps we're just control freaks.
Okay, let's get things started. I hope the result of collaborating on ideas should yield enough information to produce a software specification of sorts. Not necessarily in the strictest IEEE sense but certainly enough information so that one can go away and get something useful developed - yourself, your team or outsourced etc. In addition, if we start from complete basics perhaps total newbies will be able to follow along and get something useful from it. Indeed, I don't mind calling myself a newbie! At the same time, I will be writing some code in Java. As such, a lot of discussions will be pertinent to Java and Java components. Hopefully this won't put off others who are developing in different languages. I'm thinking of starting a project on Javaforge.com to host the code I develop etc. I don't want to use Sourceforge because it sucks right now! If anyone else has a better suggestion than Javaforge.com please point me in the right direction. Thanks. As I said before, I will be going very slowly. This is one big learning curve yeah!
I think all the low-cost third party products are useless because they focused too much on the non-technical user. The generality hurts people. The high-cost products worth using are, well, high-cost. Plus, we probably both have more time than money. There's very few people here on ET who're already wealthy. In between contracts, trading losses, and trying to stay employed, who can afford a $1000+ product but the people who're already wealthy? That's roughly 1% of my pre-tax income. It's even worse after taxes. The whole self-authoring of an ATS is personal, too. I'm somewhat annoyed with career-mobility as a software engineer. Consider that money has time value. They teach this in basic financial calculus classes. They say money now is worth money * exp( rT ) at some time T in the future. The formal acknowledgement of this in finance is the concept of interest. I believe that, along with pricing models for financial instruments, there are pricing models for programmers. In our profession, most employers don't compensate us in the form of that "interest". For example, suppose you work in C++ for some hedge fund and you work on user interfaces. During that time period, perhaps a technology like C# takes over and becomes the dominant technology. The employer might give you a raise to compensate for inflation, but he will never give you a raise or bonus to compensate for your devaluation at a time T in the future on account of the opportunity cost you incur by not knowing the ins and outs of C#. You paid for this cost by being locked into your employer's project. We programmers are like money in that we get less valuable over time, and the finance guys are winning because they get us at "market rate" and never actually help us account for skill loss over time. Here in Manhattan, I get, depending on a bonus, anywhere from $90,000 - $120,000 annual salary in an average job. People laugh at me and act as if I'm being ridiculous, but I feel like this is poverty and I'm starting to believe I should receive welfare assistance. The long hours from 60 to 80 hours a week, staying in the office until midnight, trying to balance personal programming projects on the side, etc. In the event of a contract ending or a job loss, the hedge funds around here make you take all kinds of exams, provide transcripts, take IQ tests, etc in order to get a new job. This is _after_ you've already proven you graduated from an expensive, good school with a reputation. Employers around here put you through so many hoops to work for them, and then when you do get employed by them, you implement things like log managers or you support some user interface for some trader. None of these tasks justify the intense sort of scrutiny you undergo during the interview process. As if it required an 800 in math on the SAT to implement a linked list. The compensation is only slightly higher to account for increased prozac and therapy consumption, and there's no shortage of headhunters around here to feed you lies about what you can earn. Hedge funds almost never put you in front of the money management. While you may provide a framework for quantitative models to execute on, the hedge fund will never invest in your education to understand those quantitative models or take you anywhere beyond just earning your measly $90,000-$120,000. In the event you work on any feeds or data distribution systems, it's usually correcting the problems of lesser programmers. In essence, it's totally a losing game for the Wall St. programmer in my opinion. I think the Wall St. programmer has a slight edge on the Silicon Valley programmer in that he might possibly make the transitation to being a successful trader, but that's about it. What is the natural course of action for a programmer then, really? The forum has answered this in that we see all these programmers with financial experience attempting ATS construction. This means to endure the trials and tribulations and go with a ground-up approach. To use something like AMIBroker or QuantDeveloper .NET is almost a form of shame, I think. I'm not asserting that these are inferior products. I am, however, saying that programmers have to account for their devaluation. Even if there's some degree of wheel-reinventing going on, it is our responsibility to help ourselves progress.
I couldn't agree more. That's why I gave up programming many years ago. I find programming jobs for other people are generally soul destroying activities. I refuse to make money for other people anymore. However, as a trader, I came to the conlusion that my programming experience despite being very mediocre was actually possibly an edge over retail traders that didn't have that knowledge. Yes, its interesting that a large percentage of ET members seem to have a technical background. Perhaps not that surprising. Interesting perspective. I just think it's about time we had a solid open source ATS framework - even if it means re-inventing the wheel a bit. Actually, nothing would give me greater pleasure than putting all of the commercial vendors out of business! But that's a separate matter and not likely to happen any time soon.