Yes, I developed from the scratch my own system. (In Java. But please no discussions about all this no-knowledge BS like GC etc. . With the right JRE it *is* even used for real-time military grade applications like radars, avionics etc. ... ). I don't want to start discussing elementary technical details, since there are already a lot of threads on this board about that ... But let me say what I mean under basic: 1. Your data frequency is low enough for standard db storage (I don't want to start this discussion, but for most retail brokers it does not make much sense to go lower than 1s resolution as the execution anyway lags roughly 1/2-x secs) 2. Your system is *not* able to automatically solve eventual problems/discrepancies (for example a disconnect at an unfortunate moment) but rather requires user intervention (therefore just notifies an admin via mail/sms or whatever) 3. No GUI BS, but just a socket based command & control console Such a basic engine is simple and does not take long ... but number of hours strongly depend on the skill/talent/knowledge of the programmer. A good one can do this in less than a week ...
First, thanks for a useful post. This thread has been going astray. Agree with all the 3 points above. I am surprised you mention a week, I would have thought at least couple of months to develop a basic backtesting and execution without GUI framework. But then my IT skills are fairly limited. Some specific questions regarding your setup: What DB do you use? Do you store 1 second resolution data in that DB? For how many markets do you store data for? Do you store market depth? What specific features have you implemented that are beyond what Multicharts or Ninja has? I am a long-term MC user but my biggest gripe with MC is that I have outgrown their capability. So, I need to start looking into enhancing or writing a program suited to my needs.
Actually, for my purposes it is sufficient to store long-term only 1 min resolution (~150 contracts) and keep everything else in memory. A simple mysql db is sufficient for that. As I said before, the trading engine and backtesting are two separate things. Using for the latter something like R or Mathematica is far more time efficient and sufficient for many purposes. Also, coding the strategy once in a high-level framework and then again in a lower-level framework in order to implement in the trading engine is quite useful to uncover stupid mistakes which screwed up the previous backtests ... For instance, one of the things I couldn't do with off the shell products, as far as I know, are interactions between different strategies like internalization of orders and risk distribution.
Thanks Nab for sharing your experience. I need to expand my backtesting and execution framework. This is what I am thinking as a start for streamlining my systematic trading framework. For backtesting, I will continue to use Multicharts and MATLAB. For auto-execution, portfolio construction and global risk configuration, I am going to construct a prototype in excel+vba, with 3 months of data loaded in vba from db. I will probably start with setting up a 1 minute database for 10 futures contracts. I might use sqlserver express edition or Access for that matter - again for a start and link this db to excel. After loading last 3 months data from the database into vba, I will stream the real time 1 minute data into excel. Next step would be to code all my MC strategies into VBA. Finally, construct the trading strategies portfolio and then execution on IB using this excel+vba interface. Sounds like old school, but should get the job done for my needs. Your comments are welcome.
Amibroker's symbol database and data storage facility is very clumsy to deal with. No rolling of futures contracts, no detailed exchange settings for margin nor anything related to how to maintain futures and options contracts. So, let me ask you if you need all those interfaces why not directly coding something on your own in one language that you yourself understand from the ground up and can extend at your own liberty. You are the one who does not know what he is talking about. I am well aware of the interfaces others wrote to extend Amibroker. It does not change the fact of the shortcomings above and also that its management of broker and data connectivity is very badly integrated. It suffices for keeping the app alive for a day or two but it cannot handle missing fills, erroneous executions, which you cannot avoid from time to time, very well at all. Let's stick to facts not fiction.
if someone is not professional enough to thoroughly learn a programming language, if one is not willing to seriously invest in this endeavor then maybe this is the wrong professional for such person. Fact remains that you cannot make serious money with a half-baked platform that limits you at thousand different ends. As NetTecture correctly pointed out there are retail platforms out there that are so badly coded up that they indeed do result in different backtest results over the same data sets, even without any errors on the user's end. Come on, you cannot deny the work that someone like him has put into for a year or so. Its an affront. I have spoken to him and he comes across as honest and knows what he is talking about. And I believe I also know what I am talking about. What ax I have to grind with retail platforms? I have nothing to sell to anyone here, nor do I derive benefit from belittling retail packages. I merely try to do newbies a service by not leading them into false sales promises by those retail vendors. They got the basics right (well some not even) but to make money in this competitive market you need to get more than just the basics right.
I never claimed so. Amibroker is maybe the best out of the whole pack of retail platforms and the most reasonably priced one for beginners but that is it. It also is not primed for serious systematic trading. If you cannot read between the lines and get when someone is being ironic then that is not my fault.
See this is where you really do not understand the picture, so it seems: This industry, this market is maybe one of the hardest industries in the world to make money. If you are after easy money you are better off in tons of other industries. This is a highly competitive industry where hedge funds and banks plank down hundreds of millions of dollars to get things to work the way they want it to work. This system is not designed for an average "normal" programmer to succeed. Dude, I have worked for over 10 years as professional exotic options market maker (caps, floors, swaptions, and FI structured notes), traded proprietary books on the options and cash equity, and futures side, have traded for over 4 years exclusively foreign exchange products, have known C++ since university, deem myself an advanced C# programmer with intricate knowledge of the CLR and .Net platform and I myself need to work hard every single day, sit for hours in front of the screen, need to constantly re-innovate for what? To STAY COMPETITIVE. You nor anyone else makes money with pipe dreams. If you think you can beat me, or anyone else who spends much more than I could ever imagine to extract alpha in this industry, with a 500 dollar platform or with a little programming here and there then you are absurd. This is not career for a "normal" programmer. Sorry to deliver you this shocking news. If you want to succeed then you gotta work your ass off and work harder than you ever have. The rewards are there but they are not laying on the ground for lazy people to pick up. Period.
good you mention Deltix. I wanted to list it earlier but the name did not come to mind. I actually talked to those guys because I think they have a very well made platform, I almost decided to go with them years ago rather than writing my own but in the end they could not deliver what I was after. And they are slightly overpriced in regards to their deliverables (their full platform starts at around a 100k USD, which is ok if the product supports it but I did not get the feel it did. Similarly for KDB/Kx, they charge an arm and a leg for their database and while I think its top notch I do not believe it should be priced at such ridiculous levels. IT departments with heads who do not have to pay out of their own pockets do not care as long as they have a juicy budget in good times, all they care about is their next promotion by having delivered a high performance database that trading desks and quants can access. Deltix is a good product I like it very much though I have not followed up with them in almost a year.
makes me laugh, a good programmer can create a workable trading engine in less than a week? Yes I can also create a facebook like page in less than a week. I guess we have very different definitions regarding "workable" ;-)