I've only had a cursory glance at the details of HDF before Christmas, so at this point I don't really know what I don't know! I'm thinking more about how to cleanly structure datasets to support things like storing multiple timeframes for a given symbol or storing options data for a given root symbol, dealing with gaps in data etc. I'll be doing some more research on HDF today/tommorrow and will probably then have some more specific issues I'll be soliciting advice on. Thanks.
The documents are very complete. I don't know a thing about the Java interface, so I obviously can't help you specifically there. The tutorial is the place to start. Then, the user guide is very detailed. The API docs are concise. Even though it's written in C, it lends itself to an object oriented design in it's structure. It's correct to say that it's just like a directory/file (tree) model extended into the file itself. Including attribute meta-data. After you've given it a look-see, feel free to ask me anything I can help you with.
Okay, I've been playing around with HDF5 a little bit. More specifically the HDF5 java products: http://www.hdfgroup.org/hdf-java-html/ The Java GUI tool: http://www.hdfgroup.org/hdf-java-html/hdfview/index.html makes getting up to speed with HDF5 all the more easier. You can interactively create HDF files, add/remove/edit groups, datasets etc. Miracles of miracles, the installer for HDFView sorted everything out and there were no classpath or JNI issues to resolve. The program just ran. Now come the questions for those of you that are HDF users: 1) Do you use multiple HDF files, or put everything in one HDF file with multiple groups? 2) What naming conventions have you used for files and groups to deal with symbols and different periodicity. 3) How have you defined your datasets for quote and trade tick data. 4) Data Layout and Compression - do you use contiguous storage or chunked with compression?
Just a suggestion here, but I notice you're starting to accumulate more and more tasks related to data... might I suggest a Data Manager? You mentioned that the Strategy would be able to select which data provider to use, not sure that's the best OO design. Why should the Strategy be concerned with where it gets its data? There can only be one "best" feed for each data type (quote/trade/depth). I can see the case for knowing which broker to use, but feeds can be consolidated to a single Object which would make the ATS that much more robust and lessen the load of the Strategy Object. My 2c.
I apologize for not responding earlier. The thread didn't spark quite as much community involvement as I had hoped for. So, I haven't been to ET recently and I've put the development on the backburner. Instead I've been brushing up on some technical knowledge etc. so that I may better architect and specify a solution for my purposes. Agree. A data manager of sorts was implied. The pertinent points may have been lost or buried in the pages gone by. Effectively, the strategy is coded to a data feed interface not a concrete data feed instance. The specific data feed a strategy uses can be configured at delploy time (via a deployment descriptor/configuration file) or even at runtime as deemed fit. The flexibility to allow for different data feeds may not be neccessary for most people however. Your 2 cents are welcome.
The answer to "Why are we doing this?" or why I would be doing this is because of my personal lack of trust of brokers. If you trust your broker, it only looks like you want more features. These features can probably be built on top of existing API's. I don't want the broker to see my strategy and manipulate the market. I have read that by definition, a "market maker" is allowed to manipuilate the market. Have you seen "noise" or "spikes" in a chart? That's BS. Especially if it only happens to your account (which I have seen people complain about). I would not want to "expose" my stop loss to my broker, just to be spiked out. If you want an open source project that may be able to save you from reinventing the wheel, check out https://humaitrader.dev.java.net/ which is now http://www.aiotrade.com/ As for me, a small fish in an ocean, I am currently testing strategies in MetaTrader under MQL with demo accounts (assuming a lack of broker interest in my source and compiled strategies -- which may be an incorrect assumption), but I intend to only "expose" buys and sells to my broker when I go live.
Hi there - great thread! I'm by no means a FIX expert, but I have been researching it recently with view to a similar level of implementation. I'm not sure if the following will be useful to you but thought I'd chime in just in case... There is an open-source FIX engine here http://www.quickfixengine.org that is well supported. Also, you may want to consider restricting your FIX implementation to an earlier version to begin with, such as 4.1 - and then build on the knowledge to work up to supporting the later versions. (The specification for FIX 4.1 is a *much* easier read than later versions ) Finally, http://www.fixionary.com is a great resource for searching the different versions of the spec.
Glad you like it, I hope there are one or two things you might find of use buried in it. I haven't been updating the thread recently for a variety of reasons but I haven't entirely abandoned the effort either. Please feel free to add your own design decisions and commentary to the thread even if it's old stuff. Thanks for the info. QuickFIX is what I was using or more specifically QuickFIX/J, the Java version: http://www.quickfixj.org/ I had started on the road to experimenting with the TT FIX adapter etc. but didn't get very far. It's on my list to resume at some point. Let me know how you get on and who/what you'll be connecting to and also any lessons you learn along the way!
Well, you're about the only one getting into it. Sadly there wasn't enough interest in this thread at the time combined with not being able to commit enough time to keep it going on my part. This thread is two and a half years old and some of the ideas I wrote about here are also now a bit outdated. If you're interested in open source Java ATS projects that actually have a community larger than one person why don't you take a look at: 1)Marketcetra: www.marketcetera.com (biased towards infrastructure rather than ATS) 2)ActiveQuant: www.activequant.de 3)JSystemTrader: http://groups.google.com/group/jsystemtrader For .NET there is also: 1)TradeLink: http://code.google.com/p/tradelink/