Thanks for joining the discussion T0pH4t! Your concern for extra overhead from a general purpose solution is a valid one, but only in the most extreme cases where very low latency is required. So it is not relevant to many people. For those that seek the fastest latency possible I suggest writing everything in native binaries and minimizing middleware modules, essentially writing hard to maintain, monolithic ad-hoc solutions to specific use cases. That is very expensive and out of reach for all but the well-funded institutional high-frequency traders. Every message in ZMAPI is prefixed with a byte that expresses the codec that will be used to decode the message. So far I have only used JSON codec as it's by far the easiest codec to work with and it works fine unless low latency is required. Especially those users who want to subscribe to lots of tickers simultaneously with high throughput subscription parameters I'd recommend using a binary codec. FastFIX seems okay but I haven't done a lot of research on that yet. I'll focus on that more if there is demand for that. I'm planning on sending all the PUB emissions in the connectors and middleware modules using a binary codec. The users who prefer JSON output can use codec converter middleware. That way both performance and convenience demanding users can be served. Middleware modules add a tiny bit of latency. This is negligible for most users because even if you program directly to vendor API there are typically many proprietary middleware modules already chained in the upstream so you never really can get very direct connectivity unless you are an institution with lots of funds. This problem can be alleviated in the future by writing the middleware components as native binaries, maximizing their performance. I have chosen to write them with Python for now just to get ZMAPI project up and running quickly. Python code is also relatively easy to read which will make it easier for new developers to get in touch with the code base. Native versions can be and most likely will be written in the future. ZMAPI core middleware modules are not saving any data to databases as this is a higher level functionality -- only required by some users. In other words only streaming is supported by the current set of modules. The methodology you are using can be applied by writing a data mining middleware module that collects data to a database and adds some custom commands that provide access to standardized forms of historical data. This kind of module seems like a very useful one and should actually be written. Good idea. I'm not sure if I understand the last question about data amount differences. ZMAPI provides standardized subscription definitions that include subscription speed settings.
I am referring to things like the fact that some cryto exchanges will give you the fees for orders. Some feeds will also tell you the side of a trade and some will not. Would this just be optional fields on the message format? FYI, Latency can be an issue even if you are not doing HFT depending on how many symbols you are streaming. If you stream enough symbols its possible you may not be able to normalize fast enough to keep up with the stream of data from the data provider. Some feeds I have used have put queuing mechanism in place to deal with back pressure just for this purpose.
Ah, I see. Yes, all the non-standard fields are included in the data emissions. You guessed right: they are just specified as optional fields in the specs. Yes this is the issue I mentioned in my previous reply. Queue mechanism is automatically provided by the messaging library (ZMQ). Optimizing the middleware components and using binary encoder will lead to minimum queue accumulations in those heavy-use scenarios. Thinking of the the way how this may cause variation to the actual receiving times on the downstream end I think ZMAPI should specify `timestamp` as a required field instead of optional. If the vendor does not provide it then the connector module should do it. The normalization time cost depends on the vendor. Some vendors require more work than others. In some extreme heavy-duty use cases a customized solution where there are multiple middleware normalization workers (like obcache module) spawned upon demand could be used. As long as the data is guaranteed to be timestamped by the connector module or vendor, order of emissions can always be restored/synchronized further down the stream with some effort.
Nice. Will put it into my notes. Honestly for Python really huge numbers of financial API. ZipLine. Quantopian. Hope you are not reinvented a wheel.
Hi InvBox, thanks for you message! ZMAPI has nothing to do with Python. Python has been used for implementation but it's really language agnostic. Implementation language may be changed later on for some modules. Some performance bottleneck modules will be written with a native binary outputting language, such as C. One of the modules, ctst4-acmd, is actually written with C# (due to vendor imposed restrictions). ZipLine seems to be a high-level scripting environment for building strategies in Quantopian platform? That's pretty cool but it's vendor locked to Quantopian. Am I right? Since ZipLine is an open source project, ZMAPI probably could be extended to support ZipLine front-end to make ZipLine accessible from everywhere. I'm not aware of any flexible (module chain based, language agnostic) FOSS implementations of vendor agnostic general purpose financial APIs. So I think I'm not reinventing a wheel. If you can find something like that, please let me know!
First I wish you finish your project. Second you will find about 0.0 consumers for free. Look at Quantopian. Business at the first. Open or closed sources are not much matter. Trust me.
Quantopian is a business generating revenue. ZMAPI is not a business. It's a free, open source (FOSS) project. I guess this is a strange thing in the ever so profit seeking financial space. I'm tired of that mentality and I have decided to introduce an alternative. Anyone may use ZMAPI for their business, whether it's for profit or not, or just for personal use. It's up to them. I am not trying to collect consumers. I'm trying to collect people interested to develop this further with me. I'm interested in exchanging ideas and technology with like-minded people. Project with a wide scope like this will never really be finished. For example, more connector modules can always be written even if infrastructure is otherwise considered complete. This is often the nature of community driven projects. ZMAPI already provides working MD module chains. More shall be written soon.