I think TPL Dataflow and whatever messaging framework you use are unrelated. You can use TPL in combination with Rabbit or Zero. RabbitMQ and Zero MQ are just the messaging frameworks which supply different transport layers, I only rely on them for sending messages between my different out-of-process system modules. TPL Dataflow is a framework that structures the flow and transformation of data in a larger architecture in very efficient and concurrent ways. I use TPL mostly for the merge/sort (the part where historical data feeds of different symbols are merged and then sorted in order to generate a time-stamp-ordered data stream) and of course each of my strategies runs on top of data blocks. But I also use it right behind the ZeroMQ message retrieval process: Messages are passed to the data block responsible for the particular message type so that ZeroMQ can get back at doing its job, listening to new incoming messages while the incoming message is being processed. I run on a fully concurrent framework.
Please keep in mind that I am not exclusively arguing in favor of retail platforms. I am actually the one that started this thread about building a custom platform in the first place. I am pointing out that for certain applications, retail trading platforms are sufficient. I have had excellent experiences working with Ninjatrader to work as the glue between my custom projects and brokers. It gives me a secondary, third-party verified interface to verify my activity with brokers.I have had few problems with connection stability. Also, in the rare event that I use a chart, Ninjatrader has also been nice. You are right though about back-testing - that was a mediocre experience with all the pre-built solutions I used. However, even if a retail platform could support the level of back testing a custom platform could (for instance, tick by tick data for years of data for multiple instruments), chances are that VERY few people would have the hardware to run this software. Heck, I am running on a cluster with over 90 gigs of ram and solid state drives in RAID 10. Chances are, your custom platform has more bugs than many of these retail platforms. However, the bug-scenarios do not arise in live-use because you debug the platform itself as you test the system and bugs arise. Retail platforms have to be adaptive to wider use-cases. Now, you might be saying that a good custom platform is more robust then a retail platform, and yes, it probably is. However, the "realized robustness", is probably significantly less. More unambiguously speaking, your custom platform can probably be extended to do everything that a retail platform does, but does not actually do it. Instead, it was built to accomplish certain tasks very well, and future upgrades can be made in a modular fashion. As you introduce those future modules - applied to as wide a range of use-cases as a retail platform - there will likely be just as many bugs if not more compared to a retail platform. But, those bugs will be addressed ahead of launch of tested properly because you are both the system and platform developer (another reason why I chose to build my own platform). It is likely that your custom platform is not more stable due to a sounder architecture, but rather, the capacity to work on your architecture alongside the system development process. This is the case with nearly any custom solution across nearly any project when compared to premade commercial software - not just for finance. Let's not get this mistaken: At the end of the day, I prefer a custom platform which is why I use one. Retail platforms certainly have a ton of room for improvement, but I have found cases (more specifically stated before) where they were sufficient.
Exactly...and for these 2 reasons above, I'd rather be trading with my system, applying new trading systems, and backtesting versus constantly debugging and tweaking my custom platform.
It depends a lot on the developer, for me with 15 years of experience I would always develop my own trading software. I've tried 6-7 different platforms and each one had it's own oddities. But for a beginner that totally underestimate such a big project and don't understand the basic design patterns + ignores the fine art of unit-testing, I would definitely argue to go after a retail platform.
I think this sums it up well. Taking the custom route is a decision to be weighted carefully in terms of what benefits it poses - both short term and long term. Like other aspects of trading, it is an investment, and it must be assessed if it is the one with the highest return. For me, custom yielded the highest return. Heck, just the knowledge and insight I gained from building the platform was a great return on my time investment.
I know it's an old thread. Could the poster, or maybe someone else, tell me in which of those modules the quotes are being accumulated? The strategy module is certainly needs an access to the accumulated quotes to analyse them and to generate a buy/sell signal. If the quotes accumulation is done in other module, say in data source, then how strategy module accesses the accumulated quotes? ZeroMQ is adamant about not sharing state between the modules. Thank you.