Folks, It's rather fascinating all that goes into handling automated connectivity with data providers for a trading platform. Add to that the broker connectivity and it's quite complex. Just consider some of these issues that tickzoom has uncovered and solved. The biggest challenge of connectivity to brokers and providers is "reconnecting" after any outage however brief or long it might be. So at a high level that means properly geting back fill data. And it also means resynchronizing the automated strategy with the trades. But there's far more involved in the details. For example, it makes little sense to have the automated trading system directly connect to brokers. Instead there must be execution/data service(s) which are separate from the main platform. That makes it possible to run them on the same or a different computer to share data and connectivity with different instances of the platform. That adds another level of potential connectivity issues. Now when there is an outtage anywhere in the connection, those parts which are still functioning must gracefully handle it and continue retrying at short intervals to reconnect. When it does reconnect, it must synchronize. For a data feed, it must pull the back fill data to update the local data. But is that all? Of course not, it must also switch the automated strategy back to "historical mode" meaning that it can't place real trades. This it runs the back fill data through the strategies or portfolio to bring everything current with "right now". What about trades? Of course, during a back fill period and any outage, there must be an "emergency stop" active. As soon as back fill and updating the strategy complete, the strategy must compare the positions and orders to synchronize those that need to be canceled, replaced, closed, opened, etc. Then it must switch back to real time mode to continue trading. Is that all? There's more. The data service is collecting data to a file. Well during a reconnection and back fill, the data server needs to "read" from the file to which it normally writes. That creates conflict. So it has to pause in writing, queue the data to memory while the back fill process is reading the file. Can that take very long? No! Otherwise, it would fill up memory or increase risk that a hardware failure could lose data not yet written. So that file must remain rather small. That means another process must "sweep" these data files from a cache to a more permanent data store as the begin greater than X size. Also, you advanced traders want to have more than one provider as a backup. So those data must be collected separately. And in order to automatically switchover, the platform has to recognize timeouts and delays to switch to another provider. And then switch back when necessary. Add to all this that it must handle multiple symbols, providers, and brokers at the same time. Certainly not a walk in the park. TickZoom will release an update in a couple days which has all these facilities. The only downside right now is that it still, at the moment, only supports MB Trading as a broker. But here's the plan to fix that very quickly. A plan was established to integrate with the open source TradeLink project which has free open source connections to a variety broker servers via a common interface. TradeLink already supports several but without all the facilities above. So this will be a "match made in heaven" to have TradeLink as the standard API to brokers and data vendors while TickZoom handles the rest of the platform. Josh has done great work over time in supporting and maintaining his broker connections with the standard TradeLink API. That's really a lot of work since the brokers are continually upgrading and changing stuff. His system is to use automating testing and keep subscribed and current on notices from brokers and vendors about upcoming changes to their API and required upgrades. So it will be good to support him financially to continue with that since TickZoom is commercial open source rather than free open source. It's just satisfying to have a plan to support some free open source in the trading industry. TickZOOM knows, all too well, that programmers have to feed their kids too! Sincerely, Wayne
What other systems offer such powerful back end automation? Reliable connectivity and automated reconnection, back fill, and resynchronization of strategy trades? Wayne
We can handle synchronization of strategy and brokerage in our scripts too. It took years for software vendors to understand that we needed to have the account available to the strategy via the API.. I doubt that Tradestation does that yet. They had instructions on how to stop your strategy and restart it and synch up manually a couple of years ago!! Ninjatrader and Opentick have account info available to strategies. There was a little piece of software years ago called Dynaorder, maybe about 2001-2002 era, from a guy in Italy that interfaced with Easy Language and IB that was so perfect but alas, he had health problems and he dropped it.. it was so far ahead of all the krap at the time, it interfaced TS2000 to IB perfectly. My strategy knew if the connections were up or not, what the status of the orders were and just everything I needed.. it was years before anybody caught up and some haven't yet even.... with dynaorder I could manage multi instruments and always be using the maximum the account had in it... I worked with [non trading] systems where two entities could affect each other asynchronously.. I used to flow chart in multi colors to get a handle on what all could happen..
Very cool. Eight. Given your knowledge can share any steps that we're missing above? I realized two more. 1. Potentially some error could happen (for example I goofed during testing this) and your back fill data could over lap. Or the gap data isn't fully enough to "fill" the gap properly. So tickzoom now warns that the back fill data overlapped and throws out the overlapping ticks. Secondly, it warns if the first back fill tick is more than 20 seconds from the last tick. 2. This process is delicate and must be fully and thoroughly tested through all the scenarios with every update to the software. TickZoom has automated regression testing and so I have to list all the possible scenarios and make integration tests of each of them. One issue that I want to discuss with current users is if any of this like back fill data being to short or overlapping should cause a "pull the plug" error. It reminds me of a request of someone that tickzoom have a way to remotely and quickly "pull the plug" which means have it close all positions, cancel all orders, confirm this, and shutdown running of any strategies until further notice. If fact, I hear some automated traders want off of certain popular platforms because they don't have any way to very quickly and easily "pull the plug". Anyway, there's more to this that I at first realized when originally developing this platform just for my own use. Sincerely, Wayne
I wasn't being derisive at all. I was just sort of remembering when I went through some of what you are dealing with. It would be good to have all that happen in the background. The "pull the plug" function could be wired to a big foot switch.. or a big red button on the desk marked "PANIC BUTTON" maybe jk I was a lab supervisor and technician back in the day when IBM PC's were not compatible, the DOS days.. I resolved at one point, after working with that mess for awhile that my philosophy was going to be that the machines were there to work for me, not me working for the machines. It seems like Tickzoom would fulfill that philosophy. I followed the first thread when you were in the early development stage here on ET. I liked what I saw, you were working through the design phase very nicely. Lots of people bypass the design phase either partially or completely and just start coding up stuff... that's how we get all the products that don't do what we want them to. They also tend to write off people that suggest features but you've incorporated those forward thinkers into your business model, Kudos..
Eight. Thank you sincerely and deeply. I admire people like you who can see through my admittedly poor P.R. skills and look at the quality of the software development process of tickzoom. Sincerely, Wayne