What is the benefit of analyzing real time data? If you go through all of this and come with a profitable strategy how it will be used in practice? If the real time data are in the milliseconds range you have to be closer physically to exchange to get the benefits from real time data strategy. A real time strategy is profitable in a long run if you are a broker/dealer and have the benefits of sending the order flow to the exchange. Then the speed matters a lot.
Excellent question. There are EOD traders using TickZOOM and I begged the question of why use a real time system for EOD trading. They gave me several answers: 1. Many exchanges now have over night sessions. This means that by the time they get the EOD data updates in the evening, the market has already moved on the over night session. Monitoring a real time feed allows EOD traders to get the settlement price "in real time" rather than wait for the EOD data provider hours later. 2. Some EOD traders want to trade shorter term or more sophisticated orders which can't be expressed by an ordinary stop or limit order. So they use an "emergency stop" and let an order server handle the trades. 3. Others want to respond withing milliseconds to certain events rather than wait all the wait till end of day. That's just another way of restating #2. There are more reasons but these are the most common reason for monitoring a real time data feed. Sincerely, Wayne
Actually, I am intending to analyze tick data and somewhat quasi real-time data. Real Time data is supposed to be added to the historical data store, as it comes in, rather than be accessed on slow internet speeds it can be accessed by multiple simulations at LAN speeds. I agree, but then thats what DMA technology is for. Too early for me yet to even dream of it, but I am targeting a DMA platform, where my order go directly to the execution exchange on a leased line, of course under a sub-id issued by the broker. Negative side of this would be functioning on a given exchange only, to which you have DMA access, rather than a multi-exchange execution service/broker. Actually, I am targeting running multiple simulations in parallel with actual trading. Don't really know where I will land up. Hypothetically, assume you have 100 trading strategies, built over say 25 years and finetuned over the years. My assumption is that a few of them would be profitable in one time frame, few in another, few would be good to day trade, few for swing trade & few for scalping but at any point of time most of them would be unprofitable. Another assumption is that for short or intermediate periods in time, some of the profitable strategies would become loss-making & visa versa. I see GRID based inexpensive computers running all strategies available to you in various timeframes and a master algo keeping track, what is becoming profitable and what is becoming unprofitable...and starting & stopping them as required or rather alerting the trader team that a strategy that had been unprofitable since last n years, has suddenly become profitable. Pick your permutation. All I am doing is creating a local feed & simulation/monitoring system.
If you feel like spending money, have a look at KDB+ or Vhayu. Serveral institutions (as well as data providers) use these to store and distribute historical as well as realtime timeseries data.
After just reading the white paper on Kdb+, they seem to fully understand the performance demands of real time data quite well and the appropriate ways to solve them for financial organizations. You can't get any information on the price which, of course, means it must be astronomical for institutions only. TickZOOM solves many of the same problems but it's data solution can't be called a full features "database" by itself since it doesn't support any kind of query language. TZ does have a data server which maintains a 24/7 connection to a data provider to capture raw data and write it to disk. Then you simply query symbol(s) and data range of data for a historical test. In historical mode, TZ returns the data requested. However, if you're in real time mode, it first runs from the start date using historical data. Then it switches to real time mode and feeds the data simultaneously to disk and through the engine. That's the essence of what's needed for historical and real time data processing. TickZOOM can benefit from some enhancements to move the data from cache to permanent store automatically and optimize the search time using an index to find the start time for a query. Those are on the queue and prioritized appropriately with other items. But otherwise it provides the same type of performance and speed and kdb+ which means returning data in milliseconds for processing even with very large queries spanning years. Sincerely, Wayne
Of course, the data server runs close to the exchange. We run in real time with exchange. Some users of TZ simply run on their local PC which can lose them upwards of 90 to 100 milliseconds of lag. But they usually seem satisfied with that since they're not running high frequency trading. The central issue isn't simply the speed or lag of the data but the total turn around of exchange data, decision, order, and confirmation to reduce slippage between the tick that triggered the trade and the actual fill price. Good point. Nobody who's anybody sends orders to the brokers or exchanges when doing anything intraday. TickZOOM, for example, uses signal order processing which inherently means that the broker never sees real stop or limit orders. Instead, they only see an "emergency stop" and market orders or at the money limit orders.
Do you mean multiple different strategy instances on the same symbol(s)? I think you do. TZ defines "strategy instance" as a specific strategy with it's parameters running on a particular symbol. So a MA strategy with length of 5 on ES will be a different strategy instance that a MA strategy with a length of 10 on ES. That seems to be the next frontier in trading. TZ makes this very easy by internally implementing "signal order processing". (Still assuming these strategy run on the same symbols) That's a nice theory but, in practice, it fails because these all trade on the same brokerage account. In contrast, makes very little sense to create separate brokerage accounts for every strategy to segregate the strategy trades. That's because of minimum balances per account and the loss of advantage from the single, common account portfolio. It seems that what must build instead is a system that can run all the strategies individually to provide separate trade results and statistics but, at the same time, correctly combine the trades from all the strategies for each symbol on a single brokerage account. Additionally, it makes sense in this scenario to use portfolio rules to allow certain percentages of the portfolio to strategies depending on their performance. Why build that from scratch when you can get open source platform that already does all of this so you can focus on writing and testing your actual strategies and start making money? Additionally, think of the advantage of a community of users on the system that test and report issues or enhance it besides just you alone? A commercial open source platform can be a better way unless you find it easier to afford the many months or years of development time instead of a little cash to get a system like this already built that you can enhance. Again, that will take many months or year of work and unless you're funded to do it full time, it will take all your weekends and evenings for a very long time. That's what I did and hope to share with others so they don't have to go through it. TZ simply charges for maintenance and support of the software going forward. Anyway, I wish you success because you're on the right track. I know from experience how hard what you're trying to do really is since I did it myself. Wayne
We agree on your points about high frequency trading. But, we're clearly miscommunicating because there's no "high frequency stuff" involved in anything in my discussion. Let's define high frequency trading to make this clear. Frequency, of course, is a measure of the time between recurring events. High frequency trading refers to having a very short time between each completed trade. How short? If you search the net, that term has come to mean extremely short times, on the order round turn entry and exit within seconds. Trading less often is generally called intraday trading when trades occur every 30 minutes or hours on average. EOD trading means your trades may last several days or more on average, as you know. So to use the correct terminology, all my discussion was related to intraday and EOD trading. In that scenario all the desire for speed as nothing to do with the "frequency" of trades but, instead, with the turn around time from making a trade decision on data, submitting a market order (or at the money limit order), and getting conformation. The faster turnaround can be, the lower the slippage. That is absolutely necessary for both intraday and, increasingly so, EOD traders if they execute trades other than on the close or open only. I hope this clarifies it more. Sincerely, Wayne