We've launched our US equities data today, starting with a preview of NASDAQ TotalView-ITCH historical data for our beta testers, spanning from May 1, 2018 to Sep 30, 2022. Coverage dates from Sep 30, 2022 and onwards will be released in January. All symbols, over 8,000 US stock and ETF tickers, available with just a single API call. Multiple data schemas including full order book (MBO), market depth (MBP), last sale, OHLCV aggregates, and more. Captured directly from NASDAQ extranet at our Equinix NY4 colo. Nanosecond-resolution hardware timestamps with PTP time synchronization. Multiple encodings, including CSV, JSON, and our ultra-fast compressed binary format (dbz). Support for batch flat file downloads, direct-to-application streaming, and full order book replay. Point-in-time instrument definitions. Included with our Python/C++ clients and HTTP API. See our GitHub for more details. Pricing begins at $0.45 per GB, and you only pay for what you use. Also coming soon: Real-time and intraday historical data, Cboe FX, other US equities prop feeds, and more. Message me or mention that you were "Referred from ET" when you sign up, and I'll send you an invite with free credits right away.
We're also happy to take any questions about the use of this data here. It doesn't have to be Databento-specific! I'll be joined by two of my colleagues with microstructure knowledge: - Renan Gemignani, Director of Engineering at Databento. Renan was formerly Team Lead in Equity Connectivity at Flow Traders. - Nick Macholl, Senior Software Engineer at Databento. Prior to this, Nick was a Senior Software Engineer in Test at Tower Research. Ask us anything!
My (naive) understanding of NASDAQ TotalView has been market depth is accumulated by only the top order from each MM/ECN. Is this true for Databento or are orders consolidated irrespective of the MM/ECN?
To follow up on my previous post, I've realized the answer to the question comes from the difference between TotalView and OpenView solutions NASDAQ offers. OpenView being only the top order for each MM. This leads me to another question: since this is non-display data, how will licensing work for non-professionals vs. professionals for both historical and live data? Again, my (naive) understanding is offering non-display data to professionals comes with a very large licensing fee. Will only non-professionals be allowed access to Databento live/historical? NASDAQ redistribution fees do not work on this "pay-per-gb" plan Databento will support. It seems like users who stream for a short amount of time will lead to Databento incurring losses due to the fee charge regardless amount streamed, and Databento will begin bleeding money. Am I missing something?
We're expecting a Feb 7, 2023 release date for real-time data of all of the existing datasets - CME, NASDAQ/BX/PSX TotalView-ITCH etc. We've already been capturing many of the prop feeds in real-time. The hard part that's left is adding on all of the "quality of life" features like the frontend UI, binary->{csv,sjon} transcoding, symbology etc. that comes with the territory.
Looks like you've already answered your own question. Yes, TotalView is the highest granularity feed available from NASDAQ and covers all orders at all price levels, and not just top order from each market center like the SIPs. Likewise we are expecting to release TotalView for BX, PSX; Depth for BZX/BYX/EDGA/EDX; Integated for NYSE etc., all of which will provide a full order book unlike the SIPs.
You've indeed recognized a tricky problem with real-time licensing. Because of this, our pay-per-GB model for live data is slightly different. For example, let's say non-display fees for a Cboe venue is $1,500 per month. What we do there is to pass through the venue's monthly license fees when you activate the live data. But we don't apply any upcharge unless you consume the data. So you only commit $1,500 per month and we have zero profit margin until you start consuming the data. Contrast this with nearly every other data provider - they either (a) charge their own monthly fee and also require you to contact the venue to pay your own license fees OR (b) they bundle the venue's license fees and their upcharge into a single number. We think our model is still superior: With our approach, you're paying the least when you're not using the data. Our approach also lets you hitchhike on a license you're already paying for. (a) is misleading because you might be quoted $3,500 per month and then you later contact the venue and find out you are on the hook for another $1,500 per month. (b) is not transparent because you might pay $5,000 per month and it's not immediately obvious that your vendor is collecting a huge cut. This varies with venue. Many venues, like NASDAQ, only take the position that "use" in "non-display use" applies to the execution, not the manner in which you're consuming the data. This is why many retail charting applications will receive a retail feed over its API, yet you don't have to pay a non-display license for it. On the other hand, this is also why if you have your own iLink on CME via sponsored access, you're almost automatically treated as a non-display user, even if your iLink is just dormantly hooked up on a trader's desktop click trading GUI... I think what adds extra confusion here is that most institutional data providers don't care about such nuances because they only cater to a specific class of users, so they're misclassifying your usage when they report it to the venue. License misclassification is a big problem. One common case is that we see users who operate an autospreader overpaying for their data license. Another thing we're doing differently is that we act as a vendor of record and report the entitlements, so you might only end up paying the per subscriber fees.
Hello, congrats! So you’re planning to have NYSE OpenBook and ARCA as well? If I only use historical data, are there any exchange fees? Will I need to consolidate all the books myself? For example, can I request MBP for all available books? Edit: Also. Can I request only top N levels to reduce amount of data?
Thanks. Yes, we've been capturing NYSE and ARCA. To be precise though, we are capturing the Integrated feeds. We do provide MBP synthetically constructed from the Integrated feed, which has the same semantics as OpenBook. (Keep in mind, the naming is a bit confusing: Integrated is the feed based on the newer generation Pillar infrastructure and includes both MBO and trades, whereas OpenBook is the legacy price-level feed. Based on your question, I'm fairly sure you'll prefer Integrated since it carries a superset of information.) There will be no exchange fees for any historical data of these venues. ICE/NYSE is weird in that they might require some trivial paperwork filing for approved use of historical data, and we're still working through that to make sure there is no paperwork burden to the user, but in any case they don't assess any fees for historical data. You have to consolidate all of the books yourself for now. We do have a consolidated feed of all prop feeds + TRFs in the works that takes the complexity away from you and gives the option of subscribing to the full US equity market synthetic book (including odd lots) or the protected Reg NMS quotes. But this is slated for around September 2023. It's internally referred to as "Databento SIP", for lack of obvious product name yet - it's intended to be a superior alternative to the CTA/UTP SIPs. We only expose MBP-1 and MBP-10 as convenience formats. Arbitrary MBP-N is not currently supported and will have to be done through MBO. A user can fetch MBP-1 to save costs over MBO, but MBP-10 will be more expensive than MBO. We do have reasons for this: - MBP-10 is not much larger than, say, MBP-5 because most of the book activity happens in the top levels. - We actually encourage our users to construct MBP themselves from MBO. We actually deliberately made MBO cheaper than MBP-10, because it's a win-win option: it's more efficient on bandwidth for both sides and it's more performant and information for the user to construct MBP on client side. - We plan to publish a reference order book implementation and example code so users will find it easier to use MBO.