Basically 'it' is blind trading: putting an order in through the API where you don't have live market data. When you do this via the other trading interfaces, it generates a warning. Most orders go through a simple algo that is described here and implemented here. It relies on knowing what the live Level 1 prices are, so it can switch to active execution if the price moves away or the volume indicates it will shortly move away. GAT
Ah, that's what you meant in this context. I don't know if placing an order would generate a warning message: haven't tried it myself. I do recall your blog entries about your order execution algo. To me they look "super fancy"; I implemented something simpler. On the other hand: I select contracts which are rather liquid, thus having only a small bid/ask spread, and thus am likely to get my orders filled within a minute.
Definitely works. Been doing it for years, because my IB account is in the name of an LLC, which means automatically paying the professional rates for market data. I've been using the IB Adaptive Market orders, which sound vaguely similar to your algo, but probably less smart in some ways.
Yep, just wrote a small program to connect to IB, subscribe to streaming quotes of an ICE-contract and place a limit buy order - all worked. As a benefit - the API also returns last day's close (75: "Delayed prior day's Close") once upon subscription to the contract's RT prices, so this can be used for EOD price collection (e.g. write a job that starts every day some time after the market closes - (half an hour after close ? - not sure when normally closing prices become available), subscribe to all price\carry\forward contracts of all required instruments, receive last close, save it to db, exit). I tried on March "CT" (some kind of "Cotton No2" - not sure if that's the right type ): first need to make this call: clientSocket.reqMarketDataType(3); this enables delayed prices, also IB documentation says "If live data is available, it will always be returned instead of delayed data." https://interactivebrokers.github.io/tws-api/delayed_data.html so I guess delayed can be used together with non-delayed. int ctrIdCT = 11115; IBApi.Contract contractCT = new IBApi.Contract(); contractCT.Symbol = "CT"; contractCT.SecType = "FUT"; contractCT.Exchange = "NYBOT";// SMART - didn't work contractCT.Currency = "USD"; //contract.PrimaryExch = ct.PrimaryExchange;//not needed contractCT.Expiry = GeneralHelpers.GetIBFormattedExpDate(new DateTime(2021, 03, 22)); contractCT.TradingClass = "CT";//works without that as well mktDataOptions = new List<TagValue>(); clientSocket.reqMktData(ctrIdCT, contractCT, "", false, mktDataOptions); the ticks are coming not as regular Bid\Ask but with different IDs: 66 - "Delayed Bid:", 67 - "Delayed Ask:", 68-"Delayed Last" etc. One thing: before the markets opened today, I kept getting "-1" for both delayed bid and delayed ask, but after opening I started getting normal prices. Btw, I returned back my "only > 0" filter for prices because just today I received in PAPER system some negative garbage (I think it was also -1) for SMI, which made my system think it lost a 100k and it started slashing positions . So I dunnow, I guess next time I'll just have to skip the whole period when oil or something else goes negative.. Or I'll think about it when it happens (hopefully never ) So yeah, it's definitely possible, and as tgibson11 mentioned, it's also possible to use some fancy IB orders instead of just market which might make execution not as bad.. I would have to make changes in my system to make this work though.. Also, right now my system is tracking open orders, and if the price changed back before the order is filled, it'll cancel it, this will no longer be possible with delayed prices.. So a bit worrisome to execute on a erroneous IB tick, although it's probably possible even now with non-delayed prices (I think?), and worst comes to worst, I have my daily order-count limits..
@Kernfusion thank you for reporting back how you were able to collect this data. And the challenges you see if you were to incorporate this in your current trading system implementation. Very helpful.
Awesome job. For negative prices I use a standard deviation filter that will pick up any large changes. Something you might want to consider. GAT
You pretty much nailed it there. I added MES, Nikkei 225 mini, OAT, (already had platinum), copper, EUR and AUD. Also, I finally gave in and paid for ICE data, so I added sugar and cotton as well. I'm still not sure how I feel about paying for that, but I've ran some numbers and it should more than pay for itself, and so far, for these 2 months it has (though it could have just as easily gone the other way).
I'm doing something similar right now, I currently collect data for 53 markets and trade only 20. I actually extended this project and came up with something I call my financial data infrastructure. It's a collection of datastores, jobs for populating and updating them and a Python shim for accessing the data easily. Oh of course, and backup procedures. I wanted to completely abstract databases and technologies from the whole process, so I can easily spin up a Jupyter notebook or a Python script and do some research. Currently, all it takes is: Code: from findata import Stocks # To get just some symbols tech = Stocks.daily(['TWTR', 'GOOG', 'NFLX]) # To get everything all = Stocks.daily(include_delisted=True) I have the same stuff for futures. In future, there will likely be forex and options too. I do the marking as non-tradable as simply having 0 in weight or as I call it "capital allocation". I've so far bought data (for longer term backtesting) from Norgate data (only stocks, would recommend if you need stock data) and CSI Data (only futures, would absolutely not recommend). Since all of those companies give you data by forcing you to install their Windows client that was made for Windows XP, and like you, I don't deal with Windows, I exported all of that into CSV and imported into my setup I described above. Regarding the algo, I currently use a variation of Rob's world's simplest execution algo and about 29% of my orders are filled passively, the rest are aggressive. I've been trying to get my hands on some higher frequency data to run some stats on whether it pays to pay for the data or is it better to just assume I'll pay the spread, but IB is quite stingy with minutely data. Btw, in their docs, they advertise that you can get 1 month of minutely data, but I was able to get up to a year. It's really slow, and often times fails (haven't yet figured out based on which parameters) but you only need to get it once anyway. And a quick word about IB and data. I'm surprised almost every day how wonky getting prices is, especially when the market is closed (but you can't know that). Like Rob, I don't really check for some hours, you could do that using the trading hours that IB returns, though I've seen that be incorrect too. I also like Rob's philosophy of checking for liquidity, because even if the market is open, if the spread is huge, we're probably not trading. About a month ago, some of my tests started failing because I assert that the bid price is less than ask price, IB returns them as equal, and I'm not talking one of thousand ways IB signals that the market is closed (return both prices as NaN, -1, -100, or some other random value), this was for ES, during trading hours and there were bid and ask sizes. I see the same happening in grains now, but during off hours - at 5.10PM Chicago time, I get quotes for soybeans and corn with bid == ask and sizes more than 0. For example: Code: Quote 2021-01-04 23:13:09.646737+00:00: bid: 433.0, ask: 433.0, bid_size: 17, ask_size: 18 I don't know if IB is proxying this from the exchange or if they are coming up with this, but it makes your code to determine whether the market is open a horror.
Actually, now that I think of it - Rob, I assume your previous employer had some sort of a direct connection to the exchange (I know very little about microstructure but I assume something like that exists, and costs $$$), did you see issues like these there as well? Also, I'd assume you'd purchase historical data from exchanges, is that data in general higher quality than the 3rd party? Obviously, if you can talk about these things, feel free to ignore.