I see a total failure at least once a day of the "usfuture" data farm (historical data farm is no problem). There is no way from the client application to make the gateway restart the data flow, apart manually restarting the Gateway (secure device login). The IB gateway has become of no use to me. Also, a very very bad idea adding an automatic daily re-login in the "latest" version (you guess what version it is). If you wish to insist on this nonsense, please, at least make it optional and not mandatory.
I have not seen you launch a post/thread on the tws api user group : https://groups.io/g/twsapi/topics
There is already a thread there. I do not see the point to repeat over there, since it has not been solved in more than1 year. Diversification may help. Also, I like this specific forum ("IB Technical Issues") better. Anyway, in case they may be of some help, here are the messages that the Gateway continue spitting out, and remains unable to resume the data: Code: 2019-05-07 12:57:21.125 [CG] INFO [JTS-EServerSocket-3060] - Start API message, ClientID=1 2019-05-07 12:57:21.436 [CG] INFO [JTS-QuickMktDataWrapper-81] - Farm usfuture/DISCONNECTED: Adding message to queue 923 2019-05-07 12:57:21.581 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.ar@7b72dfc0 | a()#853 | subscribe VCF | 97589608 | REGULAR | 20200514 | 2019-05-07 12:57:21.581 [CG] INFO [JTS-QuickMktDataWrapper-81] - Farm usfuture/DISCONNECTED: Adding message to queue 924 2019-05-07 12:57:21.581 [CG] INFO [JTS-Async-27] - |jtsNIA|s:97589608/REGULAR | subscribe()#120 | TWS subscribes | 2019-05-07 12:57:21.581 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.ar@7b72dfc0 | a()#871 | subscribe VSF | 97589608 | REGULAR | 2019-05-07 12:57:21.581 [CG] INFO [JTS-Async-27] - subscribing for 589 with params expiry=20200514 2019-05-07 12:57:21.581 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.y@255e0093 | a()#93 | Switch to volsf | 97589608 | REGULAR | 1 | 2019-05-07 12:57:21.581 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|t:97589608/REGULAR | subscribe()#120 | TWS subscribes | 2019-05-07 12:57:21.587 [CG] INFO [JTS-QuickMktDataWrapper-81] - Farm usfuture/DISCONNECTED: Adding message to queue 925 2019-05-07 12:57:22.961 [CG] INFO [JTS-Cleaner-31] - market data wrapper, cleaning maps 2019-05-07 12:57:24.001 [CG] INFO [NIA-Output-Queue-1] - |UNKN|MDLNAV1 | onDividendChanged()#1119 | Dividend Arrive | 17340715DividendSchedule[] | AUTO | 2019-05-07 12:57:26.081 [CG] INFO [JTS-EServerSocket-3060] - [1:142:142:1:0:0:0:ERR] Socket connection for client{1} has closed. 2019-05-07 12:57:26.081 [CG] INFO [JTS-EServerSocket-3060] - Connection terminated 2019-05-07 12:57:26.081 [CG] INFO [JTS-EServerSocketNotifier-3061] - Terminating async queue thread 2019-05-07 12:57:26.081 [CG] INFO [JTS-EServerSocket-3060] - Cleaning up [serverId: 1]... 2019-05-07 12:57:26.082 [CG] INFO [JTS-EServerSocket-3060] - Cleaning up [serverId: 1]... 2019-05-07 12:57:26.082 [CG] INFO [JTS-EServerSocket-3060] - Cleaning up [serverId: 1]... 2019-05-07 12:57:26.082 [CG] INFO [JTS-EServerSocket-3060] - ArEServer: Removing 668702198 with id 1 2019-05-07 12:57:26.082 [CG] INFO [JTS-EServerSocket-3060] - eServersChanged: 0 2019-05-07 12:57:26.083 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.ar@7b72dfc0 | a()#877 | unsubscribe VSF | 97589608 | REGULAR | 2019-05-07 12:57:26.083 [CG] INFO [JTS-EServerSocket-3060] - There are not API orders being processed. 2019-05-07 12:57:26.083 [CG] INFO [JTS-EServerSocket-3060] - SecDefRequestTimeOut: There are not timed out SecDef requests. 2019-05-07 12:57:26.083 [CG] INFO [JTS-EWriter289-3062] - [1:142:142:1:0:0:0:SYS] Writer thread terminated for socket client{1}. 2019-05-07 12:57:26.083 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.y@255e0093 | b()#129 | Switch to volcrv | 97589608 | REGULAR | 1 | 2019-05-07 12:57:26.083 [CG] INFO [JTS-EServerSocket-3060] - [1:142:142:1:0:0:0:SYS] Ending conversation with client{1} at 127.0.0.1 2019-05-07 12:57:26.083 [CG] INFO [JTS-QuickMktDataWrapper-81] - Farm usfuture/DISCONNECTED: Adding message to queue 926 2019-05-07 12:57:26.083 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|s:97589608/REGULAR | subscribe()#120 | TWS subscribes | 2019-05-07 12:57:26.083 [CG] INFO [NIA-Output-Queue-1] - subscribing for 589 with params expiry=20200514 2019-05-07 12:57:26.084 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.ar@7b72dfc0 | a()#863 | unsubscribe VCF | 97589608 | REGULAR | 20200514 | 2019-05-07 12:57:26.090 [CG] INFO [JTS-QuickMktDataWrapper-81] - Farm usfuture/DISCONNECTED: Adding message to queue 927 2019-05-07 12:57:26.091 [CG] INFO [JTS-QuickMktDataWrapper-81] - remove volsffn gentick; remaining=2 desubscribedFromTop=true desubscribedFromFrozenTop=false 2019-05-07 12:57:26.115 [CG] INFO [JTS-SocketListener-46] - State: HEADER, IsAPI: UNKNOWN 2019-05-07 12:57:26.115 [CG] INFO [JTS-SocketListener-46] - State: STOP, IsAPI: YES 2019-05-07 12:57:26.115 [CG] INFO [JTS-SocketListener-46] - ArEServer: Adding 1825496766 with id 2147483647 2019-05-07 12:57:26.115 [CG] INFO [JTS-SocketListener-46] - eServersChanged: 1 2019-05-07 12:57:26.116 [CG] INFO [JTS-EServerSocket-3064] - [2147483647:142:142:1:0:0:0:SYS] Starting new conversation with client on 127.0.0.1 2019-05-07 12:57:26.116 [CG] INFO [JTS-EServerSocketNotifier-3065] - Starting async queue thread 2019-05-07 12:57:26.116 [CG] INFO [JTS-EServerSocket-3064] - [2147483647:142:142:1:0:0:0:SYS] Server version is 142 2019-05-07 12:57:26.116 [CG] INFO [JTS-EServerSocket-3064] - [2147483647:142:142:1:0:0:0:SYS] Client version is 142 2019-05-07 12:57:26.116 [CG] INFO [JTS-EServerSocket-3064] - [2147483647:142:142:1:0:0:0:SYS] is 3rdParty true 2019-05-07 12:57:26.117 [CG] INFO [JTS-EServerSocket-3064] - Start API message, ClientID=1 2019-05-07 12:57:26.433 [CG] INFO [JTS-QuickMktDataWrapper-81] - Farm usfuture/DISCONNECTED: Adding message to queue 928 2019-05-07 12:57:26.579 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.ar@7b72dfc0 | a()#853 | subscribe VCF | 97589608 | REGULAR | 20200514 | 2019-05-07 12:57:26.579 [CG] INFO [JTS-QuickMktDataWrapper-81] - Farm usfuture/DISCONNECTED: Adding message to queue 929 2019-05-07 12:57:26.579 [CG] INFO [JTS-Async-27] - |jtsNIA|s:97589608/REGULAR | subscribe()#120 | TWS subscribes | 2019-05-07 12:57:26.579 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.ar@7b72dfc0 | a()#871 | subscribe VSF | 97589608 | REGULAR | 2019-05-07 12:57:26.579 [CG] INFO [JTS-Async-27] - subscribing for 589 with params expiry=20200514 2019-05-07 12:57:26.579 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|nia.y@255e0093 | a()#93 | Switch to volsf | 97589608 | REGULAR | 1 | 2019-05-07 12:57:26.579 [CG] INFO [NIA-Output-Queue-1] - |jtsNIA|t:97589608/REGULAR | subscribe()#120 | TWS subscribes | 2019-05-07 12:57:26.585 [CG] INFO [JTS-QuickMktDataWrapper-81] - Farm usfuture/DISCONNECTED: Adding message to queue 930 2019-05-07 12:57:27.081 [CG] INFO [NIA-Output-Queue-1] - |UNKN|MDLNAV1 | onDividendChanged()#1119 | Dividend Arrive | 17340715DividendSchedule[] | AUTO |
I just did. We will see if they fix it, and also fix the daily re-login/autorestart nonsense. I will keep this thread posted if a see solutions. As another member (cruisecontrol) rightly pointed out, all this to a programmer's eyes tells only one thing... If not solved in due time, personally, I think I will switch technology. (I do not have time to work for the broker too. Writing algos to fight market making algos is challenging enough.)
Any other broker suggestions for automated trading with full access to world markets? Am curious because I can't think of any.
Ah, yes, you are right: "in the land of the blind, the one-eyed man is king". Not my philosophy though. What automated trading? I have been doing this for about 20 years, and I am looking now at a gateway which needs manual re-login every day (to log with a device). And, not satisfied with that, an automatic daily relogin/autorestart has also been added to the "latest" version? The system worked way better several years ago than now. (Although to have the idiotic blocking re-login prompt from the gateway I almost had to fight over the phone with an IB director (see previous thread), and now this.) What "automated" trading?
Well, all cynicism aside, you answered my question. Unless you trade a very narrowly defined set of assets it's hard to find another broker out there that offers what IB does. Hence, it pays to keep bringing up the issues to IB support and also consider alternative ways such as FIX or the like.
Did you see the post in the tws/in group? "Can confirm still an issue on my side. I've just installed TWS 976, and you can set up an automatic daily restart in Global Configuration - then all you need is your app to disconnect and reconnect around the same time. Slightly easier than IBC."
From the point of view of an automated trader, most of the time, it makes no sense that the Gateway takes the initiative to shut down at a given time. There should be no point of doing that on a daily basis if it were programmed properly. It only makes people laugh about their coding skills. Anyway. Now, I will tell you the proper way to implement a relogin or autorestart feature. Talking of API automation, in general, the king rule is that the gateway is a tool for the client to communicate with IB and absolutely all communications and actions must happen through that channel. So if you add a relogin autostart feature, there has to be a command via API to trigger that, so that the client application can decide when and if to do that, in case it is necessary. In any case, the relogin or autostart process must be seamless, require no security device to relogin to the same account and absolutely raise no prompts. The blocking window warning about paper trading account should also be removed, or else nobody will be able to even start trying automation. Under no circumstance, a blocking window must ever appear. Use API to tell things! And the type of account should be communicated via API (we should not be forced to make arbitrary inference based on letters of account code). It is ridiculous that along so much unnecessary pieces of information transmitted, that one is not. In brief. - Absolutely no prompts at any time (repeat no prompts! Do not open warning windows! Use the API channel to tell things: there is a client listening, not humans!). - Everything must happen via API, including possible re-login and/or auto restart commands. There must be proper API commands, to do so. - Re-login to same account (triggered by client application) must be possible without requiring any (security) device and without any warning or prompt, outside of API communications, no matter what it is the type of account (paper/real).