Just to let you know what happened.... We were running on a gateway at the exchange which had to be changed. On Friday evening the the change was made. We tested the connection by successfully logging into the exchange. This morning, we connected w/o any problems. However, we were not receiving certain information including data from the exchange. We had to request the exchange to revert to their backup gateway. Afterwards we restarted all of our API's and were back in business. We are now waiting to hear back from the exchange to find out if they made an error on their end because on our side we saw that we were logged in and connected properly to the exchange.
Why don't IB take up my suggestion to write a monitoring app to check that quotes are coming in on time ? It should be easily done via your api. If you had the application in place, you could have noticed that preopening quotes are not coming in early and rectify the situation before your customers call you. With it you can also send out bulletins to warn you customers before hand.
>def Thank you for the detailed explanation. 'Most of us' can understand that these things can and do happen. However, I hope those at IB can understand our feelings when there is a problem, and knowing that you can't be monitoring all products on all markets, and we always wonder 'Have they noticed this?'. Would it be possible for you to consider setting up a 'Hotline' e-mail address that we could use just as soon as we notice a problem? If you're flooded with E-mails there, you would know it's not a problem on the client's side. I know you have a 'Help' address, but maybe our alerts would be mixed with less urgent mail. Just a suggestion.
still begs the question: then why the hell didn't you do the right thing and send out a bulletin about it 'this morning' when you connected. there is NO EXCUSE, KNOWING YOU HAD A PROBLEM, to not send out a bulletin. if you do not see how this is irresponsible behaviour on IB's part, then you're either not paying attention, don't care, or you've got the typical 'IB can do no wrong, it's always someone else's fault' syndrome that runs rampant @ your company. to make this even more clear: even if the fault is with the exchange (whether it was or not, i'm sure IB will blame them rather than accept responsibility)... it is IB's fault, and ONLY IB's FAULT, that you failed to send out an alert until 41 minutes after quotes normally start, and 26 minutes after trading began. you knew all that time and left your customers in the dark. shame on you.
I can confirm (I'm not going to post the email from the exchange) that the software/telco vendor they hired to handle their hardware made an input error on a configuration file. That's a fact. Does it mean I would pass blame on the exchange when it wasn't there fault - no. Could this have been handled differently on our end? Of course. You can blame me for not sending out the bulletin. Part of the delay in issuing the bulletin is that since we saw that all connections were viable we had the hopes that a restart would resolve the problem. That didn't work and as all hands on the tech side dropped what they were doing in an attempt to resolve the problem no one issued the bulletin when that didn't work. We were in the dark as well as we were connected to the exchange and confirmed with them that all API's were logged in and alive. The exchange also had no idea why we weren't getting data and we only rec'd verification of what went wrong after they found out well into the afternoon session. Your post makes it seem like we're sitting around complacent w/o a care in the world. In reality, this impacted our prop side as well and those who could assist were mobilized trying to resolve the problem. The CS desk was also swamped and while I and the rest of the staff are extremely sympathetic to our clients inability to access the exchange, I assure it was just as frustrating in our office. After issues like today, we do discuss how/what we could do better should a similar instance occur in the future. By no means to we take such an outage - no matter what the cause - lightly.
In this type of business there cannot be 100% up time, but in one week you have used up the amount of downtime I would expect in a full year. You were completely out for 45 minutes during the New York session last week, and now about the same amount of time during the Asian session. Can't you get your system more modularized? I appreciate your efforts to explain, but why did a problem at one exchange cause cash Forex to be out for almost an hour? Didn't you just deal with the exchange that was involved, and leave all your other data streams alone? Are all your APIs bundled together so that if one goes down, they all go down?
When was forex out?? I was receiving quotes well into the morning Well before the bulletin showed up and afterwards.... Perhaps it was out for you.. At 12am EST I had the normal reset issues, but nothing out of the ordinary.