Data feed overload in live testing!?

Discussion in 'Data Sets and Feeds' started by rols, Nov 9, 2005.

  1. rols


    I am developing an automated system using TWS and VB. In simulated trading the system works fine. As soon as I go live however and get portfolio feedback from TWS using the updateportfolio and reqAcctDetails() method I get jumbled data and unpredicatable results.

    It is almost as if the system can't keep up. This especially happens during the frenzy of the open when prices are moving very fast.

  2. maxpi


    There is a 50 message per second limit, could have something to do with that.
  3. I doubt that it due to TWS. I can easily monitor all NAS 100 stocks + ES, NQ, YM etc without the machine raising a sweat. (Java application).

    If by simulated trading, you mean IB's demo system, be aware that it has message rates that would put a snail to shame.

    Have you checked the CPU utilization, both of your app and the system as a whole ? Do you have any other processes consuming excessive CPU ? Are you doing excessive logging in your code , writing every tick to disk file or such like ?

    If your app is using excessive CPU, you'll need to profile it to find the offending code fragments.

    Of course it may also be just bugs and not a performance issue at all.
  4. rols


    Thanks for your replies, I have sometimes in the past hit the 50 message/sec limit but not in this case I am however logging to disk every second and I reckon that this may well be contributing.

    The trouble is it seems to happen most often when trades are being completed very rapidly and I need to make a record of this.

    Currently I am storing each completed trade as a text file so perhaps if I just keep a database connection open and record the data to this it will be able to keep up?
  5. maxpi


    Record to the database and record the whole text file after hours or something like that. Textfiles can be a problem in themselves, they can slow as they get bigger because software will read the whole file before it can append. Tradestation is like that.
  6. rols


    I have been observing more closely and it seems that after an order has been put into TWS if there is a delay in the order being filled there is also a delay in activity coming back to VB. I don't know if it momentarily freezes the whole system or not. Perhaps a few doevents will cure it. Has anybody else found similar things happening?
  7. No, I've always found that order status events are sent promptly by TWS eg 'submitted' when order reaches the exchange, 'pending' if order is a simulated order type such as trailing stop. Don't know about the VB stuff though. I never use it.
  8. I've used VB for the past 10 months with no problems whatsoever. I'll be switching to a .net based version soon but either way I don't think VB is the issue....

  9. mokwit


    Some of us have been experiencing latency with Market orders, maybe you could try a marketable limit as a workaround.