The bot made 0.75 pt. today. It sold at 12:53:41 and bought at 13:38:43. Initially, it looked like a bad time to sell.
A bot has to implement the 'nextValidId' method. One of the first things that TWS does when something connects via the API is send a next-valid-order-number, and when that is received EReader will call 'nextValidId', supplying the number as an argument. Without that number, the bot couldn't reliably place an order. After that number is used, the bot increments it for use with the next order. It appears that TWS only sends a next-valid-order-number when a bot initially connects. (I have not tried submitting orders to TWS in any other way than by incrementing the order id number as each order is submitted; however, with the 'Sample' app I did try using a number less than the one supplied by TWS and got some sort of invalid order message back.) Early in RandomBotMgr, currently about line 126, the thread sleeps for one second to allow time for the number to be received. That should use a timeout instead of arbitrarily waiting for one second. There is timeout functionality in RandomBot, which is used to wait for order confirmation. It occurred to me that, in order to implement a timeout while waiting for next-valid-order-number, I could add a 'timeout' method to the 'TimeOut' class; the 'timeout' method could be called from either RandomBot or RandomBotMgr. It wouldn't be a difficult thing to do if a method could be called with a parameter that is a class -- it would look something like: Code: boolean timeout(Class c) { int tries = MAX_TRIES; int ms = MS_TO_SLEEP; while ((tries-- > 0) && (c.eventHappened() == false)) Thread.sleep(ms); if (tries < 0) return true; // timed-out else return false; } Another argument that could be supplied to 'timeout' might be a maximum-length-of-time-to-wait value. Ha, ha! The compiler doesn't like that. Eventually, I stumbled upon the "Reflection API Trail" -- a long and winding road -- where this appears, fortunately near the beginning: "First, a note of caution. Don't use the reflection API when other tools more natural to the Java programming language would suffice. For example, if you are in the habit of using function pointers in another language, you might be tempted to use the Method objects of the reflection API in the same way. Resist the temptation! Your program will be easier to debug and maintain if you don't use Method objects. Instead, you should define an interface, and then implement it in the classes that perform the needed action." I am in the habit of using function pointers in another language, so I hope that an "interface" will let me do what is needed. What I hope to do is something like: Code: class TimeOut { ... boolean timeout(...) { ... } } class RandomBot ... { ... boolean eventHappened() { ... } boolean timedOut = TimeOut.timeout(this); } class RandomBotMgr ... { ... boolean eventHappened() { ... } boolean timedOut = TimeOut.timeout(this); } Is this possible? I may try it, later. Right now I am a little frustrated with Java.
Isn't Java wonderful? Code: interface Y { public boolean eventHappened(); } class TimeOut { boolean timeout(Y c) { if (c.eventHappened()) return true; else return false; } } class A implements Y { public boolean eventHappened() { return true; } A() { System.out.printf("A(): my eventHappened is %s\n", this.eventHappened()); } } class B implements Y { public boolean eventHappened() { return false; } B() { System.out.printf("B(): my eventHappened is %s\n", this.eventHappened()); } } public class X { public static void main(String[] args) { A a = new A(); B b = new B(); TimeOut timer = new TimeOut(); System.out.printf("a did%s time out\n", timer.timeout(a) ? "" : " not"); System.out.printf("b did%s time out\n", timer.timeout(b) ? "" : " not"); } } Is this as good as function pointers?
A review of the code revealed a bug: Code: // Does the time satisy the constraints? if ( (isLaterThan(t, t0) || coincidesWith(t, t1)) && (isEarlierThan(t, t1) || coincidesWith(t, t1)) ) break; else continue; That's in RandomBotMgr. There is a 't1' where there should be a 't0'. As is, a trade couldn't occur at the opening second.
That bug is in RandomTime.java, not RandomBotMgr.java. It's about line 53. Today, the bot bought at 13:27:32 and sold at 14:35:09, Chicago time, for a loss of 0.5 pt. If it had waited a little longer before selling ... When TWS was in the toolbar, order confirmation took more than 7 seconds; when it was out of the toolbar confirmation took about one second. The machine it runs on is on a busy network, which may account for some of the delay.
A usability issue has surfaced: if the bot is run every day with java RandomBot/Main >> log so that each day's log is appended to what was logged on previous days, then there is no indication of the day on which events in the log occurred. A workaround could be to run the bot from a script that would append some indication of the date to the log before starting the bot or the script could java RandomBot/Main > log.today To fix the problem in the bot, which is what I'm inclined to do, before line 99 of RandomBotMgr a 'logger.newday' method might be called, which would put a line like Tuesday April 1, 2007 at 03:04:05 (Japan Standard Time) into the log. It needs some more consideration -- maybe every log entry's timestamp could include the date information. Why the bot has a 'Logger' when Java has a 'Logger' class is another Java war story -- it has a lot to do with the use of 'printf' in the bot. Java's 'Logger' might facilitate internationalization of the bot, which is another usability issue.
Thinking about adding a GUI to make the bot more configurable, thinking that the GUI might be a Java program that would run the bot (a separate Java program) when the user clicked a 'run' button, after doing whatever configuration was necessary. Keeping the bot and GUI separate helps keep the bot simple. (I'd like to make the bot even simpler, not so random-trading specific. Ideally, I'd have 'main' 'BotMgr', 'SomethingBot' and 'BotAssistant' -- 'SomethingBot' would make trading decisions and might even do risk management, or risk management might be done by 'BotMgr' or even some other thread run by 'BotMgr'. 'BotAssistant' might do things like place orders, subscribe to market data, etc.) Anyway, these are the things I'd like to be able to configure, along with their priorities: Soonest: TWS client ID, port and host. (These are now configured via the command line.) Sooner: maxTrades and session hours. (These now require code modification and rebuild) Soon: A contract-details "database," to facilitate selecting a contract to trade; Trading over multiple sessions. Eventually: Choose a bot to run: random, moving-average-crossover, range-breakout, etc. I guess that all of these things could be passed via the command line when the bot is started. It might make for a lot of command line arguments though, which could be a problem on some systems. Today's log, gross 2.75 pts, which is interesting to look at on a 5-minute chart, along with an ema(4)/ema(8) (ema cross-over isn't always a good idea -- when is it a bad idea?): 08:37:24 connecting to TWS on localhost at port 7496 with id 0 Server Version:31 TWS Time at connection:20070214 07:37:20 GMT-07:00 08:37:28 TWS: (id=-1, code=2104) Market data farm connection is OK:hkfarm 08:37:28 TWS: (id=-1, code=2104) Market data farm connection is OK:usfuture 08:37:28 TWS: (id=-1, code=2104) Market data farm connection is OK:usfarm 08:37:28 TWS: (id=-1, code=2106) HMDS data farm connection is OK:ushmds2a 08:37:29 next valid order id: 169 08:37:29 round-trip interval begins at 08:37:29, ends at 15:15:00 08:37:29 scheduling buy/sell for 11:07:40/13:22:40 09:05:04 TWS: (id=-1, code=2106) HMDS data farm connection is OK:ushmds2a 11:07:43 order 169: BUY 1 ESH7, Placed at TWS 11:07:49 order 169: qty 1, rmng 0, Filled at 1458.000000 11:07:49 order 169: qty 1, rmng 0, Filled at 1458.000000 11:07:49 order 169: slept for 5750 ms. waiting for entry confirmation 11:08:57 TWS: (id=-1, code=2106) HMDS data farm connection is OK:ushmds2a 13:07:26 TWS: (id=-1, code=1100) Connectivity between IB and TWS has been lost. 13:07:34 TWS: (id=-1, code=1102) Connectivity between IB and TWS has been restored - data maintained. 13:13:08 TWS: (id=-1, code=2106) HMDS data farm connection is OK:ushmds2a 13:22:50 order 170: SELL 1 ESH7, Placed at TWS 13:22:51 order 170: qty 1, rmng 0, Filled at 1460.750000 13:22:51 order 170: qty 1, rmng 0, Filled at 1460.750000 13:22:51 order 170: slept for 1450 ms. waiting for exit confirmation 13:22:51 bot ret'd OK, entry and exit succeeded 13:22:51 disconnecting 13:22:52 main joined mgr 13:22:52 exiting 13:22:52 DEBUG: java.net.SocketException: socket closed 13:22:52 connection closed 13:22:52 connection closed The long order confirmation time occurred when TWS was not in the toolbar; the shorter one occurred when TWS was in the toolbar -- another theory bites the dust. The "HMDS data farm connection ..." messages occurred when I was refreshing a chart on TWS. Also note the IB <-> TWS connectivity loss and connectivity restoration entries. Also interesting is that the disconnect exception occurred after RandomBotMgr exited.
This may be a way to implement risk management, if there is a range-breakout bot available: 1. Run a bot 2. If the bot enters a position, run a range-breakout bot, which would place an order in the opposite direction of the step-1 bot if either side of the range were hit. What sort of system would this be if the range-breakout bot were permitted to trade in the same direction as the step-1 bot? If that were allowed, could another range-breakout bot be run to manage the position of the first range-breakout bot? If that were done, how could the position that the step-1 bot entered be risk-managed? Before going any further, it may be important to get exception handling on firm ground. That may involve narrowing the focus of exception handlers. Does an exception thrown in a thread propagate to the thread that started the exception-throwing thread, or does the runtime terminate the exception-throwing thread, or both?
This is an interesting log (today): 09:07:23 logger time zone is set to America/Chicago 09:07:23 connecting to TWS on localhost at port 7496 with id 0 Server Version:31 TWS Time at connection:20070215 08:07:15 GMT-07:00 09:07:23 TWS: (id=-1, code=2104) Market data farm connection is OK:usfuture 09:07:23 TWS: (id=-1, code=2104) Market data farm connection is OK:usfarm 09:07:23 TWS: (id=-1, code=2106) HMDS data farm connection is OK:ushmds2a 09:07:24 next valid order id: 171 09:07:24 round-trip interval begins at 09:07:24, ends at 15:15:00 09:07:24 scheduling buy/sell for 10:16:32/10:59:10 09:07:45 TWS: (id=-1, code=2104) Market data farm connection is OK:hkfarm 09:43:16 TWS: (id=-1, code=2103) Market data farm connection is broken:usfuture 09:43:16 TWS: (id=-1, code=2103) Market data farm connection is broken:usfarm 09:43:22 TWS: (id=-1, code=1100) Connectivity between IB and TWS has been lost. 09:43:56 TWS: (id=-1, code=2103) Market data farm connection is broken:hkfarm 09:48:59 TWS: (id=-1, code=2104) Market data farm connection is OK:hkfarm 09:49:02 TWS: (id=-1, code=2104) Market data farm connection is OK:usfuture 09:49:07 TWS: (id=-1, code=2104) Market data farm connection is OK:usfarm 09:49:10 TWS: (id=-1, code=1102) Connectivity between IB and TWS has been restored - data maintained. 09:51:00 TWS: (id=-1, code=2104) Market data farm connection is OK:hkfarm 09:51:03 TWS: (id=-1, code=2104) Market data farm connection is OK:usfuture 09:51:08 TWS: (id=-1, code=2104) Market data farm connection is OK:usfarm 10:16:36 order 171: BUY 1 ESH7, Placed at TWS 10:16:47 ERROR: order 171: no entry confirmation 10:16:48 FATAL: bot ret'd EntryConfirmationTimeout Two things happened: 1) there was a TWS <-> IB disconnect/reconnect; 2) the entry order wasn't confirmed (ten seconds is allowed). Looking at TWS' trades, the buy order was filled at 10:16:44. What happened? A confirmation message may have been in the network layer's buffers; if the bot had attempted a disconnect before calling logger.fatal(), would any additional info have appeared in the log? 'main', 'RandomBot' and 'RandomBotMgr' have exited, but is 'EReader' still running? Maybe, because the "API" tab still shows on TWS. Concerning the IB<->TWS disconnect/reconnect: RandomBot bot was asleep waiting to place the entry order. If RandomBotMgr had done something when the disconnect message arrived from TWS, it might have interrupted RandomBot. RandomBotMgr then would have been faced with two possibilities: RandomBot woke up early sleeping for either entry or exit order placement, unless RandomBot wasn't sleeping at the time it was interrupted (assuming that RandomBot returns to RandomBotMgr when it catches an 'interrupted' exception). If the interrupt occurred while sleeping for entry, there isn't any recovery work necessary; however, if the interrupt occurred while sleeping for exit, then there is an open position. It looks as though ES quotes aren't arriving at TWS from IB -- the ES charts look like they flat-lined about 09:57, Chicago time. The same happened with the SPX quotes from CBOE. Refreshing the charts showed that there was a lot of volume on ES when they flat-lined. Did EReader choke on all the messages from TWS about that time? Could that be why the order confirmation didn't arrive? Complications ...
re: Today's disaster The bot wasn't subscribed to market data. TWS may have choked on the high volume. I now wish that after I quit TWS I had restarted it to see if the ES data started flowing. QM was updating while the ES data was missing. A more "normal" bot would be subscribed to market data, and if it's able to do that it will also be able to get the info about TSW<->IB disconnects and whether market data farms are up or down. Doing something reasonable with that info will be challenging. The next thing I'll do is make the few lines of code that do a disconnect a method, so that it can be called from the normal path and from the path that, like the one executed today, gets an error return from the 'Callable' thread. Then, I hope to find a java debugger so that when the bot sleeps I can wake it (by interrupting it) to see that things work as expected. Without a debugger, I'll have to plant some code in the source to "simulate" the problem, rebuild, then run.