Here is test1.java (the last post exceeded the 10000 character limit): Code: import java.io.*; import java.net.*; public class test1 { public static void main(String a[]) throws Exception { Socket sock = new Socket("127.0.0.1", 6500); BufferedReader in = new BufferedReader(new InputStreamReader(sock.getInputStream())); PrintStream out = new PrintStream(sock.getOutputStream()); String query = "What's new?\n"; String reply; out.print(query); int i = 0; for (;;) { if ((reply = in.readLine()) != null) { System.out.printf("info: %s, i = %d\n", reply, i); } if (i == 1 || i == 5) out.print(query); else ; i++; } } }
Thinking about what dcraig mentioned concerning queueing: The functions in Wrapper of Main.java get called in the context of the EReader thread. The consequence of that is that any lengthy processing in those functions will delay EReader (EReader calls the EWrapper functions, which are over-ridden in Wrapper). Some queueing/buffering between EReader and any user of the tws messages could allow EReader to get back to listening for new input from tws as soon as possible, which is important when there is a lot of activity on a ticker that a tws client is subscribed to. Of course, no amount of queueing/buffering can accommodate a tws that sends messages faster than they can be consumed and/or buffered, i.e., it is possible to eventually run out of queue/buffer space, but the idea is to keep EReader busy reading the messages. There is a trade-off here, too: some consumers of tws messages may be able to get their work done in less time than it takes to queue/buffer a message, so there can be situations where queueing/buffering creates a problem rather than solving a problem. I am thinking that EReader should be the highest priority thread. EReader should also queue/buffer the messages it receives for another thread, which would run at a lower priority than EReader. That other thread could be 'Wrapper' in Main.java. Whether EReader should continue to invoke the functions in EWrapper as it does now or just post raw messages to the queue/buffer is another question.
re: queueing/buffering messages: A consideration is that this would introduce a copy of each incoming message. There may also be some OS and Java runtime buffering, which might mitigate the problem of EReader being away from looking for new input from TWS. For now, until I see a performance problem and can measure where the time is being spent, I think that I'll leave EReader alone. There is a profiler, 'hprof', that could be useful for that type of investigation. Decisions, decisions ... . Edit: Of course, some copying is implicit in the notion of distributing TWS messages to one or more clients. In the end, the best thing to do may be to modify EReader/EWrapper so that the messages are copied, possibly even modified, and buffered/queued for delivery to the clients.
Some organization is necessary. In the attachment, you'll see how I think that this project might be organized. Suggestions are welcome. Under the 'ATS' directory there are two directories: 'client' and 'server'. Under the 'client' directory there are three directories: 'chart', which would contain code for a charting client, which is that thing that visually represents market data; 'trader', which would contain code for a trading client, which is that thing that considers market data and places buy and sell orders; and 'TestJavaClient', which contains IB's API code. Under the 'server' directory there are two directories: 'broker' and 'data provider'. A 'broker' is an adapter, which provides an interface between, say, a 'trader' client and an actual broker. A 'data provider is also an adapter, which provides an interface between, say, a 'chart' and an actual data provider. The things under the 'server' directory are actually combination clients/servers: they are clients to vendors' servers and servers to this project's clients. For example, a 'chart' requests a ticker from a 'data provider', which, in turn, requests the ticker from its vendor's server, then, as price and volume information stream back, the 'data provider' delivers that information to whichever 'client' requested it. Now, vendors don't use identical communication protocols. A reason to have the 'broker' and 'data provider' adapters is to make a 'trader' independent of any particular vendor's communication protocol. A 'trader' may need to connect to a backup 'data provider' or hedge an open postition at one broker with a position at some other broker. But there must be some internal communication protocol; otherwise, our clients and servers wouldn't be able to interact. What to do? Invent a new protocol? Use an existing protocol? I've decided to use the messaging protocol that IB has defined for their API, internally, between these clients and servers. Why? It's a no-brainer, and using it permits any of the clients to plug directly into TWS -- they could dispense with the adapters. It also permits 'TestJavaClient' to be used with this 'server' software, for testing purposes, before any other 'client' is available, by just changing the port number that 'TestJavaClient' connects to (a TWS broker and/or data-provider adapter would have to listen on a different port than the one TWS listens on). Would it ever be desirable, even necessary, to add new message types beyond the ones that IB's API supports? If so, then, would a 'client' that used any of those new messages be able to continue to inter-operate with TWS? There are some comination broker/data-provider vendors. IB is one of them. Should a 'trader' who wanted to use IB for brokerage and data services connect to two server-adapters? If more than one 'trader' client is running, and both use the same broker, how does the broker-adapter associate the broker's "filled," "cancelled" or "rejected" response to the order with the 'trader' that initiated the order? Does it matter whether more than one 'trader' is working an account or should they work separate accounts? If there are problems with more than one trader working one account, would it be alright for more than one to work the same account if they traded different instruments?
The attachment? edit: it made it. That was drawn with 'xfig' and looks like the lines have been erased, at least on my browser.
I've attached dirorg1.jpg, which shows how I've started populating the 'ATS' directory. I've also changed the name of 'data provider' to 'data-provider'. The attachment just shows the directories. To fill-in the ATS/client/TestJavaClient, I copied twsapi_unixmac.jar (it works on Windows, too) to the ATS/client/TestJavaClient/original directory, then ran jar xf twsapi_unixmac.jar there. I made the 'recorder' directory with a recursive copy of the 'original' directory, but it could also be made by copying twsapi_unixmac.jar to the ATS/client/TestJavaClient/recorder directory and running 'jar xf' on it there. I plan to leave 'original' alone, but in the 'recorder' directory I'll modify EWrapper.java to capture a sequence of the messages that are sent by TWS to the API. That captured data, in a file, then can be played back, over and over, by a toy server, for development, test and debug. That capability is useful when the markets are closed.
Hi, Interesting project. Some other resources you may find helpful: * http://rubyforge.org/projects/ib-ruby/ - a Ruby implementation of the TWS api. Still incomplete and very research-project oriented at this time. * On Linux systems, you can get "true" random numbers rather than pseudorandom numbers by reading from /dev/random. The kernel collects entropy from external sources (e.g. the frequency of network packet arrival, the interval between keystrokes, etc.) and stores the data for later use. Not perfect, but a step up from a pseudorandom number generator, at least. If you really want to use Windows, you could set up a Linux box (either physically or with VMWare) and then read the random number data from the Linux box over the network. * If you *really* want true random numbers, you can buy an external hardware random number generator. There's a list at http://en.wikipedia.org/wiki/Hardware_random_number_generator#Manufacturers , and I'm sure Google will turn up many more. Good luck to you..
Raul: Thanks. I went to see the Ruby project, but could only see snippets of code. I'll have to get an svn client running, I guess. I have added a directory, 'tools', at the same level that 'client' and 'server' are on, under 'ATS'. I have placed 'test1.java' there and this, too: Code: /* * toyTws.java */ import java.io.*; import java.net.*; class Server extends Thread { String inputFile = null; String port = null; public void run() { ServerSocket ssock = null; // the server socket Socket sock = null; // the client socket RandomAccessFile raf = null; // the message input file DataInputStream dis = null; // read stream from the client DataOutputStream dos = null; // write stream to the client long length = 0; // length of the input file long read = 0; // no. of input file bytes read int b; // an input byte try { // open the input file: raf = new RandomAccessFile(inputFile, "r"); length = raf.length(); read = 0; // offer a connection: ssock = new ServerSocket(Integer.parseInt(port)); // // wait for a connection, then // read the client's version string // send the server's version string to the client // send the server's notion of the current time // read the client's id string // loop: // if necessary, rewind the message file // read a byte from the message file // send a byte to the client System.out.printf("Server: listening\n"); sock = ssock.accept(); dos = new DataOutputStream(sock.getOutputStream()); dis = new DataInputStream(sock.getInputStream()); System.out.printf("Server: client connected\n"); // get client version: System.out.printf("Client version is "); while (true) { b = dis.readByte(); if (b == 0) break; else System.out.printf("%c", b); } System.out.printf("\n"); // send server version: dos.writeBytes("99"); dos.writeByte('\0'); // send server time: dos.writeBytes("09:09:09"); dos.writeByte('\0'); // get client id System.out.printf("Client id is "); while (true) { b = dis.readByte(); if (b == 0) break; else System.out.printf("%c", b); } System.out.printf("\n"); // until an exception occurs, // read from the input file, write to the client while (true) { if (read < length) { b = raf.readByte(); read++; dos.writeByte((byte)b); /* Thread.sleep(1); // approximately 1000 cps */ } else { raf.seek(0); read = 0; } } } catch (Exception e0) { System.out.printf("Server.run: caught %s\n", e0); try { ssock.close(); } catch (Exception e1) { System.out.printf("Server.run caught %s\n", e1); } try { dis.close(); } catch (Exception e2) { System.out.printf("Server.run caught %s\n", e2); } try { raf.close(); } catch (Exception e3) { System.out.printf("Server.run caught %s\n", e3); } try { dos.close(); } catch (Exception e4) { System.out.printf("Server.run caught %s\n", e4); } try { sock.close(); } catch (Exception e5) { System.out.printf("Server.run caught %s\n", e5); } System.exit(1); } } public void fileName(String s) { inputFile = s; } public void port(String s) { port = s; } } public class toyTws { // usage: java toyTws [recordedMessagesFile [serverPort]] public static void main(String args[]) { String inputFile = null; String port = null; if (args.length == 0) { inputFile = "msgs.junk"; port = "6500"; } else if (args.length == 1) { inputFile = args[0]; } else if (args.length == 2) { inputFile = args[0]; port = args[1]; } else { System.out.printf("usage: toyTws [file [port]]\n"); System.exit(1); } Server server = new Server(); server.fileName(inputFile); // server file to read msgs from server.port(port); // server port no. to listen on server.start(); } } toyTws is a server. Once a client connects to it, it continually reads bytes from a file and sends them to the client. It is assumed that the file is a file containing a sequence of tws-to-api messages. The client I have used it with is TestJavaClient/Main, the original. To compile: javac toyTws.java and to run: java toyTws I have modified what was placed in ATS/client/TestJavaClient/recorder/com/ib/client/EReader.java That is how the recorded tws->api messages file can be obtained. I'll put it up later, maybe tomorrow night, because it needs to provide an indication that it is done writing the file, and I need to test it a little more.
At some point we have to consider the implications of what we contemplate. I have realized that I am contemplating distributing tws -- I don't mean distributing the program, but the services it provides via the api. I am a victim of the design-for-extensibility mindset (it is a way of coping with changing requirements). There are two questions: Is it possible? Is it legal? Whether it is possible depends on several things. Of course, it should be realized that one instance of tws only provides access to one account, but is it possible to let several traders trade in that one account? The answer is yes, if no more than the number of api clients that tws will permit to connect isn't exceeded. If tws were distributed, with multiple traders connected to tws indirectly via just one tws api connection a problem arises: It doesn't appear to be the case that tws supports a "request id" field in each of the requests that a client can send to tws. The consequence of this is that it may be difficult to associate requests made by an indirect client of tws with a tws response. The market data request and order request messages appear to support ticker id and order id fields, which, with some mapping between client request ids and tws response ids might support a distributed tws. The error messages from tws are another matter: there appear to be two types of error messages sent by tws: those sent in response to an invalid request and those sent asynchronously. The error messages sent asynchronously might be blindly forwarded to each client. The error messages that are a response to an invalid request would have to be associated with the client that made the request. I have tried sending bogus market data request messages and bogus order request messages (the ticker symbol was invalid in both cases) and found that the error response contains an id and an error-code. If the id is a ticker id or an order id, then it might be possible to associate such an error response message with the requesting client. Whether it is legal is a big question. A broker probably wouln't care about brokerage services being distributed, because it could be an opportunity to make more commissions. Data vendors probably do care; they make money providing data and may be concerned about how it is used; they certainly consider themselves to be the owners of the data. I now recall that a customer enters into a lot of agreements about market data, and, so, I should not proceed with distributing tws without legal advice. So, without a distributed tws, we can only have as many traders and/or charts as there are direct tws api connections. At the retail trading level this isn't a big constraint, and it will enable us to go directly to work on a bot, now that we've got the infrastructure issues out of the way. On a more positive note, it was good to find out that network programming with Java is fairly easy.