Interactive Brokers TWS API - and yet it works!

A friend asked if I could help as he was having a hard time integrating IKBR TWS API. Apparently, he's not alone:

Even after all the above pain in my day job, I'm still yet to come across an offering as crap as the TWS API. - DanWritesCode

I have been developing code for about 15 years, professionally, and have never seen a piece of garbage like the TWS Api. - puzzled_orc

I put IBKR TWS not just at the bottom of the list, it's twice as bad as the most crappy system out there. - Causal-Capital

After reviewing TWS offer, it's definitely legacy code.

The API grew to support a multitude of requests in a non-standard and non-organized fashion. But legacy code, is also code that has been running for ages, most likely providing mission-critical functionality, and there's certainly value in that.

When you publish an API you're creating a contract with the world. As soon as it's out there, you can't really change it without creating ripples across the entire ecosystem. If requirements take an unexpected curve, as they generally do, it's easy to find yourself cornered and forced to provide a less than ideal interface.

What you can definitely do, is improve the documentation and provide clear code samples with what should be best practices. While I can't really point a finger at the API as I have no idea of the challenges involved at the time, the documentation could improve.

TWS Connection

Let's see how you're supposed to integrate the API connection. Notice that per the API instructions «it is important that the main EReader object is not created until after a connection has been established», which reads like misplaced behaviour.

Erapper wrapper = new EWrapperImpl();
EReaderSignal signal = new EJavaSignal();
EClientSocket client = new EClientSocket(wrapper, signal);

client.eConnect("host", PORT, CONNECTION_ID);

We can see that:

• wrapper implements the interface where you'll get callbacks (i.e. receive things from the API, market data, fills, etc)
• client allows you to interact with TWS (i.e. send things to the API, orders, etc)

And then we have both signal and reader classes.

• signal is used to signal the reader thread that there's data to process
• reader processes incoming messages

According to IBKR documentation, you should then start reader and a launch a new thread to process the queue:

final EReader reader = new EReader(client, signal);   
reader.start();

// IBKR thread created solely to empty the messaging queue
new Thread(() -> {
    while (client.isConnected()) {
        m_signal.waitForSignal();
        try {
            reader.processMsgs();
        } catch (Exception e) {
            // ...
        }
    }
}).start();

I didn't dig up TWS code to see how it behaves internally, but from a birds-eye perspective, both signal and reader are misplaced and would be better encapsulated within the client.

The user just needs a way to pass data to TWS (client) and a way to get data from TWS (wrapper). All other low level implementation details would better belong inside the implementation.

Broken API socket connection

One of the places where my friend was having issues was in the re-connecting code. You should design to accommodate failure, much more so if you're developing an automated trading strategy.

The socket connection between your code and TWS will eventually break, so that's a good place to start. Glancing over TWS API we have that:

// creates socket connection to TWS/IBG.
client.eConnect("host", PORT, CONNECTION_ID);

// closes the socket connection and terminates its thread.
eDisconnect (bool resetState=true)

If we break the socket at the network level, like IKBR states, we get a callback in the wrapper implementation:

@Override
public void error(int id, int errorCode, String errorMsg, String advancedOrderRejectJson) {
    log.warn("error: id={}, errorCode={}, errorMsg={}, advancedOrderRejectJson={}", 
        id, errorCode, errorMsg, advancedOrderRejectJson);
}

On a broken socket (network disconnect), the above will log something like:

error: id=-1
errorCode=502
errorMsg=Couldn't connect to TWS. Confirm that "Enable ActiveX and Socket
  Clients" is enabled and connection port is the same as "Socket Port" on the
  TWS "Edit->Global Configuration...->API->Settings" menu. Live Trading ports:
  TWS: 7496; IB Gateway: 4001. Simulated Trading ports for new installations
  of version 954.1 or newer: TWS: 7497; IB Gateway: 4002, 
  advancedOrderRejectJson=null

Notice that at this point we're not connected to TWS, but client.isConnected() still returns true. While not ideal, it's documented. You must issue a eDisconnect().

// pseudo code for broken socket behaviour

client.isConnected() -> returns true
// here we break the socket
client.isConnected() -> returns true
client.eDisconnect();
wrapper.connectionClosed(); // we get a callback here
client.isConnected() -> returns false

Given the above, it would look as though we could issue a connect() to re-establish connection.

client.eConnect("host", PORT, CONNECTION_ID);
wrapper.connectAck(); // we get a callback here
wrapper.connectionClosed(); // and then we get a callback here

What's happening here?

Remember the thread that we created to empty the queue? Let's revisit that:

new Thread(() -> {
    while (client.isConnected()) { // when client disconnets it should exit
        signal.waitForSignal(); // but it's waiting on signal!
        try {
            reader.processMsgs();
        } catch (Exception e) {
            // ...
        }
    }
}).start();

Even though client.isConnected() returns false, we're stopped on waitForSignal() so the while loop won't exit!

As soon as the above thread gets a new signal, client is already connected and the above thread will not stop! This was creating havoc in my friend's code! As previously stated, all this behaviour could (should?) have been encapsulated upstream but, given that it's not, we can do it ourselves.

A proper start, stop & reconnect sequence

Per everything above, there is a meticulous sequence that you must follow for TWS to properly connect. You must create some objects, but some can only be created after specific callbacks have been received, etc.

When things start getting involved and into a spaghetti-like sequence, I recommend that you stick with the cleanest possible approach. In this case, it's a scenario where you create all required variables and another one where you reset them.

Properly encapsulate Reader

Making sure reader is properly initialized and terminated is a big part of the original issue, so we're isolating it to reduce clutter.

/**
 * Responsible for reading messages from the TWS socket.
 * <p>
 * Should be stopped when the connection to TWS is lost.
 */
public class Reader implements Runnable {
    private final AtomicBoolean running = new AtomicBoolean(true);

    private final EReaderSignal readerSignal;
    private final EReader reader;

    public Reader(EClientSocket clientSocket, EReaderSignal readerSignal) {
        this.readerSignal = readerSignal;
        reader = new EReader(clientSocket, readerSignal);
        reader.start();
    }

    @Override
    public void run() {
        while (running.get()) {
            readerSignal.waitForSignal();
            try {
                reader.processMsgs();
            } catch (Exception e) {
                // dispatch to error mechanism
            }
        }
    }

    public void stop() {
        running.set(false);
        readerSignal.issueSignal();
        reader.interrupt();
    }
}

And then the calling class may look something like this:

public class StartTwsApi {
    private static final Logger log = LoggerFactory.getLogger(Start.class);
    EWrapperImpl wrapper;
    EJavaSignal readerSignal;
    EClientSocket clientSocket;
    Executor executor = Executors.newSingleThreadExecutor();
    Reader reader;
    
    StartTwsApi() {
        initializeWrapperAndSocket();
        connectToTWS();
    }
    
    private void initializeWrapperAndSocket() {
        wrapper = new EWrapperImpl(this, this);
        readerSignal = new EJavaSignal();
        clientSocket = new EClientSocket(wrapper, readerSignal);
    }
    
    private void connectToTWS() {
        log.info("Connecting to TWS...");
        clientSocket.eConnect("localhost", 4005, 2);

        // safeguard to make sure we do connect / keep trying
        ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1);
        executorService.schedule(() -> {
            if (!clientSocket.isConnected()) {
                log.warn("Still not connected (10s). Retrying connect...");
                onTwsConnectionFailure();
            }
        }, 10, TimeUnit.SECONDS);
    }
    
    private void startReader() {
        executor.execute(reader);
    }
    
    private void stopReader() {
        if (reader != null) {
            reader.stop();
        }
    }
    
    /**
     * You'll call this method when you receive a callback from TWS 
     * stating that the connection is lost.
     * <p>
     * It stops the reader thread and re-connects to TWS.
     */
    public void onTwsConnectionFailure() {
        stopReader();
        log.info("Stopped reader. Will re-connect in 5 seconds...");
        Uninterruptibles.sleepUninterruptibly(5, TimeUnit.SECONDS);
        connectToTWS();
    }
    
    /**
     * Call this method when you get a connectAck() on TWS 
     * wrapper callback implementation
     */
    public void connectAck() {
        log.info("connectAck received. Safe to start reader thread...");
        reader = new Reader(clientSocket, readerSignal);
        startReader();
    }
    
}

My recommendation is that you abstract all this behaviour under a specific TWS module, and then pass a cleaner interface back to your actual code.

There is much more that we've delved on. Historical data, for instance, is an area where there were pretty interesting inconsistencies:

• public void historicalTicksBidAsk(int reqId, List<HistoricalTickBidAsk> ticks, boolean done)
  tickData is received in a single callback as a List of Objects (max 1000)
• public void historicalData(int reqId, Bar bar)
  barData is received as a series of multiple callbacks with a single Bar object; when all bars arrive, you get a callback on historicalDataEnd(int reqId, String startDateStr, String endDateStr).

After a brief chat on Reddit with u/fit-interaction4450 I remembered another place where dragons lurk. After connecting you'll get a callback on connectAck() which is a signal to start the reader thread, but you must wait until another callback on nextValidId(int orderId) to actually start interacting with TWS.

And while IBKR has it documented (that's all we may ask for), it still speaks volumes to the way the API grew.

Important: The IBApi.EWrapper.nextValidID callback is commonly used to indicate that the connection is completed and other messages can be sent from the API client to TWS. There is the possibility that function calls made prior to this time could be dropped by TWS.

My friend, not having read the above, solved it by adding a sleep() after the connection. When I asked why that was there, he said that sometimes messages were not being received by TWS and pausing for a bit solved it. It sure did (most of the time, anyway), but caught my attention as a sleep() without a clear intention is, by definition, a code smell.

Last thoughts:

Designing an API is hard. Designing an API to stand the test of time is even harder. While I can identify antipatterns and a lack of behaviour guidelines, I may only assume that business needs and decisions dictated the curved approach we now see. The very same decisions that made IBKR into a 45 billion company.

I don't know if there's an audience for TWS API articles. If you want more, let me know in the comments below.

My latest acquisition was the Arctis Nova 7 headphones. Sound is so crispy you can immediately tell the difference between Spotify native app vs Spotify web!

Microphone quality is also superb and has been praised in latest conference calls.

If you're in the market for a new headset, I highly recommend checking out the Nova 7.