Interactive Brokers / TWS tick capture

When writing automated trading systems (ATS) you need to get (usually buy) software and data (often bundled with software). Differently from most brokers, Interactive Brokers offer a free (assuming you have an account with them) API that makes it pretty easy to hook into their data feeds, and as a result quite a few graphing/trading packages support their interface. I've looked at a few packages and most of them do not support the features I am looking for, and quite often only support "programming" through very simplistic scripting languages. Either that, or their package/interface come at a significant up-front cost. For this and other reasons, I've decided to write my own software for writing various ATS (screenshot attached).

When/if you decide to do something similar, make sure to visit the Yahoo TWSAPI group, at http://finance.groups.yahoo.com/group/twsapi/ . There are some gurus there who knows the ins and outs of the TWS API (Interactive Broker's java-based trading application is named TWS), including (but not limited to) Richard and Tito, and with their kind help I believe I've managed to capture and interpret the TWS API events related to generating ticks.

Probably for historical reasons, the TWS API events related to ticks aren't as easy to deal with as one would hope. Ideally, the TWS API would fire of one "tick" event for every transaction, where the event would contain the price and the number of contracts traded. Unfortunately it is not that simple. I suspect through a combination of backwards compatibility and to limit the traffic between clients and Interactive Brokers' servers, a tick is not as simple as one could hope for.

The TWS API itself typically has two kinds of ticks. One is a priceTick and the other is a sizeTick. These are fired when prices or sizes change. Sometime during the evolution of the TWS API, the developers of the TWS API figured out that it would be really useful to include the size with a priceTick, and this is supported at least at the low-level (TCP packet) level of the API. What this means is that a priceTick actually do contain a price AND a size. Unfortunately, the priceTick only gets sent whenever the price changes, so if you need to keep track of volume in (near-) realtime, only capturing priceTicks will not do what you want.

There seems to be a solution however. In addition to the priceTick events, there is the sizeTick event, and also a accumulatedSizeTick event. Normally, for each priceTick event there will always be a sizeTick event as well, but if you've already made use of the "size" embedded within the priceTick event, a sizeTick event shortly after a priceTick even is mostly superflucious. At least if the size is the same as the one you got from the last priceTick event. In addition, there will sometimes be updates through accumulatedSizeTick events where there has obviously been contracts traded even if you never got any sizeTick events about them.

All this means it is not directly straightforward to generate "clean" ticks containing both price and size with each tick. The TWS application itself, however, does not seem to have any problem displaying charts which would typically depend on "clean" ticks, which means they have to use some method of generating clean ticks based on the data streams available. Theoretically, they could have access to data that would not be passed on by the TWS API itself, but that does not seem likely.

After some tinkering and help from people at the TWS API list I believe I've managed to reproduce the method used by TWS for recognizing/reconstructing clean ticks, where each tick contains both a price and size. My method for verification is nothing but simply comparing whatever TWS is displaying in real-time (quoteview) and their real-time charts, and comparing them with the ones my own application generates. But at least as far as I am able to see from my own testing, they now report identical data. There are some minor differences in volume, where one volume bar may spill over to the next volume bar, but I believe this is related to timestamping issues (since both TWS itself and my application needs to timestamp tick events using our own clocks, since a timestamp is not included with the events from TWS). Another observation is that my own application (written in C++) updates a lot faster and more often than TWS itself. This is probably related to the java-based graphics API layer that the TWS application runs under. This does mean, however, that at times my application is one or more seconds ahead of TWS itself. Whenever the tick stream pauses for a fraction of a second, however, both applications always show the same data as expected.

Anyway, a short description of how ticks are generated follows below (keep in mind that the exact method used may vary depending on which TWS API interface/layer you use):

For priceTick events from the TWS API, also extract the size parameter and generate a complete tick from this. You also need to keep track of the price and size from this tick, since they will be used by the sizeTick handler.

If you get a "normal" sizeTick event (one that simply reports the size of the last transaction), compare it to the last size from tickEvent. If the size is the same, skip this event (assume it is a duplication of the size that was reported in the priceTick event). If it is different, generate a complete tick using the current size and the last price (stored from the last priceTick event), and also add the size to an "accumulatedSize" variable that keeps track of the accumulated volume for the instrument you are tracking.

If you get an "accumulated" sizeTick event, and assuming that your accumulation variable has something in it (that you've gotten contract data previously - if you haven't, you do not have any price data to generate a tick event from), and assuming that the new accumulated size value is larger than the value you have recorded previously, generate a complete tick using the difference in the accumulated volumes (the one from the event minus the one you have stored from previuos events). Also updated accumulated size variable so you know what value to expect the next time you get an event. It is possible to get accumulated sizeTick event where the new value is less than the one stored, and if so I suggest you do not generate a tick from it, but simply update your own variable with the new updated value. It is probably a correction of some sort (possibly busted trades or similar) and you do not have enough information to deal with it reasonably.

In no way is this any guarantee that this is the "correct" or "official" method of how to get clean ticks from TWS, but for now at least it is the closest method I have found. Also remember that even if the method I have outlined seems to match TWS' own methods, I have no information about how "correct" their own implementation is. I would expect that the people with inside knowledge would try to create something as good as possible, and I've based my own implementation on this belief, which may or may not be correct.

Update 2006-03-03:

One more thing; I believe the TWS graphing module cheats. When it's time for a new candle (as defined by time itself, e.g. a new minute on a 1 minute chart), it starts a new candle using the last price from the previous candle, so that becomes the "open" price for the new candle. That means that time discontinuitities (bar periods without ticks) and gaps in the data will not appear. This is probably a limitation of the graphing library they use.