When running a large real-time processing system, monitoring is critical. But it does more than allow you to keep an eye on your system. During development it allows you test hypotheses about how it works, how it performs when certain parameters are changed, and takes the guessing out of working with dynamic systems.

Storm, a real-time computational framework open-sourced by Twitter, is such a system and comes with a Spout, allowing messages to be streamed from a Kafka Broker.

I wrote a short program, using Python, that allows you to monitor Storm Kafka spouts using the information in Zookeeper and Kafka. Using Kazoo, a Kafka Python client, and PrettyTable the tool shows the depth of each partition in Kafka, and the state of each consuming task.

You can find the code in github.