Answer a question

Since kafka 0.8.2.0 it seems to have gotten awfully hard to track consumer lag and topic sizes

How do you track offsets (topic size) and lag in kafka? Do you increment a counter somewhere when your producer inserts a message and increment another counter when your consumer acknowledges a message?

I'm using airbnb's kafka-statsd-metrics2 - But all the metrics regarding topic size is always 0 for some reason, it may be a bug report for them, but how do you do it?

Our consumers and producers are written in python using kafka-python, they state they don't support the ConsumerCoordinator offset APIs, so I've put together a solution querying zookeeper and sending those metrics to statsd instance (seems very awkward), but I'm still missing the topic size metric.

We're using collectd to gather system metrics, I have NO experience with JMX and configuring it in collectd seems pretty convoluted, I've tried a few times, so I've found a few ways not to do it.

If you have any input I'd love to hear it, even if it is: "This belongs on x stackexchange-site"

Answers

If I understood you correctly you could use HighwaterMarkOffset from the FetchResponse. This way you will know what is the offset in the end of partition and will be able to compare it with your current acked offset or offset of the last message in this FetchResponse for example.

Details here