Answer a question

In my app I will perform some kind of health check of my Kafka cluster.

Currently I make a TopicMetadataRequest to detect dead brokers:

  Future {
    // this will fail if Kafka is unavailable
    consumer.send(new TopicMetadataRequest(Seq("health-check-topic"), 1))
  }

Unfortunately this call produces a huge network traffic, because of Cluster topology/settings.

Is there a better way to check kafka brokers? What I need is something simple like true/false indicator.

Answers

I would strongly recommend you to use Yahoo Kafka Manager, which provides all the information related to Kafka setup. (e.g. bytes sent/consumed over a time interval). This tool can also be used for managing your Kafka Cluster.

It also exposes Restful API and you can consume these API in your own application, if needed. Follow the following link to access it.

https://github.com/yahoo/kafka-manager