Answer a question

I am new to kafka, So I am just clearing my kafka concepts.

I have created a simple streaming application which is streaming data from a single topic which has two partitions. I have two instances of this application (I am saying this on the basis of same application.id in both the projects). When I started third instance of the application, I got the error. From this, I understood that application.id in kafka is being treated as consumer group id where single consumer can read from single partition of a topic and 3rd consumer does not get any, so it failed to get store against the topic.

I have also tried another scenario where I changed application.id in one of the my applications. So by doing this, third instance of the application also started working fine. So it confirmed my hypothesis that application.id is being treated as consumer group id.

But I have also noticed that group.id, client.id also exists there which is confusing me. What is purpose of using group.id, client.id in our projects, what are these properties and how they work. I have set up same group id for all three applications.

Answers

In short:

• client.id (for producer and for consumer) sets the name of an individual Kafka producer or consumer client.
• group.id sets the name of the Kafka consumer group that an individual Kafka consumer client belongs to.
• application.id is a setting only used by Kafka Streams to name an application, i.e., an application that uses the Kafka Streams library (which can be run on one or more app instances). Behind the scenes, the application.id is also used to generate group.id and client.ids. See the application.id documentation for further information.