Kafka Consumer default Group Id

Kafka Consumer default Group Id

I’m working with Apache Kafka and its Java client and I see that messages are load balanced across different Kafka Consumers belonging to the same group (i.e. sharing the same group id).
In my application I need all consumers to read all messages.
So I have several questions:

if I don’t set any group id in the Consumer Properties, what group id will the Kafka Consumer be given?
Is there a single default value?
Does the client create a random value each time?
Do I need to create a different id for each consumer to be sure that each one receives all messages?

EDIT:
Thank you for your answers.
You are correct: if one doesn’t set the consumer group id, Kafka should complain.
However, I have found out that if the group id is null, the Java client sets it to the empty string “” to avoid problems.
So apparently that is the default value I was looking for.
Surprising all my consumers, even if I don’t set their groupIds (and so they are all with groupId == “”) seem to receive all the messages the producer writes.
I still can’t explain this: any suggestions?

Solutions/Answers:

Solution 1:

if I don’t set any group id in the Consumer Properties, what group id will the Kafka Consumer be given?

The kafka consumer will not have any consumer group. Instead you will get this error : The configured groupId is invalid

Is there a single default value?

Yes, you can see the consumer.properties file of kafka for reference. The default consumer group id is: group.id=test-consumer-group

Does the client create a random value each time?

Related:  Comparing kafka-node and node-rdkafka

No, groupId seems to be mandatory for Java client starting Kafka 0.9.0.x consumers. You can refer to this JIRA: https://issues.apache.org/jira/browse/KAFKA-2648

Do I need to create a different id for each consumer to be sure that each one receives all messages?

Yes, if all consumers use the same group id, messages in a topic are distributed among those consumers. In other words, each consumer will get a non-overlapping subset of the messages. Having more consumers in the same group increases the degree of parallelism and the overall throughput of consumption. On the other hand, if each consumer is in its own group, each consumer will get a full copy of all messages.

Solution 2:

If don’t set group.id,you will get error when consume topic data.

org.apache.kafka.common.errors.InvalidGroupIdException: The configured groupId is invalid
22:08:14.132 [testAuto-kafka-consumer-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining group 
22:08:14.132 [testAuto-kafka-consumer-1] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Sending JoinGroup ({group_id=,session_timeout=15000,rebalance_timeout=300000,member_id=,protocol_type=consumer,group_protocols=[{protocol_name=range,protocol_metadata=java.nio.HeapByteBuffer[pos=0 lim=18 cap=18]}]}) to coordinator bogon:9092 (id: 2147483647 rack: null)
22:08:14.132 [testAuto-kafka-consumer-1] ERROR org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Attempt to join group  failed due to fatal error: The configured groupId is invalid
22:08:14.132 [testAuto-kafka-consumer-1] ERROR org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer - Container exception

Solution 3:

I have same problem. And took some time to research this issue.
The project spring-cloud-stream will check whether you have set the group id for consumer. If not, spring-cloud-stream will create a random value as group id.
Please refer the method createConsumerEndpoint in class KafkaMessageChannelBinder.

Image

References