How to view and set offsets.retention.minutes using kafka-configs


How to view and set offsets.retention.minutes using kafka-configs

From the documentation, this is a broker config, but details are thin.
1) What is the correct way to set this to indefinite? If I do, will I still be able to delete consumer groups and their associated offsets manually?
2) Can I set individual consumer groups to have differing retention?
3) How do I view the retention globally, or for a specific consumer group? Can I view this and get a number even if I have not set this before?


Solution 1:

1) Kafka Docs mention that the valid values for offsets.retention.minutes are


which means that you cannot set offsets.retention.minutes=-1 but you can set it to a fairly large integer in order to keep the offsets for a long period.

2) I would assume that you are asking whether you can set different values for offsets.retention.minutes on a topic level; As far as I know, this is not possible. You can only modify this configuration on a broker level, i.e. inside file.

I am afraid that you cannot avoid a rolling restart as you cannot change offsets.retention.minutes using;

> bin/ --zookeeper localhost:2181 --entity-type brokers --entity-name 0 --alter --add-config offsets.retention.minutes=200800

Error while executing config command requirement failed: Unknown Dynamic Configuration 'retention.minutes'.
java.lang.IllegalArgumentException: requirement failed: Unknown Dynamic Configuration 'retention.minutes'.
        at scala.Predef$.require(Predef.scala:224)
        at kafka.server.DynamicConfig$$anonfun$kafka$server$DynamicConfig$$validate$1.apply(DynamicConfig.scala:101)
        at kafka.server.DynamicConfig$$anonfun$kafka$server$DynamicConfig$$validate$1.apply(DynamicConfig.scala:100)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at kafka.server.DynamicConfig$.kafka$server$DynamicConfig$$validate(DynamicConfig.scala:100)
        at kafka.server.DynamicConfig$Broker$.validate(DynamicConfig.scala:59)
        at kafka.admin.AdminUtils$.changeBrokerConfig(AdminUtils.scala:555)
        at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:105)
        at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:68)
        at kafka.admin.ConfigCommand.main(ConfigCommand.scala)

3) Since this is a configuration on the broker level, you can view the value in file or from your broker’s log files. For example,

 grep offsets.retention.minutes /path/to/your/kafka-broker.log

Note that even if you haven’t configured offsets.retention.minutes, its value defaults to 10080.