Apache Kafka 4.2 brings native Queue-style messaging to the platform through “Share Groups” (KIP-932). This allows multiple consumers to process messages from the same partition concurrently without the traditional one-consumer-per-partition constraint.
Traditionally, a Kafka Consumer Group assigns a single consumer to a specific partition. If a slow message or a poison pill stalls that consumer, the entire partition waits. Kafka 4.2 changes this: [1]
- Concurrent Consumption: Share groups allow multiple consumers to pull tasks from the exact same partition.
- Exactly-Once Delivery: Kafka brokers track message acknowledgment. If a consumer crashes or fails to process an event within the lease time, the broker re-queues the message to be safely picked up by another consumer in the share group.
- Dynamic Scaling: You are no longer bound by the number of partitions. You can easily scale up consumer instances to manage heavy, task-oriented workloads.
Other Kafka 4.2 highlights:
Transaction Version 2: Enhances exactly-once-semantics (EOS) with strict epoch validation, eliminating rare race conditions during transaction failures.
Kafka Streams Improvements: The update brings native Dead Letter Queue (DLQ) support to exception handlers and server-side rebalancing.