AWS Lambda adds queue consumption mode for Kafka using Share Groups

Lambda can now consume Kafka topics as queues using Kafka Share Groups. The feature works with self-managed Kafka 4.2+ clusters and managed offerings such as Confluent Cloud.

In classic Kafka consumption, parallelism is capped by partition count, and one slow record blocks its partition. With Share Groups, multiple consumers pull records from the same partitions. Each record is acknowledged individually, has a visibility timeout and is retried automatically, which is SQS-style semantics on top of Kafka.

This matters for AI and agent workloads where per-message processing time varies widely, such as calling an LLM per event. Teams can scale out Lambda concurrency without repartitioning topics. It also narrows the gap between Kafka and SQS for teams that want a single event backbone.

The caveat is that teams need Kafka 4.2+, and ordering guarantees within a partition no longer apply in queue mode. Check whether workloads depend on ordering before switching.