Kafka Consumer Groups & Replay

You can produce to orders and consume as group inventory. That was Post 3.

Tonight is the reason teams pick Kafka over a queue that forgets:

1. Email and Inventory both need every OrderPlaced — without fighting over who “owns” the message.
2. Inventory reserved the wrong warehouse for order-9912 — you replay from the log instead of asking checkout to happen twice.

Same topic, different group ids = fan-out. Same group id = workers sharing partitions. Replay = move the bookmark, read again.

Prior reading: Why Kafka Exists · Core Vocabulary · First Produce & Consume

The two jobs of a consumer group

A consumer group is a named team of readers (inventory, email, analytics). Kafka tracks committed offsets per group per partition.

That one sentence unlocks two designs that look the same in code and mean opposite things:

Click both modes. Same three partitions. Different story.

Sticky idea: Queues often delete after one ack. Kafka keeps the log so many groups can read it. Inside one group, partitions are the unit of parallelism — not “spawn infinite pods and hope.”

Try fan-out on your laptop

Use the same Docker broker from Post 3. Topic orders with a few messages already in it (produce a couple if the log is empty).

Terminal A — Inventory:

docker exec -it $(docker compose ps -q kafka) \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders \
  --from-beginning \
  --property print.key=true \
  --property key.separator=: \
  --group inventory

Terminal B — Email (new group, same topic):

docker exec -it $(docker compose ps -q kafka) \
  /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders \
  --from-beginning \
  --property print.key=true \
  --property key.separator=: \
  --group email

Both terminals print the same keys. Neither steals from the other. That is fan-out with real bytes.

Produce one more line in a third terminal while both consumers run. Both should show it. Two bookmarks advanced independently.

Scale one team (same group id)

Create (or recreate) the topic with three partitions so assignment is visible:

docker exec -it $(docker compose ps -q kafka) \
  /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic orders-scaled \
  --partitions 3 --replication-factor 1

Start two consumers with the same --group inventory-scaled on orders-scaled. Produce several messages with different keys (order-1, order-2, …). Watch which terminal prints which keys — each partition sticks to one member of the group.

Add a third consumer in the same group. Kafka rebalances: partitions are reassigned so work is shared again. A fourth consumer on a three-partition topic sits idle until someone leaves. More pods than partitions does not mean more throughput for that group.

In code the only new idea is the shared groupId:

// npm i kafkajs
import { Kafka } from "kafkajs";

const kafka = new Kafka({
  clientId: "inventory-worker",
  brokers: ["localhost:9092"],
});

// Same groupId on every Inventory replica
const consumer = kafka.consumer({ groupId: "inventory" });
await consumer.connect();
await consumer.subscribe({ topic: "orders", fromBeginning: false });

await consumer.run({
  eachMessage: async ({ partition, message }) => {
    const orderId = message.key?.toString();
    // reserve stock — must tolerate duplicates (at-least-once)
    console.log({ partition, offset: message.offset, orderId });
  },
});

Email is a different groupId (email). Same topic subscription. Separate progress.

Replay — when the handler was wrong

The log is the source of truth for what happened. Your consumer is responsible for what you did about it. When side effects are wrong, you fix the code and read the facts again.

Walk the lab:

Reset offsets with the CLI

Stop the consumers in that group first (or you will fight live commits).

docker exec -it $(docker compose ps -q kafka) \
  /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group inventory \
  --describe

You should see partitions, committed offsets, and lag.

Reset partition 0 to a specific offset (example: re-read from offset 1):

docker exec -it $(docker compose ps -q kafka) \
  /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group inventory \
  --topic orders:0 \
  --reset-offsets --to-offset 1 \
  --execute

Or jump to the start of retained data:

docker exec -it $(docker compose ps -q kafka) \
  /opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group inventory \
  --topic orders \
  --reset-offsets --to-earliest \
  --execute

Then start the consumer again without inventing new produce traffic. You are reading history you already have.

Retention matters. If the segment aged out, replay cannot invent bytes back. Know your topic retention before you promise “we can always rebuild.”

Idempotency matters. Replay re-delivers. Upsert by orderId, ignore duplicate emails, use an “already processed” store — something that makes a second pass safe. Exactly-once tooling exists; most teams win first with boring idempotent handlers (we go deep on that later in the series).

Rebalances in one paragraph

When a member joins or leaves a group (deploy, crash, scale), Kafka reassigns partitions among remaining members. That pause is a rebalance, not “Kafka is down.” Consumers stop briefly, get new assignments, resume from committed offsets. Hot partitions, sticky assignors, and lag spikes during deploys are Post 6. Tonight: expect rebalances when you change group membership; do not treat them as mystical outages.

Myths that waste a night

What this taught you

Habits worth keeping

1. Name groups after the workload — inventory, email, not consumer-1.
2. Size partitions for the busiest group’s parallelism — fan-out groups do not multiply partition need the same way.
3. Reset only when consumers in that group are stopped — live commits race your reset.
4. Design for replay on day one — idempotent side effects beat heroic “exactly once” settings.

What to remember

Consumer groups turn one durable log into many independent readers — or into a scaled worker pool — depending on whether the group id matches.

Replay is not “please send it again.” It is “I moved my bookmark; give me those offsets again.”

Next: Kafka vs HTTP / RabbitMQ / SQS — when the durable log is the right tool, and when a queue or a plain POST is enough.