Module 3 — Asynchronous Processing

Consumer Groups & Scaling

Scale message processing horizontally with consumer groups while ensuring each message is processed exactly once.

1The Pizza Delivery Analogy

Simple Analogy
A pizza shop has multiple delivery drivers (consumers) in a group. When an order comes in, only ONE driver takes it—they don't all deliver the same pizza!

Each driver handles different orders, but they all work for the same shop (group). If one driver is busy, another takes the next order.

A Consumer Group is a set of consumers that work together to process messages from a queue/topic. Each message is delivered to only ONE consumer in the group, enabling parallel processing without duplication.

2How Consumer Groups Work

📨
Messages
→
Consumer Group A
C1
C2
C3

Each message goes to exactly ONE consumer in the group

Key Rules:

  • ✓Each message delivered to ONE consumer per group
  • ✓Multiple groups can each receive all messages (pub/sub)
  • ✓Consumers in a group share the workload
  • ✓If a consumer dies, others in the group take over

3Partitions & Consumer Assignment

In Kafka-style systems, topics are split into partitions. Each partition is assigned to one consumer in the group.

Partition 0
1
2
3
↓
C1
Partition 1
1
2
3
↓
C2
Partition 2
1
2
3
↓
C3

3 partitions → 3 consumers (optimal)

Scaling Limit

You can't have more consumers than partitions in a group. Extra consumers sit idle. Plan partition count based on expected max consumers.

4Scaling Scenarios

6
partitions
2
consumers
Each consumer handles 3 partitions
6
partitions
6
consumers
Each consumer handles 1 partition (optimal)
6
partitions
8
consumers
6 active, 2 idle (wasted resources)
6
partitions
1
consumers
1 consumer handles all 6 (bottleneck)

5Rebalancing

When consumers join or leave, partitions are rebalanced across remaining consumers.

1
Consumer Joins/Leaves
New consumer added, or existing one crashes/shuts down
2
Group Coordinator Detects
Heartbeat timeout or explicit leave notification
3
Stop Processing
All consumers pause and commit their offsets
4
Reassign Partitions
Coordinator redistributes partitions among active consumers
5
Resume Processing
Each consumer starts processing its new partition assignment
Minimize Rebalancing

Use static membership (group.instance.id in Kafka) for short restarts—consumer rejoins without triggering full rebalance.

6Multiple Consumer Groups

Different groups can independently consume the same messages. Each group tracks its own offset.

Topic
→
C
C
Analytics Group
Aggregates metrics
C
C
C
Notification Group
Sends alerts
C
Archive Group
Stores to S3

All 3 groups receive ALL messages independently

7Key Takeaways

1Consumer groups enable horizontal scaling—add consumers to increase throughput.
2Each message goes to exactly one consumer per group.
3Partitions limit scaling—can't have more active consumers than partitions.
4Rebalancing redistributes work when consumers join/leave—minimize it.
5Multiple groups can independently consume the same messages (fan-out).
6Monitor consumer lag (offset behind) to detect scaling needs.