Module 3 - Async Processing
Kafka vs RabbitMQ vs SQS
Choosing the right message broker for your use case.
1Quick Overview
Simple Analogy
Kafka: A super-highway for events. High throughput, stores everything, replay anytime.
RabbitMQ: A smart post office. Routes messages flexibly, confirms delivery.
SQS: A managed mailbox. Simple, serverless, AWS-native.
RabbitMQ: A smart post office. Routes messages flexibly, confirms delivery.
SQS: A managed mailbox. Simple, serverless, AWS-native.
2Comparison Table
| Aspect | Kafka | RabbitMQ | AWS SQS |
|---|---|---|---|
| Model | Log-based streaming | Traditional queue | Managed queue |
| Throughput | 1M+ msg/s | ~50K msg/s | ~3K msg/s (standard) |
| Message Retention | Days/Forever | Until consumed | 14 days max |
| Replay | Yes | No | No |
| Ordering | Per partition | FIFO possible | FIFO available |
| Operations | Self-managed or Confluent | Self-managed | Fully managed |
3When to Use Each
Apache Kafka
- ✓ High-throughput event streaming
- ✓ Event sourcing and replay
- ✓ Multiple consumers need same data
- ✓ Log aggregation
RabbitMQ
- ✓ Complex routing requirements
- ✓ Request-response patterns
- ✓ Lower volume, flexible routing
- ✓ Need message priority
AWS SQS
- ✓ AWS-native workloads
- ✓ Simple queue needs
- ✓ Serverless architectures
- ✓ No ops overhead desired
4Decision Guide
Need to replay messages?Kafka
Complex routing logic?RabbitMQ
Serverless on AWS?SQS
Millions of events/sec?Kafka
Request-reply pattern?RabbitMQ
Zero operations?SQS (or Confluent Cloud)
5Key Takeaways
1Kafka: High throughput, event streaming, replay, log-based
2RabbitMQ: Flexible routing, traditional messaging, lower latency
3SQS: Fully managed, AWS-native, simple and reliable
4Choose based on throughput, replay needs, and operational preferences
5For AWS workloads without complex needs, SQS is often the right choice
?Quiz
1. You need to replay events from yesterday. Which broker?
2. Building a serverless app on AWS with simple queue needs?