Module 3 - Async Processing
Delivery Guarantees
At-most-once, at-least-once, exactly-once: understanding message delivery semantics.
1The Mail Analogy
Simple Analogy
At-most-once: Send a postcard. It might get lost, but you won't resend. Fast but unreliable.
At-least-once: Send registered mail. Keep resending until you get confirmation. Might deliver duplicates.
Exactly-once: Send with tracking and deduplication. Guaranteed one delivery-expensive but perfect.
At-least-once: Send registered mail. Keep resending until you get confirmation. Might deliver duplicates.
Exactly-once: Send with tracking and deduplication. Guaranteed one delivery-expensive but perfect.
2The Three Guarantees
At-Most-Once
Fire and forget. Message may be lost. No retries.
Use case: Metrics, logs where loss is acceptable
+ Fast, simple- Data loss possible
At-Least-Once
Retry until acknowledged. May deliver duplicates.
Use case: Most business events, notifications
+ No data loss- Duplicates possible
Exactly-Once
Each message processed exactly once. Hardest to achieve.
Use case: Financial transactions, billing
+ No loss or duplicates- Complex, expensive
3How Each Works
At-Most-Once
1. Send message
2. Don't wait for ACK
3. Never retry
At-Least-Once
1. Send message
2. Wait for ACK
3. If no ACK, retry (may cause duplicates)
4. Consumer must be idempotent
Exactly-Once
1. Assign unique message ID
2. Consumer tracks processed IDs
3. Reject duplicates based on ID
4. Use transactions or idempotency keys
4Achieving Exactly-Once
True exactly-once is impossible in distributed systems due to network partitions. We achieve effectively exactly-once through idempotency and deduplication.
Idempotency Keys
Include unique ID with each message. Consumer checks before processing.
Transactional Outbox
Write message to DB in same transaction as business logic.
Kafka Transactions
Kafka's exactly-once semantics with producer transactions.
Deduplication Table
Store processed message IDs, reject duplicates.
5Key Takeaways
1At-most-once: fast but may lose messages
2At-least-once: reliable but may duplicate (most common)
3Exactly-once: achieved via idempotency, not magic
4Idempotent consumers make at-least-once safe
5True exactly-once is effectively exactly-once through deduplication
?Quiz
1. Your payment service processes charges. Which guarantee?
2. How do you handle duplicates with at-least-once?