Module 3 — Asynchronous Processing

Producer-Consumer Pattern

A fundamental pattern where producers create work items and consumers process them independently.

8 min read

1The Factory Assembly Line

Simple Analogy

Think of a car factory. The engine department (producer) makes engines and puts them on a conveyor belt. The assembly department (consumer) picks up engines when ready and installs them in cars.

Neither department waits for the other. The conveyor belt (queue) buffers the difference in their speeds.

The Producer-Consumer Pattern decouples the creation of work (producing) from its execution (consuming). A shared buffer (queue) holds work items, allowing producers and consumers to operate at different rates.

2Pattern Components

📤

Producer(s)

Create messages

→

📦

Queue/Buffer

Holds messages

→

📥

Consumer(s)

Process messages

Producer

• Create work items/messages
• Push to queue
• Handle backpressure
• Don't wait for processing

Queue

• Store messages safely
• FIFO ordering (usually)
• Handle concurrent access
• Persist until consumed

Consumer

• Pull from queue
• Process messages
• Acknowledge completion
• Handle failures/retries

3Scaling Patterns

Single Producer, Single Consumer

→

Simplest form. Good for low throughput.

Single Producer, Multiple Consumers

→

Scale processing by adding consumers. Each message processed once.

Multiple Producers, Multiple Consumers

→

Full scale. Many services produce, many workers consume.

Scaling Rule

Add consumers when queue depth grows. Remove when queue stays empty. Auto-scaling based on queue length is common.

4Implementation Considerations

Message Ordering

Does order matter?

FIFO queues guarantee order but limit throughput. Standard queues are faster but may reorder. Use message timestamps if order matters.

At-Least-Once vs Exactly-Once

How many times is each message processed?

Most queues provide at-least-once (may duplicate on retry). Design consumers to be idempotent—processing twice should be safe.

Backpressure

What if queue is full?

Producer should handle: retry with backoff, drop messages, or block. Don't overwhelm the queue.

Poison Messages

What if a message always fails?

After N retries, move to Dead Letter Queue. Don't let one bad message block the entire queue.

5Code Example

Producer (Node.js + SQS)

const AWS = require('aws-sdk');
const sqs = new AWS.SQS();

async function produce(message) {
  await sqs.sendMessage({
    QueueUrl: process.env.QUEUE_URL,
    MessageBody: JSON.stringify(message),
    MessageAttributes: {
      'Type': { DataType: 'String', StringValue: 'order' }
    }
  }).promise();
  
  console.log('Message sent:', message.id);
}

Consumer (Node.js + SQS)

async function consume() {
  while (true) {
    const result = await sqs.receiveMessage({
      QueueUrl: process.env.QUEUE_URL,
      WaitTimeSeconds: 20,  // Long polling
      MaxNumberOfMessages: 10
    }).promise();
    
    for (const msg of result.Messages || []) {
      try {
        await processMessage(JSON.parse(msg.Body));
        await sqs.deleteMessage({
          QueueUrl: process.env.QUEUE_URL,
          ReceiptHandle: msg.ReceiptHandle
        }).promise();
      } catch (err) {
        console.error('Processing failed:', err);
        // Message will be retried after visibility timeout
      }
    }
  }
}

6Key Takeaways

1Producer-Consumer decouples creation from processing via a shared queue.

2Scale by adding consumers—queue depth is your scaling signal.

3Design consumers to be idempotent—handle duplicate messages safely.

4Use Dead Letter Queues for messages that repeatedly fail.

5Long polling is more efficient than frequent short polls.

6Always ACK after processing, not before—avoid data loss.