Event-Driven Architecture: When to Use Queues vs. Event Streams

1. Executive Summary & Core Conflict

As software architectures transition away from synchronous request-response monoliths toward decoupled, asynchronous microservices, messaging infrastructure becomes the central circulatory system of the enterprise. When evaluating how services should communicate asynchronously, engineering teams frequently conflate Message Queues (such as RabbitMQ, Amazon SQS, and ActiveMQ) with Event Streams (such as Apache Kafka, Redpanda, and AWS Kinesis).

Both technologies are loosely labeled as "message brokers" or "pub/sub systems." This superficial semantic overlap leads to serious architectural blunders: using Kafka to orchestrate transactional worker jobs, or using RabbitMQ to construct historical audit ledgers and real-time analytical event pipelines.

The core conflict between queues and streams lies in message lifecycle, state ownership, and consumption semantics.

A message queue is an ephemeral, broker-managed coordination tool engineered to distribute work to competing consumers and delete messages immediately upon successful acknowledgment. In contrast, an event stream is a distributed, append-only, immutable commit log where consumer groups independently maintain their own read offsets over persistent history.

Selecting the wrong primitive introduces catastrophic production consequences: unrecoverable partition head-of-line blocking, catastrophic consumer group rebalancing storms, or broker memory collapse under backpressure. Architects must select messaging systems based on concrete structural requirements rather than vague decoupled aspirations.

2. Deep Technical Breakdown

Message Queues: Broker-Centric Task Coordination

In a traditional message queue architecture (e.g., RabbitMQ operating over AMQP 0-9-1), the broker is intelligent, stateful, and responsible for message delivery lifecycle:

1. Routing and Exchanges: Producers publish messages to an Exchange (Direct, Fanout, Topic, or Headers), which inspects routing keys and binds messages to discrete queues.
2. Competing Consumers: Multiple worker instances subscribe to the same queue. The broker load-balances individual messages across connected workers using round-robin or prefetch configurations (basic.qos).
3. Destructive Consumption: When a worker successfully processes a message, it transmits a basic.ack. The broker immediately purges the message from memory and disk. If a worker crashes before sending an acknowledgment, the broker re-queues the message for another worker.
4. Per-Message Control: Queues support per-message TTL, message prioritization (e.g., Priority Queues), and automatic dead-letter exchange (DLX) routing upon repeated rejection (basic.nack).

Because the broker tracks per-message state across thousands of active consumers, broker memory consumption scales linearly with in-flight unacknowledged messages.

Event Streams: Distributed Append-Only Commit Logs

Event stream systems (e.g., Apache Kafka, Redpanda) invert the relationship: the broker is simple, fast, and dumb, while the client consumer is intelligent and stateful.

1. Partitioned Commit Log: A Kafka Topic is divided into one or more append-only, ordered physical logs called Partitions. Producers append serialized byte records to the tail of a partition. Each record receives a monotonically increasing integer identifier called an Offset.
2. Zero-Copy Performance: Brokers do not inspect or parse payloads. They leverage OS-level page caches and the Linux sendfile() system call to transfer byte buffers directly from disk cache to the network socket, avoiding user-space memory copies and achieving millions of messages per second.
3. Immutable Persistence: Messages are not deleted when consumed. Partitions retain data based on time (e.g., 7 days) or size limits (e.g., 500 GB), independent of consumer state.
4. Consumer Groups and Offset Tracking: Consumers organize into Consumer Groups. Each partition in a topic is assigned to exactly one consumer instance within a group, guaranteeing strict ordering within that partition. Consumers periodically commit their current read offset back to an internal metadata topic (__consumer_offsets).

PRODUCER ---> Partition Key "user_42" ---> Hash(Key) % 3 = Partition 1
========================================================================
PARTITION 0: [0][1][2][3][4][5] ---> Read by Consumer Group A (Worker 1)
PARTITION 1: [0][1][2][3][4][5][6] -> Read by Consumer Group A (Worker 2)
PARTITION 2: [0][1][2][3] ----------> Read by Consumer Group A (Worker 3)
========================================================================
* Consumers pull batches and update their local offset pointers independently.
* Consumer Group B can replay historical events from Offset 0 at any time.

Delivery Guarantees: The Reality of Effectively-Once

Both architectures must handle network drops, process crashes, and hardware failures:

• At-Most-Once: The consumer commits the offset or acks the message before processing. If the consumer crashes during execution, the message is permanently lost.
• At-Least-Once: The consumer commits the offset or acks after successfully persisting work. If a crash occurs before the ack completes, the message is redelivered. This is the standard production default and mandates strict consumer idempotency.
• Effectively-Once: Achieved not by magic broker protocols, but by pairing idempotent producer IDs and transaction sequence numbers with atomic database outbox commits or deduplication tables (INSERT ... ON CONFLICT DO NOTHING).

3. Production Edge Cases & Failure Modes

Edge Case 1: The Partition Poison Pill (Head-of-Line Blocking)

In Kafka, strict ordering is enforced within each partition. If an unhandled exception or deserialization error occurs on record offset 1042, the consumer cannot simply "skip" or ack that single record while continuing down the log. If the consumer crashes and restarts, it resumes reading from offset 1042, creating an infinite crash loop (poison pill). This blocks all subsequent events on that partition indefinitely.

Mitigation: Implement a non-blocking Dead Letter Topic (DLT) pattern. When an unrecoverable processing failure occurs, publish the failed record alongside error context to a dedicated error topic (orders-dlt), commit the partition offset, and continue processing.

Edge Case 2: Consumer Group Rebalance Storms

In Kafka, when a consumer instance pauses processing longer than max.poll.interval.ms (often caused by an expensive downstream database transaction or long-running Garbage Collection pause), the Kafka coordinator presumes the node is dead. The coordinator triggers a stop-the-world rebalance, revoking partition assignments from all instances in the group, redistributing partitions, and halting stream ingestion cluster-wide for 30 to 90 seconds.

Mitigation: Decouple stream polling from task execution using a worker thread pool, increase max.poll.interval.ms to match worst-case downstream latency, or adopt Kafka's Cooperative Sticky Assignor (CooperativeStickyAssignor) to avoid stop-the-world revocations.

Edge Case 3: Broker Memory Exhaustion under Backpressure

In RabbitMQ, if consumer services degrade or network partitions occur, unconsumed messages accumulate in RAM. When broker memory hits vm_memory_high_watermark, RabbitMQ blocks incoming publisher TCP connections and begins paging messages from RAM to disk. Disk paging degrades broker I/O throughput by orders of magnitude, stalling the entire messaging topology.

Mitigation: Configure Quorum Queues or Lazy Queues (x-queue-mode: lazy) in RabbitMQ to stream messages directly to disk upon arrival, eliminating RAM-based saturation cliffs.

4. Comparative Markdown Table

5. Pragmatic Decision Framework / Checklist

Use this decision matrix when choosing between queues and streams:

Choose a Message Queue (RabbitMQ / SQS) when:

• Individual work units require granular per-message acknowledgment, failure retries, and discrete Dead Letter Queue routing.
• Tasks take unpredictable, highly variable processing times (e.g., 50ms to 45 seconds). In streams, a 45-second job would block the entire partition.
• You require advanced routing patterns (e.g., binding queues dynamically based on wildcards and complex header rules).
• The number of worker consumers needs to scale dynamically up and down (e.g., from 5 to 500 pods) without repartitioning infrastructure.

Choose an Event Stream (Kafka / Redpanda) when:

• The system requires event sourcing, real-time change data capture (CDC), or analytical event fan-out across multiple downstream microservices.
• Historical replay is a business requirement (e.g., rebuilding search indexes or retraining machine learning models from day zero).
• Sustained throughput exceeds 50,000 messages per second and requires sequential disk I/O performance.
• Processing relies on strict chronological ordering for specific business entities (e.g., transactions for account_id must execute in sequence).

Operational Health Checklist:

• For Kafka: Partition count must be planned upfront. Formula: $$\text{Partitions} = \max\left(\frac{\text{Target Throughput}}{\text{Producer Throughput}}, \frac{\text{Target Throughput}}{\text{Consumer Throughput}}\right)$$
• For RabbitMQ: Enforce basic.qos(prefetch_count) on all consumers (typically 10 to 100). Never run with unbounded prefetch (prefetch = 0), which invites OOM crashes.
• Monitor Consumer Lag (Kafka) or Queue Depth (RabbitMQ) as tier-1 operational alerts. A steadily rising lag indicates consumer capacity failure.