PaperlayerView exact revision on arXiv

Source-linked AI summary

Kafka versus RabbitMQ

Philippe Dobbelaere, Kyumars Sheykh Esmaili

arXiv:1709.00333v1cs.DCcs.PF

TL;DR

The paper asks how two widely adopted pub/sub systems, Kafka and RabbitMQ, compare and which architecture best fits particular requirements. It establishes a common framework, performs qualitative and empirical comparisons, and examines distinct features and combined use cases. The results characterize trade-offs in latency, throughput, scalability, replication, and system selection, while acknowledging that the empirical study is not exhaustive.

  • Problem

    The paper addresses how Kafka and RabbitMQ compare and which system best fits particular requirements, beyond difficult-to-generalize or outdated ad-hoc recommendations.

  • Method

    The paper establishes a common pub/sub comparison framework and uses qualitative analysis, empirical experiments, feature comparisons, and use-case analysis.

  • Results

    Replication reduces performance by 50% for RabbitMQ and 75% for Kafka, while the systems show different latency, throughput, and scaling trade-offs.

  • Takeaways & Limitations

    The framework supports selecting RabbitMQ, Kafka, or a combination according to latency, throughput, storage, routing, and scaling requirements.

  • Takeaways & Limitations

    The empirical study is not exhaustive and requires considering how the systems are used within larger application architectures.

Abstract

from arXiv · show

Publish/subscribe is a distributed interaction paradigm well adapted to the deployment of scalable and loosely coupled systems. Apache Kafka and RabbitMQ are two popular open-source and commercially-supported pub/sub systems that have been around for almost a decade and have seen wide adoption. Given the popularity of these two systems and the fact that both are branded as pub/sub systems, two frequently asked questions in the relevant online forums are: how do they compare against each other and which one to use? In this paper, we frame the arguments in a holistic approach by establishing a common comparison framework based on the core functionalities of pub/sub systems. Using this framework, we then venture into a qualitative and quantitative (i.e. empirical) comparison of the common features of the two systems. Additionally, we also highlight the distinct features that each of these systems has. After enumerating a set of use cases that are best suited for RabbitMQ or Kafka, we try to guide the reader through a determination table to choose the best architecture given his/her particular set of requirements.

1 INTRODUCTION

The paper addresses how Kafka and RabbitMQ compare and which to choose by developing a holistic framework for qualitative and quantitative comparison. It also considers each system’s distinct features and use-case suitability.

  • Kafka and RabbitMQ are widely adopted open-source and commercially supported publish/subscribe systems.Both systems have existed for almost a decade and are used by enterprise companies.
  • Their different histories and design goals produce distinct architectural models.RabbitMQ routes messages through exchanges into queues, whereas Kafka appends messages to topic-specific disk logs that consumers pull through indexes.
  • The paper responds to recurring questions about how Kafka and RabbitMQ compare and which system to use.
  • Existing web recommendations are difficult to generalize because they often overlook broader context and may become stale as systems evolve.
  • The paper establishes a common pub/sub framework and applies qualitative and empirical comparisons to common features.It additionally identifies unique features, suitable use-case classes, and options for combining the systems.

2 BACKGROUND: PUB/SUB SYSTEMS

Pub/sub systems decouple producers and consumers while providing routing, delivery, ordering, and other quality-of-service guarantees. Their performance and scalability depend on trade-offs involving latency, throughput, storage, reliability, and distribution.

  • Core functionality: Pub/sub supports scalable, loosely coupled systems by decoupling publishers and subscribers across entity, time, and synchronization dimensions.These dimensions mean participants need not know one another, be active simultaneously, or synchronously block execution.
  • Core functionality: Routing logic determines whether and where producer messages reach consumers, balancing subscription flexibility against performance.Topic-based routing uses publisher-supplied topics and may support wildcards or hierarchies, whereas content-based routing filters message fields and metadata.
  • Quality of service: Delivery guarantees range from at most once to exactly once, while ordering ranges from none through partitioned ordering to global order.Stronger delivery and ordering guarantees require additional recovery, synchronization, or transaction resources and can reduce performance.
  • Quality of service: Distributed pub/sub systems must address reliability, availability, transactions, scalability, latency, and throughput as distinct design concerns.Reliability depends on redundant components, connections, and data; transactions group messages into atomic units; scalability covers growing producers, consumers, topics, and messages.
  • Performance: Latency measures delay through an architecture, whereas throughput measures packets or bytes transported per unit time and can be increased with parallel resources.Latency depends on serial processing steps, storage, persistence, ordering, copying, metadata handling, and consumer dequeueing.
  • Performance: In a simple pipeline, latency and throughput are inversely proportional, and efficiency or scalability can conflict with other desirable properties.The comparison therefore treats performance as a multidimensional trade-off rather than a single objective.

3 HIGH-LEVEL DESCRIPTION

Kafka was built at LinkedIn for high-throughput, distributed event pipelining, using partitioned persistent logs and replayable messages. RabbitMQ implements AMQP through modular exchanges and queues, emphasizing interoperable and flexible message routing.

  • Kafka: Kafka originated at LinkedIn as a centralized event-pipelining platform replacing disparate point-to-point integration systems.Its design targeted high-volume scale-out, long consumer backlogs, batch consumption, and multiple consumers with low overhead.
  • Kafka: Kafka is organized around topics spread across brokers, where each partition is an ordered write-ahead log persisted to disk and readable by many consumers.Additional consumers have very low overhead, while partitions determine the unit and maximum degree of consumer parallelism.
  • Kafka: Kafka distributes production, brokering, and consumption across machine clusters, with each partition providing ordered delivery to consumers.Messages from different partitions are not globally ordered, and consumers can replay messages instead of removing them from the log.
  • Kafka: Kafka improves throughput through batching, persistent data structures, and operating-system page-cache behavior for sequential log reads and buffered writes.Its storage layout appends published messages to the final segment file of each partition.
  • RabbitMQ: RabbitMQ is an AMQP implementation whose binary protocol was designed for interoperability among independently implemented asynchronous messaging systems.AMQP originated through cooperation involving JPMorgan Chase and Red Hat, while RabbitMQ was developed independently in Erlang.
  • RabbitMQ: AMQP divides brokering between exchanges that route messages and queues that store and deliver them, using sequential frames over multiplexed TCP channels.Queue durability is implementation-dependent, with disk-backed and memory-only storage possible.

4 COMMON FEATURES: QUALITATIVE COMPARISON

Kafka and RabbitMQ both buffer messages for later or slower consumption, but their designs differ in time decoupling and routing flexibility. RabbitMQ offers sophisticated extensible routing, whereas Kafka provides more limited producer-controlled partition routing.

  • Buffering: Both Kafka and RabbitMQ can buffer large message batches for later consumption or for consumers operating much more slowly than producers.This common capability supports time-decoupled processing across both systems.
  • Buffering: RabbitMQ keeps buffered messages in DRAM when possible, then moves them to disk without a DRAM copy after memory is exhausted, severely affecting performance.This behavior constrains RabbitMQ when backlog size exceeds available memory.
  • Buffering: Kafka was specifically designed for varied consumption rates and is better positioned to support a wider scale of time decoupling.The comparison links this advantage to Kafka’s handling of differing producer and consumer rates.
  • Routing: RabbitMQ provides sophisticated routing through AMQP exchange types, including flexible wildcard topic routing and content-based header routing.Its exchange model supports multipart topics with * and # wildcards and can route using message content.
  • Routing: RabbitMQ permits custom exchanges, enabling application-specific routing such as community-provided load-balancing exchanges.The extensibility of exchange definitions broadens routing beyond the stock exchange types.
  • Routing: Kafka supports more limited topic routing in which producers select partitions randomly or through a user-specified partitioning function.Partition-by-key hashing can preserve ordering within a partition, while the partition function can be overridden by the user.

4.3 Delivery Guarantees

RabbitMQ and Kafka provide different delivery-guarantee mechanisms, especially for acknowledgments, failure recovery, and ordering. Stronger durability and no-loss behavior can require replication, disk persistence, or reduced throughput.

  • Failure behavior: At-least-once recovery can disrupt packet order, whereas RabbitMQ can redeliver lost messages in order without resending an entire failed batch.Kafka preserves order only under specified conditions, and multiple-partition delivery cannot preserve order generally.
  • Acknowledgment mechanisms: RabbitMQ confirms messages after routing and acceptance, while Kafka appends them to a partition log on broker nodes.The systems differ in how ownership transfers from producer to broker and, for RabbitMQ, to the consumer.
  • Acknowledgment mechanisms: RabbitMQ can publish batches with individual ACK/NACK replies indicating that messages were safely fsynced to storage.This confirmation mechanism avoids relying exclusively on heavyweight AMQP transactions for publication safety.
  • Acknowledgment mechanisms: Kafka acknowledgment levels range from best effort to leader or quorum receipt, but these acknowledgments do not necessarily indicate that data was committed to disk.The selected setting controls the trade-off between delivery assurance and performance.
  • Failure behavior: Kafka’s default non-replicated configuration may lose messages during failures because acknowledgments are sent before fsync, although this can be changed at a throughput cost.The limitation applies specifically when Kafka runs without replication and uses its default configuration.

4.4 Ordering Guarantees

RabbitMQ preserves order for flows using a single AMQP channel and reorders retransmissions internally, whereas Kafka preserves order only within partitions. Kafka’s inter-batch ordering requires at most one outstanding produce request, limiting performance.

  • RabbitMQ ordering: RabbitMQ conserves order for flows using a single AMQP channel and reorders retransmitted packets inside its queue logic.Using a load balancer across different channels removes the ordering relation between packets.
  • Kafka ordering: Kafka conserves order only inside a partition, where each batch either passes or fails as a unit.Ordering is not guaranteed across partitions.
  • Kafka ordering: Kafka requires at most one outstanding produce request to conserve inter-batch order, which reduces maximum performance.The ordering condition applies within a partition when preserving order across batches.

4.5 Availability

Both systems provide availability through replication, but their replication configurations differ. RabbitMQ requires explicit mirrored-queue setup, while Kafka requires a sufficiently high replication factor and remains exposed to partition-related split-brain issues.

  • Replication: Both RabbitMQ and Kafka provide availability via replication.Replication is the shared availability mechanism identified for both systems.
  • RabbitMQ: RabbitMQ clusters can replicate exchange and binding information but do not automatically create mirrored queues.Mirrored queues must be explicitly configured when queues are created.
  • Kafka: Kafka availability requires running the system with a suitably high replication factor.Replication-based architectures can experience split-brain problems during fault-induced network partitions.

4.6 Transactions

RabbitMQ and Kafka differ in transaction support and atomicity guarantees. RabbitMQ extends AMQP transactions but still provides limited atomicity, while Kafka does not currently support transactions.

  • RabbitMQ: RabbitMQ transactions apply to publishes, acknowledgments, and rejection, but AMQP guarantees atomicity only for transactions involving a single queue.RabbitMQ still provides no atomicity guarantee even for a single-queue transaction if a commit fault exposes only part of the publishes.
  • Kafka: Kafka currently does not support transactions, although a proposal to add the feature in future releases has been adopted.The passage describes planned support rather than an available transaction mechanism.

4.7 Multicast

RabbitMQ and Kafka both support multicast, but they place different state-management responsibilities in the system. RabbitMQ tracks consumer-specific queues and message ownership, whereas Kafka retains one topic copy and relies on consumers to fetch messages by index.

  • RabbitMQ supports multicast through a dedicated queue per consumer while retaining one copy of message bodies across queues.It maintains per-queue indexes and metadata, increasing bindings rather than duplicating message bodies.
  • RabbitMQ can determine when messages are safe to flush because it tracks which consumers have taken ownership of them.
  • Kafka maintains one non-replicated copy per topic and lets each consumer fetch messages independently by message index.Kafka does not track when all consumers have taken ownership, so retention follows a configurable time or size limit.
  • Adding RabbitMQ nodes is transparent to consumers, but redistributing existing queue masters requires manual intervention.New nodes can become masters for newly created queues and accept channels for existing exchanges and queues.
  • Adding Kafka nodes can redistribute existing partitions online, but consumer groups require updated partition-to-consumer mappings.A new replica catches up before the old partition replica is removed.

5 COMMON FEATURES: QUANTITATIVE COMPARISON

The quantitative comparison evaluates RabbitMQ and Kafka empirically across latency, throughput, delivery guarantees, and availability. Both provide millisecond-scale latency, but Kafka is more sensitive to cache misses and reliability overhead, while throughput depends strongly on record size, routing, partitions, topics, and delivery mode.

  • Latency: Both systems deliver millisecond-level low latency, but Kafka’s latency rises sharply when reads require disk access.RabbitMQ latency remains below 10 ms up to medium load, whereas Kafka can reach around 100 ms when reading from disk.
  • Latency: 30% cache-miss reads increased Kafka latency by more than an order of magnitude.
  • Latency: Kafka’s at-least-once latency is about twice its at-most-once latency from cache, while RabbitMQ’s reliability-level latency is nearly unchanged.
  • Throughput: Larger records reduce throughput for both systems, while Kafka’s byte throughput is almost linear in record size.RabbitMQ throughput decreases as packet size grows, and Kafka identifies packet copying as the dominant operation for byte throughput.
  • Throughput: Kafka throughput scales with topic count only up to about 8 producers in the experimental setup, after which performance diminishes.
  • Throughput: Kafka partition throughput tapers after about 10 partitions and peaks at 200 in the experimental setup.The optimal point depends on the system’s core, DRAM, and performance characteristics.
  • Throughput: 50% lower throughput was observed for RabbitMQ in at-least-once mode, compared with its best-effort scenario.
  • Throughput: Kafka’s at-least-once throughput decreased by 50% to 75% compared with best effort.

6 DISTINCT FEATURES

Kafka and RabbitMQ provide distinct operational features that shape their suitability for different deployments. Kafka emphasizes durable replayable streams and ecosystem tooling, while RabbitMQ emphasizes interoperability, routing, federation, management, and resource controls.

  • Kafka features: Kafka stores messages on disk and purges them automatically by retention time or topic disk quota.
  • Kafka features: Kafka’s stateless consumer model lets consumers replay long-term stored messages, supporting downstream fault tolerance.
  • Kafka features: Kafka Connect and Kafka Streams extend the ecosystem for scalable data movement and stream processing.
  • Kafka features: Kafka log compaction retains at least the latest value for each message key within a topic partition, supporting change-feed use cases.
  • RabbitMQ features: RabbitMQ supports AMQP interoperability, additional MQTT and STOMP protocols, federated exchanges, Shovel, and virtual hosts for isolated environments.
  • RabbitMQ features: RabbitMQ provides management visibility, consumer/message state tracking, optional disk-free routing, flow control, queue limits, and message or queue TTLs.

7 PREFERRED USE CASES

Kafka suits simple-routing, high-throughput, durable, replayable, and stream-processing workloads, while RabbitMQ suits complex routing, RPC, realtime filtering, and low-latency dataflow. Some requirements are best met by combining both systems, with architecture ordered according to latency, storage, throughput, and routing needs.

  • Kafka use cases: Kafka fits simple-routing workloads when throughput per topic exceeds what RabbitMQ can handle, such as an event firehose.Kafka also provides a scalable loading path for high-throughput Big Data platforms.
  • Kafka use cases: Kafka’s durability and efficient multicast let it connect batch and streaming services, while its log-centric design supports externally exposed database change feeds.Kafka Streams and Samza further support stateful, fault-tolerant stream processing.
  • RabbitMQ use cases: RabbitMQ is suited to message routing and RPC, including correlation IDs and direct replies that avoid dedicated reply queues.Its routing capabilities also support edge/core message-routing scenarios.
  • RabbitMQ use cases: RabbitMQ is preferred for complex filtering, stronger ordering, and routing-intensive workloads, including realtime processing and geographic routing.The paper also identifies RabbitMQ as suitable for dataflow graphs with sub-5 ms latency for most packets and up to 40Kpps on single nodes.
  • Combined use: A combination is appropriate when neither system alone covers the requirements, with RabbitMQ→Kafka favoring latency plus long-term storage and Kafka→RabbitMQ favoring high total throughput plus complex routing.The systems may also run in parallel when merging existing Kafka- and RabbitMQ-based architectures.
  • Decision framework: The determination table maps feature combinations to architectural choices but necessarily oversimplifies decisions that require considering all relevant dimensions.Architects are advised to evaluate the broader comparison before selecting an architecture.

8 CONCLUSION

Kafka and RabbitMQ share the pub/sub paradigm but embody different designs: RabbitMQ emphasizes flexible, classic messaging, whereas Kafka emphasizes high-throughput distributed logs and replayability. The comparison finds low latency in both, different throughput scaling behavior, substantial replication costs, and cases where combining the systems is preferable.

  • Architectural distinction: RabbitMQ uses exchanges and bindings for flexible routing, while Kafka uses partitioned persisted logs for high throughput, replayability, and consumers with varying speeds.Kafka relies on batching, persistent data structures, and the OS page cache; RabbitMQ emphasizes in-memory handling and nearly empty queues.
  • Comparison framework: The paper establishes a framework for positioning Kafka and RabbitMQ through qualitative and quantitative comparison.
  • Performance: Both systems achieve mean/median latency around 10 ms, but Kafka’s at-least-once latency is about twice as large and disk reads can increase it by up to an order of magnitude.
  • Performance: In the basic single-node setup, RabbitMQ outperforms Kafka in throughput, while increasing Kafka partitions significantly improves performance and increasing RabbitMQ producer/channel count helps only moderately.
  • Performance: Replication reduces performance by 50% for RabbitMQ and 75% for Kafka, while both systems can scale across nodes through flow partitioning.
  • Decision guidance: Architects should consider dimensions beyond efficiency, including qualitative features and distinct system capabilities, before deciding.
  • Decision guidance: A combined Kafka–RabbitMQ architecture may be the best option when requirements are not covered by either system alone.
Loading 1709.00333v1…