Message Queue Architecture: Apache Pulsar vs Kafka
Apache Kafka revolutionized event streaming by implementing a append-only distributed log model where message ordering is guaranteed within partition topic logs. However, as enterprise event streaming workloads scale to millions of topics, petabyte retention windows, and multi-tenant environments, Kafka's coupled storage-and-compute architecture faces operational challenges during partition rebalancing and storage expansion.
Apache Pulsar introduces a next-generation cloud-native messaging architecture that decouples stateless serving nodes (Pulsar Brokers) from stateful storage nodes (Apache BookKeeper). This architectural separation enables instant partition rebalancing, native multi-tenancy, and automated tiered storage to cloud object stores (S3/GCS). This guide compares Apache Pulsar segment-based storage against Apache Kafka topic partitions, analyzing multi-tenancy, geo-replication, Pulsar Functions, and latency SLAs.
Mental Model: Monolithic Log Segments (Kafka) vs Tiered Storage Architecture (Pulsar)
In Apache Kafka, topic partitions are physically bound to specific broker disk drives. Adding a new broker node requires copying multi-gigabyte partition log files across the network, creating heavy CPU and disk I/O contention on production clusters.
Apache Pulsar's Decoupled Architecture separates compute from storage:
1. Stateless Brokers: Handle client TCP connections, publish/subscribe routing, and caching without storing topic log data on local disks.
2. Stateful BookKeeper Ensemble: Stores topic log data in fine-grained, ledger-based Ledgers and Ledger Segments distributed across BookKeeper bookies (Bookies). For event-driven messaging patterns, review building event driven microservices apache pulsar and building realtime data pipelines apache flink.
Quick reference
- Decouples stateless broker compute nodes from stateful Apache BookKeeper storage nodes.
- Eliminates partition rebalancing data copy overhead when adding or removing broker nodes.
- Offloads historical log segments automatically to cheap S3/GCS object storage.
- Supports million-topic scalability per cluster without degrading ZooKeeper / KRaft metadata performance.
- Powers mission-critical messaging platforms at Tencent, Yahoo, Splunk, and CoreConcept.
Remember this
Choose Apache Pulsar for decoupled storage architecture, multi-tenancy, and instant partition scaling.
Apache BookKeeper Segmented Storage & Instant Partition Rebalancing
Pulsar divides topic partitions into small, fixed-size Ledger Segments (e.g. 100MB or 1 hour duration):
1# Pulsar Broker & BookKeeper Segment Configuration2managedLedgerMaxEntriesPerLedger: 500003managedLedgerMinLedgerRolloverInSeconds: 6004offloadAutoTriggerSizeThresholdBytes: 10737418240 # Auto-offload to S3 after 10GB5offloadDriver: 'aws-s3'6s3ManagedLedgerOffloadBucket: 'pulsar-tiered-storage'When a BookKeeper node runs low on storage capacity, new ledger segments are automatically assigned to available bookies in the ensemble without moving existing historical segments.
Quick reference
- Ledger segment chunks allow fine-grained distribution of topic data across BookKeeper bookies.
- Adding new bookie nodes instantly increases cluster storage capacity without data migration.
- Tiered storage transparently shifts cold log segments to S3/GCS, reducing SSD storage costs by 80%.
- BookKeeper journal files ensure low write latency via sequential SSD append writes.
- Protects message durability with configurable Write Quorum (AckQuorum) replicas.
Remember this
Use Apache BookKeeper ledger segments to scale storage capacity instantly without copying data.
Multi-Tenancy, Geo-Replication, & Pulsar Functions Stream Processing
Apache Pulsar was built natively for enterprise multi-tenancy and global edge deployments:
- Hierarchical Topic Naming: Topics are structured as persistent://tenant/namespace/topic (e.g. persistent://finance/billing/payments).
- Native Geo-Replication: Configures asynchronous or synchronous cross-datacenter topic replication natively at the namespace level without deploying external MirrorMaker connectors.
- Pulsar Functions: Lightweight serverless event processing runtime built directly into Pulsar brokers (Python, Java, Go):
Quick reference
- Hierarchical topic namespace structure (tenant/namespace/topic) enforces isolation and ACLs.
- Native multi-datacenter geo-replication operates without external replication plugins.
- Pulsar Functions provide lightweight serverless transformations without extra Flink clusters.
- Flexible message consumption models support both Queueing (Shared/Key_Shared) and Streaming (Exclusive/Failover).
- Enforces tenant quota limits (storage, bandwidth, topic count) at runtime.
Remember this
Deploy Pulsar native multi-tenancy and geo-replication for multi-tenant enterprise event routing.
Benchmark Throughput, Latency SLAs, & Operability Tradeoffs
Evaluating Pulsar vs Kafka performance requires understanding workload characteristics:
- Tail Latency SLAs: Pulsar consistently delivers lower 99.9th percentile write latency under high concurrency due to BookKeeper's dedicated Journal SSD drives.
- Operational Complexity: Kafka is simpler to operate for small clusters (KRaft mode removes ZooKeeper). Pulsar requires managing Brokers, BookKeeper Bookies, and ZooKeeper / Metadata Store, requiring Kubernetes Operators (pulsar-operator) for production lifecycle management.
Quick reference
- Pulsar delivers lower p99 write latency SLAs under heavy concurrent publish traffic.
- Kafka offers simpler 2-tier operations (Brokers + KRaft) for smaller streaming architectures.
- Pulsar Key_Shared subscription mode enables out-of-order message processing per consumer.
- Kafka provides a wider ecosystem of third-party connectors and managed cloud offerings.
- Establishes a resilient messaging foundation tailored to organizational scale.
Remember this
Benchmark tail latency requirements and operational overhead when choosing between Pulsar and Kafka.
Key takeaway
To test Apache Pulsar locally, run docker run -d -p 6650:6650 -p 8080:8080 apachepulsar/pulsar:latest bin/pulsar standalone. Test messaging via bin/pulsar-client produce persistent://public/default/my-topic --messages 'hello'.
Related Articles
Explore this topic