Event-Driven Systems: Apache Pulsar
While traditional event streaming platforms like Apache Kafka store message logs directly on local broker disks, coupling message routing compute with physical storage creates severe operational pain points. Rebalancing partition replicas when scaling Kafka clusters requires copying hundreds of gigabytes across network links, stalling throughput and causing cluster-wide latency spikes.
Apache Pulsar is a cloud-native, enterprise-grade distributed messaging and streaming platform. By decoupling stateless message routing Brokers from durable ledger storage nodes (Apache BookKeeper), Pulsar enables instantaneous, zero-rebalance scaling, native multi-tenancy, and active-active geo-replication. This guide details Pulsar's architecture, BookKeeper storage ledgers, subscription consumption modes, and serverless Pulsar Functions.
Mental Model: Monolithic Kafka Brokers vs Decoupled Pulsar Brokers & BookKeeper Bookies
Coupling compute and storage in event streaming platforms forces operators to scale expensive storage disks whenever CPU network routing capacity is saturated.
Apache Pulsar Decoupled Architecture separates processing from persistence:
1. Stateless Brokers: Handle client connections, authentication, topic routing, and message caching. Brokers hold no persistent state on disk. 2. Apache BookKeeper (Bookies): A distributed ledger storage layer optimized for low-latency sequential writes. Ledgers are segmented into small fragments and distributed across available Bookie nodes automatically. For event streaming architecture, review building event driven microservices nats jetstream and building realtime data pipelines apache flink.
Quick reference
- Stateless brokers scale compute independently from Apache BookKeeper storage ledgers.
- Zero-rebalance cluster expansion: adding a new Bookie node immediately accepts new ledger writes.
- Segmented topic ledgers distribute storage evenly across storage nodes without partition copying.
- Delivers sub-5 millisecond write latency with strict durability guarantees.
- Powers cloud-native streaming infrastructure at Tencent, Yahoo, Splunk, and CoreConcept.
Remember this
Deploy Apache Pulsar for decoupled compute-storage scaling and zero-rebalance cluster expansion.
Multi-Tenancy, Namespace Isolation, & Hierarchical Topic Naming
Unlike platforms requiring separate clusters per team, Pulsar was engineered for multi-tenant enterprise isolation via hierarchical URLs:
persistent://tenant-name/namespace-name/topic-name
- Tenants: Correspond to organizational business units (e.g. finance, logistics), providing role-based access control (RBAC) and storage quota limits.
- Namespaces: Group related topics for operational management (e.g. finance/payments). Rate limits, retention policies, geo-replication clusters, and encryption keys are configured at the namespace level.
Quick reference
- Hierarchical topic URLs (persistent://tenant/namespace/topic) enforce multi-tenant isolation.
- Namespace-level policies manage storage quotas, retention periods, and encryption keys.
- Native TLS authentication and RBAC restrict tenant topic publish/subscribe permissions.
- Prevents noisy-neighbor resource starvation across multi-department corporate clusters.
- Consolidates enterprise messaging into a single unified Pulsar cluster topology.
Remember this
Structure Pulsar topics using tenants and namespaces for enterprise security and resource isolation.
Flexible Subscription Modes: Exclusive, Failover, Shared, & Key_Shared
Pulsar unifies message queuing (RabbitMQ style) and event streaming (Kafka style) into a single API through 4 consumption modes:
1. Exclusive: Only 1 consumer connects to the topic subscription (strict ordering). 2. Failover: 1 active consumer receives messages; backup consumers take over instantly if the active consumer drops. 3. Shared (Worker Queue): Messages are distributed Round-Robin across multiple parallel consumers to scale processing throughput beyond topic partition counts. 4. Key_Shared: Messages with identical keys are routed to the same consumer instance, combining parallel worker scalability with key-level ordering.
Quick reference
- Exclusive and Failover modes guarantee strict FIFO message consumption ordering.
- Shared mode acts as a high-throughput worker queue without partition count limitations.
- Key_Shared routes matching key messages to the same consumer for ordered parallel processing.
- Consumer groups can switch subscription modes dynamically without modifying topic schemas.
- Simplifies architecture by eliminating the need for separate RabbitMQ and Kafka clusters.
Remember this
Select Exclusive, Shared, or Key_Shared subscription modes to unify streaming and worker queues.
Pulsar Functions Serverless Stream Processing & Geo-Replication
Pulsar eliminates the need for external stream processing frameworks (like Flink or Spark) for simple event transformations through Pulsar Functions:
1# Python Pulsar Function2def process(input_event, context):3 if input_event["amount"] > 10000:4 context.publish("persistent://finance/fraud/high-value", input_event)5 return input_event### Global Geo-Replication
Pulsar natively replicates messages asynchronously across geographically distributed clusters (e.g. us-east to eu-central) at the namespace level, providing reliable disaster recovery without external mirror makers.
Quick reference
- Pulsar Functions execute lightweight serverless Python, Java, or Go transformations inside brokers.
- Eliminates external stream processing dependencies for filtering, routing, and enrichment.
- Native active-active geo-replication synchronizes topics across global cloud regions asynchronously.
- Tiered Storage automatically offloads historical ledgers to low-cost S3 / GCS object buckets.
- Ensures high-availability disaster recovery across multi-region cloud deployments.
Remember this
Use Pulsar Functions for lightweight serverless event transformations and native geo-replication.
Key takeaway
To test Apache Pulsar locally, run docker run -it -p 6650:6650 -p 8080:8080 apachepulsar/pulsar:latest bin/pulsar standalone. Publish messages using pulsar-client produce my-topic --messages 'hello'.
Related Articles
Explore this topic