“We already have pub/sub messaging infrastructure in our platform. Why are you asking for a streaming infrastructure? Use our pub/sub messaging infrastructure” – Platform Product Manager
Streaming and Messaging Systems are different. The use-cases are different.
Both streaming and messaging systems use the pub-sub pattern with producers posting messages and consumers subscribing. The subscribed consumers may choose to poll or get notified. Consumers in streaming systems generally poll the brokers, and the brokers push messages to consumers in messaging systems. Engineers use streaming systems to build data processing pipelines and messaging systems to develop reactive services. Both systems support delivery semantics (at least once, exactly once, at most once) of the messages. Brokers in streaming systems are dumber than messaging systems that build routing and filtering intelligence in the brokers. Streaming systems are faster than messaging systems due to a lack of routing and filtering intelligence 🙂
Let’s look at the top three critical differences in detail:
#1: Data Structures
In streaming, the data structure is a stream, and in messaging, the data structure is a queue.
“Queue” is FIFO (First In First Out) data structure. Once a consumer consumes an element, it is removed from the queue, reducing the queue size. A consumer cannot fetch the “third” element from the queue. Queues don’t support random access. E.g., A queue of people waiting to board a bus.
“Stream” is a data structure that is partitioned for distributed computing. If a consumer reads an element from a stream, the stream size does not reduce. The consumer can continue to read from the last read offset within a stream. Streams support random access; the consumer may choose to seek any reading offset. The brokers managing streams keep the state of each consumer’s reading offset (like a bookmark while reading a book) and allow consumers to read from the beginning, the last read offset, a specific offset, or the latest. E.g., a video stream of movies where each consumer resumes at a different offset.
In streaming systems, consumers refer to streams as Topics. Multiple consumers can simultaneously subscribe to topics. In messaging systems, the administrator configures the queues to send messages to one consumer or numerous consumers. The latter pattern is called a Topic used for notifications. A Topic in the streaming system is always a stream, and it’s always a queue in a messaging system.
Both stream and queue data structures order the elements in a sequence, and the elements are immutable. These elements may or may not be homogenous.
Queues can grow and shrink with publishers publishing and consumers consuming, respectively. Streams can grow with publishers publishing messages and do not shrink with consumers consuming. However, streams can be compacted by eliminating duplicates (on keys).
#2: Distributed (Cluster) Computing Topology
Since a single consumer consumes an element in a queue in a load-balancing pattern, the fetch must be from the central (master) node. The consumers may be in multiple nodes for distributed computing. The administrator configures the master broker node to store and forward data to other broker nodes for resiliency; however, it’s a single master active-passive distributed computing paradigm.
In the notification (topic) pattern, multiple consumers on a queue can consume filtered content to process data in parallel. The administrator configures the master node to store and forward data to other broker nodes that serve consumers. The publishers publish to a single master/leader node, but consumers can consume from multiple nodes. This pattern is the CQRS (Command Query Responsibility Segregation) pattern of distributing computing.
The streaming pattern is similar to the notification pattern w.r.t. distributed computing. Unlike messaging, partition keys break streams into shards/partitions, and the lead broker replicates these partitions to other brokers in the cluster. The leader election process selects a broker as a leader/master for a given shard/partition, and shard/partition replications serve multiple consumers in the CQRS pattern. The consumers read streams from the last offset, random offset, beginning, or latest.
If the leader fails, either a passive slave can take over, or the cluster elects a new leader from existing slaves.
#3: Routing and Content Filtering
In messaging systems, the brokers implement the concept of exchanges, where the broker can route the messages to different endpoints based on rules. The consumers can also filter content delivered to them at the broker level.
In streaming systems, the brokers do not implement routing or content filtering. Consumers may filter content, but utility libraries in the consumer filter out the content after the broker delivers the content to the consumer.
Tabular Differences View
| Category | Streaming | Messaging |
| Support Publish and Subscribe Paradigm | Yes | Yes |
| Polling vs. Notification | Polling by Consumers | Notification by Brokers to consumers |
| Use Case | Data Processing Pipelines | Reactive (micro)services |
| Delivery Semantics Supported | at-most-once at-least-once exactly-once | at-most-once at-least-once exactly-once |
| Intelligent Broker | No | Yes |
| Data Structure | Stream | Queue |
| Patterns | CQRS | Content-Based Routing/Filtering Worker (LB) Distribution Notification CQRS |
| Data Immutability | Yes | Yes |
| Data Retention | Yes. Not deleted after delivery. | No. Deleted after delivery. |
| Data compaction | Yes. Key de-duplication. | N/A |
| Data Homogeneity | Heterogenous by Default. Supports schema checks on data outside the broker. | Heterogenous by Default. |
| Speed | Faster than Messaging | Slower than Streaming |
| Distributed Computing Topology | Broker cluster with single master per stream partition and consumers consuming from multiple brokers with data replicated across brokers | Broker cluster with single master per topic/queue. Active-passive broker configuration for the load-balancing pattern. Data replicated across brokers for multiple consumer distribution. |
| State/Memory | Brokers remember the consumers’ bookmark (state) in the stream | Consumers always consume from time-of-subscription (latest only) |
| Hub-and-Spoke Architecture | Yes | Yes |
| Vendors/Services (Examples) | Kafka Azure Event Hub AWS Kinesis | RabbitMQ Azure Event Grid AWS SQS/SNS |
| Domain Model | A stream of GPS positions of a moving car | A queue to buy train tickets |
Visual Differences View
Summary
Use the right tool for the job. Use messaging systems for event-driven services and streaming systems for distributed data processing.
Digital Journey 