Redis Streams for Durable Message Queues: A Complete Guide
Redis Streams provide a powerful way to implement durable message queues using the same Redis instance already running in your environment. No extensions, modules, or hacks required. This isn’t your typical pub/sub messaging - Redis Streams offer ordered messages that persist until explicitly cleared, making them perfect for reliable message processing.
What Makes Redis Streams Different
Unlike traditional Redis pub/sub, Redis Streams maintain message order based on delivery time and persist messages until they’re explicitly removed. Each message can contain multiple data values, giving you flexibility in what information you store and process.
LINQPad makes it incredibly easy to experiment with Redis Streams. You can quickly test different queue patterns, consumer group configurations, and message processing logic without setting up complex infrastructure. The interactive nature of LINQPad lets you see exactly what’s happening with your messages in real-time.
Download LINQPad Today
Get started with the most powerful .NET Rapid Progress Tool (RPT) available.
Get LINQPad PremiumPowerful .NET acceleration. Code at the speed of thought.
Two Main Queue Patterns
Redis Streams support two primary queue patterns that you can implement with just a few lines of code:
Fanout Queues: These broadcast the same message to all consumers listening to the stream. Perfect for scenarios where multiple systems need to react to the same event.
Round Robin Queues: When paired with Consumer Groups, these allow you to scale up processing by having multiple consumers pull messages from the same queue. Each message goes to only one consumer in the group, enabling parallel processing.
Consumer Groups and Message Acknowledgment
Consumer Groups are a game-changer for reliable message processing. Redis tracks which messages are pending and which have been delivered. Your consumers can acknowledge messages, ensuring they’re only processed once and allowing Redis to maintain delivery state.
The StackExchange.Redis client provides full support for all these features, making it straightforward to implement robust message processing in your C# applications.
Getting Redis Running
Redis has a Linux-first design, which can make it challenging to run on Windows. Thankfully, Docker solves this problem completely. You can use a simple Docker Compose file to run both Redis and Redis Commander for visual exploration of your streams.
Create a docker-compose.yml file with the following configuration:
services:
redis:
restart: always
container_name: redis2
hostname: redis
image: redis
ports:
- "6379:6379"
redis-commander:
container_name: redis-commander2
hostname: redis-commander
image: rediscommander/redis-commander:latest
restart: always
environment:
- REDIS_HOSTS=local:redis:6379
ports:
- "8081:8081"
Run docker-compose up -d to start both services. Redis will be available on localhost:6379, and Redis Commander will provide a web interface at http://localhost:8081 for exploring your streams visually.
Practical Implementation
Let’s look at a complete example that demonstrates Redis Streams in action. This code creates a producer that publishes messages every two seconds and a consumer group that processes messages every 2.5 seconds:
using ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("localhost");
IDatabase db = redis.GetDatabase();
string streamName = "updates";
string groupName = "avg";
bool keyExists = await db.KeyExistsAsync(streamName);
bool groupExists = keyExists &&
(await db.StreamGroupInfoAsync(streamName)).All(x => x.Name != groupName);
bool shouldCreateConsumerGroup = keyExists == false && groupExists == false;
if (shouldCreateConsumerGroup)
{
await db.StreamCreateConsumerGroupAsync(streamName, groupName, "0-0", true);
}
var producerTask = Task.Run(async () =>
{
// every 2 seconds
// put a message onto the stream
var random = new Random();
while (!QueryCancelToken.IsCancellationRequested)
{
var entries = new NameValueEntry[]
{
new ("obj", 123),
new ("temp", random.Next(50, 65)),
new ("time", DateTimeOffset.Now.ToUnixTimeSeconds()),
};
await db.StreamAddAsync(streamName, entries);
await Task.Delay(2000);
}
});
Dictionary<string, string> ParseResult(StreamEntry entry) =>
entry.Values.ToDictionary(x => x.Name.ToString(), x => x.Value.ToString());
var consumerGroupReadTask = Task.Run(async () =>
{
double count = default;
string id = string.Empty;
while (!QueryCancelToken.IsCancellationRequested)
{
if (!string.IsNullOrEmpty(id))
{
var values = new RedisValue[] { id };
await db.StreamAcknowledgeAsync(streamName, groupName, id);
id = string.Empty;
}
var result = await db.StreamReadGroupAsync(streamName, groupName, $"{groupName}-1", ">", 1);
result.Select(ParseResult).ToList().Dump("consumer group read " + count++);
await Task.Delay(2500);
}
});
Console.WriteLine("waiting for exit...");
while (QueryCancelToken.IsCancellationRequested == false) { }
await Task.WhenAll(producerTask, consumerGroupReadTask);
Key Components Explained
The producer task creates messages with multiple data fields - an object ID, temperature reading, and timestamp. These messages are added to the stream every two seconds.
The consumer group task demonstrates proper message acknowledgment. After processing a message, it acknowledges it to Redis, ensuring the message is marked as delivered and won’t be processed again.
The ParseResult function converts the Redis stream entries into a more readable dictionary format, making it easy to work with the message data in your application logic.
When to Use Redis Streams
Redis Streams excel in scenarios where you need reliable message processing with ordering guarantees. They’re perfect for event sourcing, log aggregation, real-time analytics, and any situation where you need to ensure messages are processed exactly once.
The combination of durability, ordering, and consumer groups makes Redis Streams a compelling alternative to more complex message queue systems like RabbitMQ or Apache Kafka for many use cases.
Getting Started
To begin experimenting with Redis Streams, start with the Docker setup mentioned earlier. The visual interface provided by Redis Commander will help you understand how messages flow through your streams and how consumer groups manage message distribution.
LINQPad’s interactive environment makes it easy to modify the producer and consumer logic, test different acknowledgment strategies, and explore the various Redis Streams commands available through the StackExchange.Redis client.
Redis Streams provide a powerful, simple solution for durable messaging that leverages infrastructure you likely already have. With proper consumer group management and message acknowledgment, you can build reliable, scalable message processing systems with minimal complexity.
Want to practice these concepts hands-on? Try LINQPad—it’s the fastest way to experiment with code, test database queries, and build your confidence for interviews. For more in-depth learning, check out our LINQPad Fundamentals course.