Redis Data Structures Reference

This page provides a comprehensive reference of all Redis data structures used by Korvet.

Korvet uses Redis Streams exclusively to implement Kafka semantics:

Redis Key Layout for Topic "orders" with 3 Partitions (using default keyspace korvet)

Partition Streams:
  ├─ korvet:stream:orders:0 (Stream)
  │    ├─ 1234567890123-0: {key: "order-123", value: "...", timestamp: "1234567890"}
  │    └─ 1234567890123-1: {key: "order-124", value: "...", timestamp: "1234567891"}
  │
  ├─ korvet:stream:orders:1 (Stream)
  │    └─ 1234567890456-0: {key: "order-125", value: "..."}
  │
  └─ korvet:stream:orders:2 (Stream)
       └─ 1234567890789-0: {key: "order-126", value: "..."}

Consumer Groups (Redis Streams native):
  ├─ korvet:stream:orders:0 has consumer group "my-group"
  │    └─ Managed by Redis: XGROUP, XREADGROUP, XACK
  │
  └─ korvet:stream:orders:1 has consumer group "my-group"
       └─ Managed by Redis: XGROUP, XREADGROUP, XACK

Stream Keys

Each Kafka topic partition maps to a single Redis Stream:

{keyspace}:stream:{topic}:{partition}         # Stream: message log

Examples (using default keyspace korvet):

korvet:stream:orders:0        # Topic "orders", partition 0
korvet:stream:orders:1        # Topic "orders", partition 1
korvet:stream:payments:0      # Topic "payments", partition 0

Topic Metadata Keys

Topic metadata is stored in Redis using the following keys:

{keyspace}:topic:{topic-name}          # Hash: topic configuration and metadata
{keyspace}:topics                      # Set: all topic names
{keyspace}:topic-ids                   # Hash: topic ID to name mapping

Examples (using default keyspace korvet):

korvet:topic:orders                    # Hash with fields: id, name, partitions, retentionTime, etc.
korvet:topics                          # Set containing: "orders", "payments", "users"
korvet:topic-ids                       # Hash mapping topic IDs to names

Topic Hash Fields:

id                    # Topic UUID (Kafka topic ID)
name                  # Topic name
partitions            # Number of partitions
retentionTime         # Retention time in milliseconds
retentionBytes        # Retention size in bytes
compression           # Compression type (none, gzip, snappy, lz4, zstd)
valueType             # Value type (JSON, BYTES)
offsetSequenceBits    # Number of bits for sequence in offset encoding
messageBytes          # Average message size in bytes

Offset Encoding

Kafka offsets are stateless - encoded from Redis Stream entry IDs:

Entry ID format:  {timestamp}-{sequence}
Kafka offset:     (timestamp << N) | sequence

Where N = number of bits for sequence (typically 10-16 bits)

Example:

Entry ID:     "1234567890123-5"
Offset:       (1234567890123 << 10) | 5 = 1264197008485893

Consumer Groups

Consumer groups use Redis Streams native consumer groups:

# Create consumer group
XGROUP CREATE korvet:orders:0 my-group 0

# Read as group member
XREADGROUP GROUP my-group consumer-1 COUNT 100 STREAMS korvet:orders:0 >

# Acknowledge (commit offset)
XACK korvet:orders:0 my-group {entryId}