This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.12.5!

Redis Data Structures Reference

This page provides a comprehensive reference of all Redis data structures used by Korvet.

This page is for contributors and operators debugging storage layout. It is not required for normal client usage.

Korvet uses Redis Streams for message delivery, plus Redis-backed metadata structures for topics and explicit committed offsets:

Redis Key Layout for Topic "orders" with 3 Partitions (using default keyspace korvet)

Partition Streams:
  ├─ korvet:stream:orders:0 (Stream)
  │    ├─ 1234567890123-0: {key: "order-123", value: "...", timestamp: "1234567890"}
  │    └─ 1234567890123-1: {key: "order-124", value: "...", timestamp: "1234567891"}
  │
  ├─ korvet:stream:orders:1 (Stream)
  │    └─ 1234567890456-0: {key: "order-125", value: "..."}
  │
  └─ korvet:stream:orders:2 (Stream)
       └─ 1234567890789-0: {key: "order-126", value: "..."}

Consumer Groups (Redis Streams native):
  ├─ korvet:stream:orders:0 has consumer group "my-group"
  │    └─ Managed by Redis: XGROUP, XREADGROUP, XACK
  │
  └─ korvet:stream:orders:1 has consumer group "my-group"
       └─ Managed by Redis: XGROUP, XREADGROUP, XACK

Committed Offsets:
  ├─ korvet:commit:{streamKey}:{groupId} -> "44"
  └─ Used by OffsetCommit / OffsetFetch as the explicit committed-offset store

Stream Keys

Each Kafka topic partition maps to a single Redis Stream:

{keyspace}:stream:{topic}:{partition}         # Stream: message log

Examples (using default keyspace korvet):

korvet:stream:orders:0        # Topic "orders", partition 0
korvet:stream:orders:1        # Topic "orders", partition 1
korvet:stream:payments:0      # Topic "payments", partition 0

Topic Metadata Keys

Topic metadata is stored in Redis using the following keys:

{keyspace}:topic:{topic-name}          # Hash: topic configuration and metadata
{keyspace}:topics                      # Set: all topic names
{keyspace}:topic-ids                   # Hash: topic ID to name mapping

Examples (using default keyspace korvet):

korvet:topic:orders                    # Hash with fields: id, name, partitions, retentionTime, etc.
korvet:topics                          # Set containing: "orders", "payments", "users"
korvet:topic-ids                       # Hash mapping topic IDs to names

Topic Hash Fields:

id                    # Topic UUID (Kafka topic ID)
name                  # Topic name
partitions            # Number of partitions
retentionTime         # Retention time in milliseconds
retentionBytes        # Retention size in bytes
compression           # Compression type (none, gzip, snappy, lz4, zstd)
valueType             # Value type (JSON, BYTES)
offsetSequenceBits    # Number of bits for sequence in offset encoding
messageBytes          # Average message size in bytes

Offset Encoding

Kafka offsets are stateless - encoded from Redis Stream entry IDs:

Entry ID format:  {timestamp}-{sequence}
Kafka offset:     (timestamp << N) | sequence

Where N = number of bits for sequence (typically 10-16 bits)

Example:

Entry ID:     "1234567890123-5"
Offset:       (1234567890123 << 10) | 5 = 1264197008485893

Consumer Groups

Consumer groups use Redis Streams native consumer groups for delivery state, plus a separate committed-offset store for explicit Kafka commits:

# Create consumer group
XGROUP CREATE korvet:orders:0 my-group 0

# Read as group member
XREADGROUP GROUP my-group consumer-1 COUNT 100 STREAMS korvet:orders:0 >

# Acknowledge delivered entries
XACK korvet:orders:0 my-group {entryId}

# Persist committed Kafka offset separately
SET korvet:commit:korvet:stream:orders:0:my-group 44