For the latest stable version, please use Korvet 0.19!

Redis Data Structures Reference

This page provides a comprehensive reference of all Redis data structures used by Korvet.

This page is for contributors and operators debugging storage layout. It is not required for normal client usage.

Korvet uses Redis Streams for message delivery, plus Redis-backed metadata structures for topics and explicit committed offsets:

Redis Key Layout for Topic "orders" with 3 Partitions (using default keyspace korvet)

Partition Streams:
  ├─ korvet:storage:local:orders:0 (Stream)
  │    ├─ 1234567890123-0: {value: <bytes>, key: <bytes>, headers: <bytes>, timestamp: <ascii>}
  │    └─ 1234567890123-1: {value: <bytes>, timestamp: <ascii>}
  │
  ├─ korvet:storage:local:orders:1 (Stream)
  │    └─ 1234567890456-0: {value: <bytes>, key: <bytes>, timestamp: <ascii>}
  │
  └─ korvet:storage:local:orders:2 (Stream)
       └─ 1234567890789-0: {value: <bytes>, timestamp: <ascii>}

Consumer Groups (Redis Streams native):
  ├─ korvet:storage:local:orders:0 has consumer group "my-group"
  │    └─ Managed by Redis: XGROUP, XREADGROUP, XACK
  │
  └─ korvet:storage:local:orders:1 has consumer group "my-group"
       └─ Managed by Redis: XGROUP, XREADGROUP, XACK

Committed Offsets:
  ├─ korvet:broker:commit:{<topic>:<partition>:<groupId>} -> "44"
  │    └─ Curly braces are a Redis hash tag, co-locating the key with its stream in cluster mode
  └─ Used by OffsetCommit / OffsetFetch as the explicit committed-offset store

Each Kafka record is stored as a single Redis Stream entry whose body breaks the record out into separate, directly-readable fields: value, key, headers (all headers in one self-delimiting blob, preserving order and duplicate keys), and timestamp (ASCII decimal). A field is omitted when its component is absent (a missing value is a tombstone), so a keyless, headerless record carries just {value, timestamp}. The value field holds the record value verbatim and is what a non-Kafka client reads directly via XRANGE — unless the topic’s storage.compression.type compresses that field.

Stream Keys

Each Kafka topic partition maps to a single Redis Stream:

korvet:storage:local:{topic}:{partition}         # Stream: message log

Examples (using default keyspace korvet):

korvet:storage:local:orders:0        # Topic "orders", partition 0
korvet:storage:local:orders:1        # Topic "orders", partition 1
korvet:storage:local:payments:0      # Topic "payments", partition 0

This is the logical partition stream key. With tiered storage, the physical Redis streams append a trailing segment id (korvet:storage:local:orders:0:0, …:0:1, …). See Stream Structure.

Topic Metadata Keys

Topic metadata is stored in Redis using a single RedisJSON document:

{keyspace}:topics                      # JSON object keyed by topic name

Examples (using default keyspace korvet):

korvet:topics                          # JSON object containing "orders", "payments", "users"

Topic JSON Shape:

{
  "orders": {
    "name": "orders",
    "id": "X5M48X5pT0q1Qgz7RdfQ5w",
    "config": {
      "partitions": 3,
      "retentionTime": 604800000,
      "retentionBytes": -1,
      "compression": "none",
      "storageCompression": "lz4",
      "valueType": "AUTO",
      "offsetSequenceBits": 14
    }
  }
}

Topic JSON Fields:

id                    # Topic UUID (Kafka topic ID)
name                  # Topic name, also used as the parent JSON object key
config                # Topic configuration object
config.partitions            # Number of partitions
config.retentionTime         # Retention time in milliseconds
config.retentionBytes        # Retention size in bytes
config.compression           # Compression type (none, gzip, snappy, lz4, zstd)
config.storageCompression    # Compression type used for Redis storage
config.valueType             # Value type (AUTO, JSON, RAW)
config.offsetSequenceBits    # Number of bits for sequence in offset encoding

Offset Encoding

Kafka offsets are stateless - encoded from Redis Stream entry IDs:

Entry ID format:  {timestamp}-{sequence}
Kafka offset:     (timestamp << N) | sequence

Where N = number of bits for sequence (range 1-16 bits, default 14)

Example (with the default offsetSequenceBits=14):

Entry ID:     "1234567890123-5"
Offset:       (1234567890123 << 14) | 5 = 20227160311775237

Consumer Groups

Consumer groups use Redis Streams native consumer groups for delivery state, plus a separate committed-offset store for explicit Kafka commits:

# Create consumer group
XGROUP CREATE korvet:storage:local:orders:0 my-group 0

# Read as group member
XREADGROUP GROUP my-group consumer-1 COUNT 100 STREAMS korvet:storage:local:orders:0 >

# Acknowledge delivered entries
XACK korvet:storage:local:orders:0 my-group {entryId}

# Persist committed Kafka offset separately
SET korvet:broker:commit:{orders:0:my-group} 44