Redis Data Structures Reference

This page provides a reference of all Redis keys and data structures used by Korvet.

This page is for contributors and operators debugging storage layout. It is not required for normal client usage.

Keyspace Reference

All keys live under a configurable namespace prefix (default korvet). In the patterns below, <angle brackets> denote placeholders; curly braces {…​} are literal characters in the key, used as Redis Cluster hash tags so related keys map to the same slot. <stream> denotes a topic-partition stream key of the form <topic>:<partition>.

Key pattern Type Purpose

Registry and metadata

<ns>:topics

JSON

Topic registry: one document keyed by topic name (see Topic Metadata Keys)

<ns>:metrics:topics

JSON

Per-topic metrics document (message counts, sizes) maintained by the metrics store

<ns>:schemas

JSON

Schema registry subjects and versions

Broker coordination and security

<ns>:broker:nodes

JSON

Broker node registry: one document mapping broker id to host, port, rack, and heartbeat timestamp

<ns>:broker:credentials:<username>

JSON

SASL credentials for one user (mechanism, password hash, server key, salt, iterations)

<ns>:broker:acls:<principal>

JSON

ACL rules for one principal, mapping topic names to granted operations

<ns>:broker:producer-id

String

Atomic counter (INCR) allocating monotonically increasing producer IDs

Consumer groups and committed offsets

<ns>:broker:commit:<groupId>

Hash

Committed offsets for one consumer group: the group field holds the canonical group ID, every other field maps a stream key to its committed offset (see Consumer Groups)

<ns>:broker:commit-idx

Search index

FT.CREATE index over the group field of the committed-offset hashes, used for durable group discovery

<ns>:broker:commit:{<stream>:<groupId>}

String

Legacy per-group-per-partition committed offset. Only the migrate command reads or writes this layout; the braces are a literal hash tag

Local storage

<ns>:storage:local:<stream>

Stream

Message log for one topic partition (see Stream Keys). With tiered storage, physical streams append a segment id: <ns>:storage:local:<stream>:<segment>

Tiered storage

<ns>:storage:tiered:segments:<stream>

JSON

Segment state for one partition: array of segment descriptors (id, status, offsets, location)

<ns>:storage:tiered:group:{<stream>}:cursor

Hash

Per-group read cursors for a tiered partition: field is the group ID, value is the last-read message ID

<ns>:storage:tiered:group:{<stream>}:pel:<groupId>

Stream

Pending-entry list for one tiered consumer group; entries reference segment IDs awaiting acknowledgment

<ns>:storage:worker:leader

String

Storage-worker leader lock: holder token written with SET NX PX and renewed with PEXPIRE

Iceberg catalog (under the configured catalog key prefix, default <ns>:storage:tiered:iceberg)

<prefix>:catalog

JSON

Catalog root registry: namespaces, table name to UUID mapping, and properties

<prefix>:table:<uuid>

JSON

Per-table metadata: table name and a map of version number to metadata file location
The tiered cursor and PEL keys wrap <stream> in a literal {…​} hash tag so all coordination keys for one partition share a cluster slot. Committed-offset group hashes are deliberately not hash-tagged so they spread across slots.

Korvet uses Redis Streams for message delivery, plus Redis-backed metadata structures for topics and explicit committed offsets:

Redis Key Layout for Topic "orders" with 3 Partitions (using default keyspace korvet)
Partition Streams:
  ├─ korvet:storage:local:orders:0 (Stream)
  │    ├─ 1234567890123-0: {value: <bytes>, key: <bytes>, headers: <bytes>, timestamp: <ascii>}
  │    └─ 1234567890123-1: {value: <bytes>, timestamp: <ascii>}
  │
  ├─ korvet:storage:local:orders:1 (Stream)
  │    └─ 1234567890456-0: {value: <bytes>, key: <bytes>, timestamp: <ascii>}
  │
  └─ korvet:storage:local:orders:2 (Stream)
       └─ 1234567890789-0: {value: <bytes>, timestamp: <ascii>}

Consumer Groups (Redis Streams native):
  ├─ korvet:storage:local:orders:0 has consumer group "my-group"
  │    └─ Managed by Redis: XGROUP, XREADGROUP, XACK
  │
  └─ korvet:storage:local:orders:1 has consumer group "my-group"
       └─ Managed by Redis: XGROUP, XREADGROUP, XACK

Committed Offsets:
  └─ korvet:broker:commit:my-group (Hash)
       ├─ group    -> "my-group"
       ├─ orders:0 -> "44"
       └─ orders:1 -> "17"
Each Kafka record is stored as a single Redis Stream entry whose body breaks the record out into separate, directly-readable fields: value, key, headers (all headers in one self-delimiting blob, preserving order and duplicate keys), and timestamp (ASCII decimal). A field is omitted when its component is absent (a missing value is a tombstone), so a keyless, headerless record carries just {value, timestamp}. The value field holds the record value verbatim and is what a non-Kafka client reads directly via XRANGE — unless the topic’s storage.compression.type compresses that field.

Stream Keys

Each Kafka topic partition maps to a single Redis Stream:

korvet:storage:local:<topic>:<partition>         # Stream: message log

Examples (using default keyspace korvet):

korvet:storage:local:orders:0        # Topic "orders", partition 0
korvet:storage:local:orders:1        # Topic "orders", partition 1
korvet:storage:local:payments:0      # Topic "payments", partition 0
This is the logical partition stream key. With tiered storage, the physical Redis streams append a trailing segment id (korvet:storage:local:orders:0:0, …​:0:1, …​). See Stream Structure.

Topic Metadata Keys

Topic metadata is stored in Redis using a single RedisJSON document:

<keyspace>:topics                      # JSON object keyed by topic name

Examples (using default keyspace korvet):

korvet:topics                          # JSON object containing "orders", "payments", "users"

Topic JSON Shape:

{
  "orders": {
    "name": "orders",
    "id": "X5M48X5pT0q1Qgz7RdfQ5w",
    "config": {
      "partitions": 3,
      "retentionTime": 604800000,
      "retentionBytes": -1,
      "compression": "none",
      "storageCompression": "lz4",
      "valueType": "AUTO",
      "offsetSequenceBits": 14
    }
  }
}

Topic JSON Fields:

id                    # Topic UUID (Kafka topic ID)
name                  # Topic name, also used as the parent JSON object key
config                # Topic configuration object
config.partitions            # Number of partitions
config.retentionTime         # Retention time in milliseconds
config.retentionBytes        # Retention size in bytes
config.compression           # Compression type (none, gzip, snappy, lz4, zstd)
config.storageCompression    # Compression type used for Redis storage
config.valueType             # Value type (AUTO, JSON, RAW)
config.offsetSequenceBits    # Number of bits for sequence in offset encoding

Offset Encoding

Kafka offsets are stateless - encoded from Redis Stream entry IDs:

Entry ID format:  {timestamp}-{sequence}
Kafka offset:     (timestamp << N) | sequence

Where N = number of bits for sequence (range 1-16 bits, default 14)

Example (with the default offsetSequenceBits=14):

Entry ID:     "1234567890123-5"
Offset:       (1234567890123 << 14) | 5 = 20227160311775237

Consumer Groups

Consumer groups use Redis Streams native consumer groups for delivery state, plus a separate committed-offset store for explicit Kafka commits:

# Create consumer group
XGROUP CREATE korvet:storage:local:orders:0 my-group 0

# Read as group member
XREADGROUP GROUP my-group consumer-1 COUNT 100 STREAMS korvet:storage:local:orders:0 >

# Acknowledge delivered entries
XACK korvet:storage:local:orders:0 my-group {entryId}

# Persist committed Kafka offsets separately (one hash per group)
HSET korvet:broker:commit:my-group group my-group orders:0 44

OffsetCommit and OffsetFetch operate on the per-group hash: the group field holds the canonical group ID (indexed by korvet:broker:commit-idx for durable group discovery), and every other field maps a stream key (<topic>:<partition>) to its committed offset.

Earlier releases stored one string per group and partition (korvet:broker:commit:{orders:0:my-group}, with a literal hash tag). Only the migrate command still reads and writes that legacy layout.