This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.19!

Redis Data Structures Reference

This page provides a reference of all Redis keys and data structures used by Korvet.

This page is for contributors and operators debugging storage layout. It is not required for normal client usage.

Keyspace Reference

All keys live under a configurable namespace prefix (default korvet). In the patterns below, <angle brackets> denote placeholders; curly braces {…} are literal characters in the key, used as Redis Cluster hash tags so related keys map to the same slot. <stream> denotes a topic-partition stream key of the form <topic>:<partition>.

Key pattern Type Purpose

Key pattern	Type	Purpose
Registry and metadata
`<ns>:topics`	JSON	Topic registry: one document keyed by topic name (see Topic Metadata Keys)
`<ns>:metrics:topics`	JSON	Per-topic metrics document (message counts, sizes) maintained by the metrics store
`<ns>:schemas`	JSON	Schema registry subjects and versions
Broker coordination and security
`<ns>:broker:nodes`	JSON	Broker node registry: one document mapping broker id to host, port, rack, and heartbeat timestamp
`<ns>:broker:credentials:<username>`	JSON	SASL credentials for one user (mechanism, password hash, server key, salt, iterations)
`<ns>:broker:acls:<principal>`	JSON	ACL rules for one principal, mapping topic names to granted operations
`<ns>:broker:producer-id`	String	Atomic counter (`INCR`) allocating monotonically increasing producer IDs
Consumer groups and committed offsets
`<ns>:broker:commit:<groupId>`	Hash	Committed offsets for one consumer group: each field maps a stream key to its bare numeric offset so older brokers remain compatible (see Consumer Groups)
`<ns>:broker:commit-timestamps:{<slot-tag>}:<groupId>`	Hash	Commit timestamps for one consumer group: each field maps a stream key to `<offset>\|<commit-epoch-millis>`. The generated tag places this hash in the canonical offset hash’s Redis Cluster slot for atomic updates
`<ns>:broker:commit-groups`	Set	Canonical group IDs that have committed offsets, used for durable group discovery
`<ns>:broker:commit:{<stream>:<groupId>}`	String	Legacy per-group-per-partition committed offset. Only the migrate command reads or writes this layout; the braces are a literal hash tag
Local storage
`<ns>:storage:local:<stream>`	Stream	Message log for one topic partition (see Stream Keys). With tiered storage, physical streams append a segment id: `<ns>:storage:local:<stream>:<segment>`
Tiered storage
`<ns>:storage:tiered:segments:<stream>`	JSON	Segment state for one partition: array of segment descriptors (id, status, offsets, location)
`<ns>:storage:tiered:group:{<stream>}:cursor`	Hash	Per-group read cursors for a tiered partition: field is the group ID, value is the last-read message ID
`<ns>:storage:tiered:group:{<stream>}:pel:<groupId>`	Stream	Pending-entry list for one tiered consumer group; entries reference segment IDs awaiting acknowledgment
`<ns>:storage:worker:leader`	String	Storage-worker leader lock: holder token written with `SET NX PX` and renewed with `PEXPIRE`
Iceberg catalog (under the configured catalog key prefix, default `<ns>:storage:tiered:iceberg`)
`<prefix>:catalog`	JSON	Catalog root registry: namespaces, table name to UUID mapping, and properties
`<prefix>:table:<uuid>`	JSON	Per-table metadata: table name and a map of version number to metadata file location

Registry and metadata

<ns>:topics

JSON

Topic registry: one document keyed by topic name (see Topic Metadata Keys)

<ns>:metrics:topics

JSON

Per-topic metrics document (message counts, sizes) maintained by the metrics store

<ns>:schemas

JSON

Schema registry subjects and versions

Broker coordination and security

<ns>:broker:nodes

JSON

Broker node registry: one document mapping broker id to host, port, rack, and heartbeat timestamp

<ns>:broker:credentials:<username>

JSON

SASL credentials for one user (mechanism, password hash, server key, salt, iterations)

<ns>:broker:acls:<principal>

JSON

ACL rules for one principal, mapping topic names to granted operations

<ns>:broker:producer-id

String

Atomic counter (INCR) allocating monotonically increasing producer IDs

Consumer groups and committed offsets

<ns>:broker:commit:<groupId>

Hash

Committed offsets for one consumer group: each field maps a stream key to its bare numeric offset so older brokers remain compatible (see Consumer Groups)

<ns>:broker:commit-timestamps:{<slot-tag>}:<groupId>

Hash

Commit timestamps for one consumer group: each field maps a stream key to <offset>|<commit-epoch-millis>. The generated tag places this hash in the canonical offset hash’s Redis Cluster slot for atomic updates

<ns>:broker:commit-groups

Set

Canonical group IDs that have committed offsets, used for durable group discovery

<ns>:broker:commit:{<stream>:<groupId>}

String

Legacy per-group-per-partition committed offset. Only the migrate command reads or writes this layout; the braces are a literal hash tag

Local storage

<ns>:storage:local:<stream>

Stream

Message log for one topic partition (see Stream Keys). With tiered storage, physical streams append a segment id: <ns>:storage:local:<stream>:<segment>

Tiered storage

<ns>:storage:tiered:segments:<stream>

JSON

Segment state for one partition: array of segment descriptors (id, status, offsets, location)

<ns>:storage:tiered:group:{<stream>}:cursor

Hash

Per-group read cursors for a tiered partition: field is the group ID, value is the last-read message ID

<ns>:storage:tiered:group:{<stream>}:pel:<groupId>

Stream

Pending-entry list for one tiered consumer group; entries reference segment IDs awaiting acknowledgment

<ns>:storage:worker:leader

String

Storage-worker leader lock: holder token written with SET NX PX and renewed with PEXPIRE

Iceberg catalog (under the configured catalog key prefix, default <ns>:storage:tiered:iceberg)

<prefix>:catalog

JSON

Catalog root registry: namespaces, table name to UUID mapping, and properties

<prefix>:table:<uuid>

JSON

Per-table metadata: table name and a map of version number to metadata file location

The tiered cursor and PEL keys wrap <stream> in a literal {…} hash tag so all coordination keys for one partition share a cluster slot. Committed-offset group hashes are deliberately not hash-tagged so they spread across slots.

Korvet uses Redis Streams for message delivery, plus Redis-backed metadata structures for topics and explicit committed offsets:

Redis Key Layout for Topic "orders" with 3 Partitions (using default keyspace korvet)

Partition Streams:
  ├─ korvet:storage:local:orders:0 (Stream)
  │    ├─ 1234567890123-0: {value: <bytes>, key: <bytes>, headers: <bytes>, timestamp: <ascii>}
  │    └─ 1234567890123-1: {value: <bytes>, timestamp: <ascii>}
  │
  ├─ korvet:storage:local:orders:1 (Stream)
  │    └─ 1234567890456-0: {value: <bytes>, key: <bytes>, timestamp: <ascii>}
  │
  └─ korvet:storage:local:orders:2 (Stream)
       └─ 1234567890789-0: {value: <bytes>, timestamp: <ascii>}

Consumer Groups (Redis Streams native):
  ├─ korvet:storage:local:orders:0 has consumer group "my-group"
  │    └─ Managed by Redis: XGROUP, XREADGROUP, XACK
  │
  └─ korvet:storage:local:orders:1 has consumer group "my-group"
       └─ Managed by Redis: XGROUP, XREADGROUP, XACK

Committed Offsets:
  ├─ korvet:broker:commit:my-group (Hash)
  │    ├─ orders:0 -> "44"
  │    └─ orders:1 -> "17"
  ├─ korvet:broker:commit-timestamps:{<slot-tag>}:my-group (Hash)
  │    ├─ orders:0 -> "44|1752775672000"
  │    └─ orders:1 -> "17|1752775672500"
  └─ korvet:broker:commit-groups (Set)
       └─ "my-group"

Each Kafka record is stored as a single Redis Stream entry whose body breaks the record out into separate, directly-readable fields: value, key, headers (all headers in one self-delimiting blob, preserving order and duplicate keys), and timestamp (ASCII decimal). A field is omitted when its component is absent (a missing value is a tombstone), so a keyless, headerless record carries just {value, timestamp}. The value field holds the record value verbatim and is what a non-Kafka client reads directly via XRANGE — unless the topic’s storage.compression.type compresses that field.

Stream Keys

Each Kafka topic partition maps to a single Redis Stream:

korvet:storage:local:<topic>:<partition>         # Stream: message log

Examples (using default keyspace korvet):

korvet:storage:local:orders:0        # Topic "orders", partition 0
korvet:storage:local:orders:1        # Topic "orders", partition 1
korvet:storage:local:payments:0      # Topic "payments", partition 0

This is the logical partition stream key. With tiered storage, the physical Redis streams append a trailing segment id (korvet:storage:local:orders:0:0, …:0:1, …). See Stream Structure.

Topic Metadata Keys

Topic metadata is stored in Redis using a single RedisJSON document:

<keyspace>:topics                      # JSON object keyed by topic name

Examples (using default keyspace korvet):

korvet:topics                          # JSON object containing "orders", "payments", "users"

Topic JSON Shape:

{
  "orders": {
    "name": "orders",
    "id": "X5M48X5pT0q1Qgz7RdfQ5w",
    "config": {
      "partitions": 3,
      "retentionTime": 604800000,
      "retentionBytes": -1,
      "compression": "none",
      "storageCompression": "lz4",
      "valueType": "AUTO",
      "offsetSequenceBits": 14
    }
  }
}

Topic JSON Fields:

id                    # Topic UUID (Kafka topic ID)
name                  # Topic name, also used as the parent JSON object key
config                # Topic configuration object
config.partitions            # Number of partitions
config.retentionTime         # Retention time in milliseconds
config.retentionBytes        # Retention size in bytes
config.compression           # Compression type (none, gzip, snappy, lz4, zstd)
config.storageCompression    # Compression type used for Redis storage
config.valueType             # Value type (AUTO, JSON, RAW)
config.offsetSequenceBits    # Number of bits for sequence in offset encoding

Offset Encoding

Kafka offsets are stateless - encoded from Redis Stream entry IDs:

Entry ID format:  {timestamp}-{sequence}
Kafka offset:     (timestamp << N) | sequence

Where N = number of bits for sequence (range 1-16 bits, default 14)

Example (with the default offsetSequenceBits=14):

Entry ID:     "1234567890123-5"
Offset:       (1234567890123 << 14) | 5 = 20227160311775237

Consumer Groups

Consumer groups use Redis Streams native consumer groups for delivery state, plus a separate committed-offset store for explicit Kafka commits:

# Create consumer group
XGROUP CREATE korvet:storage:local:orders:0 my-group 0

# Read as group member
XREADGROUP GROUP my-group consumer-1 COUNT 100 STREAMS korvet:storage:local:orders:0 >

# Acknowledge delivered entries
XACK korvet:storage:local:orders:0 my-group {entryId}

# Persist committed Kafka offsets separately (one hash per group)
HSET korvet:broker:commit:my-group orders:0 44
HSET korvet:broker:commit-timestamps:{<slot-tag>}:my-group orders:0 "44|1752775672000"
SADD korvet:broker:commit-groups my-group

OffsetCommit and OffsetFetch operate on the canonical per-group offset hash. The Admin API joins those offsets with the parallel, co-slotted timestamp hash. The korvet:broker:commit-groups set provides durable group discovery without scanning.

The commit timestamp is the latest commit observed by a timestamp-aware broker. During a rolling upgrade, commits handled exclusively by an older broker cannot advance or invalidate that observation. Untimestamped writes through the current store, offset deletion, and migration replacement clear it.

Earlier releases stored one string per group and partition (korvet:broker:commit:{orders:0:my-group}, with a literal hash tag). Only the migrate command still reads and writes that legacy layout.