For the latest stable version, please use Korvet 0.19!

Redis Data Structures Reference

This page provides a comprehensive reference of all Redis data structures used by Korvet.

This page is for contributors and operators debugging storage layout. It is not required for normal client usage.

Korvet uses Redis Streams for message delivery, plus Redis-backed metadata structures for topics and explicit committed offsets:

Redis Key Layout for Topic "orders" with 3 Partitions (using default keyspace korvet)
Partition Streams:
  ├─ korvet:storage:local:orders:0 (Stream)
  │    ├─ 1234567890123-0: {key: "order-123", value: "...", timestamp: "1234567890"}
  │    └─ 1234567890123-1: {key: "order-124", value: "...", timestamp: "1234567891"}
  │
  ├─ korvet:storage:local:orders:1 (Stream)
  │    └─ 1234567890456-0: {key: "order-125", value: "..."}
  │
  └─ korvet:storage:local:orders:2 (Stream)
       └─ 1234567890789-0: {key: "order-126", value: "..."}

Consumer Groups (Redis Streams native):
  ├─ korvet:storage:local:orders:0 has consumer group "my-group"
  │    └─ Managed by Redis: XGROUP, XREADGROUP, XACK
  │
  └─ korvet:storage:local:orders:1 has consumer group "my-group"
       └─ Managed by Redis: XGROUP, XREADGROUP, XACK

Committed Offsets:
  ├─ korvet:broker:commit:{groupId}:{topic}:{partition} -> "44"
  └─ Used by OffsetCommit / OffsetFetch as the explicit committed-offset store

Stream Keys

Each Kafka topic partition maps to a single Redis Stream:

korvet:storage:local:{topic}:{partition}         # Stream: message log

Examples (using default keyspace korvet):

korvet:storage:local:orders:0        # Topic "orders", partition 0
korvet:storage:local:orders:1        # Topic "orders", partition 1
korvet:storage:local:payments:0      # Topic "payments", partition 0

Topic Metadata Keys

Topic metadata is stored in Redis using a single RedisJSON document:

{keyspace}:topics                      # JSON object keyed by topic name

Examples (using default keyspace korvet):

korvet:topics                          # JSON object containing "orders", "payments", "users"

Topic JSON Shape:

{
  "orders": {
    "name": "orders",
    "id": "X5M48X5pT0q1Qgz7RdfQ5w",
    "config": {
      "partitions": 3,
      "retentionTime": 604800000,
      "retentionBytes": -1,
      "compression": "none",
      "storageCompression": "lz4",
      "valueType": "AUTO",
      "offsetSequenceBits": 10,
      "averageMessageBytes": 1024
    }
  }
}

Topic JSON Fields:

id                    # Topic UUID (Kafka topic ID)
name                  # Topic name, also used as the parent JSON object key
config                # Topic configuration object
config.partitions            # Number of partitions
config.retentionTime         # Retention time in milliseconds
config.retentionBytes        # Retention size in bytes
config.compression           # Compression type (none, gzip, snappy, lz4, zstd)
config.storageCompression    # Compression type used for Redis storage
config.valueType             # Value type (JSON, BYTES)
config.offsetSequenceBits    # Number of bits for sequence in offset encoding
config.averageMessageBytes   # Average message size in bytes

Offset Encoding

Kafka offsets are stateless - encoded from Redis Stream entry IDs:

Entry ID format:  {timestamp}-{sequence}
Kafka offset:     (timestamp << N) | sequence

Where N = number of bits for sequence (typically 10-16 bits)

Example:

Entry ID:     "1234567890123-5"
Offset:       (1234567890123 << 10) | 5 = 1264197008485893

Consumer Groups

Consumer groups use Redis Streams native consumer groups for delivery state, plus a separate committed-offset store for explicit Kafka commits:

# Create consumer group
XGROUP CREATE korvet:storage:local:orders:0 my-group 0

# Read as group member
XREADGROUP GROUP my-group consumer-1 COUNT 100 STREAMS korvet:storage:local:orders:0 >

# Acknowledge delivered entries
XACK korvet:storage:local:orders:0 my-group {entryId}

# Persist committed Kafka offset separately
SET korvet:broker:commit:my-group:orders:0 44