For the latest stable version, please use Korvet 0.19!

Storage

Korvet uses Redis Streams as its primary storage layer, with optional tiered storage to an Apache Iceberg table on an object store for long-term archival.

Storage Architecture

Korvet supports two storage configurations:

Redis-Only Storage (Default)

The default configuration uses Redis Streams exclusively:

  • Primary storage: All messages stored in Redis Streams

  • Persistence: Redis AOF and RDB for durability

  • Consumer groups: Built-in support for coordinated consumption

  • Performance: Sub-millisecond read/write latency

  • Retention: Configurable time and size-based retention (applied at write time)

This is the recommended configuration for most use cases.

Tiered Storage (Local → Remote)

For long-term data retention and cost optimization, Korvet can be configured with 2 storage tiers:

  • Local tier: Recent messages in Redis Streams for lowest latency

  • Remote tier (also known as the cold tier): Sealed segments archived to an Apache Iceberg table on S3 (or any Hadoop-compatible filesystem)

  • Automatic archival: The built-in storage worker continuously moves messages from local to remote tier

  • High throughput: Achieves 100k+ messages/second to S3 with parallel streams

How It Works

Redis-Only Storage

  1. Produce: Messages are written to Redis Streams using XADD

  2. Retention: Retention policies are applied at write time using MAXLEN and MINID arguments

  3. Consume: Consumers read messages using XREAD (standalone) or XREADGROUP (consumer groups)

  4. Persistence: Redis handles durability through AOF/RDB snapshots

Tiered Storage

  1. Produce: Messages are written to Redis Streams (local tier)

  2. Archive: The built-in storage worker continuously drains sealed segments from Redis into an Apache Iceberg table on the remote tier

  3. Consume: Consumers read transparently across tiers — the broker reads from local or remote depending on which segment holds the offset

  4. Cleanup: The storage worker enforces per-topic retention on both tiers

Per-Topic Tiered Storage Configuration

Tiered storage is controlled at the topic level using Kafka-compatible configuration:

  • remote.storage.enable=true - Enable tiered storage for a topic (Kafka KIP-405)

  • local.retention.ms - Time to keep in local tier before moving to remote

  • retention.ms - Total retention across all tiers

Example: 1 hour local, 6 days remote (7 days total):

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true,local.retention.ms=3600000,retention.ms=604800000

See Topic Configuration for full details.

Stream Key Format

Each Kafka topic partition maps to one or more Redis Streams (segments), keyed by a trailing segment number that increases from 0:

korvet:storage:local:{topic}:{partition}:{segmentId}

Examples (using default keyspace korvet):

korvet:storage:local:orders:0:0      # Topic "orders", partition 0, segment 0
korvet:storage:local:orders:1:0      # Topic "orders", partition 1, segment 0
korvet:storage:local:payments:0:2    # Topic "payments", partition 0, segment 2

See Stream Structure for how segments are rolled.

Message Encoding

Kafka records are decomposed into Redis Stream fields:

  • Key: Stored in __key field (if present)

  • Headers: Stored as __header.{name} fields

  • Value: Encoding depends on value type:

    • JSON: Top-level fields are flattened into separate stream fields

    • Raw bytes: Stored as single __value field

See Message Format for details.

Benefits

  • Performance: Sub-millisecond latency for local tier operations

  • Simplicity: Redis-only mode requires no additional infrastructure

  • Reliability: Redis persistence ensures data durability

  • Scalability: Handle millions of messages per second

  • Cost optimization: Optional remote tier reduces storage costs for long-term retention

  • Flexibility: Choose between simplicity (Redis-only) and cost optimization (tiered)

Next Steps