For the latest stable version, please use Korvet 0.19!

Storage

Korvet uses Redis Streams as its primary storage layer, with optional tiered storage to per-segment Parquet files on an object store for long-term archival.

Storage Architecture

Korvet supports two storage configurations:

Redis-Only Storage (Default)

The default configuration uses Redis Streams exclusively:

  • Primary storage: All messages stored in Redis Streams

  • Persistence: Redis AOF and RDB for durability

  • Consumer groups: Built-in support for coordinated consumption

  • Performance: Sub-millisecond read/write latency

  • Retention: Configurable time and size-based retention (applied at write time)

This is the recommended configuration for most use cases.

Tiered Storage (Local → Remote)

For long-term data retention and cost optimization, Korvet can be configured with 2 storage tiers:

  • Local tier: Recent messages in Redis Streams for lowest latency

  • Remote tier (also known as the cold tier): Sealed segments archived as Parquet files on S3 (or any Hadoop-compatible filesystem)

  • Automatic archival: Built-in archival service continuously moves messages from local to remote tier

  • High throughput: Achieves 100k+ messages/second to S3 with parallel streams

How It Works

Redis-Only Storage

  1. Produce: Messages are written to Redis Streams using XADD

  2. Retention: Retention policies are applied at write time using MAXLEN and MINID arguments

  3. Consume: Consumers read messages using XREAD (standalone) or XREADGROUP (consumer groups)

  4. Persistence: Redis handles durability through AOF/RDB snapshots

Tiered Storage

  1. Produce: Messages are written to Redis Streams (local tier)

  2. Archive: Built-in archival service continuously drains sealed segments from Redis into per-segment Parquet files on the remote tier

  3. Consume: Consumers read transparently across tiers — the broker reads from local or remote depending on which segment holds the offset

  4. Cleanup: The trimmer enforces per-topic retention on both tiers

Per-Topic Tiered Storage Configuration

Tiered storage is controlled at the topic level using Kafka-compatible configuration:

  • remote.storage.enable=true - Enable tiered storage for a topic (Kafka KIP-405)

  • local.retention.ms - Time to keep in local tier before moving to remote

  • retention.ms - Total retention across all tiers

Example: 1 hour local, 6 days remote (7 days total):

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true,local.retention.ms=3600000,retention.ms=604800000

See Topic Configuration for full details.

Stream Key Format

Each Kafka topic partition maps to a single Redis Stream:

korvet:storage:local:{topic}:{partition}

Examples (using default keyspace korvet):

korvet:storage:local:orders:0        # Topic "orders", partition 0
korvet:storage:local:orders:1        # Topic "orders", partition 1
korvet:storage:local:payments:0      # Topic "payments", partition 0

Message Encoding

Kafka records are decomposed into Redis Stream fields:

  • Key: Stored in __key field (if present)

  • Headers: Stored as __header.{name} fields

  • Value: Encoding depends on value type:

    • JSON: Top-level fields are flattened into separate stream fields

    • Raw bytes: Stored as single value field

See Message Format for details.

Benefits

  • Performance: Sub-millisecond latency for local tier operations

  • Simplicity: Redis-only mode requires no additional infrastructure

  • Reliability: Redis persistence ensures data durability

  • Scalability: Handle millions of messages per second

  • Cost optimization: Optional remote tier reduces storage costs for long-term retention

  • Flexibility: Choose between simplicity (Redis-only) and cost optimization (tiered)

Next Steps