For the latest stable version, please use Korvet 0.12.5!

Warm Storage (Redis Flex)

The warm tier provides an intermediate storage layer between hot (RAM) and cold (Delta Lake) tiers, using disk-backed Redis for cost-effective storage of less frequently accessed data.

Overview

The warm storage tier:

  1. Uses Redis Flex or a secondary Redis cluster with disk-backed storage

  2. Receives messages from the hot tier via the warm archival service

  3. Provides lower-latency access than cold storage while reducing RAM costs

  4. Integrates seamlessly with the Korvet tiered reader for transparent data access

When to Use Warm Storage

Consider enabling warm storage when:

  • You need to retain data longer than RAM allows cost-effectively

  • Cold storage latency (S3/HDFS) is too high for your use case

  • You want a gradual data migration path: hot → warm → cold

  • You’re using Redis Flex with flash storage for cost optimization

Architecture

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Hot Tier      │     │   Warm Tier     │     │   Cold Tier     │
│  (Redis RAM)    │────▶│  (Redis Flex)   │────▶│  (Delta Lake)   │
│                 │     │                 │     │                 │
│  Recent data    │     │  Older data     │     │  Archived data  │
│  < minutes      │     │  minutes-hours  │     │  hours-forever  │
└─────────────────┘     └─────────────────┘     └─────────────────┘
        │                       │                       │
        └───────────────────────┴───────────────────────┘
                    Tiered Reader (transparent access)

Configuration

Enable warm storage by configuring a separate Redis connection:

korvet:
  # Main Redis (hot tier - RAM)
  redis:
    uri: redis://hot-redis:6379
    pool-size: 8

  # Warm Redis (disk-backed)
  warm-redis:
    uri: redis://flex-redis:6379
    pool-size: 8
    # cluster: true  # Enable if using Redis Cluster

  server:
    storage:
      # Warm tier archiver settings
      near:
        keyspace: korvet  # Optional: use different keyspace
        archiver:
          consumer-group: korvet-warm-archiver
          batch-size: 10000
          block-duration-ms: 100
          read-workers: 1
          stream-refresh-interval-ms: 30000

Configuration Properties

Property Default Description

korvet.warm-redis.uri

none

Redis URI for warm tier (e.g., redis://flex:6379)

korvet.warm-redis.host

none

Redis host (alternative to URI)

korvet.warm-redis.port

6379

Redis port

korvet.warm-redis.cluster

false

Enable Redis Cluster mode

korvet.warm-redis.username

none

Redis username for authentication

korvet.warm-redis.password

none

Redis password for authentication

korvet.warm-redis.pool-size

same as main

Connection pool size

Archiver Configuration

Property Default Description

korvet.server.storage.near.keyspace

same as main

Keyspace prefix for warm tier streams

korvet.server.storage.near.archiver.consumer-group

korvet-near-archiver

Consumer group for warm archiver

korvet.server.storage.near.archiver.batch-size

10000

Messages per batch

korvet.server.storage.near.archiver.read-workers

1

Number of reader threads

Per-Topic Retention Configuration

Control when data moves from hot to warm tier using topic-level configuration:

# Enable tiered storage and set hot tier retention to 1 hour
kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true,local.retention.ms=3600000,near.retention.ms=86400000,retention.ms=604800000

This configures:

  • Hot tier: 1 hour (local.retention.ms=3600000)

  • Warm tier: 1 day (near.retention.ms=86400000)

  • Cold tier: ~6 days (implicit: retention.ms - local.retention.ms - near.retention.ms)

  • Total retention: 7 days (retention.ms=604800000)

Configuration Default Description

remote.storage.enable

false

Enable tiered storage for this topic (required for warm/cold archiving)

local.retention.ms

-2

Time to keep in hot tier before moving to warm. -2 = use total retention.ms

near.retention.ms

-2

Time to keep in warm tier before moving to cold. -2 = skip warm tier
Data only moves to warm tier when remote.storage.enable=true AND local.retention.ms is set to a value other than -2.

How It Works

  1. Warm Archival Service reads messages from hot tier streams using XREADGROUP

  2. Messages are written to warm tier using XADD with the same message ID

  3. Original message IDs are preserved for consistent offset tracking

  4. Tiered Reader automatically falls back to warm tier when data is not in hot tier

Integration with Cold Storage

When both warm and cold tiers are configured:

  • Cold archiver reads from warm tier (not hot tier)

  • Creates a 3-tier pipeline: hot → warm → cold

  • Reduces load on hot tier for archival operations

korvet:
  warm-redis:
    uri: redis://flex:6379

  server:
    storage:
      # Cold tier
      path: s3a://bucket/korvet/delta

      # Warm tier archiver
      near:
        archiver:
          consumer-group: korvet-warm-archiver

Next Steps