For the latest stable version, please use Korvet 0.19!

Remote Storage (Apache Iceberg on object store)

Korvet includes a built-in storage worker that offloads sealed Redis stream segments to an Apache Iceberg table backed by Parquet files on S3.

In Korvet tiered storage terminology, the local tier uses Redis Streams and the remote tier is an Apache Iceberg table. Korvet uses local/remote in its configuration and APIs.

Overview

The remote tier is a single Apache Iceberg table named segments. The storage worker:

  1. Reads sealed LOCAL segments from each topic-partition’s segment metadata in Redis

  2. Streams the segment contents from Redis (one page resident at a time) into an Iceberg data file

  3. Appends and commits the data file to the Iceberg table, then marks the Redis segment as offloaded

  4. Runs in-process as part of the Korvet server; there is no separate archive daemon

The remote tier is a standard Iceberg table: external query engines (Spark, Trino, Athena, DuckDB, …​) can read it directly. Iceberg manages its own metadata and manifest files under the table location. Each offloaded write commits as one Iceberg append; if the storage worker crashes before the commit, the Redis segment stays sealed-but-not-offloaded and the next scan re-streams the (still untouched) Redis segment.

Configuration

Enable remote storage by setting korvet.storage.remote.path in your application.yml:

korvet:
  storage:
    remote:
      path: s3://my-bucket/korvet
      s3:
        region: us-west-1
    worker:
      tick-interval: 1m
Setting korvet.storage.remote.path makes the cold tier available. The leader-locked storage worker is enabled by default and rolls eligible segments, offloads sealed segments, and enforces local and remote retention. Topics are archived only when they also have remote.storage.enable=true.

Storage Properties

Property Default Description

korvet.storage.remote.path

required

Cold-tier root URI. Currently supports s3://.

korvet.storage.remote.s3.region

unset

AWS region for the S3 object store.

korvet.storage.remote.s3.endpoint

unset

Optional endpoint URL for S3-compatible stores such as MinIO or LocalStack.

korvet.storage.remote.s3.path-style-access

unset

Use path-style addressing. Required for most non-AWS S3-compatible stores.

korvet.storage.remote.s3.access-key-id

unset

Static access-key id. Prefer IAM roles in production.

korvet.storage.remote.s3.secret-access-key

unset

Static secret access key. Prefer IAM roles in production.

Maintenance Properties

Property Default Description

korvet.storage.worker.enabled

true

Enables the storage worker in this JVM.

korvet.storage.worker.tick-interval

1m

Tick cadence for the storage worker loop.

korvet.storage.worker.lease-duration

2m

Redis leader-lock lease duration. Must exceed tick-interval.

Per-Topic Retention Configuration

Control when data moves from the local tier to the remote tier using topic-level configuration:

Configuration Default Description

remote.storage.enable

false

Enable tiered storage for this topic (Kafka KIP-405 standard).

local.retention.ms

-2

Time to keep in the local tier. -2 means use total retention.ms (Kafka KIP-405).

local.retention.bytes

-2

Size to keep in the local tier. -2 means use total retention.bytes (Kafka KIP-405).

retention.ms

604800000 (7 days)

Total retention across all tiers.

Remote-tier retention is implicit: retention.ms - local.retention.ms.

Example: Keep 1 day local and the rest remote (1 year total):

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true,retention.ms=31536000000,local.retention.ms=86400000

Storage Format

Table Layout

The remote tier is a single Iceberg table named segments, partitioned by stream_key (identity) and sorted by ts. Iceberg owns the on-disk layout under korvet.storage.remote.path: a metadata/ directory for table metadata and manifests, plus Parquet data files. Each offloaded segment is written as one data file named:

segment-<segmentId>-<uuid>.parquet

The trailing UUID keeps each write unique so a retried offload never collides with a file left by a prior failed attempt.

Parquet Schema

Each row is one message. The Iceberg schema is:

required STRING stream_key;
required LONG   segment_id;
required STRING message_id;
required LONG   ts;
required MAP<STRING, BINARY> fields;
Column Type Description

stream_key

STRING

Local stream key of the source partition (the table partition column).

segment_id

LONG

Sealed segment number the message came from.

message_id

STRING

Full Redis stream message id (e.g. 1708956789000-0).

ts

LONG

Message timestamp in epoch milliseconds (the table sort key).

fields

MAP<STRING, BINARY>

Opaque key/value map. The storage layer has no Kafka knowledge — Kafka headers and the value blob are just entries in the map.

Compression

Data files use Iceberg’s default Parquet compression. Row-group and target file sizes are configurable via korvet.storage.remote.iceberg.row-group-size and korvet.storage.remote.iceberg.target-file-size (both default 128MB).

AWS Authentication

The cold tier writes through Iceberg’s S3FileIO. Configure common S3 settings under korvet.storage.remote.s3; otherwise the AWS SDK default credential provider chain is used.

Credential providers

Common production choices:

Provider Use when

EC2, ECS, or EKS node roles

Leave static credentials unset and let the AWS SDK use instance metadata.

EKS IAM Roles for Service Accounts (IRSA)

Leave static credentials unset. AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are injected into the pod and picked up by the SDK.

Static access key + secret

Use korvet.storage.remote.s3.access-key-id and korvet.storage.remote.s3.secret-access-key for development or S3-compatible stores.

IRSA (EKS IAM Roles for Service Accounts)

korvet:
  storage:
    remote:
      path: s3://my-bucket/korvet
      s3:
        region: us-east-1

When IRSA is configured on the cluster, AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are injected into the pod and picked up automatically.

IAM role (EC2/ECS/EKS nodes)

korvet:
  storage:
    remote:
      path: s3://my-bucket/korvet
      s3:
        region: us-west-1

Static credentials (dev / MinIO)

korvet:
  storage:
    remote:
      path: s3://my-bucket/korvet
      s3:
        region: us-west-1
        endpoint: http://localhost:9000
        path-style-access: true
        access-key-id: minioadmin
        secret-access-key: minioadmin

Performance

The storage worker achieves high throughput when archiving to same-region S3:

Configuration Throughput Notes

Single stream

~32,000 msg/s

Baseline

4 streams (parallel)

~115,000 msg/s

Near-linear scaling

See Remote Storage Benchmarks for detailed results.

Performance tips

  • Same-region S3: deploy Korvet in the same AWS region as your bucket.

  • Multiple partitions: archival parallelism is per (topic, partition); more partitions = more archive concurrency.

  • Segment size: larger sealed segments produce larger Parquet files with better compression and fewer object-store operations.

Next Steps