For the latest stable version, please use Korvet 0.19!

Remote Storage (Parquet on object store)

Korvet includes a built-in storage offload worker that archives sealed Redis stream segments to per-segment Parquet files on S3.

In Korvet tiered storage terminology, the local tier uses Redis Streams and the remote tier uses Parquet files. Korvet uses local/remote in its configuration and APIs.

Overview

The remote tier is per-segment Parquet files. The storage offload worker:

  1. Reads sealed LOCAL segments from each topic-partition’s manifest

  2. Streams the segment contents from Redis (one page resident at a time) into a single Parquet file

  3. Flips the manifest entry from LOCAL to REMOTE — the manifest is the commit point

  4. Runs in-process as part of the Korvet server; there is no separate archive daemon

There is no mirror stream, no transaction log, and no multi-file atomic commit. One sealed segment maps to one Parquet file; the manifest tier-flip is the single commit unit. If the storage offload worker crashes before the flip, the manifest entry stays LOCAL and the next scan re-streams the (still untouched) Redis segment.

Cold storage is broker-private: Korvet controls the path layout, schema, and footer caching, and there is no expectation that external query engines read these files directly.

Configuration

Enable remote storage by setting korvet.storage.remote-path in your application.yml:

korvet:
  storage:
    remote-path: s3://my-bucket/korvet
    s3:
      region: us-west-1
    offload:
      tick-interval: 1m
Setting korvet.storage.remote-path makes the cold tier available. The leader-locked storage offload worker is enabled by default and rolls eligible segments, offloads sealed segments, and enforces local and remote retention. Topics are archived only when they also have remote.storage.enable=true.

Storage Properties

Property Default Description

korvet.storage.remote-path

required

Cold-tier root URI. Currently supports s3://.

korvet.storage.s3.region

unset

AWS region for the S3 object store.

korvet.storage.s3.endpoint

unset

Optional endpoint URL for S3-compatible stores such as MinIO or LocalStack.

korvet.storage.s3.path-style-access

unset

Use path-style addressing. Required for most non-AWS S3-compatible stores.

korvet.storage.s3.access-key-id

unset

Static access-key id. Prefer IAM roles in production.

korvet.storage.s3.secret-access-key

unset

Static secret access key. Prefer IAM roles in production.

Maintenance Properties

Property Default Description

korvet.storage.offload.enabled

true

Enables the storage offload worker in this JVM.

korvet.storage.offload.tick-interval

1m

Tick cadence for the offload loop.

korvet.storage.offload.lease-duration

2m

Redis leader-lock lease duration. Must exceed tick-interval.

Per-Topic Retention Configuration

Control when data moves from the local tier to the remote tier using topic-level configuration:

Configuration Default Description

remote.storage.enable

false

Enable tiered storage for this topic (Kafka KIP-405 standard).

local.retention.ms

-2

Time to keep in the local tier. -2 means use total retention.ms (Kafka KIP-405).

local.retention.bytes

-2

Size to keep in the local tier. -2 means use total retention.bytes (Kafka KIP-405).

retention.ms

604800000 (7 days)

Total retention across all tiers.

Remote-tier retention is implicit: retention.ms - local.retention.ms.

Example: Keep 1 day local and the rest remote (1 year total):

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true,retention.ms=31536000000,local.retention.ms=86400000

Storage Format

File Layout

One Parquet file per sealed segment. The default path mapper produces:

<root>/<streamKey>/<segmentId>.parquet

There is no transaction-log directory, no _delta_log/, no manifest file inside the object-store prefix — the manifest lives in Redis.

Parquet Schema

Each file contains the messages of a single sealed segment with this schema:

required STRING id;
required MAP<STRING, BINARY> fields;
Column Type Description

id

STRING

Full Redis stream message id (e.g. 1708956789000-0).

fields

MAP<STRING, BINARY>

Opaque key/value map. The storage layer has no Kafka knowledge — Kafka headers and the value blob are just entries in the map.

Compression

Files are written with SNAPPY compression. Most segments fit in a single row group; parquet-mr splits row groups automatically for unusually large segments.

AWS Authentication

The cold tier writes through Iceberg’s S3FileIO. Configure common S3 settings under korvet.storage.s3; otherwise the AWS SDK default credential provider chain is used.

Credential providers

Common production choices:

Provider Use when

EC2, ECS, or EKS node roles

Leave static credentials unset and let the AWS SDK use instance metadata.

EKS IAM Roles for Service Accounts (IRSA)

Leave static credentials unset. AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are injected into the pod and picked up by the SDK.

Static access key + secret

Use korvet.storage.s3.access-key-id and korvet.storage.s3.secret-access-key for development or S3-compatible stores.

IRSA (EKS IAM Roles for Service Accounts)

korvet:
  storage:
    remote-path: s3://my-bucket/korvet
    s3:
      region: us-east-1

When IRSA is configured on the cluster, AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are injected into the pod and picked up automatically.

IAM role (EC2/ECS/EKS nodes)

korvet:
  storage:
    remote-path: s3://my-bucket/korvet
    s3:
      region: us-west-1

Static credentials (dev / MinIO)

korvet:
  storage:
    remote-path: s3://my-bucket/korvet
    s3:
      region: us-west-1
      endpoint: http://localhost:9000
      path-style-access: true
      access-key-id: minioadmin
      secret-access-key: minioadmin

Performance

The storage offload worker achieves high throughput when archiving to same-region S3:

Configuration Throughput Notes

Single stream

~32,000 msg/s

Baseline

4 streams (parallel)

~115,000 msg/s

Near-linear scaling

See Remote Storage Benchmarks for detailed results.

Performance tips

  • Same-region S3: deploy Korvet in the same AWS region as your bucket.

  • Multiple partitions: archival parallelism is per (topic, partition); more partitions = more archive concurrency.

  • Segment size: larger sealed segments produce larger Parquet files with better compression and fewer object-store operations. Tune topic-level segment-time, segment-bytes, and average-message-bytes settings.

Next Steps