For the latest stable version, please use Korvet 0.19!

Remote Storage (Parquet on object store)

Korvet includes a built-in archival service that continuously archives sealed Redis stream segments to per-segment Parquet files on an object store or filesystem.

In tiered storage terminology, the local tier (Redis Streams) is also called the hot tier, and the remote tier (Parquet files) is also called the cold tier. Korvet uses local/remote in its configuration and APIs.

Overview

The remote tier is per-segment Parquet files. The archival service:

  1. Reads sealed LOCAL segments from each topic-partition’s manifest

  2. Streams the segment contents from Redis (one page resident at a time) into a single Parquet file

  3. Flips the manifest entry from LOCAL to REMOTE — the manifest is the commit point

  4. Runs in-process as part of the Korvet server; there is no separate archiver daemon

There is no mirror stream, no transaction log, and no multi-file atomic commit. One sealed segment maps to one Parquet file; the manifest tier-flip is the single commit unit. If the archiver crashes before the flip, the manifest entry stays LOCAL and the next scan re-streams the (still untouched) Redis segment.

Cold storage is broker-private: Korvet controls the path layout, schema, and footer caching, and there is no expectation that external query engines read these files directly.

Configuration

Enable remote storage by setting korvet.storage.remote.path in your application.yml:

korvet:
  storage:
    remote:
      path: s3a://my-bucket/korvet
      hadoop-properties:
        fs.s3a.region: us-west-1
  archiver:
    enabled: true
    scan-interval: 5s
  trimmer:
    enabled: true
    scan-interval: 5m
Setting korvet.archiver.enabled=true requires korvet.storage.remote.path to be set. Topics are archived only when they also have remote.storage.enable=true.

Storage Properties

Property Default Description

korvet.storage.remote.path

required

Cold-tier root URI. Any scheme supported by Hadoop’s FileSystem — typically file:// for tests and s3a:// for production.

korvet.storage.remote.hadoop-properties

{}

Map of Hadoop Configuration keys forwarded to the underlying FileSystem. Use these for credentials, region, endpoint, etc. (fs.s3a.region, fs.s3a.endpoint, fs.s3a.aws.credentials.provider, …).

korvet.storage.remote.footer-cache-capacity

4096

LRU capacity for the Parquet footer cache used by readers.

korvet.storage.remote.row-group-size

1MB

Target Parquet row group size.

korvet.storage.remote.page-size

1MB

Target Parquet page size.

Archiver Properties

Property Default Description

korvet.archiver.enabled

unset

Enables the archiver worker in this JVM. Requires korvet.storage.remote.path.

korvet.archiver.initial-delay

5s

Delay between startup and the first manifest scan.

korvet.archiver.scan-interval

5s

Interval between manifest scans.

korvet.archiver.scan-page-size

1000

Maximum messages read per XRANGE page when draining a sealed segment.

Trimmer Properties

Property Default Description

korvet.trimmer.enabled

unset

Enables the trimmer worker in this JVM.

korvet.trimmer.initial-delay

30s

Delay between startup and the first trim scan.

korvet.trimmer.scan-interval

5m

Interval between trim scans.

korvet.trimmer.timeout

30s

Per-stream timeout applied to a single trim operation.

Per-Topic Retention Configuration

Control when data moves from the local tier to the remote tier using topic-level configuration:

Configuration Default Description

remote.storage.enable

false

Enable tiered storage for this topic (Kafka KIP-405 standard).

local.retention.ms

-2

Time to keep in the local tier. -2 means use total retention.ms (Kafka KIP-405).

local.retention.bytes

-2

Size to keep in the local tier. -2 means use total retention.bytes (Kafka KIP-405).

retention.ms

604800000 (7 days)

Total retention across all tiers.

Remote-tier retention is implicit: retention.ms - local.retention.ms.

Example: Keep 1 day local and the rest remote (1 year total):

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true,retention.ms=31536000000,local.retention.ms=86400000

Storage Format

File Layout

One Parquet file per sealed segment. The default path mapper produces:

<root>/<streamKey>/<segmentId>.parquet

There is no transaction-log directory, no _delta_log/, no manifest file inside the object-store prefix — the manifest lives in Redis.

Parquet Schema

Each file contains the messages of a single sealed segment with this schema:

required STRING id;
required MAP<STRING, BINARY> fields;
Column Type Description

id

STRING

Full Redis stream message id (e.g. 1708956789000-0).

fields

MAP<STRING, BINARY>

Opaque key/value map. The storage layer has no Kafka knowledge — Kafka headers and the value blob are just entries in the map.

Compression

Files are written with SNAPPY compression. Most segments fit in a single row group; parquet-mr splits row groups automatically for unusually large segments.

AWS Authentication

The archival service writes through Hadoop’s s3a:// filesystem. Configure credentials by adding Hadoop properties to korvet.storage.remote.hadoop-properties.

Credential providers

The credential chain is configured via fs.s3a.aws.credentials.provider. Common choices:

Provider Use when

org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider

Running on EC2, ECS, or EKS with node-level IAM roles (IMDS).

com.amazonaws.auth.WebIdentityTokenCredentialsProvider

EKS IAM Roles for Service Accounts (IRSA), using AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE.

org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

Static access key + secret (dev, MinIO, LocalStack).

com.amazonaws.auth.DefaultAWSCredentialsProviderChain

Auto-detect — environment, system properties, profile, IMDS, etc.

IRSA (EKS IAM Roles for Service Accounts)

korvet:
  storage:
    remote:
      path: s3a://my-bucket/korvet
      hadoop-properties:
        fs.s3a.region: us-east-1
        fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider

When IRSA is configured on the cluster, AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are injected into the pod and picked up automatically.

IAM role (EC2/ECS/EKS nodes)

korvet:
  storage:
    remote:
      path: s3a://my-bucket/korvet
      hadoop-properties:
        fs.s3a.region: us-west-1
        fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider

Static credentials (dev / MinIO)

korvet:
  storage:
    remote:
      path: s3a://my-bucket/korvet
      hadoop-properties:
        fs.s3a.region: us-west-1
        fs.s3a.endpoint: http://localhost:9000
        fs.s3a.path.style.access: "true"
        fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
        fs.s3a.access.key: minioadmin
        fs.s3a.secret.key: minioadmin

Performance

The archival service achieves high throughput when archiving to same-region S3:

Configuration Throughput Notes

Single stream

~32,000 msg/s

Baseline

4 streams (parallel)

~115,000 msg/s

Near-linear scaling

See Remote Storage Benchmarks for detailed results.

Performance tips

  • Same-region S3: deploy Korvet in the same AWS region as your bucket.

  • Multiple partitions: archival parallelism is per (topic, partition); more partitions = more archive concurrency.

  • Segment size: larger sealed segments produce larger Parquet files with better compression and fewer object-store operations. Tune korvet.storage.local.max-messages-per-segment and max-open-segment-age.

Next Steps