This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.19!

Remote Storage (Apache Iceberg on object store)

Korvet includes a built-in storage worker that offloads sealed Redis stream segments to Apache Iceberg tables backed by Parquet files on S3.

In Korvet tiered storage terminology, the local tier uses Redis Streams and the remote tier is a set of Apache Iceberg tables, one per topic. Korvet uses local/remote in its configuration and APIs.

Overview

The remote tier holds one Apache Iceberg table per topic, named after the topic with separator characters (., , :) rewritten to (topic orders.created.v1 becomes table orders_created_v1). All partitions of a topic share its table. The storage worker:

  1. Reads sealed LOCAL segments from each topic-partition’s segment metadata in Redis

  2. Streams the segment contents from Redis (one page resident at a time) into an Iceberg data file

  3. Appends and commits the data file to the Iceberg table, then marks the Redis segment as offloaded

  4. Runs in-process as part of the Korvet server; there is no separate archive daemon

Each topic’s table is a standard Iceberg table: external query engines (Spark, Trino, Athena, DuckDB, …​) can read it directly. Iceberg manages its own metadata and manifest files under the table location. Each offloaded write commits as one Iceberg append; if the storage worker crashes before the commit, the segment is swept back from OFFLOADING to SEALED at the start of the next leader tick and the offload scan re-streams the (still untouched) Redis segment (see Stranded offload recovery).

Configuration

Enable remote storage by setting korvet.storage.remote.path in your application.yml:

korvet:
  storage:
    remote:
      path: s3://my-bucket/korvet
      s3:
        region: us-west-1
    worker:
      tick-interval: 1m

For local development and tests, point the cold tier at a local directory instead of an object store. No s3.* settings are needed:

korvet:
  storage:
    remote:
      path: file:///var/lib/korvet/cold
Setting korvet.storage.remote.path makes the cold tier available. The leader-locked storage worker is enabled by default and rolls eligible segments, offloads sealed segments, and enforces local and remote retention. Topics are archived only when they also have remote.storage.enable=true.

Storage Properties

Property Default Description

korvet.storage.remote.path

required

Cold-tier root URI. Supports s3:// for object storage and file:// for a local-filesystem warehouse (useful for local development and tests without MinIO/S3). Iceberg resolves the FileIO from the URI scheme.

korvet.storage.remote.s3.region

unset

AWS region for the S3 object store.

korvet.storage.remote.s3.endpoint

unset

Optional endpoint URL for S3-compatible stores such as MinIO or LocalStack.

korvet.storage.remote.s3.path-style-access

unset

Use path-style addressing. Required for most non-AWS S3-compatible stores.

korvet.storage.remote.s3.access-key-id

unset

Static access-key id. Prefer IAM roles in production.

korvet.storage.remote.s3.secret-access-key

unset

Static secret access key. Prefer IAM roles in production.

Maintenance Properties

Property Default Description

korvet.storage.worker.enabled

true

Enables the storage worker in this JVM.

korvet.storage.worker.tick-interval

1m

Tick cadence for the storage worker loop.

korvet.storage.worker.lease-duration

2m

Redis leader-lock lease duration. Must exceed tick-interval.

Per-Topic Retention Configuration

Control when data moves from the local tier to the remote tier using topic-level configuration:

Configuration Default Description

remote.storage.enable

false

Enable tiered storage for this topic (Kafka KIP-405 standard).

local.retention.ms

-2

Time to keep in the local tier. -2 means use total retention.ms (Kafka KIP-405).

local.retention.bytes

-2

Size to keep in the local tier. -2 means use total retention.bytes (Kafka KIP-405).

retention.ms

604800000 (7 days)

Total retention across all tiers.

Remote-tier retention is implicit: retention.ms - local.retention.ms.

Example: Keep 1 day local and the rest remote (1 year total):

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true,retention.ms=31536000000,local.retention.ms=86400000

Storage Format

Table Layout

The remote tier holds one Iceberg table per topic, under an Iceberg namespace named after korvet.namespace (default korvet) so instances sharing a warehouse keep separate tables. Each table is partitioned by stream_key (identity) — so each topic partition’s rows land in their own Iceberg partition — and sorted by message_ts. Iceberg owns the on-disk layout under korvet.storage.remote.path: a metadata/ directory for table metadata and manifests, plus Parquet data files. Each offloaded segment is written as one data file named:

segment-<segmentId>-<uuid>.parquet

The trailing UUID keeps each write unique so a retried offload never collides with a file left by a prior failed attempt.

Iceberg Schema

Each row is one message. The hot-tier stream entry is decoded into analytics-friendly columns so external engines can query the key, value, headers, and timestamps directly. The Iceberg schema is:

required STRING                      stream_key;
required LONG                        segment_id;
required STRING                      message_id;
required LONG                        message_ts;
required LONG                        kafka_timestamp;
optional BINARY                      key;
optional BINARY                      value;
optional LIST<STRUCT<                       // headers
           required STRING  header_key,
           optional BINARY  header_value>>;
Column Type Description

stream_key

STRING

Local stream key of the source partition (the table partition column).

segment_id

LONG

Sealed segment number the message came from.

message_id

STRING

Full Redis stream message id (e.g. 1708956789000-0).

message_ts

LONG

Redis stream-id millisecond component (append-time ordering); the table sort key.

kafka_timestamp

LONG

Producer record timestamp in epoch milliseconds, or -1 when none was preserved.

key

BINARY

Record key, decoded from the stream entry. Null when the record has no key.

value

BINARY

Record value, decoded from the stream entry. Null for a tombstone.

headers

LIST<STRUCT>

Record headers, preserving order and duplicate keys. Each entry has a required header_key (STRING) and an optional header_value (BINARY).

Compression

Data files use Iceberg’s default Parquet compression. Row-group and target file sizes are configurable via korvet.storage.remote.iceberg.row-group-size and korvet.storage.remote.iceberg.target-file-size (both default 128MB).

Read concurrency

Each server runs remote Iceberg reads on a bounded executor. Configure its maximum concurrent reads with korvet.storage.remote.iceberg.read.threads (default 4) and its waiting backlog with korvet.storage.remote.iceberg.read.queue-capacity (default 64). Requests beyond both bounds fail immediately instead of accumulating after callers time out.

Increase the thread count only when object-store latency and server resources justify more concurrent reads. Keep the queue small enough that queued work can begin before the calling request’s timeout.

During shutdown, Korvet waits for active remote reads, inspections, offloads, and deletes before closing the Iceberg catalog. This deliberately favors completing catalog operations safely over a shorter shutdown; a slow object store can therefore extend shutdown up to its client-side operation timeout.

AWS Authentication

The cold tier writes through Iceberg’s S3FileIO. Configure common S3 settings under korvet.storage.remote.s3; otherwise the AWS SDK default credential provider chain is used.

Credential providers

Common production choices:

Provider Use when

EC2, ECS, or EKS node roles

Leave static credentials unset and let the AWS SDK use instance metadata.

EKS IAM Roles for Service Accounts (IRSA)

Leave static credentials unset. AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are injected into the pod and picked up by the SDK.

Static access key + secret

Use korvet.storage.remote.s3.access-key-id and korvet.storage.remote.s3.secret-access-key for development or S3-compatible stores.

IRSA (EKS IAM Roles for Service Accounts)

korvet:
  storage:
    remote:
      path: s3://my-bucket/korvet
      s3:
        region: us-east-1

When IRSA is configured on the cluster, AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are injected into the pod and picked up automatically.

IAM role (EC2/ECS/EKS nodes)

korvet:
  storage:
    remote:
      path: s3://my-bucket/korvet
      s3:
        region: us-west-1

Static credentials (dev / MinIO)

korvet:
  storage:
    remote:
      path: s3://my-bucket/korvet
      s3:
        region: us-west-1
        endpoint: http://localhost:9000
        path-style-access: true
        access-key-id: minioadmin
        secret-access-key: minioadmin

Performance

The storage worker achieves high throughput when archiving to same-region S3:

Configuration Throughput Notes

Single stream

~32,000 msg/s

Baseline

4 streams (parallel)

~115,000 msg/s

Near-linear scaling

See Remote Storage Benchmarks for detailed results.

Performance tips

  • Same-region S3: deploy Korvet in the same AWS region as your bucket.

  • Multiple partitions: archival parallelism is per (topic, partition); more partitions = more archive concurrency.

  • Segment size: larger sealed segments produce larger Parquet files with better compression and fewer object-store operations.

Remote Segment Metadata Inspection

The Admin API and UI provide read-only inspection of remote segment Iceberg metadata, allowing you to verify offloaded segments and diagnose remote storage issues without querying the Iceberg table directly.

Admin API Endpoint

When requesting segment details through the Admin API (GET /api/v1/topics/{name}/partitions/{partition}/segments/{segmentId}), offloaded segments include a remoteStore field describing the segment’s Iceberg data files: the file format (PARQUET), the Iceberg table and catalog, and each data file’s location, bucket, object key, size, and record count. See Storage Segments for the full response format.

UI Integration

The web console displays remote segment metadata on the Storage → Segments & Retention tab. Expand an offloaded segment to see its Iceberg details, including format, table, and per-data-file size and record count.

Inspection Behavior

  • Read-only: Inspection never creates or modifies Iceberg namespaces, tables, snapshots, or files

  • Cached: Results are cached for 10 seconds (configurable via korvet.admin.segment-inspection.ttl)

  • Timeout-guarded: Inspection runs on a bounded daemon thread pool and times out after 5 seconds (configurable via korvet.admin.segment-inspection.timeout)

  • Graceful degradation: If remote inspection fails or times out, the API returns local segment metadata with remoteStore: null (HTTP 200, not an error)

  • Concurrent request deduplication: Multiple concurrent requests for the same segment share a single in-flight inspection

Configuration

Remote segment inspection is enabled by default when korvet.storage.remote.path is configured. Tune inspection behavior under korvet.admin.segment-inspection:

Property Default Description

korvet.admin.segment-inspection.timeout

5s

Maximum time to wait for Iceberg catalog and object store I/O before degrading to remoteStore: null.

korvet.admin.segment-inspection.ttl

10s

Time to reuse a completed inspection result. Zero disables caching. Inspections are keyed on (streamKey, segmentId, version), so offload, re-offload, or delete invalidates the cache immediately.

korvet.admin.segment-inspection.threads

8

Maximum number of concurrent inspections. Threads are started on demand up to this many and released once idle.

korvet.admin.segment-inspection.queue-capacity

8

Maximum queued inspection tasks. When threads and queue are full, new requests degrade immediately.

Use Cases

  • Verify offload completion: Confirm sealed segments are successfully archived to the remote tier

  • Diagnose missing data: Check Iceberg file counts and record totals when reads return fewer messages than expected

  • Monitor remote storage health: Detect object store or catalog connectivity issues from remoteStore: null responses

  • Validate retention policies: Verify remote segment retention aligns with retention.ms configuration

Next Steps