For the latest stable version, please use Korvet 0.19!

Configuration

Korvet is configured using Spring Boot’s standard configuration mechanisms.

Configuration Files

Configuration can be provided via:

  • application.yml or application.properties

  • Environment variables

  • Command-line arguments

Basic Configuration

korvet:
  broker:
    port: 9092
    host: 0.0.0.0
  redis:
    uri: redis://localhost:6379

Redis Configuration

Configure the Redis connection:

korvet:
  redis:
    uri: redis://localhost:6379
    username: default
    password: ${REDIS_PASSWORD}

You can also embed credentials in the URI:

korvet:
  redis:
    uri: redis://username:password@redis.example.com:6379

For TLS/SSL connections, use the rediss:// scheme:

korvet:
  redis:
    uri: rediss://redis.example.com:6379
    username: default
    password: ${REDIS_PASSWORD}

For Redis Cluster:

korvet:
  redis:
    uri: redis://node1:6379,node2:6379,node3:6379
    cluster: true
    username: default
    password: ${REDIS_PASSWORD}

Topic Configuration

Configure topic behavior using a list of glob patterns. Patterns are evaluated in declared order and combined first-match-wins per field, so place more specific patterns ahead of a * catch-all.

korvet:
  topics:
    - name: "order.*"        # Override defaults for topics matching this pattern
      retention-time: 1h
    - name: "*"              # Catch-all defaults
      auto-create: false     # Automatically create matching topics when they don't exist
      partitions: 1          # Default number of partitions
      retention-time: 7d     # Default retention

When auto-create is disabled, topics must be created explicitly using the Kafka AdminClient or command-line tools before they can be used.

Offset Encoding Configuration

Korvet uses a stateless encoding scheme to convert Redis Stream entry IDs (timestamp-sequence pairs) into Kafka offsets. The offset-sequence-bits setting controls how many bits are allocated for the sequence number portion of the offset.

korvet:
  topics:
    - name: "*"
      offset-sequence-bits: 10  # Default: 10 bits (supports up to ~1 million messages/second)

Understanding Offset Encoding

Redis Stream entry IDs have the format {timestamp}-{sequence} (e.g., 1732896000000-0). Korvet encodes these into Kafka offsets using bit-packing:

Kafka offset = (timestamp << offset-sequence-bits) | sequence

When to Change This Setting

The default value of 10 bits is suitable for most use cases, supporting up to ~1 million messages/second per partition.

You should consider changing this setting if:

Scenario Recommended Value Maximum Throughput

Standard throughput
(up to 1M msg/sec)

10 (default)

~1 million messages/second

High throughput
(1M - 16M msg/sec)

14

~16 million messages/second

Very high throughput
(>16M msg/sec)

16

~65 million messages/second

Low throughput
(<100K msg/sec)

8

~256,000 messages/second

Trade-offs

Lower values (8-10 bits):

  • ✅ Smaller offset deltas between messages

  • ✅ Better compatibility with Kafka clients that have offset delta limits

  • ❌ Lower maximum throughput per partition

Higher values (14-16 bits):

  • ✅ Higher maximum throughput per partition

  • ❌ Larger offset deltas between messages from different milliseconds

  • ❌ May exceed Kafka’s maximum offset delta limit (Integer.MAX_VALUE) for messages far apart in time

Maximum Offset Delta Constraint

Kafka has a constraint that the offset delta between consecutive messages in a fetch response cannot exceed Integer.MAX_VALUE (2,147,483,647).

With different sequence bit settings, messages from different milliseconds have different offset deltas:

Sequence Bits Offset Delta/ms Max Time Span

8 bits

256

~97 days

10 bits (default)

1,024

~24 days

14 bits

16,384

~36 hours

16 bits

65,536

~9 hours
If you use higher sequence bits (14-16), ensure your consumers fetch messages regularly to avoid exceeding the offset delta limit.

Example Configurations

For a high-throughput logging system:

korvet:
  topics:
    - name: "*"
      offset-sequence-bits: 14  # Support up to ~16 million messages/second

For a low-throughput event system with long retention:

korvet:
  topics:
    - name: "*"
      offset-sequence-bits: 8  # Lower offset delta, supports longer time spans (~256K msg/sec max)

Consumer Group Configuration

Configure consumer group rebalancing behavior:

korvet:
  broker:
    rebalance-delay: 3s  # Delay before completing rebalance (default: 3s)

The rebalance-delay setting controls how long Korvet waits before completing a consumer group rebalance to allow more members to join. Increase this value for larger consumer groups, especially in Kubernetes environments where pods may start at different times.

TLS Configuration

Enable TLS for the Kafka protocol endpoint:

korvet:
  broker:
    tls: true
    cert-file: /path/to/server.crt
    key-file: /path/to/server.key
    key-password: ${KEY_PASSWORD}

Remote Storage Configuration

Enable tiered storage to archive sealed segments as Parquet files on an object store. Credentials, region, endpoint, etc. are passed through hadoop-properties as Hadoop S3A keys:

korvet:
  storage:
    remote:
      path: s3a://my-bucket/korvet  # s3a://... or file:///abs/path
      hadoop-properties:
        fs.s3a.region: us-east-1
        fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
  archiver:
    enabled: true
Remote storage is enabled when korvet.archiver.enabled=true; that requires korvet.storage.remote.path to be set.

For static credentials:

korvet:
  storage:
    remote:
      path: s3a://my-bucket/korvet
      hadoop-properties:
        fs.s3a.region: us-east-1
        fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
        fs.s3a.access.key: ${AWS_ACCESS_KEY_ID}
        fs.s3a.secret.key: ${AWS_SECRET_ACCESS_KEY}

For LocalStack or MinIO:

korvet:
  storage:
    remote:
      path: s3a://my-bucket/korvet
      hadoop-properties:
        fs.s3a.region: us-east-1
        fs.s3a.endpoint: http://localhost:4566
        fs.s3a.path.style.access: "true"
        fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
        fs.s3a.access.key: test
        fs.s3a.secret.key: test
Remote storage is optional. If not configured, all reads will be served from Redis Streams (local storage only).

See Remote Storage for complete configuration options and per-topic retention settings.

Redis Metrics Configuration

Korvet can optionally enable Lettuce command latency metrics to track Redis operation performance.

These metrics are disabled by default to minimize overhead. Enable them when you need detailed Redis performance monitoring. 
korvet:
  redis:
    metrics:
      enabled: true  # Enable Lettuce command metrics (default: false)
      histogram: false  # Enable histogram buckets for percentiles (default: false)
      local-distinction: false  # Track per connection vs per host (default: false)
      max-latency: 5m  # Maximum expected latency (default: 5 minutes)
      min-latency: 1ms  # Minimum expected latency (default: 1 millisecond)

Configuration Properties:

  • enabled: Enable Lettuce command latency metrics (default: false)

  • histogram: Enable histogram buckets for aggregable percentile approximations (default: false)

  • local-distinction: Track metrics per connection instead of per host/port (default: false)

  • max-latency: Maximum expected latency for histogram buckets (default: 5m, only applies when histogram is enabled)

  • min-latency: Minimum expected latency for histogram buckets (default: 1ms, only applies when histogram is enabled)

When enabled, Lettuce will publish two timer metrics:

  • lettuce.command.firstresponse - Time to first response from Redis

  • lettuce.command.completion - Time to complete Redis command

Each metric includes tags: command (e.g., XADD, XREAD), local (if local-distinction enabled), and remote (Redis server address).

For more details, see Lettuce Redis Client Metrics.

Environment Variables

All configuration can be set via environment variables using Spring Boot’s relaxed binding. Property paths are converted to uppercase with underscores:

# Broker configuration
export KORVET_BROKER_HOST=0.0.0.0
export KORVET_BROKER_PORT=9092
export KORVET_BROKER_ID=0

# Redis configuration
export KORVET_REDIS_URI=redis://redis.example.com:6379
export KORVET_REDIS_USERNAME=default
export KORVET_REDIS_PASSWORD=secret

# Redis metrics (optional)
export KORVET_REDIS_METRICS_ENABLED=false
export KORVET_REDIS_METRICS_HISTOGRAM=false
export KORVET_REDIS_METRICS_LOCAL_DISTINCTION=false
export KORVET_REDIS_METRICS_MAX_LATENCY=5m
export KORVET_REDIS_METRICS_MIN_LATENCY=1ms

# Topic configuration (pattern-based; index 0 is the catch-all here)
export KORVET_TOPICS_0_NAME='*'
export KORVET_TOPICS_0_AUTO_CREATE=true
export KORVET_TOPICS_0_PARTITIONS=1

# Remote storage configuration (optional)
export KORVET_ARCHIVER_ENABLED=true
export KORVET_STORAGE_REMOTE_PATH=s3a://my-bucket/korvet
export KORVET_STORAGE_REMOTE_HADOOP_PROPERTIES_FS_S3A_REGION=us-east-1
export KORVET_STORAGE_REMOTE_HADOOP_PROPERTIES_FS_S3A_AWS_CREDENTIALS_PROVIDER=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
# For static credentials, also set:
# export KORVET_STORAGE_REMOTE_HADOOP_PROPERTIES_FS_S3A_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
# export KORVET_STORAGE_REMOTE_HADOOP_PROPERTIES_FS_S3A_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Next Steps