For the latest stable version, please use Korvet 0.12.5!

Configuration

Korvet is configured using Spring Boot’s standard configuration mechanisms.

Configuration Files

Configuration can be provided via:

  • application.yml or application.properties

  • Environment variables

  • Command-line arguments

Basic Configuration

korvet:
  server:
    port: 9092
    host: 0.0.0.0
  redis:
    uri: redis://localhost:6379

Redis Configuration

Configure the Redis connection:

korvet:
  redis:
    uri: redis://localhost:6379
    username: default
    password: ${REDIS_PASSWORD}

You can also embed credentials in the URI:

korvet:
  redis:
    uri: redis://username:password@redis.example.com:6379

For TLS/SSL connections, use the rediss:// scheme:

korvet:
  redis:
    uri: rediss://redis.example.com:6379
    username: default
    password: ${REDIS_PASSWORD}

For Redis Cluster:

korvet:
  redis:
    uri: redis://node1:6379,node2:6379,node3:6379
    cluster: true
    username: default
    password: ${REDIS_PASSWORD}

Topic Configuration

Configure topic behavior:

korvet:
  server:
    auto-create-topics: true  # Automatically create topics when they don't exist
    default-partitions: 1     # Default number of partitions for auto-created topics

When auto-create-topics is disabled, topics must be created explicitly using the Kafka AdminClient or command-line tools before they can be used.

Offset Encoding Configuration

Korvet uses a stateless encoding scheme to convert Redis Stream entry IDs (timestamp-sequence pairs) into Kafka offsets. The offset-sequence-bits setting controls how many bits are allocated for the sequence number portion of the offset.

korvet:
  server:
    offset-sequence-bits: 10  # Default: 10 bits (supports up to ~1 million messages/second)

Understanding Offset Encoding

Redis Stream entry IDs have the format {timestamp}-{sequence} (e.g., 1732896000000-0). Korvet encodes these into Kafka offsets using bit-packing:

Kafka offset = (timestamp << offset-sequence-bits) | sequence

When to Change This Setting

The default value of 10 bits is suitable for most use cases, supporting up to ~1 million messages/second per partition.

You should consider changing this setting if:

Scenario Recommended Value Maximum Throughput

Standard throughput
(up to 1M msg/sec)

10 (default)

~1 million messages/second

High throughput
(1M - 16M msg/sec)

14

~16 million messages/second

Very high throughput
(>16M msg/sec)

16

~65 million messages/second

Low throughput
(<100K msg/sec)

8

~256,000 messages/second

Trade-offs

Lower values (8-10 bits):

  • ✅ Smaller offset deltas between messages

  • ✅ Better compatibility with Kafka clients that have offset delta limits

  • ❌ Lower maximum throughput per partition

Higher values (14-16 bits):

  • ✅ Higher maximum throughput per partition

  • ❌ Larger offset deltas between messages from different milliseconds

  • ❌ May exceed Kafka’s maximum offset delta limit (Integer.MAX_VALUE) for messages far apart in time

Maximum Offset Delta Constraint

Kafka has a constraint that the offset delta between consecutive messages in a fetch response cannot exceed Integer.MAX_VALUE (2,147,483,647).

With different sequence bit settings, messages from different milliseconds have different offset deltas:

Sequence Bits Offset Delta/ms Max Time Span

8 bits

256

~97 days

10 bits (default)

1,024

~24 days

14 bits

16,384

~36 hours

16 bits

65,536

~9 hours
If you use higher sequence bits (14-16), ensure your consumers fetch messages regularly to avoid exceeding the offset delta limit.

Example Configurations

For a high-throughput logging system:

korvet:
  server:
    offset-sequence-bits: 14  # Support up to ~16 million messages/second

For a low-throughput event system with long retention:

korvet:
  server:
    offset-sequence-bits: 8  # Lower offset delta, supports longer time spans (~256K msg/sec max)

TLS Configuration

Enable TLS for the Kafka protocol endpoint:

korvet:
  server:
    tls: true
    cert-file: /path/to/server.crt
    key-file: /path/to/server.key
    key-password: ${KEY_PASSWORD}

Cold Storage Configuration

Enable tiered storage with Delta Lake for archival of old messages:

korvet:
  storage:
    threshold: 6h  # Messages older than this are read from cold storage
    path: s3a://my-bucket/korvet/delta  # S3 path or file:///path/to/delta for local
    s3:
      region: us-east-1
      credentials:
        type: iam  # Use IAM role for authentication

For static credentials instead of IAM role:

korvet:
  storage:
    threshold: 6h
    path: s3a://my-bucket/korvet/delta
    s3:
      region: us-east-1
      credentials:
        type: static
        access-key-id: ${AWS_ACCESS_KEY_ID}
        secret-access-key: ${AWS_SECRET_ACCESS_KEY}

For LocalStack or MinIO:

korvet:
  storage:
    threshold: 6h
    path: s3a://my-bucket/korvet/delta
    s3:
      region: us-east-1
      endpoint: http://localhost:4566  # LocalStack endpoint
      credentials:
        type: static
        access-key-id: test
        secret-access-key: test
Cold storage is optional. If threshold is not configured, all reads will be served from Redis Streams (hot storage only).

Environment Variables

All configuration can be set via environment variables using Spring Boot’s relaxed binding. Property paths are converted to uppercase with underscores:

# Server configuration
export KORVET_SERVER_HOST=0.0.0.0
export KORVET_SERVER_PORT=9092
export KORVET_SERVER_BROKER_ID=0

# Redis configuration
export KORVET_REDIS_URI=redis://redis.example.com:6379
export KORVET_REDIS_USERNAME=default
export KORVET_REDIS_PASSWORD=secret

# Topic configuration
export KORVET_SERVER_AUTO_CREATE_TOPICS=true
export KORVET_SERVER_DEFAULT_PARTITIONS=1

# Cold storage configuration (optional)
export KORVET_STORAGE_THRESHOLD=6h
export KORVET_STORAGE_PATH=s3a://my-bucket/korvet/delta
export KORVET_STORAGE_S3_REGION=us-east-1
export KORVET_STORAGE_S3_CREDENTIALS_TYPE=iam
# For static credentials:
# export KORVET_STORAGE_S3_CREDENTIALS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
# export KORVET_STORAGE_S3_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Next Steps