Configuration Reference
Complete configuration reference for Korvet.
Configuration Properties
Server Configuration
| Property | Default | Description |
|---|---|---|
|
|
Host address to bind the server |
|
|
Port to listen on |
|
|
Broker ID for this server |
|
|
Redis keyspace prefix for all keys |
|
host |
Advertised host for clients (if not set, uses |
|
port |
Advertised port for clients (if not set, uses |
|
|
Netty boss thread pool size |
|
|
Netty worker thread pool size (0 = 2 × CPU cores) |
|
|
Maximum request size |
|
|
Maximum bytes per partition in fetch requests |
|
|
Maximum time to wait for fetch requests |
|
|
Automatically create topics when they don’t exist |
|
|
Default partitions for auto-created topics |
|
|
Bits for sequence number in offset encoding (1-16) |
|
|
Default time-based retention for topics |
|
|
Default size-based retention (-1 = unlimited) |
|
|
Default compression type (none, gzip, snappy, lz4, zstd) |
|
|
Default compression for values at rest in Redis |
|
|
Default value storage strategy ( |
|
|
Enable remote/tiered storage by default for newly created topics |
|
inherit |
Default local-tier retention time. If unset, inherits from |
|
|
Default local-tier retention size. |
|
|
Use approximate Redis |
|
|
Delay before completing consumer group rebalance |
|
|
Maximum pending bytes before applying backpressure to producers |
|
|
Threshold for resuming reads after backpressure (should be < max-pending-bytes) |
|
|
Maximum RediSearch page size used by the hot-tier |
TLS Configuration
| Property | Default | Description |
|---|---|---|
|
|
Enable TLS for the Kafka protocol endpoint |
|
none |
Server certificate file (PEM format) |
|
none |
Server private key file (PEM format) |
|
none |
Password for the private key file |
|
none |
Trusted CA certificates file (PEM format) |
|
|
Require client certificate authentication |
Redis Configuration
| Property | Default | Description |
|---|---|---|
|
none |
Redis connection URI (e.g., |
|
|
Redis host (used if URI not set) |
|
|
Redis port (used if URI not set) |
|
none |
Redis username |
|
none |
Redis password |
|
|
Enable Redis Cluster mode |
|
none |
Connection timeout |
|
|
Connection pool size for Redis stream operations |
|
|
Connection pool size for Redis metadata operations (bucket index, tiered group coordination, Delta group state) |
|
|
Maximum time to wait for a connection when pool is exhausted. When the pool is full, requests will wait up to this duration before timing out. Supports duration formats like |
|
CPU cores |
Lettuce I/O thread pool size |
Redis Metrics Configuration
| Property | Default | Description |
|---|---|---|
|
|
Enable Lettuce command latency metrics |
|
|
Enable histogram buckets for percentiles |
|
|
Track metrics per connection vs per host |
|
|
Maximum expected latency for histograms |
|
|
Minimum expected latency for histograms |
Storage Configuration
Remote Tier (Delta Lake)
Cold storage is enabled automatically when korvet.storage.remote.path is configured.
| Property | Default | Description |
|---|---|---|
|
none |
Delta Lake storage path (e.g., |
|
none |
AWS region (required for S3) |
|
none |
Custom S3 endpoint (for MinIO or LocalStack) |
|
|
Credential type: |
|
|
Maximum time a foreground request waits for cold index bootstrap |
|
|
Whether to rebuild cold indexes during startup |
|
|
Scanned files between cold index rebuild progress log messages |
|
|
Maximum number of cold-index segment entries examined per Redis Search lookup |
|
|
Maximum records written per archive write |
|
|
Minimum undelivered archival-group backlog before workers start buffering a stream |
|
|
Archive partial batches when the archival cursor age reaches this threshold |
|
|
Parquet files written per Delta commit |
|
|
Maximum records written into a single Parquet file before the writer rolls over |
|
|
Target payload budget for each Parquet file so cold storage defaults avoid undersized Delta files |
|
|
How often the archiver checks for data to move from hot to cold storage |
|
|
Number of archiver worker threads |
|
not set |
How often cold-storage retention cleanup runs |
See Cold Storage Configuration for complete configuration details and S3 performance tuning.
Default Configuration
The following shows the default application.yml configuration file:
# Korvet Server Configuration
#
# All properties have sensible defaults defined in code.
# Override using environment variables (e.g., KORVET_REDIS_URI) or this file.
# See docs/modules/ROOT/pages/reference/configuration.adoc for full reference.
spring:
application:
name: korvet
# Korvet Configuration
# Uncomment and modify properties to override defaults.
# korvet:
# # Main Redis (hot tier) configuration
# # Defaults: host=localhost, port=6379, pool-size=8, metadata-pool-size=8,
# # archival-source-pool-size=8, committed-offset-pool-size=8
# redis:
# uri: redis://localhost:6379
# # Alternative: specify host/port instead of URI
# # host: localhost
# # port: 6379
# # timeout: 5s
# # cluster: false # Enable for Redis Cluster (OSS Cluster API)
# # username: default
# # password: secret
# # pool-size: 8 # Stream operations pool
# # metadata-pool-size: 8 # Metadata/state operations pool
# # archival-source-pool-size: 8 # Dedicated hot-tier archival read pool
# # committed-offset-pool-size: 8 # Consumer offset storage pool
# # io-thread-pool-size: 4
# # pool-max-wait: 10s # Shared hot-path + metadata pool acquisition wait
# # archival-source-pool-max-wait: 10s # Dedicated archival source pool acquisition wait
# # committed-offset-pool-max-wait: 10s # Dedicated committed-offset pool acquisition wait
#
# server:
# # Network configuration (defaults: host=0.0.0.0, port=9092)
# # host: 0.0.0.0
# # port: 9092
# # broker-id: 0
# # keyspace: korvet
# # advertised-host: localhost
# # advertised-port: 9092
#
# # Thread pool configuration
# # boss-threads: 1
# # worker-threads: 0 # 0 = use Netty default (2 * CPU cores)
#
# # Request size limits
# # max-request-size: 100MB
# # partition-max-bytes: 1MB
# # fetch-max-wait: 500ms
# # bucket-index-search-limit: 1000
#
# # TLS/SSL configuration
# # tls: true
# # cert-file: /path/to/server.crt
# # key-file: /path/to/server.key
# # key-password: secret
# # trust-cert-file: /path/to/ca.crt
# # client-auth-required: false
#
# # Consumer group and producer backpressure tuning
# # rebalance-delay: 3s
# # max-pending-bytes: 100MB
# # resume-pending-bytes: 50MB
#
# # Topic defaults (used when creating new topics)
# topics:
# auto-create: false
# partitions: 1
# offset-sequence-bits: 10
# retention-time: 7d
# retention-bytes: -1 # -1 = unlimited
# compression: none
# storage-compression: lz4
# value-type: auto
# remote-storage-enabled: false
# # local-retention-time: 6h
# # local-retention-bytes: -2 # -2 = inherit retention-bytes
#
# storage:
# # Local Storage Configuration (Redis hot tier)
# local:
# bucketing:
# # Time-bucketed streams to avoid large stream keys.
# # Streams are rotated into time-based buckets, limiting key size.
# enabled: true # enabled by default
# duration: 10m # Typical: 5m, 10m, 30m
#
# # Remote Storage Configuration (Delta Lake cold tier)
# # When path is set, tiered storage is enabled automatically.
# # Topics with remote.storage.enable=true will use cold storage for archived data.
# remote:
# # path: s3a://my-bucket/korvet/delta # or file:///tmp/korvet
# # s3:
# # region: us-east-1
# # access-key-id: AKIA...
# # secret-access-key: ...
# # index:
# # bootstrap-enabled: true
# # bootstrap-timeout: 3s
# # progress-interval: 25 # scanned files between rebuild progress log messages
# # cache-size: 10000 # max cached stream bucket lists in the local client cache
# # archiver: # Runs automatically when path is set
# # interval: 5m # How often idle workers re-check source streams
# # batch-size: 100000 # Records per archive write
# # lag-messages-threshold: 100000 # Start archiving when undelivered group lag reaches this size
# # lag-duration-threshold: 5m # Or start when the archival cursor has become stale for this long
# # files-per-commit: 10 # Parquet files per Delta commit
# # workers: 1 # Number of archiver threads
# # writer:
# # messages-per-file: 100000 # Guardrail cap per Parquet file
# # target-file-size-bytes: 134217728 # 128 MB target payload budget per Parquet file
# Spring Boot Actuator Configuration
management:
endpoints:
web:
exposure:
include: health,info,prometheus,metrics,s3diagnostics,loggers
endpoint:
health:
show-details: always
prometheus:
metrics:
export:
enabled: true
# Logging Configuration
logging:
level:
root: INFO
# Example: raise or lower Korvet component verbosity explicitly when needed.
# com.redis.korvet: DEBUG
# Suppress Netty internal messages (native transport detection, buffer allocation, etc.)
io.netty.bootstrap: ERROR
io.lettuce.core.resource.Transports: ERROR
io.netty.util.internal: WARN
io.netty.buffer: WARN
# Suppress Delta Lake internals so rebuild visibility comes from Korvet-owned logs.
io.delta.kernel.internal: WARN
io.delta.standalone.internal: WARN
io.delta.storage: WARN
shadedelta.org.apache.parquet: WARN
shadedelta.org.apache.parquet.hadoop.InternalParquetRecordReader: WARN
org.apache.hadoop.util.NativeCodeLoader: ERROR
org.apache.parquet: WARN
org.apache.parquet.hadoop.InternalParquetRecordReader: WARN
org.apache.hadoop.io.compress.CodecPool: WARNEnvironment Variables
All configuration can be set via environment variables using Spring Boot’s relaxed binding:
# Server Configuration
export KORVET_SERVER_HOST=0.0.0.0
export KORVET_SERVER_PORT=9092
export KORVET_SERVER_BROKER_ID=0
export KORVET_SERVER_KEYSPACE=korvet
export KORVET_SERVER_ADVERTISED_HOST=localhost
export KORVET_SERVER_ADVERTISED_PORT=9092
# Thread pool configuration
export KORVET_SERVER_BOSS_THREADS=1
export KORVET_SERVER_WORKER_THREADS=0 # 0 = use Netty default (2 * CPU cores)
# Request size limits
export KORVET_SERVER_MAX_REQUEST_SIZE=100MB
export KORVET_SERVER_PARTITION_MAX_BYTES=1MB
# Fetch Configuration
export KORVET_SERVER_FETCH_MAX_WAIT=500ms # Maximum time to wait for fetch requests when no data is available
# Offset Encoding
# Number of bits for sequence number in offset encoding (1-16)
# Default: 10 (supports up to ~1 million messages/second per partition)
# See xref:quick-start/configuration.adoc#_offset_encoding_configuration[Offset Encoding Configuration] for details
export KORVET_TOPICS_OFFSET_SEQUENCE_BITS=10
# Topic Configuration
export KORVET_TOPICS_AUTO_CREATE=true
export KORVET_TOPICS_PARTITIONS=1
export KORVET_TOPICS_RETENTION_TIME=7d
export KORVET_TOPICS_RETENTION_BYTES=-1 # -1 = unlimited
export KORVET_TOPICS_COMPRESSION=none # none, gzip, snappy, lz4, zstd
export KORVET_TOPICS_STORAGE_COMPRESSION=lz4
export KORVET_TOPICS_VALUE_TYPE=auto
export KORVET_TOPICS_REMOTE_STORAGE_ENABLED=false
# Consumer Group Configuration
export KORVET_SERVER_REBALANCE_DELAY=3s # Delay before completing rebalance (default: 3s)
# Backpressure Configuration
export KORVET_SERVER_MAX_PENDING_BYTES=100MB # Max pending bytes before backpressure (default: 100MB)
export KORVET_SERVER_RESUME_PENDING_BYTES=50MB # Resume threshold after backpressure (default: 50MB)
# TLS Configuration (optional)
export KORVET_SERVER_TLS=true
export KORVET_SERVER_CERT_FILE=/path/to/server.crt
export KORVET_SERVER_KEY_FILE=/path/to/server.key
export KORVET_SERVER_KEY_PASSWORD=secret
export KORVET_SERVER_TRUST_CERT_FILE=/path/to/ca.crt
export KORVET_SERVER_CLIENT_AUTH_REQUIRED=false
# Redis Connection
export KORVET_REDIS_URI=redis://localhost:6379
export KORVET_REDIS_HOST=localhost
export KORVET_REDIS_PORT=6379
export KORVET_REDIS_USERNAME=default
export KORVET_REDIS_PASSWORD=secret
export KORVET_REDIS_CLUSTER=false
export KORVET_REDIS_TIMEOUT=5s # Connection timeout (optional)
export KORVET_REDIS_POOL_SIZE=8 # Connection pool size (default: 8)
export KORVET_REDIS_IO_THREAD_POOL_SIZE=4 # Lettuce I/O thread pool size (default: number of CPUs)
# Redis Metrics (optional)
# Enables Lettuce command latency metrics published to Micrometer
export KORVET_REDIS_METRICS_ENABLED=false # Default: false
export KORVET_REDIS_METRICS_HISTOGRAM=false # Default: false
export KORVET_REDIS_METRICS_LOCAL_DISTINCTION=false # Default: false
export KORVET_REDIS_METRICS_MAX_LATENCY=5m # Default: 5m
export KORVET_REDIS_METRICS_MIN_LATENCY=1ms # Default: 1ms
# Remote Storage (optional) - enabled when path is set
export KORVET_STORAGE_REMOTE_PATH=s3a://my-bucket/korvet/delta
export KORVET_STORAGE_REMOTE_S3_REGION=us-east-1
export KORVET_STORAGE_REMOTE_S3_ENDPOINT=http://localhost:4566 # Optional: for LocalStack/MinIO
export KORVET_STORAGE_REMOTE_S3_CREDENTIALS_TYPE=iam # or 'static'
export KORVET_STORAGE_REMOTE_S3_CREDENTIALS_ROLE_ARN=arn:aws:iam::123456789012:role/KorvetReaderRole
# For static credentials:
export KORVET_STORAGE_REMOTE_S3_CREDENTIALS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export KORVET_STORAGE_REMOTE_S3_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# S3 Performance Tuning (optional)
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_MAX_CONNECTIONS=96 # Default: 96
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_MAX_THREADS=64 # Default: 64
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_READAHEAD_RANGE=256K # Default: 256K
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_INPUT_POLICY=normal # normal, sequential, random
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_FAST_UPLOAD=true # Default: true
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_FAST_UPLOAD_BUFFER=disk # disk, bytebuffer, array
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_FAST_UPLOAD_ACTIVE_BLOCKS=8 # Default: 8
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_BLOCK_SIZE=64M # Default: 64M
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_MULTIPART_SIZE=64M # Default: 64M
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_CONNECTION_TIMEOUT=15000 # ms, Default: 15000
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_SOCKET_TIMEOUT=60000 # ms, Default: 60000
# Cold index configuration (optional, used when storage path is set)
export KORVET_STORAGE_REMOTE_INDEX_BOOTSTRAP_ENABLED=true # Default: true
export KORVET_STORAGE_REMOTE_INDEX_BOOTSTRAP_TIMEOUT=3s # Default: 3s
export KORVET_STORAGE_REMOTE_INDEX_PROGRESS_INTERVAL=25 # Scanned files between progress log messages
export KORVET_STORAGE_REMOTE_INDEX_SEARCH_LIMIT=1 # Default: 1
# Cold tier archiver configuration (optional, used when storage path is set)
export KORVET_STORAGE_REMOTE_ARCHIVER_BATCH_SIZE=1000 # Default: 1000
export KORVET_STORAGE_REMOTE_ARCHIVER_LAG_MESSAGES_THRESHOLD=1000 # Default: 1000
export KORVET_STORAGE_REMOTE_ARCHIVER_LAG_DURATION_THRESHOLD=1m # Default: 1m
export KORVET_STORAGE_REMOTE_ARCHIVER_INTERVAL=5s # Default: 5s
export KORVET_STORAGE_REMOTE_ARCHIVER_WORKERS=1 # Default: 1
# Cold tier retention cleanup configuration (optional)
export KORVET_STORAGE_REMOTE_ARCHIVER_RETENTION_INTERVAL=1h # Default: 1h
# Logging
export LOGGING_LEVEL_COM_REDIS_KORVET=INFO
