|
This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.12.5! |
Configuration Reference
Complete configuration reference for Korvet.
Configuration Properties
Broker Configuration
| Property | Default | Description |
|---|---|---|
|
|
Host address to bind the server |
|
|
Port to listen on |
|
|
Broker ID for this server |
|
|
Redis keyspace prefix for broker metadata, topic registry, and committed offsets |
|
host |
Advertised host for clients (if not set, uses |
|
port |
Advertised port for clients (if not set, uses |
|
|
Netty boss thread pool size |
|
|
Netty worker thread pool size (0 = 2 × CPU cores) |
|
|
Maximum request size |
|
|
Maximum bytes per partition in fetch requests |
|
|
Maximum time to wait for fetch requests |
|
|
Enable SASL authentication for the Kafka protocol listener using Redis-backed credentials in the broker keyspace |
|
|
Automatically create topics when they don’t exist |
|
|
Default partitions for auto-created topics |
|
|
Bits for sequence number in offset encoding (1-16) |
|
|
Average message size estimate used when translating byte-based retention into message-count trimming. |
|
|
Default time-based retention for topics |
|
|
Default size-based retention (-1 = unlimited) |
|
|
Default compression type (none, gzip, snappy, lz4, zstd) |
|
|
Default compression for values at rest in Redis |
|
|
Default value storage strategy ( |
|
|
Enable remote/tiered storage by default for newly created topics |
|
inherit |
Default local-tier retention time. If unset, inherits from |
|
|
Default local-tier retention size. |
|
none |
Default bucket duration for newly created topics. When unset, topics are unbucketed unless configured explicitly. |
|
|
Delay before completing consumer group rebalance |
|
|
Maximum pending bytes before applying backpressure to producers |
|
|
Threshold for resuming reads after backpressure (should be < max-pending-bytes) |
Runtime Services
| Property | Default | Description |
|---|---|---|
|
|
Comma-separated list of services this process should run. Supported values are |
TLS Configuration
| Property | Default | Description |
|---|---|---|
|
|
Enable TLS for the Kafka protocol endpoint |
|
none |
Server certificate file (PEM format) |
|
none |
Server private key file (PEM format) |
|
none |
Password for the private key file |
|
none |
Trusted CA certificates file (PEM format) |
|
|
Require client certificate authentication |
Redis Configuration
| Property | Default | Description |
|---|---|---|
|
none |
Redis connection URI (e.g., |
|
|
Redis host (used if URI not set) |
|
|
Redis port (used if URI not set) |
|
none |
Redis username |
|
none |
Redis password |
|
|
Enable Redis Cluster mode |
|
none |
Connection timeout |
|
|
Connection pool size for Redis committed-offset operations |
|
|
Maximum time to wait for a connection when pool is exhausted. When the pool is full, requests will wait up to this duration before timing out. Supports duration formats like |
|
CPU cores |
Lettuce I/O thread pool size |
Local Storage Configuration
| Property | Default | Description |
|---|---|---|
|
inherits `korvet.redis.uri` |
Optional Redis URI override for hot storage |
|
|
Optional Redis host override for hot storage |
|
|
Optional Redis port override for hot storage |
|
inherits `korvet.redis.username` |
Optional Redis username override for hot storage |
|
inherits `korvet.redis.password` |
Optional Redis password override for hot storage |
|
|
Optional Redis Cluster mode override for hot storage |
|
inherits `korvet.redis.timeout` |
Optional Redis timeout override for hot storage |
|
inherits `korvet.redis.io-thread-pool-size` |
Optional Lettuce I/O thread pool override for hot storage |
|
|
Redis keyspace prefix for hot-tier metadata and coordination |
|
|
Connection pool size shared by hot-stream and storage-metadata operations |
|
|
Maximum acquisition wait for the shared hot-storage pool |
|
|
Enable fail-fast protection for hot-tier writes after Redis reports max-memory OOM errors |
|
|
How long hot-tier writes fail fast after a Redis max-memory OOM trips the breaker |
|
|
Maximum frequency for repeated "breaker still open" write rejection logs |
|
shared fallback |
Optional shared fallback for the local-storage circuit breaker when |
Remote Storage Configuration
| Property | Default | Description |
|---|---|---|
|
|
Redis keyspace prefix for remote-tier metadata and Delta coordination |
Redis Metrics Configuration
| Property | Default | Description |
|---|---|---|
|
|
Enable Lettuce command latency metrics |
|
|
Enable histogram buckets for percentiles |
|
|
Track metrics per connection vs per host |
|
|
Maximum expected latency for histograms |
|
|
Minimum expected latency for histograms |
Storage Configuration
Remote Tier
Cold storage is enabled automatically when korvet.storage.remote.path is configured.
| Property | Default | Description |
|---|---|---|
|
none |
Cold storage base path (e.g., |
|
none |
AWS region (required for S3) |
|
none |
Custom S3 endpoint (for MinIO or LocalStack) |
|
|
Credential type: |
|
|
Maximum records written per archive write |
|
|
Archive a stream once its oldest unarchived message reaches this age |
|
|
How often the planner scans Redis streams for archival work |
|
|
How long a claimed pending archival entry may remain idle before the archiver tries to reclaim it |
|
|
Maximum number of streams this process archives concurrently |
|
|
Maximum records written into a single Parquet file before the writer rolls over |
|
|
Target payload budget for each Parquet file so cold storage defaults avoid undersized files |
See Cold Storage Configuration for complete configuration details and S3 performance tuning.
Default Configuration
The following shows the default application.yml configuration file:
Unresolved include directive in modules/ROOT/pages/reference/configuration.adoc - include::../../examples/application.yml[]Environment Variables
All configuration can be set via environment variables using Spring Boot’s relaxed binding:
# Broker Configuration
export KORVET_BROKER_HOST=0.0.0.0
export KORVET_BROKER_PORT=9092
export KORVET_BROKER_BROKER_ID=0
export KORVET_BROKER_KEYSPACE=korvet
export KORVET_BROKER_ADVERTISED_HOST=localhost
export KORVET_BROKER_ADVERTISED_PORT=9092
# Thread pool configuration
export KORVET_BROKER_BOSS_THREADS=1
export KORVET_BROKER_WORKER_THREADS=0 # 0 = use Netty default (2 * CPU cores)
# Request size limits
export KORVET_BROKER_MAX_REQUEST_SIZE=100MB
export KORVET_BROKER_PARTITION_MAX_BYTES=1MB
# Fetch Configuration
export KORVET_BROKER_FETCH_MAX_WAIT=500ms # Maximum time to wait for fetch requests when no data is available
# Offset Encoding
# Number of bits for sequence number in offset encoding (1-16)
# Default: 10 (supports up to ~1 million messages/second per partition)
# See xref:quick-start/configuration.adoc#_offset_encoding_configuration[Offset Encoding Configuration] for details
export KORVET_TOPICS_DEFAULTS_OFFSET_SEQUENCE_BITS=10
# Topic Configuration
export KORVET_TOPICS_AUTO_CREATE=true
export KORVET_TOPICS_DEFAULTS_PARTITIONS=1
export KORVET_TOPICS_DEFAULTS_RETENTION_TIME=7d
export KORVET_TOPICS_DEFAULTS_RETENTION_BYTES=-1 # -1 = unlimited
export KORVET_TOPICS_DEFAULTS_COMPRESSION=none # none, gzip, snappy, lz4, zstd
export KORVET_TOPICS_DEFAULTS_STORAGE_COMPRESSION=lz4
export KORVET_TOPICS_DEFAULTS_VALUE_TYPE=auto
export KORVET_TOPICS_DEFAULTS_REMOTE_STORAGE_ENABLED=false
# Consumer Group Configuration
export KORVET_BROKER_REBALANCE_DELAY=3s # Delay before completing rebalance (default: 3s)
# Runtime services (optional)
# Defaults to broker,archiver. Without korvet.storage.remote.path, archiver is skipped.
export KORVET_SERVICES=broker,archiver
# Backpressure Configuration
export KORVET_BROKER_MAX_PENDING_BYTES=100MB # Max pending bytes before backpressure (default: 100MB)
export KORVET_BROKER_RESUME_PENDING_BYTES=50MB # Resume threshold after backpressure (default: 50MB)
# TLS Configuration (optional)
export KORVET_BROKER_TLS=true
export KORVET_BROKER_CERT_FILE=/path/to/server.crt
export KORVET_BROKER_KEY_FILE=/path/to/server.key
export KORVET_BROKER_KEY_PASSWORD=secret
export KORVET_BROKER_TRUST_CERT_FILE=/path/to/ca.crt
export KORVET_BROKER_CLIENT_AUTH_REQUIRED=false
# Redis Connection
export KORVET_REDIS_URI=redis://localhost:6379
export KORVET_REDIS_HOST=localhost
export KORVET_REDIS_PORT=6379
export KORVET_REDIS_USERNAME=default
export KORVET_REDIS_PASSWORD=secret
export KORVET_REDIS_CLUSTER=false
export KORVET_REDIS_TIMEOUT=5s # Connection timeout (optional)
export KORVET_REDIS_POOL_SIZE=8 # Connection pool size (default: 8)
export KORVET_REDIS_IO_THREAD_POOL_SIZE=4 # Lettuce I/O thread pool size (default: number of CPUs)
# Redis Metrics (optional)
# Enables Lettuce command latency metrics published to Micrometer
export KORVET_REDIS_METRICS_ENABLED=false # Default: false
export KORVET_REDIS_METRICS_HISTOGRAM=false # Default: false
export KORVET_REDIS_METRICS_LOCAL_DISTINCTION=false # Default: false
export KORVET_REDIS_METRICS_MAX_LATENCY=5m # Default: 5m
export KORVET_REDIS_METRICS_MIN_LATENCY=1ms # Default: 1ms
# Remote Storage (optional) - enabled when path is set
export KORVET_STORAGE_REMOTE_PATH=s3a://my-bucket/korvet/delta
export KORVET_STORAGE_REMOTE_S3_REGION=us-east-1
export KORVET_STORAGE_REMOTE_S3_ENDPOINT=http://localhost:4566 # Optional: for LocalStack/MinIO
export KORVET_STORAGE_REMOTE_S3_CREDENTIALS_TYPE=iam # or 'static'
export KORVET_STORAGE_REMOTE_S3_CREDENTIALS_ROLE_ARN=arn:aws:iam::123456789012:role/KorvetReaderRole
# For static credentials:
export KORVET_STORAGE_REMOTE_S3_CREDENTIALS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export KORVET_STORAGE_REMOTE_S3_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# S3 Performance Tuning (optional)
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_MAX_CONNECTIONS=96 # Default: 96
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_MAX_THREADS=64 # Default: 64
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_READAHEAD_RANGE=256K # Default: 256K
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_INPUT_POLICY=normal # normal, sequential, random
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_FAST_UPLOAD=true # Default: true
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_FAST_UPLOAD_BUFFER=disk # disk, bytebuffer, array
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_FAST_UPLOAD_ACTIVE_BLOCKS=8 # Default: 8
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_BLOCK_SIZE=64M # Default: 64M
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_MULTIPART_SIZE=64M # Default: 64M
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_CONNECTION_TIMEOUT=15000 # ms, Default: 15000
export KORVET_STORAGE_REMOTE_S3_PERFORMANCE_SOCKET_TIMEOUT=60000 # ms, Default: 60000
# Cold tier archiver configuration (optional, used when storage path is set)
export KORVET_ARCHIVER_BATCH_SIZE=100000 # Default: 100000
export KORVET_ARCHIVER_LAG_MESSAGES_THRESHOLD=100000 # Default: 100000
export KORVET_ARCHIVER_LAG_DURATION_THRESHOLD=5m # Default: 5m
export KORVET_ARCHIVER_INTERVAL=5s # Default: 5s
export KORVET_ARCHIVER_WORKERS=1 # Default: 1
export KORVET_ARCHIVER_FILES_PER_COMMIT=10 # Default: 10
# Logging
export LOGGING_LEVEL_COM_REDIS_KORVET=INFO
