This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.19!

Deployment

This guide covers deploying Korvet in production environments.

Deployment shapes

A Korvet JVM runs the broker (Kafka listener and group coordinator) and the leader-locked storage worker by default. The storage worker rolls eligible segments, offloads sealed segments, and enforces local and remote retention.

Disable a component explicitly only when a deployment needs a single-role JVM.

Shape Configuration Components started

All-in-one (default)

no component flags required

broker + storage worker

Broker-only pod

KORVET_STORAGE_WORKER_ENABLED=false

broker

Storage-worker pod

KORVET_BROKER_ENABLED=false

storage worker

Single-role pod

set one of KORVET_BROKER_ENABLED / KORVET_STORAGE_WORKER_ENABLED to false

the selected role

Custom

set KORVET_BROKER_ENABLED and/or KORVET_STORAGE_WORKER_ENABLED

the components whose flag is true

korvet.storage.remote.path makes the cold tier available. Without it, Korvet runs local-only on Redis Streams.

Docker Deployment

Single Instance

docker run -d \
  --name korvet \
  -p 9092:9092 \
  -e JAVA_OPTS="-Xms2g -Xmx2g -XX:MaxDirectMemorySize=512m" \
  -e KORVET_REDIS_URI=redis://redis.example.com:6379 \
  -e KORVET_REDIS_USERNAME=default \
  -e KORVET_REDIS_PASSWORD=${REDIS_PASSWORD} \
  redisfield/korvet:latest server

Use JAVA_OPTS to pass heap settings and additional JVM flags to the container at startup.

Tune JVM memory through JAVA_OPTS, not JAVA_TOOL_OPTIONS. The broker bakes -XX:MaxDirectMemorySize=512m into its default launch arguments, and those defaults are placed after JAVA_TOOL_OPTIONS on the command line — so a MaxDirectMemorySize set via JAVA_TOOL_OPTIONS is silently overridden. Only JAVA_OPTS (applied last) overrides the baked default.

Direct buffer memory lives outside the heap, so size the container so that memory limit >= -Xmx + MaxDirectMemorySize + ~512m native/metaspace/stack overhead. With -Xmx2g and MaxDirectMemorySize=512m, use a container memory limit of at least 4Gi. Increase MaxDirectMemorySize for high fan-in workloads (many concurrent consumers, e.g. Spark/Flink) and raise the container limit to match.

Docker Compose

services:
  redis:
    image: redis:8.6
    command: redis-server --requirepass ${REDIS_PASSWORD}
    ports:
      - "6379:6379"

  korvet:
    image: redisfield/korvet:latest
    command: server
    ports:
      - "9092:9092"
    environment:
      JAVA_OPTS: "-Xms2g -Xmx2g -XX:MaxDirectMemorySize=512m"
      KORVET_REDIS_URI: redis://redis:6379
      KORVET_REDIS_PASSWORD: ${REDIS_PASSWORD}
    depends_on:
      - redis

Multiple Listeners

When in-network containers (Kafka UI, sidecar consumers) and host-side processes (an IDE-launched producer, a Python script on the laptop) both need to reach the same broker, a single advertised endpoint cannot serve both networks. Additional named listeners (korvet.broker.listeners) solve this the same way Kafka’s advertised.listeners does: each listener binds its own port and advertises its own endpoint, and clients receive the endpoint of the listener they connected through.

services:
  korvet:
    image: redisfield/korvet:latest
    command: server
    ports:
      - "29092:29092"   # host-side clients
    environment:
      KORVET_REDIS_URI: redis://redis:6379
      # Primary listener: in-network clients connect to korvet:9092 (Docker DNS)
      KORVET_BROKER_ADVERTISED_HOST: korvet
      KORVET_BROKER_ADVERTISED_PORT: "9092"
      # Additional listener: host clients connect to localhost:29092 (port-forwarded)
      KORVET_BROKER_LISTENERS_0_NAME: host
      KORVET_BROKER_LISTENERS_0_PORT: "29092"
      KORVET_BROKER_LISTENERS_0_ADVERTISED_HOST: localhost
      KORVET_BROKER_LISTENERS_0_ADVERTISED_PORT: "29092"

In-network clients bootstrap against korvet:9092; host clients bootstrap against localhost:29092. Each listener falls back like the primary one when its advertised endpoint is unset: the advertised host defaults to the bind host (or localhost when binding to 0.0.0.0), and the advertised port defaults to the bound port.

Listener names must be unique, and broker is reserved for the primary listener. In a multi-broker deployment give every broker the same listener names: each broker publishes its per-listener advertised endpoints through the broker registry, so metadata responses resolve every broker’s endpoint for the listener a client connected through (brokers that do not serve a listener fall back to their primary endpoint). TLS and SASL settings are broker-wide and apply to every listener.

Kubernetes Deployment

For single-broker or simple deployments, use a standard Deployment. For multi-broker clusters with proper broker discovery, see Multi-Broker Deployment.

Simple Deployment (Single Broker)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: korvet
spec:
  replicas: 1  # Single broker
  selector:
    matchLabels:
      app: korvet
  template:
    metadata:
      labels:
        app: korvet
    spec:
      containers:
      - name: korvet
        image: redisfield/korvet:latest
        args: ["server"]
        ports:
        - containerPort: 9092
        env:
        - name: KORVET_REDIS_URI
          value: redis://redis-service:6379
        - name: KORVET_BROKER_HOST
          value: 0.0.0.0
        - name: KORVET_BROKER_PORT
          value: "9092"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2000m"
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
The readiness probe includes Redis connectivity: a broker that cannot reach Redis reports not-ready and is removed from load-balancer rotation until connectivity is restored. Liveness only reflects the application itself, so a Redis outage does not restart pods.

Service Manifest

apiVersion: v1
kind: Service
metadata:
  name: korvet-service
spec:
  selector:
    app: korvet
  ports:
  - protocol: TCP
    port: 9092
    targetPort: 9092
  type: LoadBalancer
For multi-broker deployments with high availability, use a StatefulSet instead. See Kubernetes StatefulSet.

Multi-Broker Deployment

Korvet supports multi-broker deployments where multiple Korvet instances share the same Redis backend. This provides high availability and load distribution while maintaining full data consistency.

Architecture Overview

In a multi-broker deployment:

  • All brokers connect to the same Redis instance or cluster

  • Brokers automatically discover each other via the Broker Registry stored in Redis

  • Kafka clients can bootstrap against any broker and receive metadata listing all live brokers in the cluster

  • Partition leadership is spread across brokers (see Partition Leadership), so clients distribute produce and fetch traffic through the standard Kafka protocol — no load balancer is required

  • All message data lives in shared Redis, so a partition served by one broker today can be served by another immediately after a leadership change

                 ┌───────────────────────┐
                 │     Kafka Clients     │
                 │ bootstrap.servers=    │
                 │ korvet-0,korvet-1,... │
                 └───────────┬───────────┘
                             │  (direct connections,
                             │   routed by metadata)
        ┌────────────────────┼────────────────────┐
        │                    │                    │
        ▼                    ▼                    ▼
┌───────────────┐   ┌───────────────┐   ┌───────────────┐
│   Korvet 0    │   │   Korvet 1    │   │   Korvet 2    │
│  id=0         │   │  id=1         │   │  id=2         │
│  port=9092    │   │  port=9092    │   │  port=9092    │
└───────┬───────┘   └───────┬───────┘   └───────┬───────┘
        │                   │                   │
        └───────────────────┼───────────────────┘
                            │
                            ▼
                ┌───────────────────────┐
                │        Redis          │
                │  (Streams + Registry) │
                └───────────────────────┘

Broker Configuration

Each broker requires a unique id and proper network configuration:

korvet:
  namespace: korvet         # Must be the same across all brokers

  broker:
    id: 0                  # Unique ID for this broker (0, 1, 2, etc.)
    host: 0.0.0.0          # Listen on all interfaces
    port: 9092             # Kafka protocol port
    advertised-host: korvet-0.korvet.default.svc.cluster.local  # Hostname clients use
    advertised-port: 9092  # Port clients use

  redis:
    uri: redis://redis:6379  # Same Redis for all brokers
All brokers in a cluster must use the same korvet.namespace and connect to the same Redis instance.

Broker Discovery

Korvet uses a Broker Registry stored in Redis for automatic broker discovery:

  • Each broker registers itself on startup with its ID, host, and port

  • Brokers send periodic heartbeats (every 10 seconds by default)

  • Entries are considered stale 30 seconds after their last heartbeat and are then ignored

  • Kafka clients receive all live registered brokers in metadata responses

  • Registration fails fast when the broker id is already held by a live broker at a different endpoint: the starting broker stops instead of silently corrupting registry membership and coordinator/leadership hashing, and it does not unregister the conflicting live broker. Re-registering the same endpoint (for example after a pod restart) and taking over a stale entry are both allowed.

Redis keys used by the registry:

korvet:broker:nodes   # Single hash: field = broker id, value = host/port/rack + heartbeat timestamp
There is no Redis TTL on the key. Staleness is determined client-side by comparing each entry’s heartbeat timestamp against the 30-second threshold.

Partition Leadership

Although every partition is stored in shared Redis and any broker is technically capable of serving it, each topic partition is assigned a single leader broker, and clients spread traffic across the cluster by following leadership through the standard Kafka protocol:

  • The leader for a partition is selected by rendezvous hashing of the topic name and partition number over the live broker set. The hash is stable across JVMs, so every broker independently computes the same assignment and no coordination is needed.

  • Metadata responses advertise that leader for each partition (with all live brokers listed as replicas and in-sync replicas), and Kafka clients send produce and fetch requests directly to the leader — this is how load is distributed across brokers.

  • Leadership is enforced: a produce or fetch request for a partition led by another broker is rejected with NOT_LEADER_OR_FOLLOWER, which prompts the client to refresh its metadata and reroute.

Rendezvous hashing localizes membership-change churn: when a broker joins or leaves, only the partitions that broker wins or loses change leader. Clients handle the change transparently with a metadata refresh, and no data moves — all data stays in shared Redis — but expect a brief burst of NOT_LEADER_OR_FOLLOWER retries for the affected partitions during scaling or failover.

Docker Compose Example

services:
  redis:
    image: redis:8.6
    ports:
      - "6379:6379"

  korvet-0:
    image: redisfield/korvet:latest
    command: server
    ports:
      - "9092:9092"
    environment:
      KORVET_BROKER_ID: 0
      KORVET_BROKER_HOST: 0.0.0.0
      KORVET_BROKER_ADVERTISED_HOST: localhost
      KORVET_BROKER_ADVERTISED_PORT: 9092
      KORVET_REDIS_URI: redis://redis:6379

  korvet-1:
    image: redisfield/korvet:latest
    command: server
    ports:
      - "9093:9092"
    environment:
      KORVET_BROKER_ID: 1
      KORVET_BROKER_HOST: 0.0.0.0
      KORVET_BROKER_ADVERTISED_HOST: localhost
      KORVET_BROKER_ADVERTISED_PORT: 9093
      KORVET_REDIS_URI: redis://redis:6379

  korvet-2:
    image: redisfield/korvet:latest
    command: server
    ports:
      - "9094:9092"
    environment:
      KORVET_BROKER_ID: 2
      KORVET_BROKER_HOST: 0.0.0.0
      KORVET_BROKER_ADVERTISED_HOST: localhost
      KORVET_BROKER_ADVERTISED_PORT: 9094
      KORVET_REDIS_URI: redis://redis:6379

Clients can connect using multiple bootstrap servers:

kafka-console-producer --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --topic test

Kubernetes StatefulSet

For Kubernetes, use a StatefulSet to ensure each broker gets a unique, stable identity. The StatefulSet provides:

  • Stable network identity: Each pod gets a predictable DNS name (korvet-0, korvet-1, etc.)

  • Ordered deployment: Pods are created sequentially, ensuring broker registration order

  • Stable storage: PersistentVolumeClaims are retained across pod restarts (if needed)

Complete Kubernetes Manifests

# ConfigMap for shared configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: korvet-common
data:
  KORVET_BROKER_HOST: "0.0.0.0"
  KORVET_BROKER_PORT: "9092"
  KORVET_NAMESPACE: "korvet"
  KORVET_BROKER_REBALANCE_DELAY: "5s"  # Allow time for consumers to join in K8s
---
# Headless service for StatefulSet DNS
apiVersion: v1
kind: Service
metadata:
  name: korvet
  labels:
    app: korvet
spec:
  clusterIP: None
  selector:
    app: korvet
  ports:
  - port: 9092
    name: kafka
  - port: 8080
    name: actuator
---
# LoadBalancer service for external access
apiVersion: v1
kind: Service
metadata:
  name: korvet-lb
spec:
  type: LoadBalancer
  selector:
    app: korvet
  ports:
  - port: 9092
    targetPort: 9092
    name: kafka
---
# StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: korvet
spec:
  serviceName: korvet
  replicas: 3
  podManagementPolicy: Parallel  # Start all pods simultaneously
  selector:
    matchLabels:
      app: korvet
  template:
    metadata:
      labels:
        app: korvet
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      terminationGracePeriodSeconds: 30
      initContainers:
      # Extract broker ID from pod name (korvet-0 -> 0, korvet-1 -> 1, etc.)
      - name: init-broker-id
        image: busybox:1.36
        command:
        - sh
        - -c
        - |
          ORDINAL=${HOSTNAME##*-}
          echo "KORVET_BROKER_ID=${ORDINAL}" > /config/broker.env
          echo "KORVET_BROKER_ADVERTISED_HOST=${HOSTNAME}.korvet.${NAMESPACE}.svc.cluster.local" >> /config/broker.env
          echo "Broker ID: ${ORDINAL}, Advertised Host: ${HOSTNAME}.korvet.${NAMESPACE}.svc.cluster.local"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: config-volume
          mountPath: /config
      containers:
      - name: korvet
        image: redisfield/korvet:latest
        command:
        - sh
        - -c
        - |
          # Source the broker-specific config
          export $(cat /config/broker.env | xargs)
          # Start the application
          exec java --sun-misc-unsafe-memory-access=allow -jar /app/korvet.jar
        ports:
        - containerPort: 9092
          name: kafka
        - containerPort: 8080
          name: actuator
        envFrom:
        - configMapRef:
            name: korvet-common
        - secretRef:
            name: korvet-redis-credentials
            optional: true
        env:
        - name: KORVET_REDIS_URI
          value: redis://redis:6379
        - name: KORVET_BROKER_ADVERTISED_PORT
          value: "9092"
        # JVM tuning for containers. This manifest launches `java -jar` directly (no distribution
        # start script), so there are no baked applicationDefaultJvmArgs to override and JAVA_OPTS
        # would not be expanded by the command above — JAVA_TOOL_OPTIONS is auto-applied by the JVM
        # and is the right place here. Keep -Xmx + MaxDirectMemorySize below the container limit.
        - name: JAVA_TOOL_OPTIONS
          value: "-Xms2g -Xmx2g -XX:+UseG1GC -XX:MaxDirectMemorySize=512m --sun-misc-unsafe-memory-access=allow"
        resources:
          requests:
            memory: "2Gi"
            cpu: "500m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        volumeMounts:
        - name: config-volume
          mountPath: /config
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 3
        startupProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 30
      volumes:
      - name: config-volume
        emptyDir: {}

Redis Credentials Secret

If Redis requires authentication, create a secret:

kubectl create secret generic korvet-redis-credentials \
  --from-literal=KORVET_REDIS_PASSWORD=your-password

Scaling the Cluster

Scale up or down with:

# Scale to 5 brokers
kubectl scale statefulset korvet --replicas=5

# Scale down to 3 brokers
kubectl scale statefulset korvet --replicas=3

When scaling down, brokers are removed in reverse order (highest ID first). The broker registry stops advertising an entry once its last heartbeat is older than 30 seconds.

Pod Disruption Budget

For high availability, configure a PodDisruptionBudget:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: korvet-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: korvet

External Access

For clients outside the Kubernetes cluster, you have several options:

Option 1: LoadBalancer Service (shown above)

Clients connect to the LoadBalancer IP. All traffic is distributed across brokers.

Option 2: NodePort Service

apiVersion: v1
kind: Service
metadata:
  name: korvet-nodeport
spec:
  type: NodePort
  selector:
    app: korvet
  ports:
  - port: 9092
    targetPort: 9092
    nodePort: 30092

Option 3: Ingress with TCP support (e.g., NGINX Ingress Controller)

Configure TCP services in the ingress controller’s ConfigMap.

Monitoring in Kubernetes

Korvet exposes Prometheus metrics at /actuator/prometheus. With the annotations in the StatefulSet, Prometheus will automatically scrape metrics. When korvet.admin.security-enabled=true (the default) the endpoint requires HTTP Basic admin credentials — configure the scrape’s basic_auth, or set korvet.admin.anonymous-prometheus=true to keep it open.

For Grafana dashboards, query metrics like:

  • korvet_broker_produce_seconds_count - Total produce requests

  • korvet_broker_fetch_seconds_count - Total fetch requests

  • korvet_broker_request_seconds_count - Kafka API request rate, by api_key and result

  • korvet_broker_backpressure_connections - Connections currently under backpressure

Consumer Group Coordination

In multi-broker deployments, consumer group coordination is handled specially:

  • The group coordinator is selected using rendezvous hashing based on the group ID, the same scheme used for partition leadership

  • Clients are directed to the correct coordinator via the FindCoordinator response

  • Committed offsets are stored in Redis, so any surviving broker can serve offset commits/fetches

Partition leadership uses the same rendezvous hashing over the live broker set, so when a broker joins or leaves, only the partitions and groups owned by that broker move to a new owner. The rest of the cluster keeps its leaders and coordinators, avoiding cluster-wide NOT_LEADER_OR_FOLLOWER churn and mass rebalances on every membership change.

Broker Failover Semantics

A broker crash or restart is detected when its registry heartbeat goes stale (30 seconds by default). Failover is automatic but not fully transparent:

  • Messages and committed offsets live in Redis and survive: no data is lost, and consumers resume from their last committed offset on the new coordinator.

  • Consumer group sessions and assignments are held in the coordinator broker’s memory. Groups coordinated by the failed broker go through a full re-join and rebalance on the broker that takes over. Groups coordinated by surviving brokers are unaffected.

  • Idempotent producer sequence state is held in the partition leader’s memory. After a leadership move, producer deduplication state resets: retries of batches sent before the failover are no longer detected as duplicates and may be written twice. Producers writing to partitions whose leadership did not move are unaffected.

Messages delivered between the last committed offset and the crash are re-delivered after the rebalance, which matches Kafka’s at-least-once semantics.

Best Practices

Broker IDs

Use sequential IDs starting from 0 (0, 1, 2, …​). Each broker must have a unique ID.

Advertised Listeners

Always configure advertised-host and advertised-port to the address clients should use to connect. This is especially important in Docker/Kubernetes where internal and external addresses differ.

Load Balancing

A load balancer is not required. Kafka clients discover all brokers from metadata and connect to partition leaders directly, so traffic is distributed by the protocol itself (see Partition Leadership). A TCP load balancer (not HTTP) is only useful as a stable bootstrap address; each broker must still advertise its own routable address, not the load balancer’s. Do not put all client traffic through a single address advertised by every broker — leadership routing depends on clients reaching specific brokers.

Bootstrap Servers

Instead of a load balancer, simply list multiple brokers in bootstrap.servers for fault-tolerant bootstrapping:

bootstrap.servers=korvet-0:9092,korvet-1:9092,korvet-2:9092
Redis High Availability

For production, use a highly available Redis deployment, such as Redis Enterprise, to ensure the storage layer is also highly available.

High Availability

For production deployments:

  • Multiple instances: Run 3+ Korvet instances for redundancy

  • Redis HA: Use a highly available Redis deployment, such as Redis Enterprise, for HA

  • Health checks: Configure liveness and readiness probes

  • Graceful shutdown: Allow time for in-flight requests to complete

Active-Active Databases (CRDB)

Korvet can run on a Redis Enterprise Active-Active database with a region-pinned topic design: every topic is written from exactly one region, and consumers everywhere read all regions' topics. This is the same pattern used for active/active Kafka with MirrorMaker 2 (cluster-prefixed remote topics), with the Active-Active database providing the cross-region replication.

The one rule that everything else follows from: a topic must only ever be produced to through brokers in a single region. Kafka requires each partition to be a single totally ordered log; an Active-Active database merges per-region stream logs, so writing the same topic from two regions breaks offset ordering (visible as non_monotonic_offset produce or fetch errors, and messages that consumers never see). Writing each topic from one region keeps its log single-source and totally ordered, while the database replicates it to all other regions for consumption.

Topology

Active-Active deployment with region-pinned topics

  • Run an independent Korvet broker fleet in each region, connected to that region’s Active-Active instance.

  • Suffix topic names with the owning region, for example orders-region1 and orders-region2.

  • Producers write to their local region’s topic through their local brokers.

  • Consumers subscribe to all regional variants — for example with a pattern subscription on orders-.* — through their local brokers. Topics created in other regions replicate automatically and are consumable locally, including consumer-group reads.

Rules and caveats

  • One writing region per topic. Enforcement is operational today: nothing stops a misconfigured producer from writing a topic through the wrong region’s brokers, so guard this with client configuration and ACLs.

  • One region per consumer group name. Committed offsets replicate between regions; the same group name consuming in two regions would overwrite each other’s offsets. Suffix group names by region, or pin each group to one region’s brokers.

  • Per-key ordering is per-region. A key produced in two regions lands in two topics, so its messages have no total order. Applications that need per-key ordering must route each key to a single region.

  • Group failover is at-least-once. In-flight delivery state (pending-entry lists, acknowledgements) is region-local. A consumer group redirected to another region resumes from its last replicated committed offset and re-reads anything after it.

  • Run the storage worker for a topic only in its owning region. Retention and segment management for a topic should not run in two regions concurrently; disable the storage worker on fleets that do not own topics, or scope each region’s worker to its own topics.

  • Region failover: if a region becomes unavailable, its producers reconnect to a surviving region’s brokers and write to that region’s topics — never to the failed region’s topics. The failed region’s topics stop growing and their unreplicated tail is delivered, in order, when the region returns.

Scaling

Korvet can be scaled horizontally:

  • Stateless: Each instance shares state via Redis

  • Protocol-native routing: Clients discover brokers and partition leaders via Kafka metadata; no load balancer is needed (see Partition Leadership)

  • Add brokers: Simply start new instances with unique broker IDs and routable advertised addresses; they register themselves and take over a share of partition leadership

Adding brokers scales connection handling and broker-side CPU (for example compression), but all brokers share one Redis backend — once Redis is the bottleneck, scale Redis shards rather than adding brokers.

Next Steps