This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.19!

Resilience & Chaos Testing

How Korvet behaves under infrastructure failures, the automated chaos test suite that validates this behaviour, and a runbook for operators responding to common failure scenarios.

Resilience model

Korvet is a stateless broker: all durable state (topics, messages, consumer offsets, producer IDs) lives in Redis. A Korvet process holds only in-flight request state and cached metadata. Two consequences follow:

Broker restarts are cheap and lossless. A restarted broker rebuilds its view from Redis, so any record that was acknowledged before the restart remains readable.
Korvet’s availability tracks Redis. When Redis is unreachable, Korvet cannot durably write or read, so it fails affected requests rather than acknowledging data it cannot persist. This is by design: an acknowledgement always means the data is in Redis — on the master; see the durability model below for what happens to acknowledged data when the master itself fails over.

Graceful shutdown

When a broker shuts down (e.g., during a rolling update or pod restart), it performs a graceful drain before closing connections:

New requests are rejected with a retriable BROKER_NOT_AVAILABLE error, signaling clients to discover and connect to other brokers.
Parked long-poll fetches complete immediately with empty successful responses, rather than forcing clients to wait up to 30 seconds for the max_wait timeout.
In-flight requests are allowed to finish — the broker waits up to korvet.broker.drain-timeout (default 5s) for produces and other active requests to complete before closing the transport.

This ensures clients receive clean responses instead of abrupt connection drops, reducing errors and retries during broker restarts. Configure the drain timeout via korvet.broker.drain-timeout (see Configuration Reference).

Durability model

A produce is acknowledged once the Redis master has accepted the write. What that guarantee is worth after a Redis failure depends entirely on how Redis itself is configured, so make the choice explicitly:

Acknowledgement contract (default). Ack = data written to the Redis master’s memory. Redis replication is asynchronous, so if the master fails and a replica is promoted, any acknowledged writes the replica had not yet received are lost. The failover loss window equals the replication lag — typically milliseconds under normal load, but unbounded if a replica falls behind or the replication link is down.
Replication-acknowledged produce (opt-in). Set korvet.storage.local.write.wait-replicas to the number of replicas that must confirm each write. The broker issues a Redis WAIT after the batch’s stream writes and only then acknowledges the produce, giving Kafka acks=all-like semantics: an acknowledged record survives the failover of the master as long as one of the acknowledging replicas is promoted. If the required replicas do not confirm within korvet.storage.local.write.wait-timeout (default 1s), the produce fails and the client retries. As with a timed-out Kafka acks=all produce, the master may already hold the write, so a retry can duplicate the record. The cost is one extra Redis round trip per write batch plus the replication latency; benchmark against your workload before enabling it broker-wide.
Redis-side settings. Match Redis persistence and replication to the durability you need: with Redis Enterprise, enable replication (and rack awareness across zones); with open-source Redis, run at least one replica and consider AOF (appendfsync everysec or always) to also survive the simultaneous loss of master and replicas. WAIT confirms replica receipt, not disk persistence — combine it with AOF where a total-outage loss of in-memory state is unacceptable.

Durability of consumer offsets and other metadata follows the same contract: they are written to the master and inherit whatever replication/persistence Redis provides.

Multi-region and disaster recovery

A Korvet deployment is anchored to one Redis database: all brokers of a deployment must use the same database, and both should live in the same region (ideally the same AZ — see the latency guidance in the runbook below). For active multi-region deployments, run a broker fleet per region on a Redis Enterprise Active-Active database with region-pinned topics — see Active-Active Databases (CRDB) for the topology and the rules to follow.

High availability is delegated to Redis within a region: run Redis with replication and automatic failover (Redis Sentinel, Redis Enterprise, or a managed equivalent). The failover chaos test validates the broker’s behaviour through exactly that event.

For disaster recovery across regions, use Redis-level mechanisms:

Backup/restore — snapshot the Redis database (RDB backups, or your managed provider’s backup feature) on a schedule that matches your recovery-point objective. To recover, restore the snapshot into a Redis database in the recovery region and start brokers against it. Everything Korvet needs (topics, messages, offsets, producer state) is in that one database; records produced after the snapshot are lost.
Passive replica ("Replica Of") promotion — a Redis Enterprise Replica Of (or plain Redis replica) in a second region can be promoted for a lower recovery-point objective. The replica must be promoted to a writable master and all brokers switched to it — never serve brokers from both regions at once.
Tiered data — segments already offloaded to object storage survive independently of Redis; cross-region S3 replication protects the remote tier. Offloaded data alone is not a full recovery source, because open/unoffloaded segments and consumer state live only in Redis.

Failure modes

Failure Broker behaviour Recovery

Failure	Broker behaviour	Recovery
Redis connection dropped	In-flight and new produce/fetch requests fail; the broker does not acknowledge writes it cannot persist. No partial or phantom acknowledgements.	Lettuce auto-reconnects when Redis returns. The next request succeeds with no operator action. Acknowledged data is intact.
Network latency to Redis	Request latency increases roughly in proportion to the added round-trip cost; throughput drops. No errors while latency stays under client timeouts.	Latency returns to normal when the network recovers. No operator action.
Network partition from Redis	Equivalent to a connection drop for the duration of the partition: affected requests fail.	Broker reconnects automatically when the partition heals and resumes serving traffic.
Broker (pod) restart	Broker drains in-flight requests gracefully before closing connections. Long-poll fetches complete immediately with empty responses, while in-flight produces and other requests are allowed to finish (bounded by `korvet.broker.drain-timeout`, default 5s). Connected clients see clean responses during shutdown and reconnect to the broker (or fail over to another broker discovered via metadata).	Restarted broker serves all data acknowledged before the restart. No data loss.
Redis master failure and replica promotion	While no master is reachable, produce and fetch requests fail — the broker never acknowledges a write it cannot persist. Once a replica is promoted and the stable endpoint (Sentinel, Redis Enterprise endpoint, or Kubernetes service) points at it, the broker’s Lettuce client reconnects and resumes serving without a restart.	Records acknowledged and replicated before the failover remain readable. With the default configuration, writes acknowledged on the old master but not yet replicated are lost — the loss window is the replication lag. With `korvet.storage.local.write.wait-replicas` ≥ 1, acknowledged writes survive promotion of an acknowledging replica. See the durability model above to choose the acknowledgement contract that fits the workload.
Remote storage (S3) outage during tiering	Local (Redis) tier is unaffected: produce and recent-data fetch continue. Offload of cold segments retries with backoff; reads of already-offloaded segments fail until S3 returns.	Offload resumes when S3 recovers; no data is lost because segments are only deleted from the local tier after a successful offload.

Redis connection dropped

In-flight and new produce/fetch requests fail; the broker does not acknowledge writes it cannot persist. No partial or phantom acknowledgements.

Lettuce auto-reconnects when Redis returns. The next request succeeds with no operator action. Acknowledged data is intact.

Network latency to Redis

Request latency increases roughly in proportion to the added round-trip cost; throughput drops. No errors while latency stays under client timeouts.

Latency returns to normal when the network recovers. No operator action.

Network partition from Redis

Equivalent to a connection drop for the duration of the partition: affected requests fail.

Broker reconnects automatically when the partition heals and resumes serving traffic.

Broker (pod) restart

Broker drains in-flight requests gracefully before closing connections. Long-poll fetches complete immediately with empty responses, while in-flight produces and other requests are allowed to finish (bounded by korvet.broker.drain-timeout, default 5s). Connected clients see clean responses during shutdown and reconnect to the broker (or fail over to another broker discovered via metadata).

Restarted broker serves all data acknowledged before the restart. No data loss.

Redis master failure and replica promotion

While no master is reachable, produce and fetch requests fail — the broker never acknowledges a write it cannot persist. Once a replica is promoted and the stable endpoint (Sentinel, Redis Enterprise endpoint, or Kubernetes service) points at it, the broker’s Lettuce client reconnects and resumes serving without a restart.

Records acknowledged and replicated before the failover remain readable. With the default configuration, writes acknowledged on the old master but not yet replicated are lost — the loss window is the replication lag. With korvet.storage.local.write.wait-replicas ≥ 1, acknowledged writes survive promotion of an acknowledging replica. See the durability model above to choose the acknowledgement contract that fits the workload.

Remote storage (S3) outage during tiering

Local (Redis) tier is unaffected: produce and recent-data fetch continue. Offload of cold segments retries with backoff; reads of already-offloaded segments fail until S3 returns.

Offload resumes when S3 recovers; no data is lost because segments are only deleted from the local tier after a successful offload.

Automated chaos test suite

Two suites in module korvet-test cover the failure modes above.

ChaosEngineeringIntegrationTest injects failures into the network path between the broker and Redis using Toxiproxy. Redis runs on a private Docker network and all broker traffic is routed through Toxiproxy, so a test can sever the connection, add latency, or simulate a partition while the broker keeps running. In this suite Redis state is never destroyed, which lets every scenario assert that acknowledged data survives the failure.

RedisFailoverChaosIntegrationTest covers the event the HA story delegates to Redis: loss of the Redis master itself. It runs a master and a streaming replica, routes the broker through a Toxiproxy listener that stands in for the deployment’s stable endpoint, kills the master, promotes the replica (REPLICAOF NO ONE), and repoints the endpoint — the same sequence Sentinel, Redis Enterprise, or an operator performs. Acknowledged writes are fenced onto the replica with WAIT before the failover, mirroring the durability boundary Redis HA actually offers.

Scenario	Assertion
Redis connection cut during produce	Sends during the outage fail fast (graceful degradation); every acknowledged record stays readable; sends after recovery succeed.
Redis connection cut during consume	The consumer does not crash — polls return no records during the outage — and reads every record after recovery.
Latency injection	Added round-trip latency is observable on a produce cycle; the broker returns to normal once the latency is removed.
Network partition then heal	The broker reconnects after the partition heals and serves a full produce/consume cycle.
Broker restart	Data acknowledged before the restart is still readable from a freshly started broker.
Redis master failover with replica promotion	Produce fails while no master is reachable (no phantom acknowledgements); after the replica is promoted and the endpoint repointed, the broker recovers through its existing connection with no restart, and every record acknowledged and replicated before the failover is still readable.

Scenario

Assertion

Redis connection cut during produce

Sends during the outage fail fast (graceful degradation); every acknowledged record stays readable; sends after recovery succeed.

Redis connection cut during consume

The consumer does not crash — polls return no records during the outage — and reads every record after recovery.

Latency injection

Added round-trip latency is observable on a produce cycle; the broker returns to normal once the latency is removed.

Network partition then heal

The broker reconnects after the partition heals and serves a full produce/consume cycle.

Broker restart

Data acknowledged before the restart is still readable from a freshly started broker.

Redis master failover with replica promotion

Produce fails while no master is reachable (no phantom acknowledgements); after the replica is promoted and the endpoint repointed, the broker recovers through its existing connection with no restart, and every record acknowledged and replicated before the failover is still readable.

The chaos tests are tagged @Tag("chaos"). Because they sleep through simulated outages they are slower than regular integration tests, so they are excluded from the default integrationTest suite and run in a dedicated chaosTest task (and its own CI workflow).

Run the suite:

./gradlew chaosTest

The suite requires a Docker daemon (Testcontainers pulls the Redis and Toxiproxy images).

Scenarios validated outside this suite

Two scenarios from the resilience story need an environment Testcontainers cannot provide and are exercised by end-to-end suites instead:

S3 503 errors during tiering — covered by the remote-storage integration tests against a MinIO/S3 endpoint (korvet-storage-tiered-iceberg). The offload path retries on transient 5xx responses and only removes a local segment after a confirmed offload.
Kubernetes pod eviction — covered by deploying to a cluster and deleting the broker pod while a client produces. Because state is in Redis, the client reconnects (to the rescheduled pod or another replica) and no acknowledged data is lost.

Runbook

Redis is unreachable

Symptoms: produce/fetch requests fail or time out; broker logs show Lettuce reconnect attempts; redis_client metrics show connection errors.

Confirm Redis health directly: redis-cli -h <host> -p <port> ping.
Check network reachability from the broker host to Redis (firewall, security groups, DNS).
If Redis is up and reachable, the broker reconnects automatically — no restart needed. Verify recovery by producing a test record.
If Redis is down, restore it. Korvet resumes serving as soon as the connection is re-established. Acknowledged data is intact.

Elevated produce/fetch latency

Symptoms: client-observed latency rises; broker-to-Redis round-trip metrics increase.

Inspect network latency between the broker and Redis.
Check Redis-side load (slow commands, CPU, memory pressure).
Latency clears on its own once the network or Redis recovers; no Korvet action is required. If it persists, scale Redis or move the broker closer to Redis (same AZ/region).

Network partition between broker and Redis

Symptoms: sustained request failures with no Redis-side errors; broker cannot reach Redis.

Treat as "Redis is unreachable" above — the failure mode and recovery are identical.
The broker reconnects automatically when the partition heals; confirm with a test produce.

Broker (pod) restart

Symptoms: clients briefly see BROKER_NOT_AVAILABLE errors as the broker drains, then reconnect; in-flight requests complete gracefully during the drain window.

No data action required — durable state is in Redis.
The broker’s graceful shutdown (configured via korvet.broker.drain-timeout, default 5s) ensures in-flight requests finish and clients receive clean responses rather than abrupt disconnects.
After restart, verify the broker is healthy (/actuator/health) and serving by producing and consuming a test record.
For zero-disruption restarts, run multiple replicas so clients fail over to the remaining brokers via a metadata refresh while one pod restarts (see Kubernetes).

Redis master failover

Symptoms: a burst of failed produce/fetch requests while the master is down, then automatic recovery once a replica is promoted; broker logs show Lettuce reconnecting.

No broker action is required if the failover mechanism (Sentinel, Redis Enterprise, managed Redis) keeps the connection endpoint stable — the broker reconnects and resumes on its own.
If brokers connect to a fixed master address instead of a stable endpoint, update it to the promoted node and restart the brokers.
Verify recovery by producing and consuming a test record.
Expect to lose only records that were acknowledged but not yet replicated at the instant the master died (asynchronous replication) — none, for topics produced with korvet.storage.local.write.wait-replicas ≥ 1 (see the durability model above). Producers using retries/idempotence will have re-sent unacknowledged records.

Remote storage (S3) outage

Symptoms: cold-segment offload stalls; reads of already-offloaded (cold) data fail; recent data still produces and consumes normally.

Confirm the S3 endpoint/credentials and bucket reachability.
Recent data on the local (Redis) tier is unaffected — produce and recent reads continue.
Offload resumes automatically when S3 recovers. No data is lost: segments are removed from the local tier only after a successful offload.
Watch the storage-worker metrics for offload backlog draining once S3 is healthy.

Stranded offload recovery

An offload that is interrupted mid-flight — the storage-worker leader dies between marking a segment OFFLOADING and committing it OFFLOADED, or the remote write fails — leaves the segment in the OFFLOADING state. The storage worker recovers these automatically: at the start of every leader tick, before scanning for offload-eligible segments, it sweeps OFFLOADING segments back to SEALED so the normal offload scan retries them. No operator action is needed, and the retry is safe: a slow ex-leader that still commits loses the versioned compare-and-set, and retried offloads write to collision-free remote filenames. Without this sweep, a stranded segment would be skipped by offload and local retention alike, pinning its Redis memory indefinitely.