This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.20!

Frequently Asked Questions

Common questions about Korvet.

General

What is Korvet?

Korvet is a Kafka-compatible streaming service backed by Redis Streams, with optional tiered storage to an Apache Iceberg table on an object store for long-term archival.

Why use Korvet instead of Kafka?

  • Simpler operations: Brokers are stateless — adding brokers moves no partition data; scale Redis shards when Redis is the bottleneck

  • Redis integration: Leverage existing Redis infrastructure and persistence

  • Low latency: Sub-millisecond read/write performance with Redis Streams

  • Optional cost optimization: Tiered storage to an Apache Iceberg table on an object store for long-term retention

Is Korvet production-ready?

Korvet is in active development. The core features are functional:

  • ✅ Kafka protocol implementation (Produce, Fetch, Consumer Groups, Admin API)

  • ✅ Redis Streams storage with retention policies

  • ✅ Async operations for high throughput

  • ✅ Metrics and observability

  • ✅ Built-in tiered storage to an Apache Iceberg table on S3 (~115k msg/s measured with 4 parallel streams to same-region S3; see the archival benchmark)

Compatibility

Which Kafka clients work with Korvet?

Any Kafka client that supports the Kafka protocol should work. Tested clients include:

  • Java: kafka-clients

  • Python: kafka-python, confluent-kafka-python

  • Go: sarama

  • Node.js: kafkajs

What Kafka features are supported?

Supported:

  • ✅ Produce API

  • ✅ Fetch API

  • ✅ Consumer Groups (JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, OffsetFetch)

  • ✅ Topic metadata

  • ✅ Admin API (CreateTopics, DeleteTopics, DescribeConfigs, DescribeCluster)

  • ✅ Compression (GZIP, SNAPPY, LZ4, ZSTD)

Not supported:

  • ❌ Transactions

  • ❌ Exactly-once semantics (at-least-once delivery only)

Does Korvet work with Redis Enterprise Active-Active (CRDB) databases?

Yes, with a region-pinned topic design: each topic is produced from exactly one region (for example orders-region1, orders-region2), and consumers subscribe to all regional variants through their local brokers. The Active-Active database replicates every region’s topics everywhere for consumption. See Active-Active Databases (CRDB) for the topology and the rules to follow.

Can I migrate from Kafka to Korvet?

Yes. korvet import topics copies topic definitions from an existing Kafka cluster and, optionally, their data (--data, preserving key, value, headers, timestamp and per-partition order). --live keeps tailing the source for a cutover without stopping producers first, and --groups migrates consumer group committed offsets, translated by record timestamp. See Importing Topics from Kafka.

Considerations:

  • The command is a dry run by default; pass --apply to write anything

  • Kafka offsets are not preserved; with --groups, stop consumers first and expect possible redelivery of records sharing the cutover timestamp

  • Transactions are not supported

  • Exactly-once semantics are not supported (at-least-once only)

Performance

What throughput can Korvet handle?

Performance depends on your Redis deployment, broker count and workload. In the daily pipeline saturation benchmark (3 brokers, Redis Enterprise with 24 shards, ~700-byte events):

  • Peak produce: ~640k messages/second, limited by Redis shard CPU rather than the brokers

  • Produce latency: broker-side p99 under 25 ms up to ~530k messages/second

  • Per broker: a single broker tops out around 200k messages/second, so scale brokers horizontally

Throughput on your own hardware can be measured with the built-in korvet bench command, which works against Korvet or any Kafka-compatible broker.

How do I scale Korvet?

Korvet is stateless and can be scaled horizontally:

  1. Run multiple Korvet instances against the same Redis, each with a unique broker ID and a routable advertised address

  2. List several brokers in the client’s bootstrap.servers

  3. Clients discover all brokers via Kafka metadata and spread traffic across partition leaders automatically — no load balancer is required

See Partition Leadership for how traffic is distributed.

Brokers scale throughput only until Redis becomes the bottleneck; past that point, add Redis shards (see the deployment sizing table).

What are the resource requirements?

Minimum:

  • CPU: 1 core

  • Memory: 512MB

  • Redis: 8.6+ with the RedisJSON (JSON.*) commands

Recommended for production:

  • CPU: 2-4 cores

  • Memory: 2-4GB

  • Redis: Cluster or Enterprise for HA

Storage

How long can I keep data in Redis?

As long as your Redis instance has capacity. Configure retention policies (the Kafka-compatible topic config keys retention.ms and retention.bytes, or the retention-time/retention-bytes keys on a korvet.topics pattern) to trim old messages. Trimming is enforced asynchronously by a leader-locked background storage worker (via Redis XTRIM), not at write time.

What happens when Redis is full?

You have several options:

  1. Increase Redis memory: Scale up your Redis instance

  2. Configure retention: Set retention.ms or retention.bytes to automatically trim old messages

  3. Enable tiered storage: Configure the optional Apache Iceberg remote tier to move old messages to object storage

  4. Redis eviction policies: Configure Redis eviction (e.g., allkeys-lru) as a last resort

Can I query archived data directly from the object store?

Yes. The remote tier is a standard Apache Iceberg table (Parquet data files plus Iceberg metadata and manifests), so external query engines such as Spark, Trino, Athena, and DuckDB can read it directly. Korvet clients also read archived data transparently through the normal Kafka fetch API — the broker reads from the remote tier on their behalf.

See Remote Storage for configuration details.

Which storage compression codec should I use?

Records are stored in a single envelope format, so the choice is the at-rest codec:

  • Structured records (JSON, logs, metrics, events) up to a few KB: use zstd. For small records, its per-topic trained dictionary (on by default for zstd topics) recovers the cross-record redundancy that per-record compression cannot see.

  • Large compressible values (roughly 4 KB and above) on CPU-constrained brokers: lz4 or snappy cost the least CPU per byte; prefer zstd when storage cost matters more than broker CPU.

  • Already-compressed or high-entropy payloads (images, encrypted data): none.

  • Values that non-Kafka clients read directly from Redis: none, since every codec requires readers to decompress.

See Choosing a Codec for the full guidance.

How much memory can storage compression save?

On structured records of 512 B to 6 KB, zstd with its default per-topic dictionary measured a 3.5x to 5x reduction in Redis memory per record (see Measured Memory Savings). The dictionary matters most for small records: at ~512 B plain zstd manages only ~1.3x, while the dictionary reaches 3.5x.

Actual ratios depend on your data. Heterogeneous topics (mixed schemas, or payloads dominated by unique IDs or base64 blobs) land lower, and high-entropy data barely compresses at all; measure on your own data before capacity planning.

Operations

How do I monitor Korvet?

Korvet exposes metrics via Prometheus and health checks via Spring Boot Actuator. See Monitoring.

How do I troubleshoot issues?

  1. Check logs (JSON format by default)

  2. Review metrics in Prometheus

  3. Check health endpoints

  4. See Troubleshooting guide

Can I run Korvet in Kubernetes?

Yes! See Deployment guide for Kubernetes manifests.

Development

How do I contribute to Korvet?

See the GitHub repository for contribution guidelines.

Where can I report bugs?

File an issue at https://github.com/redis-field-engineering/korvet-dist/issues

Is there a roadmap?

Check the GitHub issues and project boards for planned features.

Next Steps