For the latest stable version, please use Korvet 0.12.5!

Frequently Asked Questions

Common questions about Korvet.

General

What is Korvet?

Korvet is a Kafka-compatible streaming service backed by Redis Streams, with optional tiered storage to Delta Lake for long-term archival.

Why use Korvet instead of Kafka?

  • Simpler operations: No ZooKeeper, no partition rebalancing complexity

  • Redis integration: Leverage existing Redis infrastructure and persistence

  • Low latency: Sub-millisecond read/write performance with Redis Streams

  • Optional cost optimization: Tiered storage to Delta Lake for long-term retention

  • Analytics-ready: Optional Delta Lake integration for Spark/Flink analytics

Is Korvet production-ready?

Korvet is in active development. The core features are functional:

  • ✅ Kafka protocol implementation (Produce, Fetch, Consumer Groups, Admin API)

  • ✅ Redis Streams storage with retention policies

  • ✅ Async operations for high throughput

  • ✅ Metrics and observability

  • 🚧 Optional Delta Lake tiering (requires separate Flink job)

Compatibility

Which Kafka clients work with Korvet?

Any Kafka client that supports the Kafka protocol should work. Tested clients include:

  • Java: kafka-clients

  • Python: kafka-python, confluent-kafka-python

  • Go: sarama

  • Node.js: kafkajs

What Kafka features are supported?

Supported:

  • ✅ Produce API

  • ✅ Fetch API

  • ✅ Consumer Groups (JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, OffsetFetch)

  • ✅ Topic metadata

  • ✅ Admin API (CreateTopics, DeleteTopics, DescribeConfigs, DescribeCluster)

  • ✅ Compression (GZIP, SNAPPY, LZ4, ZSTD)

Not supported:

  • ❌ Transactions

  • ❌ Exactly-once semantics (at-least-once delivery only)

Can I migrate from Kafka to Korvet?

Yes, but with some considerations:

  • Transactions are not supported

  • Exactly-once semantics are not supported (at-least-once only)

  • You’ll need to re-produce historical data or use a migration tool

  • Consumer group offsets won’t transfer automatically

Performance

What throughput can Korvet handle?

Performance depends on your Redis instance and configuration. With async operations and pipelining:

  • Produce: 50,000+ messages/second per instance (tested up to 170k msg/sec)

  • Fetch: 100,000+ messages/second per instance (tested up to 313k msg/sec)

  • Latency: Sub-millisecond p50, 1-3ms p95, <100ms p99

See the load-testing sample for benchmarking tools.

How do I scale Korvet?

Korvet is stateless and can be scaled horizontally:

  1. Run multiple Korvet instances

  2. Put a load balancer in front

  3. Clients connect to any instance

What are the resource requirements?

Minimum:

  • CPU: 1 core

  • Memory: 512MB

  • Redis: 7.0+ with Streams support

Recommended for production:

  • CPU: 2-4 cores

  • Memory: 2-4GB

  • Redis: Cluster or Enterprise for HA

Storage

How long can I keep data in Redis?

As long as your Redis instance has capacity. Configure retention policies (retention.ms and retention.bytes) to automatically trim old messages at write time.

What happens when Redis is full?

You have several options:

  1. Increase Redis memory: Scale up your Redis instance

  2. Configure retention: Set retention.ms or retention.bytes to automatically trim old messages

  3. Enable tiered storage: Configure optional Delta Lake archival to move old messages to object storage

  4. Redis eviction policies: Configure Redis eviction (e.g., allkeys-lru) as a last resort

Can I query historical data in Delta Lake?

Yes, if you’ve configured optional tiered storage! Use Spark, Flink, Presto, Athena, or any Delta Lake-compatible tool to query archived messages.

Note: This requires running the separate Flink archival job (korvet-storage-writer module).

Operations

How do I monitor Korvet?

Korvet exposes metrics via Prometheus and health checks via Spring Boot Actuator. See Monitoring.

How do I troubleshoot issues?

  1. Check logs (JSON format by default)

  2. Review metrics in Prometheus

  3. Check health endpoints

  4. See Troubleshooting guide

Can I run Korvet in Kubernetes?

Yes! See Deployment guide for Kubernetes manifests.

Development

How do I contribute to Korvet?

See the GitHub repository for contribution guidelines.

Where can I report bugs?

File an issue at https://github.com/redis-field-engineering/korvet-dist/issues

Is there a roadmap?

Check the GitHub issues and project boards for planned features.

Next Steps