Frequently Asked Questions

Common questions about Korvet.

General

What is Korvet?

Korvet is a Kafka-compatible streaming service backed by Redis Streams, with optional tiered storage to Delta Lake for long-term archival.

Why use Korvet instead of Kafka?

  • Simpler operations: No ZooKeeper, no partition rebalancing complexity

  • Redis integration: Leverage existing Redis infrastructure and persistence

  • Low latency: Sub-millisecond read/write performance with Redis Streams

  • Optional cost optimization: Tiered storage to Delta Lake for long-term retention

  • Analytics-ready: Delta Lake integration for Spark, Databricks, and SQL engines

Is Korvet production-ready?

Korvet is in active development. The core features are functional:

  • ✅ Kafka protocol implementation (Produce, Fetch, Consumer Groups, Admin API)

  • ✅ Redis Streams storage with retention policies

  • ✅ Async operations for high throughput

  • ✅ Metrics and observability

  • ✅ Built-in Delta Lake archival (100k+ msg/s to S3)

Compatibility

Which Kafka clients work with Korvet?

Any Kafka client that supports the Kafka protocol should work. Tested clients include:

  • Java: kafka-clients

  • Python: kafka-python, confluent-kafka-python

  • Go: sarama

  • Node.js: kafkajs

What Kafka features are supported?

Supported:

  • ✅ Produce API

  • ✅ Fetch API

  • ✅ Consumer Groups (JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, OffsetFetch)

  • ✅ Topic metadata

  • ✅ Admin API (CreateTopics, DeleteTopics, DescribeConfigs, DescribeCluster)

  • ✅ Compression (GZIP, SNAPPY, LZ4, ZSTD)

Not supported:

  • ❌ Transactions

  • ❌ Exactly-once semantics (at-least-once delivery only)

Can I migrate from Kafka to Korvet?

Yes, but with some considerations:

  • Transactions are not supported

  • Exactly-once semantics are not supported (at-least-once only)

  • You’ll need to re-produce historical data or use a migration tool

  • Consumer group offsets won’t transfer automatically

Performance

What throughput can Korvet handle?

Performance depends on your Redis instance and configuration. With async operations and pipelining:

  • Produce: 50,000+ messages/second per instance (tested up to 170k msg/sec)

  • Fetch: 100,000+ messages/second per instance (tested up to 313k msg/sec)

  • Latency: Sub-millisecond p50, 1-3ms p95, <100ms p99

See the load-testing sample for benchmarking tools.

How do I scale Korvet?

Korvet is stateless and can be scaled horizontally:

  1. Run multiple Korvet instances

  2. Put a load balancer in front

  3. Clients connect to any instance

What are the resource requirements?

Minimum:

  • CPU: 1 core

  • Memory: 512MB

  • Redis: 7.0+ with Streams support

Recommended for production:

  • CPU: 2-4 cores

  • Memory: 2-4GB

  • Redis: Cluster or Enterprise for HA

Storage

How long can I keep data in Redis?

As long as your Redis instance has capacity. Configure retention policies (retention.ms and retention.bytes) to automatically trim old messages at write time.

What happens when Redis is full?

You have several options:

  1. Increase Redis memory: Scale up your Redis instance

  2. Configure retention: Set retention.ms or retention.bytes to automatically trim old messages

  3. Enable tiered storage: Configure optional Delta Lake archival to move old messages to object storage

  4. Redis eviction policies: Configure Redis eviction (e.g., allkeys-lru) as a last resort

Can I query historical data in Delta Lake?

Yes, if you’ve enabled tiered storage! Use Spark, Databricks, Presto, Athena, or any Delta Lake-compatible tool to query archived messages.

See Cold Storage for configuration details.

Should I use JSON flattening or RAW storage with compression?

It depends on your data structure and message size:

Use JSON flattening (default for JSON objects):

  • Shallow JSON with many top-level fields (10+)

  • Small messages (<1KB)

  • When you need field-level access in Redis

  • Varied field values (UUIDs, timestamps, user data)

Use RAW + ZSTD compression:

  • Deeply nested JSON (>2-3 levels)

  • Large messages (>10KB)

  • Repetitive data patterns (logs, telemetry)

  • Binary formats (Protocol Buffers, Avro)

Example: For log messages with deep nesting, RAW+ZSTD can reduce Redis memory usage by 18x compared to JSON flattening.

See Value Mapper Selection for detailed guidance and performance comparisons.

How much memory can storage compression save?

Based on production testing with 100KB messages:

  • ZSTD: Up to 51x compression (100KB → 2KB)

  • LZ4: Up to 14x compression (100KB → 7KB)

  • GZIP: Up to 26x compression (100KB → 4KB)

  • SNAPPY: Up to 10x compression (100KB → 10KB)

Actual compression ratios depend on your data. Repetitive data (logs, structured text) compresses better than random data (images, encrypted data).

Use ZSTD for best compression ratio with good performance.

Operations

How do I monitor Korvet?

Korvet exposes metrics via Prometheus and health checks via Spring Boot Actuator. See Monitoring.

How do I troubleshoot issues?

  1. Check logs (JSON format by default)

  2. Review metrics in Prometheus

  3. Check health endpoints

  4. See Troubleshooting guide

Can I run Korvet in Kubernetes?

Yes! See Deployment guide for Kubernetes manifests.

Development

How do I contribute to Korvet?

See the GitHub repository for contribution guidelines.

Where can I report bugs?

File an issue at https://github.com/redis-field-engineering/korvet-dist/issues

Is there a roadmap?

Check the GitHub issues and project boards for planned features.

Next Steps