For the latest stable version, please use Korvet 0.12.5!

Kafka Compatibility

Korvet implements a subset of the Kafka protocol for compatibility with existing clients and tools.

Supported APIs

API Status Notes

Produce

✅ Supported

Send messages to topics

Fetch

✅ Supported

Read messages from topics

Metadata

✅ Supported

Topic and partition information

ApiVersions

✅ Supported

Protocol version negotiation

Consumer Groups

✅ Supported

JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, OffsetFetch

Transactions

❌ Not planned

Use Redis transactions instead

Admin API

✅ Supported

CreateTopics, DeleteTopics, DescribeConfigs, DescribeCluster, ListGroups, DescribeGroups

Kafka Version Compatibility

Korvet uses the Apache Kafka client library version 3.9.1 and is compatible with Kafka clients from version 2.8.0 and later.

Client Compatibility

  • Minimum supported client version: 2.8.0

  • Recommended client version: 3.9.x

  • Kafka client library: 3.9.1

Kafka clients are backward compatible, so newer clients (3.x, 4.x) can connect to Korvet without issues.

Protocol Features

Korvet implements Kafka protocol features equivalent to Kafka 2.8.0+, including:

  • Produce API (v0-v9)

  • Fetch API (v0-v13)

  • Consumer Group Protocol (JoinGroup, SyncGroup, Heartbeat, LeaveGroup)

  • Offset Management (OffsetCommit, OffsetFetch)

  • Topic Administration (CreateTopics, DeleteTopics)

  • Metadata API

Compression

Korvet supports all Kafka compression types:

  • NONE: No compression (default)

  • GZIP: Good compression ratio, higher CPU usage

  • SNAPPY: Balanced compression and speed

  • LZ4: Fast compression, lower CPU usage

  • ZSTD: Best compression ratio, moderate CPU usage

How Compression Works

Korvet implements server-side compression:

  1. Producer side: Kafka clients can send compressed or uncompressed messages. Korvet automatically decompresses incoming batches and stores individual records uncompressed in Redis Streams.

  2. Consumer side: When consumers fetch messages, Korvet compresses the response based on the topic’s compression configuration (not the producer’s compression setting).

  3. StorageTier: Records are always stored uncompressed in Redis Streams for better queryability and to avoid decompression overhead in the storage layer.

Configuring Compression

Compression is configured per-topic using the compression.type setting:

# Set compression for a topic (requires Admin API support)
kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter \
  --add-config compression.type=lz4

The default compression type is NONE.

Benefits

  • Network bandwidth: Compression reduces the amount of data transferred between Korvet and consumers

  • Flexibility: Different topics can use different compression algorithms based on their data characteristics

  • Compatibility: Works transparently with all Kafka clients

Limitations

  • Replication factor: Always 1 (Redis provides persistence)

  • Transactions: Not supported

  • Exactly-once semantics: Not supported (at-least-once delivery)

Client Configuration

Most Kafka client configurations work with Korvet. Some settings are ignored:

  • acks: Always treated as acks=1

  • replication.factor: Ignored (always 1)

  • min.insync.replicas: Ignored

Testing Compatibility

You can test Korvet with your existing Kafka applications by simply changing the bootstrap.servers configuration to point to Korvet.

Next Steps