This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.19!

Kafka Compatibility

Korvet implements a subset of the Kafka protocol for compatibility with existing clients and tools.

Supported APIs

API Status Notes

Produce

✅ Supported

Send messages to topics

Fetch

✅ Supported

Read messages from topics

Metadata

✅ Supported

Topic and partition information

ApiVersions

✅ Supported

Protocol version negotiation

Consumer Groups

✅ Supported

JoinGroup, SyncGroup, Heartbeat, LeaveGroup, OffsetCommit, OffsetFetch

Transactions

❌ Not planned

Use Redis transactions instead

Admin API

✅ Supported

CreateTopics, CreatePartitions, DeleteTopics, DescribeConfigs, AlterConfigs, IncrementalAlterConfigs, DescribeCluster, ListGroups, DescribeGroups, DeleteGroups

Idempotent Producers

✅ Supported

InitProducerId API for idempotent producer support (non-transactional)

Kafka Version Compatibility

Korvet uses the Apache Kafka client library version 3.9.2 and is compatible with Kafka clients from version 2.8.0 and later.

Client Compatibility

  • Minimum supported client version: 2.8.0

  • Recommended client version: 3.9.x

  • Kafka client library: 3.9.2

Kafka clients are backward compatible, so newer clients (3.x, 4.x) can connect to Korvet without issues.

Protocol Features

Korvet implements Kafka protocol features equivalent to Kafka 2.8.0+, including:

  • Produce API (v0-v11; record batches must use the modern magic v2 format — legacy magic v0/v1 batches are rejected with INVALID_RECORD)

  • Fetch API (v0-v12, see Supported Version Matrix)

  • Consumer Group Protocol (JoinGroup, SyncGroup, Heartbeat, LeaveGroup)

  • Offset Management (OffsetCommit, OffsetFetch)

  • Topic Administration (CreateTopics, CreatePartitions, DeleteTopics)

  • Metadata API

Supported Version Matrix

Each API’s advertised max version is pinned explicitly to what the broker actually implements, rather than tracking whatever the bundled kafka-clients library defines. Kafka clients negotiate versions through the ApiVersions response, so any compatible client transparently uses the advertised version; features gated on a newer version fail fast on the client with UnsupportedVersionException instead of misbehaving on the broker.

API Advertised versions Cap rationale

Produce

v0-v11

Transactional produce is not supported (transactional batches require a transaction coordinator, see Limitations).

Fetch

v0-v12

v13+ switches to topic-ID-based fetching (KIP-516) with incremental fetch sessions and follower fetching, none of which Korvet implements: topics are identified by name, each Fetch request is served statelessly, and there are no follower replicas.

Metadata

v0-v12

Request-side topic IDs (v10+) are not resolved; clients only fetch by topic ID on the Fetch path, which is capped at v12.

ApiVersions

v0-v4

FindCoordinator

v0-v2

v3+ restricts the coordinator host/port response fields.

JoinGroup

v0-v9

SyncGroup

v0-v5

Heartbeat

v0-v4

LeaveGroup

v0-v2

v3+ responses carry per-member errors (Members[]) that the broker does not produce.

OffsetCommit

v0-v9

committedLeaderEpoch is accepted but not persisted; OffsetFetch reports -1 (no KIP-320 fencing, see Leader Epoch).

OffsetFetch

v0-v9

Committed leader epoch is always -1.

ListOffsets

v0-v6

The v7+ timestamp sentinels (MAX_TIMESTAMP, EARLIEST_LOCAL_TIMESTAMP, LATEST_TIERED_TIMESTAMP) are not implemented; capping lets clients fail fast instead of receiving INVALID_REQUEST.

CreateTopics

v0-v7

CreatePartitions

v0-v3

DeleteTopics

v0-v6

DescribeConfigs

v0-v4

Config synonyms and documentation are not returned.

AlterConfigs

v0-v2

IncrementalAlterConfigs

v0-v1

ListGroups

v0-v5

DescribeGroups

v0-v5

DeleteGroups

v0-v2

DescribeCluster

v0-v1

The v1 EndpointType field (KIP-919) is ignored; there is no separate controller endpoint.

InitProducerId

v0-v2

v3+ adds the KIP-360 epoch-bump handshake (client-supplied producerId/producerEpoch), which the broker does not implement; idempotent producers reinitialize instead. Transactional IDs are rejected.

SaslHandshake

v0-v1

SaslAuthenticate

v0-v2

ConsumerGroupHeartbeat

v0

Only advertised when the KIP-848 consumer protocol is enabled; v1+ (e.g. regex subscriptions) is not implemented.

ConsumerGroupDescribe

v0

Only advertised when the KIP-848 consumer protocol is enabled.

The matrix is enforced in code: ApiVersionsHandler advertises these exact ranges, and ApiVersionsHandlerTest pins both the advertised matrix and the bundled library’s per-API latest versions, so a kafka-clients upgrade cannot silently raise an advertised version without an explicit review.

Leader Epoch for Stale Cache Detection

Korvet maintains a cluster-wide registry epoch that helps Kafka clients detect when their cached metadata is stale after broker membership changes:

  • Registry epoch: A shared counter incremented whenever the live broker set changes (broker join, leave, or stale heartbeat expiry)

  • Advertised as leader epoch: Each partition’s leader epoch in Metadata responses reflects the current registry epoch

  • Not-leader fetch responses: Include currentLeader with the leader ID and epoch (Fetch v12+) so misrouted clients refresh metadata immediately

Single-broker mode: Always uses epoch 0.

Multi-broker clusters: The epoch increments on any membership change:

  • A new broker joins the cluster

  • An existing broker unregisters

  • A stale broker heartbeat expires

This allows clients to detect when their metadata cache is outdated and refresh it, reducing the number of misrouted requests after broker churn.

This is stale-cache detection only. Full KIP-320 leader epoch fencing (validating client-supplied currentLeaderEpoch to reject reads from older epochs) is not yet implemented.

Compression

Korvet supports all Kafka compression types:

  • NONE: No compression (default)

  • GZIP: Good compression ratio, higher CPU usage

  • SNAPPY: Balanced compression and speed

  • LZ4: Fast compression, lower CPU usage

  • ZSTD: Best compression ratio, moderate CPU usage

How Compression Works

Korvet implements server-side compression:

  1. Producer side: Kafka clients can send compressed or uncompressed batches. Korvet automatically decompresses incoming batches into individual records before storing them.

  2. Consumer side: When consumers fetch messages, Korvet compresses the response based on the topic’s compression.type configuration (not the producer’s compression setting).

  3. At rest: The producer’s Kafka batch compression is not retained — each record is stored as its own Redis Stream entry. The storage backend may then apply its own configurable at-rest compression to each record’s value field (korvet.storage.local.compression.codec, default none), independently of the Kafka compression.type used on the wire.

Configuring Compression

Compression is configured per-topic using the compression.type setting:

# Set compression for a topic (requires Admin API support)
kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter \
  --add-config compression.type=lz4

The default compression type is NONE.

Benefits

  • Network bandwidth: Compression reduces the amount of data transferred between Korvet and consumers

  • Flexibility: Different topics can use different compression algorithms based on their data characteristics

  • Compatibility: Works transparently with all Kafka clients

Limitations

  • Replication factor: Always 1 (Redis provides persistence)

  • Transactions: Not supported

  • Exactly-once semantics: Not supported (at-least-once delivery). Streaming frameworks that implement exactly-once sinks with Kafka transactions (for example Flink’s KafkaSink with DeliveryGuarantee.EXACTLY_ONCE) fail at startup; use their at-least-once mode instead (see Apache Flink Integration).

  • Consumer group membership: Held in broker memory. Committed offsets are durable in Redis, so after a broker restart ListGroups and DescribeGroups report groups with committed offsets in the Empty state (with no member details) until clients rejoin.

Kafka Streams Compatibility

Korvet supports the broker-side features Kafka Streams relies on for group coordination and state:

  • Consumer group protocol, including the CooperativeStickyAssignor used by Kafka Streams for task assignment

  • Automatic creation of internal topics (repartition and changelog topics) via the Admin API

  • Log compaction (cleanup.policy=compact) for changelog topics

  • Idempotent producers (InitProducerId)

Kafka Streams applications configured with processing.guarantee=at_least_once are expected to work against Korvet.

Applications using processing.guarantee=exactly_once_v2 (the Kafka Streams default in recent versions) are not supported, because Korvet has no transaction coordinator: AddPartitionsToTxn, AddOffsetsToTxn, EndTxn, and TxnOffsetCommit are not implemented. This work is tracked in issue #36.

Client Configuration

Most Kafka client configurations work with Korvet. Some settings are ignored:

  • acks: Always treated as acks=1

  • replication.factor: Ignored (always 1)

  • min.insync.replicas: Ignored

Testing Compatibility

You can test Korvet with your existing Kafka applications by simply changing the bootstrap.servers configuration to point to Korvet.

Next Steps