For the latest stable version, please use Korvet 0.19!

Design Decisions

This page documents key design decisions made in the Kafka-to-Redis mapping.

This page is for contributors and operators who need implementation rationale and tradeoffs.

Offset Mapping Strategy

Challenge: Kafka uses sequential integer offsets (0, 1, 2…​), while Redis Streams use timestamp-based entry IDs (e.g., 1234567890123-0).

Selected Approach: Stateless Encoding

Kafka offsets are computed directly from Redis Stream entry IDs using bit-packing:

Entry ID format:  {timestamp}-{sequence}
Kafka offset:     (timestamp << sequenceBits) | sequence

Tuning: sequenceBits is set per topic via offset-sequence-bits. For defaults, limits, and how to configure it, see the configuration reference; the rationale and tradeoffs follow.

  • sequenceBits: Number of bits allocated for sequence number (default: 14, max: 16)

  • Maximum sequence per millisecond: 2^sequenceBits - 1 (16,383 for 14 bits)

  • Redis keeps incrementing the sequence within the same millisecond; it does not roll over to the next timestamp. If more than 2^sequenceBits entries are written to one partition in a single Redis millisecond, the sequence exceeds the encodable range. The broker detects this unencodable entry, rejects the produce, and deletes the affected entries, so sequenceBits must be large enough to cover the partition’s peak per-millisecond write burst.

  • sequenceBits is a two-sided trade-off:

    • Higher values raise per-millisecond write capacity (2^bits), reducing produce rejections.

    • Higher values make offsets sparser, so a single fetch record-batch can span only 2^(31-bits) ms before it must be split (Kafka’s batch lastOffsetDelta is int32). The default of 14 bits allows 16,384 records/ms (~16.8M/s) and keeps a fetch batch coherent across ~2.2 min.

Advantages:

  • No extra storage: No hash mappings needed

  • Stateless: Offset ↔ Entry ID conversion is pure computation

  • Monotonic: Offsets increase monotonically like Kafka

  • Efficient: O(1) conversion in both directions

Trade-offs:

  • Offsets are large numbers (not 0, 1, 2…​)

  • Requires configuring sequenceBits based on throughput

  • High-throughput partitions (>16,383 msg/ms at the default of 14 bits) need more sequence bits (up to 16)

Consumer Group Coordination

Challenge: Kafka has a complex group coordination protocol (join, sync, heartbeat, rebalance).

Approach:

  • Use Redis Streams native consumer groups for message delivery and offset tracking

  • Implement Kafka group coordinator protocol in broker (in-memory state)

  • Use XGROUP CREATE to create consumer groups

  • Use XREADGROUP for group-based message consumption

  • Use XACK to acknowledge delivered Redis entries during commit processing

  • Persist committed Kafka offsets in a dedicated committed-offset store

  • Group membership, assignments, and rebalancing are managed by broker state rather than Redis stream metadata

Retention and Cleanup

Challenge: Kafka supports time-based and size-based retention policies.

Approach:

  • Retention is enforced by a leader-locked background storage worker, not at write time and not via XADD arguments

  • Size-based retention: XTRIM {streamKey} MAXLEN {count} (exact trimming for predictable retention)

  • Time-based retention: XTRIM {streamKey} MINID {timestamp} (trim messages older than timestamp)

  • Retention configuration stored in topic metadata (RedisJSON)

  • retention.bytes is converted to a message count using the measured average message size

Implementation: A background worker periodically calls trimToRetention, which issues the XTRIM commands above against each partition stream.

Transactional Writes

Challenge: Kafka supports atomic writes across partitions.

Approach:

  • Use Redis transactions (MULTI/EXEC) for single-partition atomicity

  • Use Lua scripts for multi-partition atomic writes

  • Store transaction markers in stream entries

  • Implement idempotent producer semantics

Performance Optimization

Strategies:

  1. Pipelining: Batch multiple XADD commands in a single network round-trip

  2. Connection Pooling: Reuse Redis connections across requests

  3. Batch Reads: Use XREAD with COUNT to fetch multiple messages efficiently

  4. Group Reads: Use XREADGROUP to read from multiple streams in single call

Open Questions

Multi-Tenancy

Question: How to isolate topics across tenants?

Options:

  • Prefix all keys with tenant ID: {tenant}:{topic}:{partition}

  • Use separate Redis databases per tenant

  • Use Redis ACLs for access control

Replication

Question: How to handle Kafka’s replication factor > 1?

Options:

  • Use Redis Enterprise for replication

  • Ignore replication factor, rely on Redis persistence (AOF/RDB)

  • Implement application-level replication across Redis instances

Log Compaction

Question: How to support Kafka’s log compaction feature?

Options:

  • Background process that reads stream, deduplicates by key, writes to new stream

  • Use Redis Streams with custom compaction logic

  • Store compacted view in separate structure

Large Messages

Question: How to handle messages larger than Redis limits?

Options:

  • Store large payloads in separate keys, reference in stream entry

  • Reject messages above threshold with error response

  • Implement chunking for large messages

Related Documentation