Kafka-to-Redis Mapping Overview

This document describes how Korvet maps Kafka protocol operations to Redis Stream commands.

Overview

Korvet implements a Kafka-compatible broker using Redis Streams as the storage backend. This design document shows the mapping between Kafka client requests and Redis commands for each supported use case.

Architecture Overview

Core Mapping Concepts

Kafka to Redis Model

Kafka Concept Redis Implementation Key Pattern

Kafka Concept	Redis Implementation	Key Pattern
Topic Partition	Redis Stream	`{topic}:{partition}`
Message Offset	Stream Entry ID + Mapping	Sequential integers (0, 1, 2…)
Consumer Group	Redis Consumer Group	Same name as Kafka group
Consumer Group Offset	Hash + Consumer Group State	Tracked in Redis
Message Key/Value/Headers	Stream Entry Fields	`key`, `value`, `headers`, `timestamp`

Topic Partition

Redis Stream

{topic}:{partition}

Message Offset

Stream Entry ID + Mapping

Sequential integers (0, 1, 2…)

Consumer Group

Redis Consumer Group

Same name as Kafka group

Consumer Group Offset

Hash + Consumer Group State

Tracked in Redis

Message Key/Value/Headers

Stream Entry Fields

key, value, headers, timestamp

Key Design Principles

Offset Semantics: Kafka uses sequential integer offsets (0, 1, 2…), Redis uses timestamp-based entry IDs. We maintain bidirectional mappings.
Atomicity: Use Redis transactions (MULTI/EXEC) and Lua scripts for atomic operations.
Performance: Leverage pipelining, connection pooling, and caching for high throughput.
Compatibility: Maintain full Kafka protocol compatibility for existing clients.

Documentation Organization

The Kafka-to-Redis mapping documentation is organized by workflow and reference:

Workflows (End-to-End API Sequences)

Producer Workflow - Complete API sequence for producing messages
Consumer Workflow (Standalone) - Complete API sequence for standalone consumers
Consumer Group Workflow - Complete API sequence for group consumers

Individual API Details

Producer: Send Messages - ProduceRequest details
Consumer: Fetch Messages - FetchRequest details
Consumer: Commit Offsets - OffsetCommitRequest details
Consumer: Fetch Committed Offsets - OffsetFetchRequest details
List Offsets - ListOffsetsRequest details
Topic Management - CreateTopics, DeleteTopics, Metadata
Consumer Group Management - JoinGroup, SyncGroup, Heartbeat, LeaveGroup

Reference

Redis Data Structures - Complete reference of all Redis keys and structures
Design Decisions - Architecture decisions and open questions

Quick Reference

Common Redis Key Patterns

Pattern Purpose

Pattern	Purpose
`{keyspace}:stream:{topic}:{partition}`	Stream containing messages for a topic partition
`{keyspace}:topic:{topic}`	Hash containing topic configuration and metadata
`{keyspace}:topics`	Set containing all topic names
`{keyspace}:topic-ids`	Hash mapping topic IDs to names
`group:{group_id}:offsets`	Hash containing committed offsets for a consumer group
`group:{group_id}:members`	Hash containing active members of a consumer group
`topics`	Set containing all topic names
`topic:{topic}:partitions`	Set containing partition numbers for a topic

{keyspace}:stream:{topic}:{partition}

Stream containing messages for a topic partition

{keyspace}:topic:{topic}

Hash containing topic configuration and metadata

{keyspace}:topics

Set containing all topic names

{keyspace}:topic-ids

Hash mapping topic IDs to names

group:{group_id}:offsets

Hash containing committed offsets for a consumer group

group:{group_id}:members

Hash containing active members of a consumer group

topics

Set containing all topic names

topic:{topic}:partitions

Set containing partition numbers for a topic

Supported Kafka APIs

Produce API (API Key 0) - Write messages to topics
Fetch API (API Key 1) - Read messages from topics
ListOffsets API (API Key 2) - Find beginning/end offsets
Metadata API (API Key 3) - Discover topics and partitions
OffsetCommit API (API Key 8) - Commit consumer group offsets
OffsetFetch API (API Key 9) - Retrieve committed offsets
FindCoordinator API (API Key 10) - Locate group coordinator
JoinGroup API (API Key 11) - Join a consumer group
Heartbeat API (API Key 12) - Maintain group membership
LeaveGroup API (API Key 13) - Leave a consumer group
SyncGroup API (API Key 14) - Synchronize group state
CreateTopics API (API Key 19) - Create new topics
DeleteTopics API (API Key 20) - Delete existing topics
DeleteGroups API (API Key 42) - Delete consumer groups