Core Concepts

Korvet is a Kafka-compatible streaming service backed by Redis Streams.

Architecture Overview

Korvet implements the Kafka protocol and stores messages in Redis Streams for low-latency, durable message storage.

Topics and Partitions

Like Kafka, Korvet organizes messages into topics and partitions:

  • Topic: A logical stream of messages (e.g., "orders", "events")

  • Partition: A topic is divided into partitions for parallelism

  • StreamOffset: Each message has a unique offset within its partition

Storage

Redis Streams

All messages are stored in Redis Streams:

  • Fast reads/writes: Sub-millisecond latency

  • Consumer groups: Multiple consumers can read from the same stream

  • Persistence: Redis AOF and RDB for durability

  • Stream per partition: Each topic partition maps to a Redis Stream

  • Retention: Configurable time and size-based retention enforced at write time

Message Format

Messages in Korvet follow the Kafka format:

  • Key: Optional message key (byte array)

  • Value: Message payload (byte array)

  • Headers: Optional key-value metadata

  • Timestamp: Message timestamp

Kafka Compatibility

Korvet implements the Kafka protocol, allowing you to use existing Kafka clients:

  • Produce API: Send messages to topics

  • Fetch API: Read messages from topics

  • Consumer Groups: Coordinate multiple consumers

  • Offset Management: Track consumer progress