For the latest stable version, please use Korvet 0.12.5!

Topic Management

This guide covers creating and managing topics in Korvet.

Creating Topics

Automatic Topic Creation

By default, topics are automatically created when you first produce to them or request metadata for them. This behavior can be controlled with the auto-create-topics configuration:

korvet:
  server:
    topic-registry:
      auto-create-topics: true  # Enable/disable automatic topic creation (default: true)
      default-partitions: 1     # Default partitions for auto-created topics (default: 1)

When auto-creation is disabled, you must explicitly create topics before using them.

Explicit Topic Creation

You can create topics explicitly using the Kafka command-line tools:

kafka-topics --bootstrap-server localhost:9092 \
  --create \
  --topic my-topic \
  --partitions 3 \
  --replication-factor 1
The replication-factor parameter is accepted for compatibility but ignored, as Korvet uses Redis for storage and replication.

Creating Topics with Korvet-Specific Configuration

You can set Korvet-specific options like value.type and storage.compression during topic creation using kafka-configs:

# Create topic with storage compression and value type
kafka-topics --bootstrap-server localhost:9092 \
  --create \
  --topic my-topic \
  --partitions 3 \
  --config value.type=raw \
  --config storage.compression=zstd \
  --config retention.ms=604800000

Available Korvet-specific configurations:

  • value.type - How message values are stored (json or raw)

  • storage.compression - Compression for values at rest in Redis (none, gzip, snappy, lz4, zstd)

  • compression.type - Compression for Kafka fetch responses (none, gzip, snappy, lz4, zstd)

  • retention.ms - Time-based retention in milliseconds

  • retention.bytes - Size-based retention in bytes

For large messages or deeply nested JSON, use --config value.type=raw --config storage.compression=zstd to reduce Redis memory usage by up to 51x.

Listing Topics

List all topics:

kafka-topics --bootstrap-server localhost:9092 --list

Describing Topics

Get details about a topic:

kafka-topics --bootstrap-server localhost:9092 \
  --describe \
  --topic my-topic

Deleting Topics

Delete a topic:

kafka-topics --bootstrap-server localhost:9092 \
  --delete \
  --topic my-topic

Altering Topic Configuration

You can modify topic configuration after creation using kafka-configs.

Topics can be configured with:

  • Partitions: Number of partitions for parallelism (set during creation only)

  • Retention: Time-based (retention.ms) and size-based (retention.bytes) retention policies

  • Value Type: How message values are stored (value.type: json or raw)

  • Storage Compression: Compression for values at rest in Redis (storage.compression)

  • Protocol Compression: Compression for Kafka fetch responses (compression.type)

Altering Value Type and Storage Compression

Change value type and storage compression for an existing topic:

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter \
  --add-config value.type=raw,storage.compression=zstd

Value types:

  • json - Flatten top-level JSON fields into separate Redis Stream fields (default for JSON objects)

  • raw - Store entire value as single field (default for non-JSON data)

Storage compression types:

  • none - No compression (default)

  • gzip - Good compression ratio (26x), slower

  • snappy - Fast, moderate compression (10x)

  • lz4 - Very fast, good compression (14x)

  • zstd - Best compression ratio (51x), fast - Recommended

Use value.type=raw with storage.compression=zstd for deeply nested JSON or large messages (>10KB) to reduce Redis memory usage by up to 51x.

See Value Mapper Selection for detailed guidance on choosing between JSON flattening and RAW storage.

Altering Protocol Compression

Change protocol compression type for Kafka fetch responses:

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter \
  --add-config compression.type=lz4

Altering Multiple Configurations

You can alter multiple configurations in a single command:

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter \
  --add-config value.type=raw,storage.compression=zstd,compression.type=lz4,retention.ms=604800000

Viewing Topic Configuration

View current topic configuration:

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --describe

Protocol compression types:

  • none - No compression (default)

  • gzip - Good compression ratio, higher CPU usage

  • snappy - Balanced compression and speed

  • lz4 - Fast compression, lower CPU usage

  • zstd - Best compression ratio, moderate CPU usage

Storage compression and protocol compression are independent. You can use both together (e.g., storage.compression=zstd for Redis memory savings and compression.type=lz4 for fast network compression).

See Compression for more details on protocol compression.

Next Steps