For the latest stable version, please use Korvet 0.12.5!

Topic Management

This guide covers creating and managing topics in Korvet.

Creating Topics

Automatic Topic Creation

By default, topics are automatically created when you first produce to them or request metadata for them. This behavior can be controlled with the auto-create-topics configuration:

korvet:
  server:
    auto-create-topics: true  # Enable/disable automatic topic creation (default: true)
    default-partitions: 1     # Default partitions for auto-created topics (default: 1)

When auto-creation is disabled, you must explicitly create topics before using them.

Explicit Topic Creation

You can create topics explicitly using the Kafka command-line tools:

kafka-topics --bootstrap-server localhost:9092 \
  --create \
  --topic my-topic \
  --partitions 3 \
  --replication-factor 1
The replication-factor parameter is accepted for compatibility but ignored, as Korvet uses Redis for storage and replication.

Creating Topics with Korvet-Specific Configuration

You can set Korvet-specific options like value.type and storage.compression during topic creation using kafka-configs:

# Create topic with storage compression and value type
kafka-topics --bootstrap-server localhost:9092 \
  --create \
  --topic my-topic \
  --partitions 3 \
  --config value.type=raw \
  --config storage.compression=zstd \
  --config retention.ms=604800000

Available configurations:

  • value.type - How message values are stored (auto, json, or raw). Default: auto

  • storage.compression - Compression for values at rest in Redis (none, gzip, snappy, lz4, zstd)

  • compression.type - Compression for Kafka fetch responses (none, gzip, snappy, lz4, zstd)

  • retention.ms - Total time-based retention in milliseconds (across all tiers)

  • retention.bytes - Total size-based retention in bytes

Tiered storage configurations (when cold/warm storage is enabled at server level):

  • remote.storage.enable - Enable tiered storage for this topic (Kafka KIP-405). Default: false

  • local.retention.ms - Time to keep in hot tier (Redis RAM) before archiving. -2 = use retention.ms (Kafka KIP-405)

  • local.retention.bytes - Size to keep in hot tier before archiving. -2 = use retention.bytes (Kafka KIP-405)

  • near.retention.ms - Time to keep in warm tier before moving to cold. -2 = skip warm tier (Korvet extension)

  • near.retention.bytes - Size to keep in warm tier before moving to cold. -2 = skip warm tier (Korvet extension)

For large messages or deeply nested JSON, use --config value.type=raw --config storage.compression=zstd to reduce Redis memory usage by up to 51x.

Listing Topics

List all topics:

kafka-topics --bootstrap-server localhost:9092 --list

Describing Topics

Get details about a topic:

kafka-topics --bootstrap-server localhost:9092 \
  --describe \
  --topic my-topic

Deleting Topics

Delete a topic:

kafka-topics --bootstrap-server localhost:9092 \
  --delete \
  --topic my-topic

Altering Topic Configuration

You can modify topic configuration after creation using kafka-configs.

Topics can be configured with:

  • Partitions: Number of partitions for parallelism (set during creation only)

  • Retention: Time-based (retention.ms) and size-based (retention.bytes) retention policies

  • Value Type: How message values are stored (value.type: auto, json, or raw)

  • Storage Compression: Compression for values at rest in Redis (storage.compression)

  • Protocol Compression: Compression for Kafka fetch responses (compression.type)

Altering Value Type and Storage Compression

Change value type and storage compression for an existing topic:

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter \
  --add-config value.type=raw,storage.compression=zstd

Value types:

  • auto - Automatically detect optimal storage: JSON flattening for JSON objects, RAW for everything else (default)

  • json - Flatten top-level JSON fields into separate Redis Stream fields

  • raw - Store entire value as single field with optional compression

Storage compression types:

  • none - No compression (default)

  • gzip - Good compression ratio (26x), slower

  • snappy - Fast, moderate compression (10x)

  • lz4 - Very fast, good compression (14x)

  • zstd - Best compression ratio (51x), fast - Recommended

Use value.type=raw with storage.compression=zstd for deeply nested JSON or large messages (>10KB) to reduce Redis memory usage by up to 51x.

See Value Mapper Selection for detailed guidance on choosing between JSON flattening and RAW storage.

Altering Protocol Compression

Change protocol compression type for Kafka fetch responses:

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter \
  --add-config compression.type=lz4

Altering Multiple Configurations

You can alter multiple configurations in a single command:

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter \
  --add-config value.type=raw,storage.compression=zstd,compression.type=lz4,retention.ms=604800000

Viewing Topic Configuration

View current topic configuration:

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --describe

Protocol compression types:

  • none - No compression (default)

  • gzip - Good compression ratio, higher CPU usage

  • snappy - Balanced compression and speed

  • lz4 - Fast compression, lower CPU usage

  • zstd - Best compression ratio, moderate CPU usage

Storage compression and protocol compression are independent. You can use both together (e.g., storage.compression=zstd for Redis memory savings and compression.type=lz4 for fast network compression).

See Compression for more details on protocol compression.

Tiered Storage Configuration

When tiered storage is enabled at the server level, you can configure per-topic retention policies to control when data moves between tiers.

Enabling Tiered Storage for a Topic

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true

Configuring Tier Retention

Example: Keep 1 day in hot tier (RAM), 7 days in warm tier (disk), remainder in cold tier (S3):

kafka-configs --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic --alter \
  --add-config remote.storage.enable=true,retention.ms=31536000000,local.retention.ms=86400000,near.retention.ms=604800000

This configures:

  • Hot tier: 1 day (local.retention.ms=86400000)

  • Warm tier: 7 days (near.retention.ms=604800000)

  • Cold tier: ~357 days (implicit: retention.ms - local.retention.ms - near.retention.ms)

  • Total retention: 1 year (retention.ms=31536000000)

Tiered Storage Configuration Reference

Configuration Default Description

remote.storage.enable

false

Enable tiered storage for this topic (Kafka KIP-405)

local.retention.ms

-2

Time to keep in hot tier before archiving. -2 = use total retention.ms

local.retention.bytes

-2

Size to keep in hot tier. -2 = use total retention.bytes

near.retention.ms

-2

Time to keep in warm tier. -2 = skip warm tier (go directly to cold)

near.retention.bytes

-2

Size to keep in warm tier. -2 = skip warm tier

retention.ms

604800000

Total retention across all tiers (7 days default)
Cold tier retention is implicit and calculated as retention.ms - local.retention.ms - near.retention.ms. Data is deleted after the total retention.ms period.

See Cold Storage and Warm Storage for server-level tiered storage configuration.

Next Steps