For the latest stable version, please use Korvet 0.12.5!

Logging

Korvet uses Logback for logging with JSON output for easy parsing.

Log Format

Logs are output in JSON format by default:

{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "INFO",
  "logger": "com.redis.korvet.server.KorvetServer",
  "message": "Started KorvetServer in 2.5 seconds",
  "thread": "main"
}

Log Levels

Configure log levels via environment variables:

# Root level
export LOGGING_LEVEL_ROOT=INFO

# Korvet components (default)
export LOGGING_LEVEL_COM_REDIS_KORVET=INFO

# Spring framework
export LOGGING_LEVEL_ORG_SPRINGFRAMEWORK=WARN

# Redis client
export LOGGING_LEVEL_IO_LETTUCE=INFO

Or in application.yml:

logging:
  level:
    root: INFO
    com.redis.korvet: INFO
    org.springframework: WARN
    io.lettuce: INFO

Structured Logging

Korvet includes structured fields in logs:

  • topic: Topic name

  • partition: Partition number

  • offset: Message offset

  • clientId: Kafka client ID

  • correlationId: Request correlation ID

Log Aggregation

Filebeat

Ship logs to Elasticsearch:

filebeat.inputs:
- type: container
  paths:
    - '/var/lib/docker/containers/*/*.log'
  processors:
    - decode_json_fields:
        fields: ["message"]
        target: ""

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

Fluentd

<source>
  @type tail
  path /var/log/korvet/*.log
  pos_file /var/log/korvet/korvet.log.pos
  tag korvet
  <parse>
    @type json
  </parse>
</source>

<match korvet>
  @type elasticsearch
  host elasticsearch
  port 9200
  index_name korvet
</match>

Troubleshooting Logs

Korvet defaults to INFO logging. Raise specific components to DEBUG only while troubleshooting:

export LOGGING_LEVEL_COM_REDIS_KORVET_KAFKA=DEBUG
export LOGGING_LEVEL_COM_REDIS_KORVET_REDIS=DEBUG
export LOGGING_LEVEL_COM_REDIS_KORVET=DEBUG

Runtime Log Level Changes With Actuator

For Spring Boot applications, the usual way to change logger levels without restarting the process is the Actuator loggers endpoint. The same approach works in Kubernetes as long as the application includes Actuator and exposes /actuator/loggers on the management port.

Korvet already includes spring-boot-starter-actuator, but the loggers endpoint must be exposed before you can use it remotely. Add loggers to management.endpoints.web.exposure.include:

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus,metrics,s3diagnostics,loggers

Update a specific logger at runtime with a POST request:

{"configuredLevel":"DEBUG"}

Example:

curl -X POST http://<pod-or-service>:8080/actuator/loggers/com.redis.korvet.storage.tiered \
  -H 'Content-Type: application/json' \
  -d '{"configuredLevel":"DEBUG"}'

For Korvet troubleshooting, this is a good way to enable DEBUG only for targeted packages such as:

  • com.redis.korvet.storage.tiered

  • com.redis.korvet.storage.redis

In Kubernetes, operators commonly use kubectl port-forward to reach the management port without exposing the endpoint publicly:

kubectl port-forward pod/<korvet-pod> 8080:8080
curl -X POST http://127.0.0.1:8080/actuator/loggers/com.redis.korvet.storage.redis \
  -H 'Content-Type: application/json' \
  -d '{"configuredLevel":"DEBUG"}'

This change is runtime-only and usually does not survive a pod restart unless you also update the application configuration. If the endpoint is not exposed, or is protected by network policy or security controls, Spring Boot will not let you change logger levels remotely.

Next Steps