Troubleshooting

Common issues and solutions for Korvet.

Connection Issues

Cannot connect to Korvet

Symptoms: Kafka clients timeout or fail to connect

Solutions:

  1. Check Korvet is running:

    curl http://localhost:8080/actuator/health

  2. Verify port is accessible:

    telnet localhost 9092

  3. Check firewall rules allow port 9092

Cannot connect to Redis

Symptoms: Korvet fails to start or logs Redis connection errors

Solutions:

  1. Verify Redis is running:

    redis-cli ping

  2. Check Redis connection settings:

    export KORVET_REDIS_HOST=localhost
export KORVET_REDIS_PORT=6379

  3. Test Redis connectivity:

    redis-cli -h localhost -p 6379 ping

Performance Issues

High latency

Symptoms: Slow produce/fetch operations

Solutions:

  1. Check Redis latency:

    redis-cli --latency

  2. Monitor JVM garbage collection:

    curl http://localhost:8080/actuator/metrics/jvm.gc.pause

  3. Increase JVM heap size:

    export JAVA_OPTS="-Xmx2g -Xms2g"

Low throughput

Symptoms: Cannot achieve expected messages per second

Solutions:

  1. Enable pipelining in Redis client

  2. Increase batch size for produce operations

  3. Scale horizontally (add more Korvet instances)

  4. Use Redis Cluster for better performance

Data Issues

Messages not appearing

Symptoms: Produced messages don’t show up when consuming

Solutions:

  1. Check topic exists:

    kafka-topics --bootstrap-server localhost:9092 --list

  2. Verify partition assignment

  3. Check consumer is reading from correct offset

  4. Enable debug logging:

    export LOGGING_LEVEL_COM_REDIS_KORVET=DEBUG

Duplicate messages

Symptoms: Same message consumed multiple times

Solutions:

  1. Check consumer offset management

  2. Verify consumer group configuration

  3. Ensure proper error handling in consumer

Resource Issues

Out of memory

Symptoms: JVM crashes with OutOfMemoryError

Solutions:

  1. Increase heap size:

    export JAVA_OPTS="-Xmx4g -Xms4g"

  2. Check for memory leaks in metrics

  3. Reduce message batch sizes

  4. Enable GC logging for analysis

Redis storage full

Symptoms: Cannot produce new messages

Solutions:

  1. Check Redis memory usage:

    redis-cli info memory

  2. Use Redis XTRIM to manually trim old messages

  3. Reduce retention period for topics

  4. Increase Redis memory limit

Getting Help

If you can’t resolve the issue:

  1. Check logs with debug level enabled

  2. Collect metrics from /actuator/prometheus

  3. File an issue at https://github.com/redis-field-engineering/korvet-dist/issues

Next Steps