This version is still in development and is not considered stable yet. For the latest stable version, please use Korvet 0.19!

Monitoring

Korvet provides comprehensive monitoring through Spring Boot Actuator and Micrometer.

Health Checks

Korvet exposes health check endpoints:

# Overall health (anonymous)
curl http://localhost:8080/actuator/health

# Liveness probe (for Kubernetes, anonymous)
curl http://localhost:8080/actuator/health/liveness

# Readiness probe (for Kubernetes, anonymous)
curl http://localhost:8080/actuator/health/readiness
Health check endpoints (/actuator/health, /actuator/health/liveness, and /actuator/health/readiness) are publicly accessible without authentication, even when korvet.admin.security-enabled=true. This ensures Kubernetes health probes and monitoring systems can check service health without credentials.

Korvet enables the liveness and readiness health groups by default. Readiness includes Redis connectivity, so a broker that loses either required Redis backend stops receiving traffic. Liveness excludes Redis, so a backing-store outage does not cause a pod restart: the pod drains traffic while Redis is down and automatically resumes serving once Redis recovers and the readiness probe returns UP. This default comes from the shipped application.yml; deployments that replace it with external configuration must retain management.endpoint.health.probes.enabled=true to expose the probe sub-paths outside a Kubernetes-detected environment.

Health components

/actuator/health aggregates several subsystem indicators under components:

  • redis: Redis reachability (a PING against the broker Redis client, and — when korvet.storage.local.redis.* points local stream storage at a separate backend — the storage Redis client too). Reports DOWN when either backing store is unreachable. Results are cached for five seconds to limit load from readiness and UI polling. Connection and command timeouts are explicitly set to two seconds to ensure fast failure detection.

  • broker: local broker process liveness, sourced from the korvet.broker.up gauge. Reports DOWN when the broker is stopped or failed to start.

  • diskSpace, ping, ssl: standard Spring Boot indicators.

Because the overall status is the worst of its components, a Redis outage degrades /actuator/health to DOWN (HTTP 503) instead of leaving it UP. The admin API GET /api/v1/health wraps this tree, so its status and details.components reflect the same subsystems, and the dashboard health pill and Redis-connectivity chip degrade accordingly. The structured checks[] array on /api/v1/health additionally reports a dedicated redis-connectivity probe (and conditional storage-redis-connectivity when storage uses a separate Redis backend).

UI Health Display

The web console (/) displays overall health status in two places:

  • App bar health pill: shows UP (green) or DOWN (red)

  • Operations page: detailed health view showing all components and their individual statuses

During a Redis outage the health pill turns red, the Operations page shows the Redis components as DOWN while other components may remain UP, and operations requiring Redis fail gracefully.

The readiness probe includes a Redis connectivity check (RedisReadinessHealthIndicator) that PINGs the broker’s Redis client on each readiness probe, caching the result for 5 seconds. A broker that loses Redis connectivity reports not-ready and is removed from load-balancer rotation, preventing it from accepting requests it cannot fulfill. Liveness remains application-only, so a Redis outage does not restart pods. See Kubernetes Deployment for probe configuration.

Metrics

Metrics are exposed in Prometheus format at /actuator/prometheus.

Authentication

When korvet.admin.security-enabled=true (the default), all /actuator/** endpoints except the health probes require HTTP Basic authentication with console-user credentials. Any role — including VIEWER — can read (GET); mutating actuator calls such as POST /actuator/loggers require an ADMIN user. See Authentication for console roles.

# Authenticate with a console username and password
curl -u admin:your-password http://localhost:8080/actuator/prometheus

# Or disable authentication for Prometheus scraping (not recommended for production)
# Set korvet.admin.anonymous-prometheus=true

To allow anonymous Prometheus scraping while keeping other admin endpoints secured, set:

korvet:
  admin:
    security-enabled: true
    anonymous-prometheus: true  # Allow unauthenticated /actuator/prometheus access

When korvet.admin.security-enabled=false, all actuator endpoints are publicly accessible.

Admin API Metrics Endpoints

Korvet provides curated HTTP metrics endpoints designed for dashboard polling and operational visibility, available under /api/v1/metrics:

  • GET /api/v1/metrics: Curated snapshot with topic/group counts, active connections, backpressure connections, cumulative counters (messages, bytes, requests, rebalances, failures), and aggregated request latency percentiles.

  • GET /api/v1/metrics/topics: Per-topic produce/fetch rates (records/s, bytes/s, requests/s) and latency percentiles (p50/p95/p99). Paginated by topic name with offset/limit parameters (default 100, max 1000). Rates are per-step counter deltas from the in-process history store, so they read 0 until the first step (korvet.admin.metrics-history.interval, default 15s) completes. Each entry also carries samples, the topic’s per-step produced-traffic history over the retention window (oldest first, empty when the topic has no history series).

  • GET /api/v1/metrics/history: In-memory time-series of curated metrics snapshots for sparkline visualization. Sampled at a configurable interval (default 15s) with bounded retention (default 1h), controlled by korvet.admin.metrics-history.interval and korvet.admin.metrics-history.retention. The buffer is process-local and starts empty on restart—no storage dependency or cross-restart persistence.

  • GET /api/v1/metrics/consumer-lag: Aggregate consumer-group lag in a single call, returning per-group totalLag, maxLag, partitionsWithUnknownLag, and lastCommitTimestamp. Totals are lower bounds when some partitions have unknowable lag.

These endpoints complement the raw Prometheus export with pre-aggregated, paginated data optimized for real-time dashboard polling. See Admin API: Monitoring for full endpoint details and response schemas.

Available Metrics

Korvet Custom Metrics

  • korvet.broker.produce: Produce request latency histogram (tag: topic)

  • korvet.broker.fetch: Fetch request latency histogram (tag: topic)

  • korvet.broker.request: Kafka API request latency histogram (tags: api_key, result)

  • korvet.broker.frame_size: Kafka frame size distribution (tags: direction, api_key)

  • korvet.broker.up: Broker lifecycle gauge

  • korvet.broker.failures: Broker failure counter (tags: operation, error_type). Authentication failures arrive here with operation=auth.

  • korvet.group.lag: Per-consumer-group total lag gauge (tag: group). Sampled server-side every 15s (configurable via korvet.broker.metrics.group-lag-refresh-interval, default 15s) from ConsumerGroupService.lag(). Gauges read a cached map so Prometheus scrapes never trigger Redis fan-out. Reports NaN when lag is unknowable (metadata lookup failed, no partition lag known, or refresh failure), allowing Prometheus to drop the sample and metrics history to record a gap. Cardinality is bounded by korvet.broker.metrics.group-lag-cardinality-cap (default 1000); when exceeded, all gauges read NaN and a warning is logged. In multi-broker clusters, every node publishes its own view of cluster-wide lag; deduplicate or aggregate by instance label in Prometheus.

  • korvet.storage.read, korvet.storage.write, korvet.storage.ack: Cross-tier storage verb latency timers (tags: tier, result; read also carries mode=stream|group)

  • korvet.storage.read.messages, korvet.storage.write.messages: Cross-tier messages-per-call distribution summaries (tag: tier; read.messages also carries mode)

  • korvet.storage.archive, korvet.storage.archive.segments, korvet.storage.archive.bytes, korvet.storage.archive.failures, korvet.storage.archive.lag.\*: Cold-tier offload latency, throughput, failures, and sealed-segment backlog.

  • korvet.storage.local.pool.acquire / korvet.storage.local.pool.pending: Local Redis connection-pool meters (acquire timer carries result=success|timeout|error)

  • korvet.storage.worker.up, korvet.storage.worker.failures: Storage worker liveness gauge and failure counter (failures tagged phase, error_type). The worker’s storage I/O flows through the shared korvet.storage.{read,write,ack} timers tagged tier=remote.

See Metrics Reference for the full module-by-module catalog.

JVM and System Metrics

Standard JVM and system metrics from Micrometer:

  • jvm.memory.used: JVM memory used

  • jvm.memory.max: JVM maximum memory

  • jvm.gc.pause: Garbage collection pause time

  • jvm.threads.live: Live threads

  • process.cpu.usage: Process CPU usage

  • system.cpu.usage: System CPU usage

  • system.load.average.1m: System load average

Prometheus Configuration

Add Korvet to your Prometheus scrape config. When korvet.admin.security-enabled=true (the default), you have two options:

Option 1: HTTP Basic Authentication (Recommended)

Configure Prometheus to authenticate with console-user credentials (a read-only VIEWER user is sufficient for scraping):

scrape_configs:
  - job_name: 'korvet'
    metrics_path: '/actuator/prometheus'
    basic_auth:
      username: admin
      password: your-admin-password  # Use the actual admin password
    static_configs:
      - targets: ['korvet:8080']
Store credentials securely using Prometheus configuration management, Kubernetes secrets, or environment variable substitution. Do not commit plaintext passwords to version control.

Option 2: Anonymous Prometheus Access

Enable anonymous access for the Prometheus endpoint only:

# Korvet configuration
korvet:
  admin:
    security-enabled: true
    anonymous-prometheus: true  # Allow unauthenticated /actuator/prometheus

Then configure Prometheus without authentication:

scrape_configs:
  - job_name: 'korvet'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['korvet:8080']
The korvet.admin.anonymous-prometheus flag is used by the benchmark kustomize base to keep its ServiceMonitor credential-free. For production deployments, HTTP Basic authentication (Option 1) is recommended.

In-Process Metrics History

Korvet maintains a short-term in-process metrics history store powered by a Micrometer StepMeterRegistry. This store captures per-step metrics snapshots (default: every 15 seconds) and retains them in bounded ring buffers (default: 1 hour). The history powers the web console dashboard sparklines and the Admin API’s /api/v1/metrics/history endpoint.

How It Works

At each step interval:

  • Counters yield per-step deltas, producing natural per-second rates

  • Timers capture percentile values (p50, p95, p99) aggregated across the step

  • Gauges sample their current value

All values are stored as scalars (never raw histogram snapshots) in memory-bounded ring buffers keyed by meter ID. The store is optimized for dashboard convenience and low cardinality — it’s not a replacement for Prometheus or other long-term monitoring systems.

Configuration

Control the history store via these properties:

korvet:
  admin:
    metrics-history:
      enabled: true              # Serve the /metrics/history series
      interval: 15s              # Capture interval (step duration)
      retention: 1h              # How long to retain samples
      max-series: 10000          # Maximum tracked metric series

  • enabled: Whether GET /api/v1/metrics/history returns data. When disabled, the endpoint returns an empty series, but the in-process history store keeps capturing because /metrics/topics and /storage-stats read from it.

  • interval: How often metrics are captured. Shorter intervals provide finer-grained data but increase memory usage. Must be at least 1 millisecond.

  • retention: Total time to keep samples. Longer retention uses more memory. The effective buffer size is retention / interval samples per series.

  • max-series: Cardinality cap on per-topic and per-group meter series (e.g., per-topic produce/fetch metrics and per-group lag gauges). Curated broker-level meters are always captured. When the cap is reached, new topic- or group-tagged series are rejected with a logged warning.

API Endpoints

The metrics history store backs several Admin API endpoints:

  • GET /api/v1/metrics/history: Curated per-step metrics for dashboard sparklines. Counter-backed fields (*PerSecond) are already computed as per-second rates — no client-side delta math required. The response includes messagesInPerSecond, bytesInPerSecond, produceRequestsPerSecond, fetchRequestsPerSecond, request latency percentiles, and failure rates (brokerFailuresPerSecond, lossyRecordsPerSecond), oldest sample first.

  • GET /api/v1/metrics/topics: Per-topic produce/fetch rates (records/s, bytes/s, requests/s) and latency percentiles, averaged over the configured lookback window from the history store. Rates read as 0 until the first step completes.

  • GET /api/v1/storage-stats: Remote storage throughput rates (segmentsPerMin, bytesPerMin) derived from history-store counters.

Memory and Cardinality

The history store is an in-memory cache that resets on every restart. To keep memory usage bounded:

  • Only a curated allowlist of meter names is captured (broker request timers, produce/fetch per topic, storage I/O, worker metrics).

  • Topic- and group-tagged series are counted against max-series so unbounded topic or consumer-group creation doesn’t exhaust memory.

  • Retention is short (default 1 hour) and controlled by the retention property.

Example Prometheus Queries

See Metrics Reference for the full metric and tag vocabulary used below.

Produce latency (p99) per topic:

histogram_quantile(0.99, sum by (topic, le) (rate(korvet_broker_produce_seconds_bucket[5m])))

Fetch latency (p95) per topic:

histogram_quantile(0.95, sum by (topic, le) (rate(korvet_broker_fetch_seconds_bucket[5m])))

Broker request rate by API key and result:

sum by (api_key, result) (rate(korvet_broker_request_seconds_count[5m]))

Local Redis pool timeout rate:

rate(korvet_storage_local_pool_acquire_seconds_count{result="timeout"}[5m])

Read mix by tier:

sum by (tier, result) (rate(korvet_storage_read_seconds_count[5m]))

Broker failure rate by operation and error type:

sum by (operation, error_type) (rate(korvet_broker_failures_total[5m]))

Redis command latency (p99) by command:

histogram_quantile(0.99, sum by (command, le) (rate(lettuce_command_completion_seconds_bucket[5m])))

Consumer group lag by group:

korvet_group_lag{group="my-consumer-group"}

Consumer groups with high lag (threshold: 1000 messages):

korvet_group_lag > 1000

Monitoring Stack Setup

A complete monitoring stack with Prometheus and Grafana is available in the korvet-dist observability sample directory.

Quick Start with Docker Compose

The easiest way to set up monitoring is to include the observability stack in your docker-compose.yml:

include:
  - path/to/korvet-dist/samples/observability/monitoring.yml

services:
  redis:
    image: redis:8.6
    ports:
      - "6379:6379"

  korvet:
    image: redisfield/korvet:latest
    command: server
    ports:
      - "9092:9092"
      - "8080:8080"
    environment:
      - KORVET_REDIS_HOST=redis
      - KORVET_REDIS_METRICS_ENABLED=true
    depends_on:
      - redis

This automatically adds:

  • Prometheus on port 9090 - Metrics collection and storage

  • Grafana on port 3000 - Pre-configured dashboard for Korvet metrics

Accessing the Dashboard

  1. Start your services:

    docker compose up -d

  2. Open Grafana at http://localhost:3000

  3. The Korvet dashboard loads automatically (no login required)

Dashboard Features

The pre-built Grafana dashboard visualizes:

  • Message Rates: Real-time produce/fetch rates using irate() for instant metrics

  • Latency Percentiles: P50, P95, P99 for produce and fetch operations

  • Throughput: Ingress and egress bytes/sec

  • Redis Metrics: Command rates, latency percentiles (when korvet.redis.metrics.enabled=true)

  • JVM Metrics: Heap memory, GC pauses, thread count

  • System Metrics: CPU usage, load average, disk space

The dashboard defaults to a 5-minute time range with 5-second auto-refresh for real-time monitoring.

Standalone Setup

For production deployments, see the observability README for:

  • Prometheus configuration examples

  • Grafana datasource and dashboard provisioning

  • Customizing the dashboard

Broker Health Signals

The metrics snapshot includes operational health signals beyond basic counters:

  • backpressureConnections: Current count of connections experiencing backpressure (gauge). A sustained non-zero value indicates the broker is limiting client throughput due to downstream constraints. The dashboard surfaces this as a warning banner and tile.

  • rebalances: Cumulative count of completed consumer-group rebalances. Only complete phase events are counted, so each rebalance increments this once.

  • brokerFailures: Cumulative count of broker failures as a single aggregate number (the underlying Prometheus meter carries operation and error-type tags). Rendered as context on dashboard tiles rather than permanent alerts—use for trend analysis and post-incident investigation.

  • lossyRecords: Cumulative count of records that experienced data loss or corruption. Should remain zero in normal operation.

These signals are available in the GET /api/v1/metrics snapshot and provide early warning of degraded operation.

Alerting

Set up alerts for:

Korvet-Specific Alerts

  • High produce latency: p99 > 100ms

  • High fetch latency: p99 > 50ms

  • Backpressure active: backpressureConnections > 0 sustained for more than 1 minute

  • High consumer group lag: korvet.group.lag exceeds your threshold for a sustained period

  • Broker request error spike: result=error share of korvet.broker.request > 1% of requests

  • Redis pool contention: sustained korvet.storage.local.pool.pending growth or korvet.storage.local.pool.acquire{result="timeout"} rate

  • No produce activity: No messages produced in 5 minutes (if expected)

  • Consumer lag growth: Per-group totalLag increasing over time (query /api/v1/metrics/consumer-lag)

System Alerts

  • Memory pressure: JVM heap > 80%

  • High GC activity: Frequent or long GC pauses

  • High CPU usage: Process CPU > 80%

Logging

See Logging for log-based monitoring.

Next Steps