Metrics

Kinference exports a variety of metrics in Prometheus-compatible format. By default, these metrics are available via HTTP on port 9001 under the /metrics path.

A note on per-tenant metrics #

Metrics are optionally available on a per-tenant basis, i.e. with a tenant label. However, this can result in a large number of metrics: approximately 91 series per (model, qos, tenant) tuple. At 5 models and 3 QoS classes that is about 1,365 series per tenant.

Per-tenant metrics are off by default, but can be configured (including via an explicit allowlist) via the [metrics] tenant_labels configuration block:

As with tenants outside of the allowlist, requests without tenant information will also be bucketed under the tenant value of default.

Token usage metrics #

Token usage per model, QoS level, and tenant. Measured at prompt (input) and completion (output).

Cached tokens are tokens within the prompt that were served from the KV cache rather than requiring computation. Sophisticated token attribution / billing systems may wish to charge for only uncached prompt tokens. This field is not always reported by workers; the “absent” counter tracks the number of prompts for which it was not reported.

Latency metrics #

Response latency, per model, QoS level, and tenant. Measured at request_duration (end-to-end latency), TTFT (time to first token), and ITL (inter-token latency).

Note that TTFT and ITL metrics only populate in the presence of streaming requests.

Prefix cache metrics #

Performance of KV-cache-aware load balancing.

Prefix index health metrics #

The internal radix tree state, for diagnostic purposes.

Tokenization metrics #

Where path is one of:

bllmp_tokenize_compare_total reports result as match, divergence or error. Divergence means that Kinference’s block hashes disagree with those reported by the engine. This is a signal that local tokenization is not accurate. Note that this value is only non-zero when [profile.tokenize] mode is compare.

Routing decision metrics #

bllmp_predicted_ttft_ms is the distriubtion of predicted TTFT across all candidates scored (not just those that won). Uses the same buckets as bllmp_ttft_ms, so the predictions are comparable.

Where outcome is one of:

Configuration health metrics #

Number of InferenceWorkerSets rejected and the reason, which includes: MissingProfile, UnknownProfile, MissingKvBlockSize, InvalidKvBlockSize, MissingServiceRef, KvEventsMissingPort, and ModelMissingName.

Request buffering metrics #

Internal metrics capturing the buffering of request bodies by Kinference, for diagnosing Kinference memory usage.

Leadership election metrics #

Internal metrics corresponding to Kubernetes leader election. For diagnostic purposes.

Auth and rate limiting metrics #

Possible token_review outcomes are allowed, invalid, timeout and transport.

Possible rate_limit outcomes are allowed, commit_ok and commit_not_found.

Build identity metrics #

Always 1; the information is in the version label, which matches the output of kinference --version. Across a deployment, more than one version being active at a time is an indication of a stalled rollout.

Router service health metrics #

Performance of Pick() gRPC call.

Where outcome is one of:

Note that bllmp_pick_duration_ms is Kinference’s view of the call latency, not the client’s.

bllmp_workers reports state as either ok or unresponsive, the latter meaning the worker’s metrics scrape has been stale for over 5 seconds.

Worker tracking statistics #

KInference’s view of workers (e.g. vLLM pods)

Workers are evicted with reason block_size_mismatch when KInference receives block sizes different form the InferenceWorkerSet’s kvBlockSize configuration.

Worker saturation statistics #

These are the same values available to queuedPrompts and kvCacheUsage candidate signals in Policies, allowing saturation thresholds to be tuned from real data.