KV cache-aware load balancing

For each vLLM worker, Kinference automatically tracks the KV cache content, queue length, and other aspects of runtime state, and uses that information for load-balancing decisions.

This tracking is done automatically once Kinference is configured with an InferenceWorkerSet for the vLLM service. Note that this feature requires that vLLM be configured to emit KV cache events via ZMQ.

Kinference uses this information to provide a predicted TTFT measure per vLLM endpoint, for a specific prompt. As the name suggests, this measure is the prediction of the time-to-first-token if a specific prompt were to be sent to a specific vLLM instance.

Load balancing by predicted TTFT is the default behavior for Kinference, and in most situations will “just work”. It can also be explicitly specified in the route.select clause of inference policies.