Policy-driven inference serving

Kinference allows you to control exactly how inference requests are served in your system through declarative policies that can take into account tenant id, model requested, prompt size, semantic classification, the state of your local GPUs, and more.

Policies are expressed via InferencePolicy resources. Policies determine, for each inference request sent through Kinference:

1. Whether the request is served or rejected;
2. If served, whether it is served from a local vLLM pod or an off-cluster provider;
3. If local, which pod is targeted; and
4. Which model is actually used for the prompt (which may or may not be the model requested!)

To make this decision, policies can take into account attributes of the request, including:

1. The tenant (aka client) id
2. The size of the prompt
3. The model requested
4. Any semantic classifications of the prompt (e.g. does it contain PII, is it complex)

Policies can also take into account attributes of the system, including:

1. Availability of endpoints
2. Saturation of vLLM instance, as measured by e.g. pending queue size or KV cache usage

Combining these into one logical system allows you to express a wide variety of behaviors, including:

1. Model optimization: Send any request for GLM that is classified as simple to Qwen instead
2. Cache-aware model optimization: Send any request for GLM that is classified as simple to Qwen instead, unless it’s acutally faster to serve them from GLM (e.g. because the KV cache is already warm)
3. Burst to cloud: If our local vLLM pool is entirely saturated, send excess requests to a third-party provider
4. PII-safe burst to cloud: If our local vLLM pool is entirely saturated, send excess requests to a third-party provider, unless they contain PII, in which case fail them and have the client retry.

Policies can be written agentically and Kinference provides a CLI for local agentic validation. See Writing inference policies for a walkthrough of how to write policies.