The inference control plane for Kubernetes and vLLM
Kinference [kay-inference] turns Kubernetes and vLLM into a fully-featured distributed inference platform, with token rate limits, routing policies, cache-aware load balancing, and everything else needed to serve multiple models, tenants, and SLOs.
Solve routing, limiting, and balancing in one layer
Scalable inference across multiple workers requires routing, rate limiting, load balancing, and more. Kinference provides a unified layer that plugs into any existing vLLM deployment and combines these capabilities in a principled manner.
Build a scalable inference platform
Running vLLM is just the starting point. Kinference is the mediator between clients and your GPU fleet, giving you a single OpenAI- and Anthropic-compatible endpoint that applies policy, optimizing load, and tracks inference-specific metrics.
KV-cache-aware balancing
Cache-aware balancing optimizes inference response times and maximizes GPU utilization using each replica’s live queue and cache state.
Token rate limiting
Prevent tenants from consuming more capacity than they should.
Per-tenant cost attribution
Measure and report cached prompt token and output token usage, allowing you to measure per-tenant GPU utilization and attribute cost.
Inference "golden metrics"
Measure per-tenant TTFT, ITL, TPOT, and end-to-end latency, and evaluate against declared service objectives.
Capacity modeling
Measure GPU saturation by tenant and over time to build accurate capacity models for your GPU usage.
Semantic classification
Tag requests by complexity, PII, and other factors, and ensure they’re routed correctly.
Control everything with inference policies
Kinference’s declarative policy language allow you to control exactly how inference requests are handled, based on user, model, prompt, state of the GPU pool, and more.
Example policies
Send the simple prompts to less expensive models
Semantically classify prompts by complexity level, and route simpler prompts to cheaper models. But take into account cache usage: if it's faster to send it to an expensive model (because of caching), just do that.
Without Kinference
The model that the user selects is used directly, regardless of suitability for the model.
With Kinference
Inference platform owners can decide exactly under which situations a model is used, including by taking into account semantic classification and GPU cache overlap.
InferencePolicy
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata: { name: smart-semantic-routing,
namespace: inference }
spec:
rules:
- name: low-complexity-consider-qwen
matches:
- model: glm-5.3
tag: complexity-low
route:
consider:
- models: { names: [ qwen-3.6-27b,
glm-5.3 ] }
select: [ { minimize: PredictedTTFT } ]Reading the policy
matches:This rule applies only when the requested model is glm-5.3 and the classifier has tagged the prompt as low complexity.
consider:This rule can route to either glm-5.3 or qwen-3.6-27b.
minimize: PredictedTTFTThis rule picks whichever endpoint is the fastest, based on the predicted TTFT for the prompt. This measure incorporates both KV cache overlap and GPU queue length.
Send requests to cloud providers if GPU pool is saturated
When queue depth or cache pressure indicate GPUs are saturated, burst the excess requests to cloud providers.
Without Kinference
Requests to overloaded GPUs go into the pending queue, where they may cause client timeouts
With Kinference
Inference platform owners can decide exactly what happens when GPU pools are saturated, including sending to a third-party inference provider.
InferencePolicy
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata: { name: burst-policy,
namespace: inference }
spec:
rules:
- name: burst-to-cloud
route:
exclude:
- name: drop-overloaded-workers
drop:
backend: { type: Local }
queuedPrompts: { greaterThan: 2 }
select:
- { type: Local,
minimize: PredictedTTFT }
- { type: Provider,
order: [bedrock, databricks] }Reading the policy
exclude:Drop from consideration any workers that have more than an average of 2 pending prompts over the past 30 seconds
select:Select from the remaining endpoints, in order: local endpoints first, then third-party providers.
Ensure prompts that contain PII never leave the environment
A more sophisticated variant of burst-to-cloud. Semantically classify prompts by whether they contain PII, and burst requests to the third-party provider if the GPU pool is saturated -- unless the prompt contains PII, in which case it would be better to fail it entirely.
Without Kinference
No control over whether PII leaves the cluster.
With Kinference
Inference platform owners explicitly decide how inference that contains PII is handled.
InferencePolicy
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata: { name: pii-stays-local,
namespace: inference }
spec:
rules:
- name: serve
route:
exclude:
- name: pii-never-leaves-cluster
if: { tag: pii-detected }
drop: { backend: { type: Provider } }
select:
- { type: Provider,
order: [ databricks ] }
- { type: Local,
minimize: PredictedTTFT }
onEmpty: Reject # fail closedReading the policy
exclude:Remove all provider (third-party) endpoints if the request contains PII.
select:Select from the remaining endpoints, in order: local endpoints first, then third-party providers.
onEmpty: RejectFail closed. If nothing acceptable survives the filter, refuse the request rather than fall back to something unsafe.
Run batch work only on GPUs with room to spare
Batch and low-priority requests share the same GPUs as interactive traffic. Send them only when replicas have KV cache to serve them, and reject them when every replica is busy, so a nightly eval run can’t slow down responses for live users.
Without Kinference
Batch jobs land on the same replicas as live users, fill the KV cache, and slow down interactive responses.
With Kinference
Inference platform owners can serve batch and low-priority work on GPUs with spare KV cache capacity, and reject it when there isn’t any.
InferencePolicy
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata: { name: batch-only-to-idle-gpus,
namespace: inference }
spec:
rules:
- name: batch-only-to-idle-gpus
matches:
- qos: batch
- qos: deferrable
route:
consider: [ { type: Local } ]
exclude:
- name: busy
drop:
backend: { type: Local }
kvCacheUsage: { greaterThan: 0.7 }
select: [ { type: Local, minimize: PredictedTTFT } ]
onEmpty: RejectReading the policy
matches:Applies to batch or deferrable requests. Each entry is checked on its own, so either one matches.
exclude:Drops any local worker averaging over 70% KV cache usage in the last 30 seconds.
select:Of the workers left, picks the one with the lowest predicted TTFT.
onEmpty: RejectIf no workers are left, rejects the request so it can’t queue behind live traffic.
Provide abstract model names to clients
Application owners shouldn’t have to track checkpoint names. Give models an alias--or even group several models under the same alias--and tell Kinference how to resolve them.
Without Kinference
Every client hard-codes a model name, so swapping or adding a model means updating every client.
With Kinference
Inference platform owners can publish one stable name, change which models are behind it, and let policy pick the fastest one for each request.
InferencePolicy
# qwen-worker (InferenceWorkerSet)
models:
- name: Qwen/Qwen3.6-27B
aliases: [ standard-model, qwen3.6, qwen ]
# glm-worker (InferenceWorkerSet)
models:
- name: zai-org/GLM-5.3
aliases: [ standard-model, glm-5.3, glm ]
---
kind: InferencePool
metadata: { name: standard-pool }
spec:
destinations:
- { kind: InferenceWorkerSet, name: qwen-worker }
- { kind: InferenceWorkerSet, name: glm-worker }
---
kind: InferencePolicy
spec:
rules:
- name: fastest-standard-model
matches: [ { model: standard-model } ]
route:
select: [ { type: Local, minimize: PredictedTTFT } ]Reading the policy
aliases:Both worker sets answer to standard-model. Clients can still ask
for qwen or glm to get one model.
destinations:The pool groups both worker sets. Clients send requests to the pool.
matches:This rule applies to requests that ask for standard-model.
select:Of every worker serving the alias, picks the one with the lowest predicted TTFT.
Frequently asked questions
How does Kinference use vLLM?
vLLM is the data plane; Kinference is the control plane. Kinference builds on top of vLLM to provide a scalable, multi-tenant inference platform. Kinference ties together multiple vLLM instances into a unified platform capable of serving many models, tenants, SLOs, and more.
Is Kinference different from an AI gateway?
Yes. AI gateways send traffic across many different external inference providers, handling the peculiarities of each one. Kinference is designed for local inference running on GPUs in the cluster. Kinference can be paired with an AI gateway, which it can use when it offloads traffic from the local cluster.
Which models does Kinference support?
Kinference is model agnostic and can support any model that vLLM is capable of running.
Does my AI application need code changes?
No. Simply point your client at the Kinference service instead of the model endpoint. Model selection, fallback, and rate limiting move into Kubernetes resources managed by the platform team.
How do I run Kinference?
Kinference is a standard Kubernetes deployment. It contains an enforcement gateway that handles inference traffic. Once you deploy Kinference to your Kubernetes cluster, all you need to do is point inference requests through the Kinference enforcement gateway.
Kinference ships with Helm charts as well as a suite of metrics and dashboards in standard Prometheus and Grafana formats, so plugging it into your existing Kubernetes stack is easy.