Features
- KV cache-aware load balancing Optimize inference speed and GPU utilitization by sending prompts to the best worker. Kinference automatically tracks KV cache content, queue length, and other factors per vLLM workers to make the optimal load-balancing decision.
- Model aliases Kinference allows you to provide aliases for long-form model checkpoint names, such as "glm" for "zai-org/GLM-5.3" or "qwen-3.6" for "Qwen/Qwen3.6-35B-A3B".
- Policy-driven inference serving Kinference allows you to control exactly how inference requests are served in your system through declarative policies that can take into account tenant id, model requested, prompt size, semantic classification, the state of your local GPUs, and more.
- Production-ready metrics, monitoring, and dashboards Kinference provides a complete suite of live system metrics, including operational metrics as well as per-tenant inference metrics such as TTFT, ITL, and end-to-end latency. It also ships with a fully-featured Grafana dashboard that provides an out-of-the-box operational view of the state of your inference platform.
- Semantic classification Kinference allows you apply semantic classification to prompts, to determine attributes such as "does this request contain PII" or "is this request complex or simple?". Classification information can be used by inference policies to intelligently serve requests.
- Token-based rate limiting Kinference allows you to limit tenant inference usage by input and output token rates.