Getting started

Welcome to Kinference! 🎈

Kinference [kay-inference] is an inference control plane for Kubernetes. You deploy your models using vLLM (or in the future, SGLang and TensorRTLLM), and Kinference wraps them all into a single policy-driven inference serving platform that includes:

  • Per-tenant routing policies
  • Token rate limits
  • Model aliases (e.g. “gemma-4” instead of “google/gemma-4-12B-it”)
  • A full suite of inference metrics (including TTFT, ITL, and end-to-end latency)
  • Semantic classification (e.g. PII, prompt complexity)
  • Burst-to-cloud when the local vLLM pool is saturated
  • And much more.

Your users simply need to send their OpenAI-compatible inference requests to an endpoint you publish, and Kinference will do the rest, using the policies you’ve defined.

The latest version of Kinference is v0.1.0-alpha.7.

01 Install the Gateway API CRDs #

Note: This guide installs Kinference into the default namespace. To install into a different namespace, override the namespaces in the following helm commands.

The first step is to install the Gateway API CRDs, which provide standard mechanisms for handling traffic.

GWAPI_VERSION=1.5.1
kubectl apply --server-side --force-conflicts -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$GWAPI_VERSION/standard-install.yaml

02 Install the Kinference gateway #

Kinference has two basic components: a traffic gateway and an inference handler. Each is released independently and ships its CRDs as a separate chart, installed before the component itself.

Install the gateway CRDs and components:

helm upgrade -i gateway-crds oci://ghcr.io/buoyantio/charts/gateway-crds --devel
helm upgrade -i gateway oci://ghcr.io/buoyantio/charts/gateway --devel --wait

03 Install Kinference #

Now, install Kinference and the Kinference CRDs:

helm upgrade -i kinference-crds oci://cr.kinference.ai/kinference-ai/charts/kinference-crds --devel
helm upgrade -i kinference oci://cr.kinference.ai/kinference-ai/charts/kinference \
  --set extension.enabled=true \
  --devel --wait

04 Make sure it’s working #

kubectl get po

You should see an output like this:

NAME                               READY   STATUS    RESTARTS   AGE
gateway-cp-566cddc46d-jsqlq        1/1     Running   0          8s
kinference-7bb854cfc5-pp4t7        1/1     Running   0          8s

05 Install the CLI on your local machine #

The Kinference CLI is designed to assist in agent-guided workflows for configuring Kinference. For example, if you’re using an agent to help develop policies, the CLI will allowing you to validate potential policies before shipping them to the server.

To install the CLI, run:

curl -fsSL https://raw.githubusercontent.com/BuoyantIO/kinference-releases/main/install.sh | sh

Add it to your path with:

export PATH="$PATH:$HOME/.kinference/bin"

Finally, validate you have it with:

kinfctl --version

You should see something like:

kinfctl 0.1.0-alpha.7

You’re finally ready for action!

Next steps #

  1. Install the Kinference Grafana dashboards.
  2. Set up tenant tracking.
  3. Configure basic inference serving with Kinference.