Basic inference serving

In this guide, we’ll configure Kinference to send inference requests to your existing fleet of vLLM workers, and explore some basic inference platform and observability capabilities.

Prerequisites #

You should have Kinference installed. If not, start here.

You should also have a vLLM deployment running on your cluster. If you don’t, you’ll need to do that first. To get started with vLLM on Kubernetes, we recommend the vLLM Helm documentation.

Overview #

The basic Kinference configuration involves a few types of Kubernetes resources:

An InferenceWorkerSet that wraps each vLLM Service;
An InferencePool that aggregates these worker sets together;
A Gateway API HTTPRoute that points to the pool, and instructs Kinference’s traffic enforcement gateway component as to which endpoints to handle.

Once these Kubernetes resources have been created, traffic to Kinference’s gateway will be handled through Kinference.

(In a later guide, we’ll also create an InferencePolicy resource to provide more control over how requests are handled in the pool–but for now, creating the pool is sufficient.)

Working example #

A simple working example is as follows. Let’s assume we have a Kubernetes Service resource that points to one or more vLLM pods and is called qwen-vllm, in the default namespace.

First, the InferenceWorkerSet. Let’s call it qwen-vllm-ws:

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferenceWorkerSet
metadata:
  name: qwen-vllm-ws
  namespace: default
spec:
  profile: default
  kvBlockSize: 16

  serviceRef:
    name: qwen-vllm # name of the Service
    port: http
  kvEvents:
    port: kv-pub

  models:
    - name: Qwen/Qwen2.5-32B-Instruct-AWQ
      aliases: [qwen-2.5, qwen-simple]

This worker set names the qwen-vllm Service in its serviceRef: component, expresses that its block size is 16, and also adds “qwen-2.5” and “qwen-simple” as model aliases of the “Qwen/Qwen2.5-32B-Instruct-AWQ” model. Note that this models: block is optional and exists only to allow for aliases.

A pool that contains one worker set is relatively trivial. Let’s call it “inference-pool”:

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePool
metadata:
  name: inference-pool
  namespace: default
spec:
  destinations:
    - { kind: InferenceWorkerSet, name: qwen-vllm-ws }

At this point we have the inference side of configuration done, so we need to layer on the networking side of the config. Kinference uses the Gateway API for its configuration, so we’ll need two components: a gateway and a route.

First, the Gateway resource, which we’ll call “kinference-gw”:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kinference-gw
  namespace: default
spec:
  gatewayClassName: buoyant-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 8080

Finally, tying it all together, an HTTPRoute that references this gateway and points to the inference pool, and sends all traffic to /v1/chat/completions to it. Let’s call it “kinference-route”:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: kinference-route
spec:
  parentRefs:
    - name: kinference-gw # our gateway
      sectionName: http
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/chat/completions
      backendRefs:
        - kind: InferencePool
          group: kinference.buoyant.io
          name: inference-pool # our inference pool

And we’re done! At this point, inference requests to port 8080 on path /v1/chat/completions will go through Kinference, and will target the qwen-vllm service, with all features enabled.

Above was an example. Now let’s actually build some configuration programmatically, to reflect the actual vLLM services on your cluster.

Prerequisite: Ensure your vLLM deployments are configured correctly #

To make full use of Kinference, we need to make sure that vLLM is configured correctly. Specifically, we’ll need:

A relatively new version of vLLM (e.g. 0.28.0 or later)
KV cache event data exported via zmq
Prompt token details enabled

KV cache event data is used by Kinference for its KV cache-aware load balancing feature. To enable it, ensure a kv-events-config commandline argument is passed to vLLM at startup. It should look something like this:

kubectl get deploy/vllm -o yaml | grep -- --kv-events-config

You should see something like:

        - --kv-events-config={"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5557","replay_endpoint":"tcp://*:5558","topic":"kv@"}

If this argument is missing, you will need to add it and restart vLLM.

Similarly, in order to track token usage correctly, Kinference needs certain token counts from vLLM. You will need to ensure the --enable-prompt-tokens-details flag has been passed in:

kubectl get deploy/vllm -o yaml | grep -- --enable-prompt-tokens-details

You should see something like:

        - --enable-prompt-tokens-details

If that flag is missing, you will need to add it and restart vLLM.

Create the InferenceWorkerSets #

With our properly-configured vLLM, the first step is an InferenceWorkerSet that corresponds to the vLLM deployment.

To do this, you’ll need:

The name of the Kubernetes Service resource that fronts the vLLM deployment
The KV cache block size of the service
If you want to provide model aliases: the names of the models served.

The block size is important: Kinference does not detect this automatically, and needs it to correctly parse the KV cache event data. Most of the time, the KV cache block size can be determined from the command line arguments:

kubectl get deploy/vllm -o yaml | grep -- --block-size

You might see something like this:

        - --block-size=16

Similarly, the model name:

kubectl get deploy/vllm -o yaml | grep -- --served-model-name

You might see something like this:

        - --served-model-name=Qwen/Qwen2.5-32B-Instruct-AWQ

Given this information, we can construct an InferenceWorkerSet like so, filling in $DEPLOY, $NS, and $ALIAS with any short name of the model (e.g. “qwen-simple”). (For simplicity, in this doc we will create this and subsequent resources in the “default” namespace.)

DEPLOY=my-deployment # set to your vLLM deployment name
NS=my-namespace      # set to your vLLM namespace
ALIAS=my-alias       # set to any alias you'd like to use

MANIFEST=$(kubectl -n $NS get deploy/$DEPLOY -o yaml)
MODEL_NAME=$(echo "$MANIFEST" | grep -- --served-model-name | sed -n 's/.*=//p')
BLOCK_SIZE=$(echo "$MANIFEST" | grep -- --block-size | sed -n 's/.*=//p')

kubectl apply -f - <<EOF
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferenceWorkerSet
metadata:
  name: $DEPLOY
  namespace: default
spec:
  profile: default
  kvBlockSize: $BLOCK_SIZE

  serviceRef:
    name: $DEPLOY
    port: http
  kvEvents:
    port: kv-pub

  models:
    - {name: $MODEL_NAME, aliases: [$ALIAS]}
EOF

Your first InferenceWorkerSet has been created. Check its status with:

kubectl get iws/$DEPLOY

You should see something like:

NAME          READY   STATE      AGE
gemma-4-12b   1       Accepted   18s

If you don’t see Accepted, check the InferenceWorkerSet reference for a diagnosis.

Create an InferenceWorkerSet in this manner for every vLLM Service you would like to serve through Kinference.

Create the InferencePool #

The next step is easy. A pool is simply a collection of workersets, so we can create it like so (giving it the imaginative name “inference-pool”):

LIST=$(kubectl get --no-headers iws | awk '{printf "    - {kind: InferenceWorkerSet, name: %s}\n", $1}')
kubectl apply -f - <<EOF
apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePool
metadata:
  name: inference-pool
  namespace: default
spec:
  destinations:
$LIST
EOF

Check its status with:

kubectl get infpool/inference-pool

You should see something like:

NAME              POLICY    DESTINATIONS    STATE       AGE
inference-pool              2               Accepted    12m

If you don’t see Accepted, check the InferencePool reference for a diagnosis.

Create the Gateway and HTTPRoute #

Finally, the networking configuration to tie it all together:

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kinference-gw
  namespace: default
spec:
  gatewayClassName: buoyant-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: kinference-route
spec:
  parentRefs:
    - name: kinference-gw
      sectionName: http
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /v1/chat/completions
    backendRefs:
      - kind: InferencePool
        group: kinference.buoyant.io
        name: inference-pool
EOF

Now check the state of these resources:

kubectl get gateway,httproute

You should see something like:

NAME                                              CLASS             ADDRESS                                                                        PROGRAMMED   AGE
gateway.gateway.networking.k8s.io/kinference-gw   buoyant-gateway   k8s-default-kinferen-1dcb3455a3-39f63ac0e9cb18ca.elb.us-west-2.amazonaws.com   True         11d

NAME                                                  HOSTNAMES   AGE
httproute.gateway.networking.k8s.io/kinference-route               12d

If you don’t see PROGRAMMED of “True”, check the Gateway API troubleshooting docs.

Because we’re using the Gateway API, the creation of the Gateway resource above actually created a Service called kinference-gw-gatewayd. Let’s also confirm this service exists:

kubectl get svc/kinference-gw-gatewayd

You should see something like:

NAME                    TYPE            CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kinference-gw-gatewayd   LoadBalancer   172.20.10.100    ......        8080:30167/TCP   25s

This is the Service which will handle our inference traffic.

At this point you should have the configuration machinery ready for action: we’ve got a gateway and corresponding Service, a route that matches /v1/chat/completions and is backed by an InferencePool that contains our vLLM instances.

Let’s give it a try! Replace MODEL_NAME below and run:

kubectl port-forward svc/kinference-gw-gatewayd 8080
curl -v -d '{"model":"MODEL_NAME", "messages":[{"role":"user","content":"Hello world"}]}' -H 'Content-Type: application/json' localhost:8080/v1/chat/completions

If all is well, you’ll get a simple response from your inference pool. Congratulations, Kinference is now working!

If you haven’t already, your next steps should include:

Install the Kinference Grafana dashboards.
Set up tenant tracking.
Writing your first inference policy.