Guides

Writing inference policies

In this guide, we’ll walk you through how to write a basic inference policy.

Inference policies determine how Kinference handles inference requests. When a new request comes in, Kinference can take one of several actions:

  1. It can send it to a specific vLLM pod on the cluster.
  2. It can send it to a specific off-cluster inference provider.
  3. It can deny the request.

Policies are how you, the operator, control how this decision is made. Policies can take into account the features of the request, the tenant (aka client), the state of the vLLM pods and GPUs on cluster, the state of the remote providers, and more.

Policy generation is a great task for AI #

Policies are fully documented in the reference section, especially in the InferencePolicy CRD reference and the Policy reference. Additionally, policies can be linted and validated locally by using the kinfctl CLI tool.

Thus, our concrete recommendation is to author these policies with an AI tool such as Claude. We have seen success with prompts like the following:

Read the docs in https://kinference.ai/reference/crds and then confirm that the kinfctl binary is installed. Then, write a policy that sends all traffic to Qwen 3.6 to our local vLLM pods, until they’re saturated, and at that point send the excess traffic to Databricks. But if the prompt has PII in it, never send it externally–just fail it if the local pods are overloaded.

With a little back-and-forth the agent is typically able to generate functioning policies.

The old-fashioned way #

Of course, there are reasons why you might prefer to write a policy by hand. Let’s walk through a simple example: when Alice requests Qwen 2.5, we want to forcibly redirect her to Gemma 4.

As we know from our policy reference, policies have four basic components:

  1. Match. Whether this policuy is applied in the first place.
  2. Consider. Which inference-serving endpoints will be considered (the candidates).
  3. Exclude. Which candidates should then be excluded, based on parameters of the request and the candidates themselves.
  4. Select. How the final destination should be selected from the remaining candidates.

Each of these has intelligent defaults if unspecified.

To accomplish our goal of routing Alice to gemma-4 when she requests Qwen-2.5, we won’t need exclude. We will match requests from tenant “alice” for model “qwen-2.5” and consider only endpoints that serve “gemma-4”, and select from whatever remains by predicted TTFT. Our policy then looks like this:

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata: { name: my-policy, namespace: default }
spec:
  rules:
    - name: alice-qwen-to-gemma
      matches:
        - model: qwen-2.5
          tenant: alice
      route:
        consider:
          - models: { names: [gemma-4] }
        select: [{ type: Local, minimize: PredictedTTFT }]

Note this policy assumes that you can serve both “qwen-2.5” and “gemma-4” models in your fleet (including as aliases for the full model names.)

Once we have a policy, we need to attach it to one or more pools before it will take effect! Let’s assume we have a pool with these qwen-2.5 and gemma-4 models. To attach it, we set the policyRef component of the pools’s spec to the name of the model, like so:

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePool
metadata:
  name: inference-local
  namespace: default
spec:
  destinations:
    - { kind: InferenceWorkerSet, name: gemma-4-vllm }
    - { kind: InferenceWorkerSet, name: qwen-2-5-vllm }
  policyRef:
    - name: my-policy

If we wanted to alter this policy to allow Kinference to optionally use gemma-4 OR qwen-2.5 for any such request, and make a decision based on which model would be the fastest, we would update the consider block to be models: { names: [gemma-4, qwen-2.5] }, i.e. include both models.

If we wanted to further filter out any overloaded endpoints, then we would need to bring in an exclude: block. Let’s assume that any endpoint with more that 5 queued prompts over the past 30 seconds is considered overloaded. Then our policy becomes:

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata: { name: policy-3, namespace: default }
spec:
  rules:
    - name: alice-qwen-to-gemma-no-overloaded-endpoints
      matches:
        - model: qwen-2.5
          tenant: alice
      route:
        consider:
          - models: { names: [gemma-4, qwen-2.5] }
        exclude:
          - name: drop-by-queue-size
            drop:
              backend: { type: Local }
              queuedPrompts: { greaterThan: 4 }
        select: [{ type: Local, minimize: PredictedTTFT }]

Don’t forget to update the policyRef in your pool to point to this new policy!

Congratulations, you now have a relatively sophisticated inference policy.

Useful tips in policy development #

Two good things to know:

  1. The kinfctl CLI tool will lint and validate policies for you locally, without needing cluster access.
  2. Once you push the CRD to the cluster, the status field will report any issues and a kubectl describe will also list any warnings or errors found. See the InferencePolicy CRD reference for more.

Good luck and have fun!