InferenceWorkerSet

An InferenceWorkerSet represents a set of local inference workers (e.g. vLLM pods) that serve one model. An InferenceWorkerSet wraps a standard Kubernetes Service with inference-specific routing information including model and KV cache information.

• Version: kinference.buoyant.io/v1alpha1
• Short name: iws

InferenceWorkerSets are typically used as part of an InferencePool.

InferenceWorkerSets can also specify aliases for model names.

Example #

This InferenceWorkerSet wraps a Service named qwen-3-6-35b-a3b-vllm; specifies that it has a KV cache block size 16 and emits KV cache events on the port named “kv-pub”, and adds “qwen3.6” as an alias to the “Qwen/Qwen3.6-35B-A3B” model served.

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferenceWorkerSet
metadata:
  name: qwen-3-6-35b-a3b
  namespace: default
spec:
  profile: default
  kvBlockSize: 16
  serviceRef:
    name: qwen-3-6-35b-a3b-vllm
    port: http
  kvEvents:
    port: kv-pub
  models:
    - name: Qwen/Qwen3.6-35B-A3B
      aliases: [qwen3.6]

Specification #

profile field #

Which routing profile to use. Determines the latency-model parameters and tokenization mode.

kvBlockSize field #

Tokens per block in the KV-cache events these engines publish, e.g. 16, 256.

Note: This value is Not necessarily the same as the --block-size parameter passed in to vLLM, or supplied in cache_config_info metrics. For example, Gemma 4 reports block_size=64 in cache_config_info but publishes 256-token blocks.

kvEvents field #

The socket on which the workers publish KV cache events (BlockStored / BlockRemoved and equivalents).

If unset, KV cache information will not be used. This may result in degraded routing quality.

models field #

Model information is auto-discovered from /v1/models. This optional field is used to provide model aliases. For example:

models:
  - name: Qwen/Qwen2.5-32B-Instruct-AWQ # what the engine reports
    aliases: [qwen-2.5, qwen-simple] # and two shorter ways to ask for it

In this example, users may call the Qwen2.5-32B-Instruct-AWQ model by passing the full model name or either of the two aliases (“qwen-2.5” or “qwen-simple”).

Status fields #

Printer columns and state #

Example:

$ kubectl get iws
NAME               READY   STATE                            AGE
qwen-3-6-35b-a3b   3       Accepted                         7d
gemma-4-12b        0       Accepted,NotReady                7d
orphan             0       Accepted,NotReady,Unreferenced   19s
no-such-profile    0       UnknownProfile                   2m

READY is the number of ready endpoints; STATE is a comma-separated list of states:

Possible rejection reasons include:

Diagnoses #