InferencePool

An InferencePool represents a set of possible inference destinations that can be targeted for an inference request. It comprises zero or more InferenceWorkerSets and InferenceProviders, and optionally has an InferencePolicy attached. (If no policy is set, inference requests to this pool will follow the default inference policy.)

An InferencePool is invoked when it is backendRef target of a Gateway API HTTPRoute resource.

Version: kinference.buoyant.io/v1alpha1
Short name: infpool

Example #

This InferencePool includes InferenceWorkerSets gemma-4-12b and qwen-2-5-32b as well as the InferenceProvider databricks.

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePool
metadata:
  name: inference-all
  namespace: default
spec:
  destinations:
    - { kind: InferenceWorkerSet, name: gemma-4-12b }
    - { kind: InferenceWorkerSet, name: qwen-2-5-32b }
    - { kind: InferenceProvider, name: databricks }
  policyRef:
    name: policy-3

Specification #

Field	Type	Required	Default
`destinations`	list	no	every `InferenceWorkerSet` in this namespace
`policyRef`	object	no	serve as asked, on Local destinations

`destinations` field #

Field	Type	Required	Default
`group`	string	no	`kinference.buoyant.io`
`kind`	`InferenceWorkerSet` \| `InferenceProvider`	yes	—
`name`	string	one of `name` / `selector`	—
`namespace`	string	no	this namespace; requires a `ReferenceGrant`
`selector.matchLabels`	map (≤64)	one of `name` / `selector`	—

An empty destinations is equivalent to including every InferenceWorkerSet (but no InferenceProvider) in this namespace.

Note: A pool listing both an InferenceWorkerSet and an InferenceProvider under the same name is rejected as ambiguous.

`policyRef` field #

Field	Type	Required	Default
`group`	string	no	`kinference.buoyant.io`
`kind`	string	no	`InferencePolicy`
`name`	string	yes	—

An InferencePolicy. Omit it for the default inference policy.

Status fields #

Field	Meaning
`destinationCount`	Resolved destinations
`models[]`	What traffic on this route can reach, aggregated across destinations
`models[].name`	Canonical, client-facing name
`models[].servedBy`	Destination names serving it, across both kinds
`models[].contextWindow`	`{min, max}` across destinations
`unresolvedDestinations[]`	`{kind, name, reason}` per destination that did not resolve
`parents[]`	Gateway API status, one entry per `parentRef`
`conditions`	On each `parents[]` entry: `Accepted`, `ResolvedRefs`

Printer columns and state #

Example:

$ kubectl get infpools
NAME              POLICY     DESTINATIONS   STATE                             AGE
inference-all     policy-4   3              Accepted,UnresolvedDestinations   3h
inference-local              2              Accepted                          12m

POLICY is the name of the attached InferencePolicy, if any. DESTINATIONS is the number of resolvable destinations in the pool. STATE is a comma-separated list of states:

Token	Position	Means
`Accepted` / `NoDestinations`	primary	Whether anything resolved to route to
`UnresolvedDestinations`	qualifier	At least one `destinations` entry did not resolve
`PolicyNotFound`	qualifier	`policyRef.name` names no policy here
`PolicyRefWrongKind`	qualifier	`policyRef.group`/`kind` name something else

Diagnoses #

Symptom	Cause
`destinationCount` lower than the list	Check `unresolvedDestinations[].reason`
`reason: AmbiguousName`	Two destinations of different kinds share a name
`reason: CrossNamespaceNotGranted`	No `ReferenceGrant` in the target namespace, or the router is scoped to a single namespace and cannot see it
`reason: Incomplete`	A destination is missing `kind`, or a provider is missing a required field
`ResolvedRefs=False`, all destinations resolved	`policyRef` names a missing policy
A model shows one `servedBy` entry, not two	Canonical names diverge between a worker set and a provider
Providers listed but never picked	The attached policy never admits them.