InferenceProvider

An InferenceProvider represents an inference provider, e.g. an endpoint on a separate cluster, or a third-party endpoint accessed over the Internet. An InferenceProvider wraps a GatewayBackend with inference-specific information including model and KV cache information.

Version: kinference.buoyant.io/v1alpha1
Short name: infprov

InferenceProviders are typically used as part of an InferencePool. Like InferenceWorkerSets, they can also specify aliases for model names.

Example #

In this example, an InferenceProvider named “databricks” is made available. It uses the OpenAI API and provides “Qwen/Qwen2.5-32B-Instruct-Turbo”, which is aliased to “qwen-2.5” and “qwen-simple”.

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferenceProvider
metadata:
  name: databricks
  namespace: default
spec:
  backendRef:
    name: databricks
  api: OpenAI
  path: /serving-endpoints/vllm-endpoint/invocations
  models:
    - name: Qwen/Qwen2.5-32B-Instruct-Turbo
      aliases: [qwen-2.5, qwen-simple]
      contextWindow: 32768

Specification #

Field	Type	Required	Default
`backendRef.group`	string	no	`gateway.linkerd.io`
`backendRef.kind`	string	no	`GatewayBackend`
`backendRef.name`	string	yes	—
`backendRef.namespace`	string	no	this namespace
`api`	`OpenAI`	yes	—
`models`	list	yes	—
`models[].name`	string	yes	—
`models[].aliases`	list	no	—
`models[].contextWindow`	integer	no	—
`path`	string	no	client’s original path

`backendRef` field #

A gateway.linkerd.io/GatewayBackend resource pointing to the underlying provider.

`api` field #

Wire format the upstream speaks. Currently, the only supported value is OpenAI.

`models` field #

Models served by this provider. Required.

Field	Type	Required	Meaning
`name`	string	yes	What this vendor calls the model (an endpoint name, an ARN, a repo path)
`aliases`	list	no	Client-facing names that also resolve to `name`
`contextWindow`	integer	no	Declared token limit

Example:

models:
  - name: Qwen/Qwen2.5-32B-Instruct-Turbo # reachable by this name
    aliases: [qwen-2.5, qwen-simple] # ... or either of these names
    contextWindow: 32768
  - name: meta/llama-3.3-70b

`path` field #

An optional URL path to which Kinference will send the request at the provider, replacing the client’s original path. For example, Databricks requires inference requests on /serving-endpoints/vllm-endpoint/invocations; if path is set to that value, it will be used instead of /v1/chat/completions in the request to the provider.

If empty, the client’s original path (e.g. /v1/chat/completions) will be used as is.

Status fields #

Field	Meaning
`conditions`	`Accepted`, `ResolvedRefs`, `Ready`

Printer columns and state #

Example:

NAME         BACKEND      API      STATE                                     AGE
databricks   databricks   OpenAI   Accepted                                  7d
vertex       vertex       OpenAI   Accepted,BackendUnauthorized,Suppressed   2h

BACKEND and API are propagated from the spec. STATE is .status.state:

Token	Position	Means
`Accepted`	primary	Well-formed and resolved into a pool destination
`UnsupportedApiFormat`	primary	`api` is not `OpenAI`
`Incomplete`	primary	No `backendRef.name`, no `api`, or no `models` row carrying a `name`
`CrossNamespaceNotGranted`	primary	`backendRef` crosses namespaces with no `ReferenceGrant`
`AmbiguousName`	primary	A worker set in the same pool has this name
`BackendUnresolved` / `BackendUnauthorized`	qualifier	From the caller’s `Abort` reports
`Suppressed`	qualifier	Repeated failures have temporarily removed this candidate from the pool

Diagnoses #

Symptom	Cause
Rejected on create	`api` is not `OpenAI`; the schema enum admits nothing else
`STATE: UnsupportedApiFormat`	Same cause, on an object that bypassed admission. The router parses it and then refuses it, rather than failing to parse
`ResolvedRefs=False`	`backendRef` dangles, or a cross-namespace ref has no `ReferenceGrant`
Never picked, conditions clean	Suppressed after repeated dispatch failures, or losing on rank
Never picked, and this is its first traffic	Under a `PredictedTTFT` tier a provider carries no value and is dropped from that tier. Use `EstimatedTTFT`, or give providers their own tier
Requests fail with a model error upstream	`name` is wrong. The router substitutes it into the body verbatim
Long requests never fail-over here	`contextWindow` is below the local set’s effective window