InferencePolicy

An InferencePolicy determines how inference requests are handled within an InferencePool.

Version: kinference.buoyant.io/v1alpha1
Short name: infpol

For a reference guide to the policies themselves, see Policies.

Example #

This policy allows requests to “burst to cloud” by sending them to the databricks InferenceProvider when local endpoints are at saturation, unless the request includes PII, in which case it will be rejected rather than sent externally.

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata: { name: my-policy, namespace: default }
spec:
  rules:
    - name: local-first-then-provider-unless-pii
      route:
        exclude:
          - name: drop-local-endpoints-with-big-queues
            drop:
              backend: { type: Local }
              queuedPrompts: { greaterThan: 4 }
          - name: drop-local-endpoints-with-no-kv-headroom
            drop:
              backend: { type: Local }
              kvCacheUsage: { greaterThan: 0.9 }
          - name: ensure-pii-never-leaves-cluster
            if: { tag: [pii-confirmed, pii-unknown] }
            drop: { backend: { type: Provider } }
        select:
          - { type: Local, minimize: PredictedTTFT }
          - { type: Provider, order: [databricks] }

Specification #

Field	Type	Required	Default
`unmatched`	`ServeLocal` \| `Reject`	no	`ServeLocal`
`rules`	list	no	—

`unmatched` field #

What happens when no rules match. ServeLocal serves it as the pool’s Local destinations, ranking by predicted TTFT. Reject rejects the request.

`rules` field #

Field	Type	Required
`name`	string	yes
`matches`	list	no
`route`	object	yes

`rules`.`matches` field #

A list of predicates that determine whether this rule matches the requests. (See Policies for more.)

`rules`.`route` field #

What happens if the rule is applied. (See Policies for more.)

Field	Type	Required	Default
`consider`	list	no	the requested model on every destination
`exclude`	list	no	—
`select`	list	yes	—
`tieBreak`	list	no	first in rank order wins
`onEmpty`	`Reject` \| `FallThrough`	no	`Reject`

Status fields #

Field	Meaning
`ruleCount`	Rules compiled
`referencedBy`	Pools whose `policyRef` names this policy. Empty means nothing routes through it
`rules[].name`
`rules[].requiresClassifiers`	Derived from the tag prefixes this rule references
`rules[].worstCaseClassifierLatency`	What this rule adds to the request path
`rules[].classifierFailureMode`	Effective built-in behaviour per classifier. Not configurable, so it cannot be reviewed from the manifest
`rules[].emittableTags`	Every tag those classifiers can emit. The only place a written tag can be checked against reality
`warnings`	Compilation diagnostics. Never fatal
`conditions`	`Accepted`

Warnings #

Warnings never make a policy unusable. Each describes one term or rule that was dropped or defaulted; the rest of the policy still applies. Read them — every one of them names a way the manifest means less than it says.

kubectl -n default get infpol policy-4 -o jsonpath='{.status.warnings}' | jq

Printer columns and statuses #

Example:

$ kubectl get infpols

NAME       RULES   STATE           UNMATCHED    AGE
policy-1   1       NotReferenced   ServeLocal   3d
policy-3   1       Accepted        ServeLocal   3d
policy-4   2       NotReferenced   ServeLocal   3d

`Accepted`	Reason	Means
`True`	`Accepted`	In force on at least one pool
`False`	`NotReferenced`	Compiled, but no `InferencePool` names it in `spec.policyRef`

Diagnoses #

Symptom	Cause
`RULES` blank	No status written — the policy is in no namespace the router watches
`STATE: NotReferenced`	No pool names it. Patch the pool’s `policyRef`
Every policy shows the same `RULES`	Expected; `RULES` is compiled, not live. Read `STATE`
A rule never fires	It follows a catch-all, or matches a tag no classifier emits. Check `warnings`
Providers never picked	A `PredictedTTFT` tier with no `backends` subset keeps every local candidate, so the provider tier is unreachable
A provider is picked on its first request	Under `EstimatedTTFT` an unobserved provider stays eligible and sorts last — with no local candidate surviving, last is first
PII reached a provider	The `if` clause matched only `pii-confirmed`, and the classifier emitted `pii-unknown`
An exclusion did nothing	It used a class-scoped signal without constraining `backend.type`. Check `warnings`
Requests are refused with no obvious cause	`unmatched: Reject`, or `onEmpty: Reject` after every candidate was excluded