Policies

A policy in Kinference describes how traffic is handled. In practice, policies exist as part of an InferencePolicy; when this InferencePolicy is attached to an InferencePool, it determines how inference requests to that pool are handled.

Rules #

A policy comprises zero or more rules. Rules are evaluated in list order, and the first match wins.

Rules determine both routing and load balancing behavior: they can select not just the model that should be used but also the specific endpoint that should serve this request. This allows for sophisticated policies that can make joint tradeoffs between models and performance, e.g. “serve this simple request on the cheaper model, unless the more expensive model will actually be cheaper because of caching”.

A rule matches and is applied to a single request. Given the request, the rule evalutes a set of candidates that corresponding to potential endpoints for the request. Finally, it chooses a single endpoint to serve the request–or no endpoint.

Rules comprise two clauses:

1. Match. Whether this rule is applied to the request in the first place.
2. Route. What happens if the rule matches.

The route clauses of a rule, in turn, consists of three sub-clauses:

1. Consider. Which inference-serving endpoints will be considered (the candidates). If unset, this will default to all local endpoints that match the model in the request.
2. Exclude. Which candidates should then be excluded, based on parameters of the request and the candidates themselves. If unset, the set of candidates from the consider phase will be used as is.
3. Select. How the final destination should be selected from the remaining candidates. A select statement is required in every policy.

Putting it together structurally, we get:

rules:
  - name: your-rule-name
    matches:
      # how to determine whether the rule matches a request
    route:
      consider:
        # which candidates to consider
      exclude:
        # which candidates to filter out
      select:
        # how to choosefrom the remaining candidates
      # finally, optional tie breaking or fallthrough behavior

Requests, candidates, and attributes #

A rule is evaluated over a single request and operates on zero or more candidates. Both the request and the candidates have a set of attributes.

Attributes can be strings, numeric, sets, or durations. Durations are simple expressed as numbers with units, e.g. 5s, 200ms, 1m30s.

Request attributes #

The request for a rule has the following set of attributes.

Candidate attributes #

The candidates for a rule have the following set of attributes.

Important note: Not every candidate has every attribute. For example, Provider endpoints (those corresponding to off-cluster inference providers) do not have a PredictedTTFT attribute, as this attribute requires information such as KV cache content that is only available for local endpoints.

Classifier attributes #

When used as part of a rule, Kinference will run classifiers to provide additional information about the request.

This information is available to rules as tags in the of [type]-[degree] tags, e.g. “pii-confirmed”, “ssn-confirmed”, “complexity-low”, “code-unknown”.

Note that classifier that time out or are skipped for any reason will not emit any tags. Note also that classifiers may emit an explicit unknown tag, and a restrictive policy must match on it. For example:

if: { tag: [pii-confirmed, pii-unknown] } # not just pii-confirmed

Rule predicates #

How do we use attributes? The matches, exclude[].if, and exclude[].drop sections use a set of predicates to express filters on attributes. These predicates are constructed in the form [feature name]: [how to match]. For example, model: foo means “the model must be ‘foo’.

These predicates allow not just for exact matches, but also for set membership and for operators such as greaterThan:

Predicate operators take exactly one key. Predicates in the same entry are ANDed together, e.g. { model: qwen-2.5, tenant: alice } matches only requests from alice for qwen-2.5. To express OR, you can give one attribute a list of values (tenant: [alice, bob]), or put the predicates in separate entries of matches.

Note that a predicate for a feature that a candidate does not have always evaluates to false, regardless of whether it is negated. In other words, a candidate with no backend.name is not considered “a candidate whose backend.name is not X”.

The only predict that matches the absence of a feature is exists: false.

matches clause #

The matches clause of a rule is a list where each element in the list is a set of predicates against attributes of the request. The rule matches a request if any entry matches; a given entry matches when all of its predicates are true.

For example, this rule will match either bob, or alice selecting qwen-2.5:

matches:
  - { model: qwen-2.5, tenant: alice } # alice's requests for qwen-2.5,
  - tenant: bob # or any request from bob

If matches is unset, this rule will matches every request. In this case, it must be the final rule in the list.

route clause #

The route clause comprises consider, exclude, and select sub-clauses, as well as optional tieBreak and onEmpty specifiers.

route.consider clause #

This phase is the candidate generation phase. If unset, this rule will consider every local destination the pool allows. Empty lists are not allowed.

The consider phase may make use of the following candidate attributes:

A backends block with no fields means every destination.

Examples:

# Route a low-complexity request to a smaller model than it asked for.
consider:
  - models: { names: [gemma-4] }

# Let both compete, and let the ranking decide.
consider:
  - models: { names: [gemma-4, qwen-2.5] }

route.exclude clause #

This phase is the candidate filtering phase. It allows for removal of candidates based on both candidate attributes and request attributes. This claude takes a list of items of the form:

route.select clause #

This phase is the candidate selection phase. Order matters: each entry is a selection tier, and the first tier with a surviving candidate wins.

Tiers take the following structure:

Where:

An objective must be defined for every candidate in its tier. Note that the PredictedTTFT objective only applies to local candidates, so it cannot be used by a tier that spans both local and provider candidates.

If a minimize clause is used in a tier, a candidate with no value for the tier’s objective is dropped from that tier.

For example:

select:
  - { type: Local, minimize: PredictedTTFT }
  - { type: Provider, order: [databricks] }

route.tieBreak field #

PreferLocal or Random. Omitted on an explicit rule means the first candidate in rank order wins deterministically.

route.onEmpty field #

What happens when no tier had a survivor. Reject refuses the request; FallThrough continues to the next rule.