CRDs
Kinference policy is configured through several Kubernetes custom resources, including:
| Kind | Group | Description |
|---|---|---|
| InferenceWorkerSet | kinference.buoyant.io | A set of local inference workers for one model |
| InferenceProvider | kinference.buoyant.io | A remote inference provider |
| InferencePool | kinference.buoyant.io | A set of all possible inference destinations |
| InferencePolicy | kinference.buoyant.io | The inference policy for a pool |
Kinference also uses one Linkerd Gateway CRD to represent external endpoints:
| Kind | Group | Version | Description |
|---|---|---|---|
| GatewayBackend | gateway.linkerd.io | v1alpha1 | Where an off-cluster backend is, and how to authenticate to it |
These CRDs are designed to integrate with the Gateway API to allow for easy routing of inference requests via existing gateways. A Gateway API HTTPRoute can point to a InferencePool; this InferencePool is a collection of local and remote destinations; an optional InferencePolicy can control exactly how inference is handled for this pool.
Default inference policy #
An InferencePool contains an optional reference to an InferencePolicy. If no InferencePolicy is set for a pool, Kinference follows a specific default policy:
- All local endpoints that match the requested model are considered.
- The endpoint with the lowest expected TTFT is chosen.
Note also that no InferenceProviders will ever be selected under the default policy.