Intel GPU Base Operator for Kubernetes

CAUTION: This is an beta / non-production software, do not use on production clusters.

Intel GPU Base operator allows automatic deployment of GPU related components to enable use of Intel GPU hardware within the Kubernetes cluster.

For the production security baseline, image guidance, RBAC, webhook TLS, metrics, firmware updates, and optional integration risks, see the Security Configuration Guide.

Description

Intel GPU Base operator deploys components to a cluster to make it possible to run Intel GPU workloads on the cluster nodes. It can configure the cluster with either Device Plugin or Dynamic Resource Allocation (DRA) based approach. In addition to registering GPU resources for the cluster, it can also deploy Intel XPU-Manager. If the cluster has Prometheus enabled, operator can deploy the required Kubernetes components to integrate GPU metrics to the Prometheus database.

Operator does its best to detect cluster features and to only deploy components that are supported. For example, GPU DRA driver is only deployed from certain Kubernetes version onwards.

When the operator is requested with device plugin, resource monitoring:

apiVersion: intel.com/v1alpha1
kind: ClusterPolicy
metadata:
  name: gpu-dp
spec:
  resourceRegistration: dp
  useNFDLabeling: true
  resourceMonitoring: true

the end result will look something like this:

$ kubectl get pods -n intel-gpu-base-operator
NAME                                                          READY   STATUS    RESTARTS   AGE
gpu-dp-device-plugin-5txt5                                    2/2     Running   0          2m4s
gpu-dp-xpu-manager-hdg6b                                      2/2     Running   0          2m4s
intel-gpu-base-operator-controller-manager-749cb69d6c-kqnzw   1/1     Running   0          124m

Device Plugin GPU resources can be seen like so:

$ kubectl get node node1 -o yaml | yq .status.allocatable
...
gpu.intel.com/i915: "1"
gpu.intel.com/i915_monitoring: "1"
...

When the operator is requested with DRA, resource monitoring and with NFD:

apiVersion: intel.com/v1alpha1
kind: ClusterPolicy
metadata:
  name: gpu-dra
spec:
  resourceRegistration: dra
  useNFDLabeling: true
  resourceMonitoring: true

the end result will look something like this:

$ kubectl get pods -n intel-gpu-base-operator
NAME                                                          READY   STATUS    RESTARTS   AGE
gpu-dra-gpu-dra-rqr94                                         1/1     Running   0          88s
gpu-dra-xpu-manager-cqsbj                                     2/2     Running   0          88s
intel-gpu-base-operator-controller-manager-749cb69d6c-kqnzw   1/1     Running   0          147m

DRA GPU resource slices can be seen like so:

$ kubectl get resourceslices.resource.k8s.io
NAME                                NODE            DRIVER          POOL            AGE
node1                               node1           gpu.intel.com   node1           111s

Dependencies

Dependency	Level	Purpose	Chart integration
Cert Manager	Mandatory	The operator uses webhooks which have hooks to cert-manager's TLS apply.	None
Node Feature Discovery	Recommended	Allows scheduling Pods only to Nodes with Intel GPU hardware.	Optionally installed
Kueue	Recommended	Enables advanced Pod scheduling mechanisms.	Optionally installed
Prometheus	Recommended	Accesses GPU metrics from all Nodes from one endpoint.	None

Getting Started

The latest release version is 0.4.0. Instructions for deploying the operator via Helm are described below.

Helm deployment

The preferred installation method to the cluster via our Helm charts. Please use Helm v4 or later to avoid issues during installation.

Helm deployment is split into two charts: operator and policy. The reason for this split is to allow the operator to run cleanup before it is removed from the cluster. DRA especially is problematic as Pods using its resources (e.g. XPU Manager) will get stuck at Terminating if the DRA plugin is removed from the cluster.

The basic installation is as follows:

helm install --create-namespace --namespace "intel-gpu-base-operator" --version 0.4.0 gpu-operator \
  oci://ghcr.io/intel/intel-gpu-base-operator-chart --wait
helm install --namespace "intel-gpu-base-operator" --version 0.4.0 gpu-policy \
  oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart --set resourceRegistration=dra

This installs the operator and a DRA-enabled deployment with Intel XPU Manager. Node Feature Discovery and Kueue may be installed with the operator chart; this depends on the kueue.install and nfd.install parameters.

Example: DRA without NFD

helm install --create-namespace --namespace "intel-gpu-base-operator" --version 0.4.0 gpu-operator \
  oci://ghcr.io/intel/intel-gpu-base-operator-chart --wait
helm install --namespace "intel-gpu-base-operator" --version 0.4.0 gpu-policy \
  oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart --set resourceRegistration=dra

Example: DRA with NFD and Kueue

helm install --create-namespace --namespace "intel-gpu-base-operator" --version 0.4.0 gpu-operator \
  oci://ghcr.io/intel/intel-gpu-base-operator-chart --wait \
  --set nfd.install=true \
  --set kueue.install=true
helm install --namespace "intel-gpu-base-operator" --version 0.4.0 gpu-policy \
  oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart \
  --set resourceRegistration=dra \
  --set useNFDLabeling=true \
  --set enableKueue=true

Example: Device Plugin with NFD

helm install --create-namespace --namespace "intel-gpu-base-operator" --version 0.4.0 gpu-operator \
  oci://ghcr.io/intel/intel-gpu-base-operator-chart --wait \
  --set nfd.install=true
helm install --namespace "intel-gpu-base-operator" --version 0.4.0 gpu-policy \
  oci://ghcr.io/intel/intel-gpu-base-operator-policy-chart \
  --set resourceRegistration=dp \
  --set useNFDLabeling=true

Uninstalling the charts:

helm uninstall --namespace "intel-gpu-base-operator" gpu-policy --wait
helm uninstall --namespace "intel-gpu-base-operator" gpu-operator

See more details for the charts in the operator and policy READMEs.

Installation to an existing namespace

Installation to a pre-existing namespace needs to set createNamespace=false chart variable and drop the --create-namespace argument to prevent chart from trying to create the namespace.

Also note that DRA requires the install namespace to contain resource.kubernetes.io/admin-access=true label for the monitoring to function. If one installs the operator to an existing namespace, the namespace has to be labeled:

kubectl label ns <target namespace> resource.kubernetes.io/admin-access=true

Custom Resource (CR) fields

CR fields control how the operator configures the cluster. See the full struct for all options.

Core fields

Field	Description	Default
`spec.resourceRegistration`	Resource registration mode: `dp` (Device Plugin) or `dra` (Dynamic Resource Allocation)	—
`spec.resourceMonitoring`	Deploy XPU-Manager for per-GPU telemetry and health monitoring	`false`
`spec.useNFDLabeling`	Deploy an NFD NodeFeatureRule to label Intel GPU nodes; requires NFD to be installed	`false`
`spec.prometheusIntegration`	Create a ServiceMonitor so Prometheus scrapes XPU-Manager GPU metrics; requires Prometheus Operator	`false`
`spec.enableKueue`	Create Kueue ResourceFlavor / ClusterQueue / LocalQueue resources for GPU workload queuing	`false`
`spec.logLevel`	Default log level for all deployed components (0–4)	`2`

Device Plugin (`spec.dp`)

Used when spec.resourceRegistration: dp.

Field	Description	Default
`spec.dp.allowIDs`	Only register GPUs whose PCI device ID is in this list. Format: `['0xabcd']`. Cannot be combined with `denyIDs`	`[]`
`spec.dp.denyIDs`	Exclude GPUs whose PCI device ID is in this list. Format: `['0xabcd']`. Cannot be combined with `allowIDs`	`[]`
`spec.dp.byPathMode`	Controls which DRI by-path symlinks are exposed to containers: `single`, `all`, or `none`	`single`
`spec.dp.affinity`	Optional `k8s.io/api/core/v1` `Affinity` object applied to the device plugin daemon set	—

Dynamic Resource Allocation (`spec.dra`)

Used when spec.resourceRegistration: dra.

Field	Description	Default
`spec.dra.podHealthCheck`	Enable health check for DRA Pod	true
`spec.dra.manageBinding`	Allow DRA plugin to manage device binding between xe/i915 and vfio drivers. Needed for dynamic switching between normal and KubeVirt workloads	`false`
`spec.dra.affinity`	Optional `k8s.io/api/core/v1` `Affinity` object applied to the DRA daemon set	—

Health monitoring (`spec.health`)

Applies to both DP and DRA unless noted. Thresholds that are exceeded mark the GPU as unhealthy.

Field	Description	Default
`spec.health.coreTemperatureThreshold`	GPU core temperature limit in °C (1–130)	`100`
`spec.health.memoryTemperatureThreshold`	GPU memory temperature limit in °C (1–130)	`100`
`spec.health.checkIntervalSeconds`	How often health is evaluated in seconds (1–3600). Not supported by Device Plugin	`5`

XPU-Manager (`spec.xpu`)

Field	Description	Default
`spec.xpu.monitoringResource`	Set XPUMD resource for Device Plugin use.	`monitoring`
`spec.xpu.configMapOverride`	Name of a ConfigMap in the operator namespace containing a custom OpenTelemetry Collector `config.yaml`	—
`spec.xpu.affinity`	Optional `k8s.io/api/core/v1` `Affinity` object applied to the XPU manager daemon set	—
`spec.xpu.restartOnDeviceRecovery`	When to restart a node's XPU Manager pod so it can monitor a GPU its container cannot reach. `OnRecoveredDevice`, `Always` or `Disabled`.	`OnRecoveredDevice`
`spec.xpu.otlp`	Export metrics and logs also to an OTLP gateway. See OTLP export	—

OTLP export (`spec.xpu.otlp`)

XPU-Manager can push its metrics and logs to an OpenTelemetry Protocol (OTLP) gateway, in addition to serving the Prometheus endpoint. Requires spec.resourceMonitoring: true.

NOTE: For overall OTel configurations in the cluster, see the opentelemetry documentation for recommended configuration.

Set exactly one transport, http or grpc. The scheme of the endpoint selects whether the connection is encrypted:

Option	Transport	Endpoint example	Encryption
HTTP/HTTPS	`otlp.http` (OTLP/HTTP, protobuf)	`http://otel-gateway.monitoring:4318`	TLS with `https://`, plain text without
gRPC	`otlp.grpc` (OTLP/gRPC)	`http://otel-gateway.monitoring:4317`	TLS with `https://`, plain text without

For HTTP, the endpoint is the base URL; the signal paths (/v1/metrics, /v1/logs) are appended to it.

Both transports accept the same fields:

Field	Description
`endpoint`	URL of the OTLP gateway, `http://` or `https://`
`headers[]`	Up to 16 headers (gRPC metadata for `grpc`) added to every export request, e.g. `Authorization`. Each has a `name` and a `valueFrom` Secret key holding the complete value. Values are passed to the container as environment variables, so they never appear in the ClusterPolicy or the collector ConfigMap. Changes to the Secret take effect once the XPU-Manager pods restart
`tls.ca`	ConfigMap key with a PEM CA bundle to verify the gateway, e.g. for a private or self-signed CA. Defaults to the system CA pool of the XPU-Manager image
`tls.clientCertSecret`	`kubernetes.io/tls` Secret whose `tls.crt` and `tls.key` are presented for mutual TLS
`tls.serverName`	Overrides the host name the gateway certificate is verified against
`tls.insecureSkipVerify`	Skip gateway certificate verification. For testing only; cannot be combined with `tls.ca`

All referenced Secrets and ConfigMaps have to be in the operator namespace. Credentials can only be given as Secret references, never inline in the ClusterPolicy. If a referenced Secret, ConfigMap or key does not exist, the XPU-Manager pods do not start until it is created.

TLS

The endpoint scheme decides whether TLS is used at all:

http:// — no TLS. The operator explicitly disables TLS in the exporter, as the gRPC exporter would otherwise attempt TLS regardless of the scheme. Setting tls for an http:// endpoint is rejected. Use only towards a trusted, in-cluster collector.
https:// — TLS. The tls block is optional; without it, the gateway certificate is verified against the system CA pool of the XPU-Manager image, and the host name against the host in endpoint.

The tls fields adjust the verification and add a client certificate:

Server verification (tls.ca): The ConfigMap key is mounted read-only to the XPU-Manager container as /etc/otlp/ca/ca.crt and replaces the system CA pool. Use it for a private or self-signed CA. The key can hold several PEM certificates, e.g. the old and the new CA during a rotation. A ConfigMap is used because a CA certificate is not secret.
Host name (tls.serverName): Verifies the gateway certificate against this name instead of the endpoint host, e.g. when the endpoint is an IP address or a Service name that is not in the certificate's SANs.
Mutual TLS (tls.clientCertSecret): The tls.crt and tls.key of the kubernetes.io/tls Secret are mounted read-only (mode 0400) to /etc/otlp/client/, and presented to the gateway when it requests a client certificate. tls.crt may contain intermediate certificates after the leaf. A Secret managed by cert-manager Certificate works as-is.
tls.insecureSkipVerify: The traffic is still encrypted, but the gateway is not authenticated, so it is open to man-in-the-middle attacks and any headers can be captured by an impostor. The webhook warns about it, and it cannot be combined with tls.ca. For testing only.

Authentication

The gateway can authenticate XPU-Manager with request headers, a client certificate, or both:

Headers: For bearer tokens (Authorization: Bearer <token>), basic auth (Authorization: Basic <base64>), or API keys (e.g. X-API-Key). The Secret key has to hold the complete header value, including any Bearer or Basic prefix. The operator maps each header to an OTLP_HEADER_<index> environment variable in the XPU-Manager container (<index> being the position in headers), and the generated collector configuration only refers to it as ${env:OTLP_HEADER_<index>}. The value is therefore not visible in the ClusterPolicy, the collector ConfigMap or the DaemonSet. Header names are case-insensitive and have to be unique. With grpc, headers are sent as gRPC metadata. The webhook warns if headers are configured for an http:// endpoint, as they would be sent in plain text.
Mutual TLS: With tls.clientCertSecret, the gateway authenticates XPU-Manager by its client certificate during the TLS handshake. Requires an https:// endpoint.

All XPU-Manager pods on all nodes share the same credentials.

Rotating credentials

Changes to the otlp spec in the ClusterPolicy roll out the XPU-Manager DaemonSet automatically. Changes to the contents of the referenced Secrets and ConfigMaps do not: environment variables are only read at container start, and the collector loads the CA and client certificate files at startup. After rotating a token, a CA or a client certificate, restart the pods:

kubectl rollout restart daemonset -n intel-gpu-base-operator <cr-name>-xpu-manager

For cert-manager issued client certificates, schedule the restart within the renewal window, i.e. before the old certificate expires.

With `spec.xpu.configMapOverride`

When spec.xpu.configMapOverride is set, the operator does not generate the collector configuration, but it still sets up the header environment variables and the TLS file mounts from otlp. The override has to configure the otlp_http or otlp_grpc exporter itself and refer to them:

Item	Reference in the override
Header `headers[<index>]`	`${env:OTLP_HEADER_<index>}`
`tls.ca`	`ca_file: /etc/otlp/ca/ca.crt`
`tls.clientCertSecret`	`cert_file: /etc/otlp/client/tls.crt`, `key_file: /etc/otlp/client/tls.key`

Examples

Plain-text HTTP, e.g. to an in-cluster collector:

spec:
  resourceMonitoring: true
  xpu:
    otlp:
      http:
        endpoint: http://otel-collector.monitoring.svc:4318

HTTPS with a bearer token and a private CA:

kubectl create secret generic -n intel-gpu-base-operator otlp-auth --from-literal=authorization="Bearer <token>"
kubectl create configmap -n intel-gpu-base-operator otlp-ca --from-file=ca.crt=./ca.crt

spec:
  resourceMonitoring: true
  xpu:
    otlp:
      http:
        endpoint: https://otel-gateway.example.com:4318
        headers:
        - name: Authorization
          valueFrom:
            name: otlp-auth
            key: authorization
        tls:
          ca:
            name: otlp-ca
            key: ca.crt

gRPC with mutual TLS:

kubectl create secret tls -n intel-gpu-base-operator otlp-client --cert=client.crt --key=client.key

spec:
  resourceMonitoring: true
  xpu:
    otlp:
      grpc:
        endpoint: https://otel-gateway.example.com:4317
        tls:
          ca:
            name: otlp-ca
            key: ca.crt
          clientCertSecret:
            name: otlp-client

Kueue (`spec.kueue`)

When enableKueue: true, the operator creates Kueue ClusterQueue and LocalQueue resources to enable GPU-aware job scheduling via Kueue.

The operator currently supports one configuration for Kueue: equal division. With equal division, the operator discovers all GPU resources available in the cluster and divides them equally across the ClusterQueues defined in spec.kueue.equalResources. It also automatically creates a single ResourceFlavor that covers all GPU resource types. Other configuration options and resource division schemas will be added in the future.

Field	Description
`spec.kueue.equalResources`	List of ClusterQueues. GPU capacity is split evenly across all entries
`spec.kueue.equalResources[].name`	Name of the Kueue `ClusterQueue` to create
`spec.kueue.equalResources[].localQueues`	List of `LocalQueue` resources to create for this ClusterQueue
`spec.kueue.equalResources[].localQueues[].name`	Name of the `LocalQueue`
`spec.kueue.equalResources[].localQueues[].namespace`	Namespace in which the `LocalQueue` is created

NOTE: Kueue prevents Pods from consuming more resource than defined in ClusterQueues. Kueue does not prevent Pods in one LocalQueues from consuming all the resources in the ClusterQueue, thus leaving no resources for the other LocalQueues.

Building the operator

To build the operator without containerization.

Prerequisites:

golang (1.24+)
make

go mod vendor
make build

To build the operator container.

Prerequisites for the docker-build:

docker version 17.03+
make

make docker-build

Development builds

While developing the operator, it is sometimes required to re-generate the autogenerated code. This can be done by

make generate

If the operator receives (or loses) access rights to the cluster, the deployment files can be updated with:

make manifests

One can run the operator unit tests with:

make test

Dependencies

Cert-manager:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.4/cert-manager.yaml

Node Feature Discovery (optional):

kubectl apply -k config/nfd

Kueue (optional):

kubectl apply -k config/kueue

Kubectl deployment

NOTE: As the operator's images are not available anywhere yet, one has to build the container and store it in a registry somewhere.

To store the operator container image in the container-runtime image cache:

docker save ghcr.io/intel/intel-gpu-base-operator:devel | sudo ctr -n k8s.io image import -

To deploy the operator:

kubectl apply -k config/default

To undeploy the operator:

kubectl delete -k config/default

To deploy a Device Plugin CR:

kubectl apply -k config/samples/deviceplugin/

To deploy a DRA CR:

kubectl apply -k config/samples/dra/

WARNING: If the operator is deleted when DRA CR is configured, the XPU-Manager Pod has the tendency to get stuck in Terminating phase. This is because DRA needs to tear down the claim for the Pod to get Terminated. If that happens, the quickest fix is to redeploy the operator and custom resource, and then remove the custom resource properly.

Firmware update

The Intel GPU base operator supports updating the GPU firmware on the cluster nodes. The update is handled via a GPUFirmwareUpdate CRD. The update flow and details are explained in the FW update documentation.

OoT Kernel driver

The base operator has preliminary support to use out-of-tree Xe kernel mode driver to either add the driver where it is missing, or update the driver to a newer version.

The driver is installed via the Kernel Module Management (KMM) operator, which must be installed to the cluster. The driver containers are always built in-cluster, so a registry that the cluster can push to is also required. Currently only Ubuntu 26.04 is supported.

Details of the functionality are provided in the OOTDRIVER.md file.

Contributing

Contributions to this project are welcome as issues (bugs, enhancement requests) or via pull requests. Please review our Code of Conduct and our note on security policy.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries.

Name		Name	Last commit message	Last commit date
.github		.github
api/v1alpha1		api/v1alpha1
build		build
charts		charts
cmd		cmd
config		config
docs		docs
fwfiles		fwfiles
hack		hack
internal		internal
scripts		scripts
test		test
.gitignore		.gitignore
.golangci.yml		.golangci.yml
.trivyignore.yaml		.trivyignore.yaml
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
FWUPDATE.md		FWUPDATE.md
LICENSE		LICENSE
Makefile		Makefile
NOTICE		NOTICE
OOTDRIVER.md		OOTDRIVER.md
PROJECT		PROJECT
README.md		README.md
RECOVERY.md		RECOVERY.md
SECURITY-CONFIGURATION.md		SECURITY-CONFIGURATION.md
SECURITY.md		SECURITY.md
go.mod		go.mod
go.sum		go.sum
trivy.yaml		trivy.yaml

Navigation Menu

Repository navigation

About

Repository files navigation

Intel GPU Base Operator for Kubernetes

Description

Dependencies

Getting Started

Helm deployment

Example: DRA without NFD

Example: DRA with NFD and Kueue

Example: Device Plugin with NFD

Installation to an existing namespace

Custom Resource (CR) fields

Core fields

Device Plugin (`spec.dp`)

Dynamic Resource Allocation (`spec.dra`)

Health monitoring (`spec.health`)

XPU-Manager (`spec.xpu`)

OTLP export (`spec.xpu.otlp`)

TLS

Authentication

Rotating credentials

With `spec.xpu.configMapOverride`

Examples

Kueue (`spec.kueue`)

Building the operator

Development builds

Dependencies

Kubectl deployment

Firmware update

OoT Kernel driver

Contributing

License

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Intel GPU Base Operator for Kubernetes

Description

Dependencies

Getting Started

Helm deployment

Example: DRA without NFD

Example: DRA with NFD and Kueue

Example: Device Plugin with NFD

Installation to an existing namespace

Custom Resource (CR) fields

Core fields

Device Plugin (spec.dp)

Dynamic Resource Allocation (spec.dra)

Health monitoring (spec.health)

XPU-Manager (spec.xpu)

OTLP export (spec.xpu.otlp)

TLS

Authentication

Rotating credentials

With spec.xpu.configMapOverride

Examples

Kueue (spec.kueue)

Building the operator

Development builds

Dependencies

Kubectl deployment

Firmware update

OoT Kernel driver

Contributing

License

About

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

Device Plugin (`spec.dp`)

Dynamic Resource Allocation (`spec.dra`)

Health monitoring (`spec.health`)

XPU-Manager (`spec.xpu`)

OTLP export (`spec.xpu.otlp`)

With `spec.xpu.configMapOverride`

Kueue (`spec.kueue`)