Introduction

OpenShift Virtualization places a VM on a node exactly once, at admission time, and it does so based on the VM’s requested vCPU and memory, not on what the VM will actually do once it’s running. That single decision point creates a blind spot: a cluster can be perfectly balanced by requests and badly imbalanced by real load.

Picture four identically-sized VMs spread two-per-node across two workers. On paper, both nodes carry the same weight. In practice, if the two busy VMs happen to share a node while the two idle ones share the other, one node is starved for CPU and the other is coasting. The scheduler will never revisit that placement on its own.

Why PSI, Not Requests

The Kube Descheduler Operator closes the loop: it evicts pods, and therefore VMs, from nodes it considers overutilized, letting the scheduler try again with more headroom. The question is what “overutilized” means. If the descheduler measures by requests, the scenario above never triggers, since requests were balanced the whole time. This guide configures the descheduler to use PSI (Pressure Stall Information) instead: a kernel metric that reports the percentage of time tasks on a node spend stalled waiting for CPU, memory, or I/O. PSI measures suffering, not bookkeeping, which makes it the right eviction signal for VM workloads. This is the load-aware approach Red Hat describes in its work on rebalancing OpenShift Virtualization (linked in the references).

Prerequisites

  • A disconnected OpenShift cluster on 4.17 or newer. 4.17 is the floor because the profileCustomizations fields this guide depends on aren’t available earlier. Note this is newer than the 4.14 baseline in the related OpenShift Virtualization disconnected-install guide – if you’re following both, plan for 4.17+ from the start.
  • Two or more worker nodes advertising devices.kubevirt.io/kvm. Rebalancing needs a destination; more nodes simply make the effect more visible.
  • OpenShift Virtualization installed and the HyperConverged resource reporting Available (see the related OpenShift Virtualization disconnected install guide).
  • A quiet cluster during testing, ideally – other CPU-heavy workloads will muddy the pressure signal you’re trying to observe.
  • A bastion host with internet access, oc, and oc mirror.
  • A Fedora (or similar) containerDisk image in the internal registry, plus the Fedora base container image mirrored as well (Step 4 runs a small controller from it).

What’s Different on a Disconnected Cluster

Less than you might expect. We’re building this on a disconnected cluster because that’s the environment we’re used to working in for customer deployments; nothing about the PSI mechanics or the descheduler behavior below depends on it, so the same steps apply just as well on a connected cluster. OperatorHub is empty until you mirror the operator in, and the mirrored catalog must match the cluster’s minor version, the same two rules as any disconnected operator install. Enabling PSI itself, on the other hand, is a kernel argument delivered by MachineConfig and needs no network access at all, so that part is identical on connected and disconnected clusters. The only piece that touches the mirror is the operator.

Step 1: Enable PSI At The Kernel Level

The kernel ships with PSI accounting compiled in, but OpenShift nodes boot with it disabled. Two MachineConfigs turn it on, one per role. The worker MachineConfig also enables scheduler statistics (schedstats), which some of the underlying CPU metrics rely on; folding it into the same object means the workers only go through a single reboot cycle.

# mc-psi-master.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-master-enable-psi
spec:
  kernelArguments:
    - psi=1
# mc-psi-worker.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-enable-psi
spec:
  kernelArguments:
    - psi=1
    - schedstats=enable
    - sched.schedstats=enable

Apply both and expect a rolling reboot of every node:

$ oc apply -f mc-psi-master.yaml
$ oc apply -f mc-psi-worker.yaml
$ oc get mcp -w

Wait until both MachineConfigPools report UPDATED=True. Then confirm PSI is live by reading it straight from a worker’s kernel:

$ oc debug node/<any-worker> -- chroot /host cat /proc/pressure/cpu
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

If /proc/pressure/cpu exists and prints these two lines, the kernel side is done. If the file is missing, the node hasn’t picked up the karg yet.

Step 2: Mirror And Install The Operator

If cluster-kube-descheduler-operator already shows up in your cluster’s catalog, skip ahead to Step 3.

On the bastion, describe what to mirror. Match the catalog tag to your cluster’s minor version (4.17 shown):

# imageset-config.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17
      packages:
        - name: cluster-kube-descheduler-operator
          channels:
            - name: stable

Mirror it into the internal registry and apply the generated catalog resources:

$ oc mirror --config=imageset-config.yaml docker://<internal-registry>
$ oc apply -f oc-mirror-workspace/results-*/

Check which CSV the stable channel resolves to on your mirror:

$ oc get packagemanifest cluster-kube-descheduler-operator -n openshift-marketplace \
  -o jsonpath='{.status.channels[*].currentCSV}{"\n"}'

Install the operator by creating its namespace, an OperatorGroup, and a Subscription:

# descheduler-install.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-kube-descheduler-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
spec:
  targetNamespaces:
    - openshift-kube-descheduler-operator
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-kube-descheduler-operator
  namespace: openshift-kube-descheduler-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: cluster-kube-descheduler-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
$ oc apply -f descheduler-install.yaml

Verify the CSV reaches Succeeded (you can also confirm it in the console under Operators → Installed Operators):

$ oc get csv -n openshift-kube-descheduler-operator

Step 3: Configure The KubeDescheduler Resource For PSI

The operator manages a singleton KubeDescheduler resource that must be named cluster. Three settings do the heavy lifting here:

  • mode: Automatic – actually evict. The default Predictive mode only logs what it would do, which is useful for a dry run but makes for a much less convincing example.
  • profiles – LongLifecycle is the profile that targets utilization imbalance; the two EvictPodsWith* profiles allow VMs backed by PVCs or local storage to be candidates at all.
  • profileCustomizations.devActualUtilizationProfile: PrometheusCPUPSIPressure — the point of this guide: switch LongLifecycle from requests-based math to the CPU PSI signal scraped by cluster monitoring.
# kubedescheduler.yaml
apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
  name: cluster
  namespace: openshift-kube-descheduler-operator
spec:
  mode: Automatic
  managementState: Managed
  deschedulingIntervalSeconds: 60
  profiles:
    - LongLifecycle
    - EvictPodsWithPVC
    - EvictPodsWithLocalStorage
  profileCustomizations:
    devEnableEvictionsInBackground: true
    devActualUtilizationProfile: PrometheusCPUPSIPressure
$ oc apply -f kubedescheduler.yaml

The descheduler reads PSI through the cluster monitoring stack, so its service account needs read access to metrics. Without this binding it fails silently – the pods run, nothing ever gets evicted:

$ oc adm policy add-cluster-role-to-user cluster-monitoring-view \
  -z openshift-descheduler -n openshift-kube-descheduler-operator

Verify:

$ oc get kubedescheduler cluster -n openshift-kube-descheduler-operator \
  -o jsonpath='{.spec.mode}{"\n"}{.spec.profiles}{"\n"}'
$ oc get pods -n openshift-kube-descheduler-operator

Step 4: Protect Drained Nodes with a Pressure-Aware Tainter

There’s a gap in the loop: LongLifecycle evicts a VM off a pressured node, but nothing tells the scheduler to avoid that node, so the replacement VM can land right back where it came from. The fix is to taint pressured nodes with PreferNoSchedule – a soft signal that steers new placements away without forcing anything off.

Rather than scraping the descheduler’s logs, the small controller below goes to the same source of truth the descheduler uses: it queries CPU PSI per worker from the in-cluster Prometheus (via the Thanos querier), taints any worker whose 1-minute CPU pressure crosses a threshold, and removes the taint once the node calms down. It’s a single Python script, self-contained, and needs no packages beyond what the stock Fedora image ships.

First, a dedicated service account with exactly the permissions it needs – read and patch nodes, plus metrics read:

# psi-tainter-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: psi-tainter
  namespace: openshift-kube-descheduler-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psi-tainter
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: psi-tainter
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psi-tainter
subjects:
  - kind: ServiceAccount
    name: psi-tainter
    namespace: openshift-kube-descheduler-operator
$ oc apply -f psi-tainter-rbac.yaml
$ oc adm policy add-cluster-role-to-user cluster-monitoring-view \
  -z psi-tainter -n openshift-kube-descheduler-operator

The controller itself:

# psi-tainter.py
import json, os, ssl, time, urllib.parse, urllib.request

SA = "/var/run/secrets/kubernetes.io/serviceaccount"
API = "https://kubernetes.default.svc"
THANOS = "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query"
QUERY = 'avg by (instance) (rate(node_pressure_cpu_waiting_seconds_total[1m])) * 100'
TAINT_KEY = "vm-rebalance/cpu-pressure"
THRESHOLD = float(os.environ.get("PSI_THRESHOLD", "25"))   # percent
INTERVAL = int(os.environ.get("POLL_SECONDS", "30"))

ctx = ssl.create_default_context(cafile=f"{SA}/ca.crt")

def call(url, method="GET", body=None, content_type=None):
    token = open(f"{SA}/token").read()
    req = urllib.request.Request(url, data=body, method=method)
    req.add_header("Authorization", f"Bearer {token}")
    if content_type:
        req.add_header("Content-Type", content_type)
    with urllib.request.urlopen(req, context=ctx) as resp:
        return json.load(resp)

def pressure_by_node():
    data = call(f"{THANOS}?{urllib.parse.urlencode({'query': QUERY})}")
    out = {}
    for r in data["data"]["result"]:
        node = r["metric"]["instance"].split(":")[0]
        out[node] = float(r["value"][1])
    return out

def workers():
    sel = urllib.parse.quote("node-role.kubernetes.io/worker=")
    return call(f"{API}/api/v1/nodes?labelSelector={sel}")["items"]

def set_taints(name, taints):
    patch = json.dumps({"spec": {"taints": taints}}).encode()
    call(f"{API}/api/v1/nodes/{name}", "PATCH", patch,
         "application/strategic-merge-patch+json")

while True:
    try:
        psi = pressure_by_node()
        for node in workers():
            name = node["metadata"]["name"]
            taints = node["spec"].get("taints") or []
            tainted = any(t["key"] == TAINT_KEY for t in taints)
            value = psi.get(name, 0.0)
            if value >= THRESHOLD and not tainted:
                taints.append({"key": TAINT_KEY, "effect": "PreferNoSchedule"})
                set_taints(name, taints)
                print(f"tainted {name} (cpu psi {value:.1f}%)", flush=True)
            elif value < THRESHOLD and tainted:
                set_taints(name, [t for t in taints if t["key"] != TAINT_KEY])
                print(f"untainted {name} (cpu psi {value:.1f}%)", flush=True)
    except Exception as e:
        print(f"error: {e}", flush=True)
    time.sleep(INTERVAL)

Package it as a ConfigMap and run it as a one-replica Deployment. The image reference points at your mirrored Fedora base image:

# psi-tainter-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: psi-tainter
  namespace: openshift-kube-descheduler-operator
  labels:
    app: psi-tainter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: psi-tainter
  template:
    metadata:
      labels:
        app: psi-tainter
    spec:
      serviceAccountName: psi-tainter
      containers:
        - name: tainter
          image: <internal-registry>/fedora/fedora:latest
          command: ["python3", "/app/psi-tainter.py"]
          env:
            - name: PSI_THRESHOLD
              value: "25"
          volumeMounts:
            - name: script
              mountPath: /app
      volumes:
        - name: script
          configMap:
            name: psi-tainter
$ oc create configmap psi-tainter -n openshift-kube-descheduler-operator \
  --from-file psi-tainter.py
$ oc apply -f psi-tainter-deploy.yaml

Tune PSI_THRESHOLD to taste – 25% “some” pressure over a minute is a reasonable starting point for a test cluster; production clusters may want it higher.

Step 5: Build a Load Imbalance the Scheduler Can’t See

To prove the point, the example needs an imbalance that exists in pressure but not in requests. Six identical VMs do it: three running a sustained CPU burn, three completely idle – all initially pinned to a single node.

Create a namespace for them:

$ oc new-project vm-rebalance-example

Cordon every worker except one so the whole batch lands together:

$ oc get nodes -l node-role.kubernetes.io/worker -o name | tail -n +2 | xargs -I{} oc adm cordon {}

The busy VMs pin every guest vCPU with sha256sum /dev/zero – an infinite hashing loop that needs nothing installed, works on any minimal image, and produces steady, saturating CPU pressure:

$ for i in 1 2 3; do
cat <<EOF | oc apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-busy-$i
  namespace: vm-rebalance-example
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        cpu: { cores: 2 }
        memory: { guest: 2Gi }
        devices:
          disks:
            - name: containerdisk
              bootOrder: 1
            - name: cloudinitdisk
      volumes:
        - name: containerdisk
          containerDisk:
            image: <internal-registry>/kubevirt/fedora-container-disk:41
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |
              #cloud-config
              runcmd:
                - [ bash, -c, "for c in \$(seq \$(nproc)); do sha256sum /dev/zero & done" ]
EOF
done

The idle VMs are byte-for-byte the same size, minus the burn:

$ for i in 1 2 3; do
cat <<EOF | oc apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-idle-$i
  namespace: vm-rebalance-example
spec:
  runStrategy: Always
  template:
    spec:
      domain:
        cpu: { cores: 2 }
        memory: { guest: 2Gi }
        devices:
          disks:
            - name: containerdisk
              bootOrder: 1
      volumes:
        - name: containerdisk
          containerDisk:
            image: <internal-registry>/kubevirt/fedora-container-disk:41
EOF
done

Confirm all six VMIs are running on the one open node, then uncordon the rest so the descheduler has somewhere to send them:

$ oc get vmi -n vm-rebalance-example -o wide
$ oc get nodes -l node-role.kubernetes.io/worker -o name | xargs -I{} oc adm uncordon {}

One expectation to set: these VMs define no evictionStrategy, so an eviction here is a restart, not a live migration. The VMI is deleted and – because runStrategy: Always holds – immediately recreated on a less-pressured node. Each rebalanced VM sees a brief outage, which is fine for this kind of exercise and worth knowing before you point this at anything real.

Step 6: Watch It Rebalance

VM distribution per node, refreshed live:

$ watch 'oc get vmi -n vm-rebalance-example -o jsonpath="{range .items[*]}{.status.nodeName}{\"\n\"}{end}" | sort | uniq -c'

Confirm the descheduler is issuing real evictions (not simulating):

$ oc logs -n openshift-kube-descheduler-operator -l app=descheduler --tail=20 | grep -i evicted

Watch the tainter follow the pressure:

$ oc logs -n openshift-kube-descheduler-operator -l app=psi-tainter -f
$ oc get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

And to see the raw signal driving all of it, skip the monitoring stack entirely and read PSI at the source on each worker – this is the very number node_exporter exports and the descheduler consumes:

$ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o name); do
  echo "== $node =="
  oc debug $node -- chroot /host cat /proc/pressure/cpu
done

avg10 on the some line is the percentage of the last 10 seconds during which at least one task sat stalled waiting for CPU. Right after Step 5 the packed node should show a high avg10 while every other worker reads near zero. Re-run the loop every minute or two: as VMs move off, the hot node’s number falls, the others tick up slightly, and eventually the workers settle into roughly similar values.

If nothing moves, check three things in order: spec.mode is Automatic and not Predictive; the cluster-monitoring-view binding from Step 3 exists (without it the descheduler can’t read PSI and quietly does nothing); and the busy VMs are genuinely burning CPU – oc exec into one and run top.

Summary

The whole pipeline in one breath: two MachineConfigs switch on PSI and schedstats at the kernel (one full rolling reboot), oc mirror brings the Kube Descheduler Operator into the disconnected registry, the KubeDescheduler singleton runs LongLifecycle against PrometheusCPUPSIPressure instead of requests, a small pressure-aware tainter keeps freshly-drained nodes from being refilled, and a deliberately lopsided six-VM example lets you watch real kernel pressure, not paperwork, decide where VMs live.

Since this guide spans disconnected operator management and node-level metrics, these related articles pair well with it:

References And Further Reading

Tags: