CPU Scheduling, Quotas, Throttling & PSI

Engineering focus: An API's p99 latency spikes while node CPU usage remains below 50%. Could scheduling or Kubernetes CPU limits be responsible? How do you prove the diagnosis without guessing?

4.1 CPU requests and limits

A Kubernetes CPU request informs scheduling and relative CPU allocation; it doesn't usually reserve a dedicated physical core. A CPU limit enforces a ceiling on CPU runtime through kernel controls. Under Linux cgroups v2, cpu.weight governs relative share during contention while cpu.max enforces a bandwidth cap.

resources:
  requests:
    cpu: 500m
  limits:
    cpu: "1"

Here 500m means 0.5 CPU requested, while one CPU is the cap. CPU limits are not the same as CPU affinity or exclusive CPU-manager allocation.

4.2 How cpu.max creates quota throttling

cpu.max uses a quota and period in microseconds:

100000 100000  # about 1 CPU of runtime per period
50000 100000   # about 0.5 CPU of runtime per period
max 100000     # no hard bandwidth limit at this level

A multithreaded process can burn through quota much sooner than the wall-clock period. With a one-CPU quota and four CPU-bound threads running in parallel on four CPUs, the group can consume approximately 100 ms of combined CPU time in ~25 ms wall time, then be throttled for the remaining period.

One-CPU quota with four runnable threads

Critical: the host may have idle CPUs while this container is throttled. CPU limits are intentionally enforced at the cgroup boundary.

4.3 Reading cgroup v2 counters

cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat

Typical fields include usage_usec, nr_periods, nr_throttled, throttled_usec. Observe deltas, not only lifetime totals.

[ \text{fraction of quota periods affected} = \frac{\Delta nr_throttled}{\Delta nr_periods} ]

That is not the percentage of wall-clock time that requests were waiting. Likewise, accumulated throttle-time accounting can reflect more than one runnable thread and must be interpreted carefully.

4.4 Load average is not utilization

Linux load average summarizes tasks that are runnable or in uninterruptible sleep, over 1, 5 and 15 minutes. A load average of 40 with 70% idle CPU is possible when many tasks are blocked in D state. Investigate vmstat runnable and blocked fields, per-CPU metrics and wait channels.

uptime
vmstat 1
mpstat -P ALL 1
pidstat -u -t 1

%iowait is a CPU accounting category and is tricky to interpret on multi-core systems; it isn't a direct storage latency measure. %steal in a VM reflects virtual CPU scheduling time unavailable to the guest because the hypervisor serviced other work—not the same thing as guest cgroup throttling.

4.5 Pressure Stall Information (PSI)

PSI measures time lost because runnable/workload tasks cannot progress due to resource contention:

cat /proc/pressure/cpu
cat /proc/pressure/memory
cat /proc/pressure/io

Example:

some avg10=8.50 avg60=4.20 avg300=2.10 total=15000000

some means at least one task was stalled on the resource during measured time. Memory and I/O also support full measures of simultaneous stalls. System-level CPU full is not a generally useful signal and has been reported as zero for compatibility on documented kernels. Per-cgroup PSI can refine investigation.

Utilization answers "how busy?"; pressure answers "how much productive work is stalled?" Neither alone proves the application root cause; correlate with tail latency and traces.

4.6 Evidence-based CPU incident

Hypothetical observations:

Signal	Value
Host CPU	48%
Container CPU limit	1 core
Application workers	8 concurrent threads
Average use	650m
Throttling	Sharp increase during bursts
p50 / p99	35 ms / 900 ms

Diagnosis: test whether the cgroup's burst quota produces scheduling stalls. Compare cpu.stat and latency time series, inspect thread activity, then run a controlled load test raising or removing the cap while watching impact on neighboring workloads. Adding nodes alone may not solve a self-imposed per-container hard CPU cap.

What not to do

Don't remove all CPU limits universally based on one incident: multi-tenant workload isolation and noisy-neighbor risk remain important. Don't conclude 50% host CPU means no CPU problem; quota, affinity, priority inversion and synchronization are separate causes.

Key takeaways

Separate CPU utilization, runnable contention and quota throttling. A cgroup can consume its CPU budget in a short parallel burst even while the node has idle CPU, causing long tail latency. Inspect `cpu.max`, throttle-counter deltas, per-thread CPU, PSI and traces; then change one resource control in a controlled load test and check both application p99 and neighboring workloads.