Resource management, QoS, throttling, eviction and priority

Engineering focus: The application is Guaranteed QoS, but its container was OOMKilled on a node with plenty of RAM. Another service has poor p99 despite low host CPU. Explain both phenomena and how you'd size resources.

Requests, limits and three separate decisions

Request: Guides scheduling against node allocatable and influences resource entitlement. Limit: Is a local enforcement ceiling where implemented. Observed usage: Is what the process actually consumes. These are not interchangeable.

Resource	Request	Limit / enforcement	Failure symptom
CPU	Scheduler capacity and relative entitlement	Cgroup CPU bandwidth quota in typical configurations	Throttling, p99 spikes despite spare host CPU
Memory	Scheduler/eviction considerations	Memory cgroup hard ceiling	Local OOM kill or allocation failure
Ephemeral storage	Scheduling/accounting	Kubelet accounting/eviction subject to layout/support	DiskPressure, eviction, failed writes
PID count	Node/container resource policy	Cgroup pids controller when configured	fork()/thread creation fails

resources:
  requests:
    cpu: 500m
    memory: 512Mi
    ephemeral-storage: 1Gi
  limits:
    memory: 1Gi
    ephemeral-storage: 4Gi

If only a limit is specified for a supported resource and admission hasn't changed the object, Kubernetes generally defaults the request to the same value. ResourceQuota and LimitRange policies may further constrain or default resources.

QoS: Guaranteed, Burstable, BestEffort

• Guaranteed: CPU and memory request=limit for every relevant container; strongest common QoS class under pressure.
• Burstable: Some requests/limits exist, but Guaranteed requirements aren't satisfied.
• BestEffort: No CPU or memory requests/limits are specified for the Pod's containers.

QoS affects eviction and OOM-selection behavior, but Guaranteed is not immune to exceeding its own memory.max. Local cgroup OOM can happen even if the node has spare memory. Distinguish OOMKilled (kernel/cgroup kill) from Evicted (kubelet's node-pressure action).

CPU quota versus CPU request

Suppose a Pod is limited to one CPU but has four runnable CPU-bound threads. Parallel threads can consume a combined period quota early and be throttled until replenishment, even if the host remains partly idle. Averages hide short bursts, so inspect nr_throttled, nr_periods, throttled_usec, CPU pressure and request latency together. The throttled-period ratio isn't the percentage of request time stalled.

cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat
cat /proc/pressure/cpu

Memory charges include more than heap

Cgroup memory may include anonymous/native allocations, file cache, socket buffers, stacks and certain kernel charges. Thus a JVM heap of 600 MiB doesn't imply a container with a 1 GiB hard memory cap is safe. Use runtime telemetry and cgroup memory breakdown:

cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.stat
cat /sys/fs/cgroup/memory.events
cat /proc/pressure/memory

memory.high induces reclaim/throttling pressure; memory.max is the hard ceiling. Memory PSI and reclaim may reveal severe latency before OOM. Not every Kubernetes version/node exposes the same cgroup files, and some managed clusters restrict access.

NodePressure and eviction

Kubelet monitors node-level memory, disk and PID pressure. When thresholds are breached, it can evict Pods according to relevant usage, requests and priority. Pod phase/status may show Evicted. Kubelet eviction is not the same as scheduler preemption; the latter frees capacity for higher-priority pending Pods during scheduling.

PriorityClass and governance

Priority affects scheduling and preemption. Use it for genuinely critical workloads, with controls on who may assign high classes. Making every application high-priority removes meaningful differentiation and may increase disruption of lower-priority jobs.

Production scenario: CPU low, p99 high

Node CPU averages 45%; one Pod consumes 650m on average with a 1000m CPU limit, yet is repeatedly throttled during concurrent bursts. Validate by correlating cpu.stat deltas and request latency. A controlled higher-limit canary may improve p99, but watch neighboring workloads and node capacity. More replicas could also increase DB or connection-pool pressure, so treat scaling as one hypothesis, not an automatic fix.

Key takeaways: Requests, limits and actual consumption solve different problems. Scheduler placement uses requests; CPU limits can throttle, while memory limits can cause local OOM regardless of spare host memory. QoS is a pressure policy, not immunity. I size workloads using representative peak load, p99 latency, OOM/throttle data, native memory, dependency capacity and failure-domain headroom. I differentiate cgroup OOM, node eviction and scheduler preemption before mitigating.