Linux Internals & Production Troubleshooting
A complete practical field guide for intermediate-to-senior Linux, SRE and platform engineers.
Linux is the runtime foundation under containers and Kubernetes. This module explains what the kernel is doing, how resource controls behave, how performance degrades, and how to prove root causes using production evidence.
Scope
Examples are hypothetical and for education. Run intrusive tracing, load tests, process signals, and sysctl changes only in environments where you have permission. Provider-specific and kernel-version-dependent behavior should be verified against the deployed environment.
Prerequisites
- Basic terminal usage, processes, files and networking.
- Familiarity with containers and Kubernetes is helpful, but not required.
- A disposable Linux VM for the hands-on exercises.
Learning objectives
By the end, you should be able to explain the process and scheduler lifecycle; virtual memory and page faults; cgroups v2, OOM and QoS; CPU quotas and Pressure Stall Information; Linux filesystem I/O and durability; descriptors, sockets and epoll; and systematic production performance diagnosis.
Chapters
- Processes, threads, signals and scheduling —
fork,execve, process states, context switching, fair scheduling. - Virtual memory, page faults and reclaim — MMU/TLB, RSS/PSS, heap/stack, page cache and memory pressure.
- Cgroups v2, OOM and container memory — memory limits, hierarchical accounting, local/global OOM and Kubernetes QoS.
- CPU scheduling, quotas, throttling and PSI —
cpu.max, weights, throttling, load average and pressure. - Disk I/O, filesystems and durability — VFS, page cache, buffered/direct I/O,
fsync, IOPS, latency. - File descriptors, sockets, epoll and limits — FD tables,
EMFILE,ENFILE, TCP states and efficient event loops. - Full production performance investigation — USE/RED, evidence-first triage, resource-specific commands, reversible mitigations.
Hands-on practice
Five reproducible Linux systems labs — CPU throttling, memory, sockets, durability and incident diagnosis.
technical review exercises
Practice explaining each kernel mechanism, its observable symptoms, the commands that prove it, and the safer mitigation. Avoid interpreting a single utilization percentage as proof of saturation.
Suggested study workflow
Read each chapter, draw the Mermaid flow or sequence diagram from memory, run the commands on an isolated VM, and reason through a hypothetical marketplace incident. Finish with the 35-question challenge bank.