Production Infrastructure

Infrastructure work is diagnosis under time pressure. The skill that matters is being able to take a symptom (elevated latency, a health check flapping, a pod that will not schedule) and narrow it to a layer quickly, using tools that are already on the box. Everything else is configuration you can look up.

This path is ordered accordingly. It starts at the kernel-visible level, because if you cannot see what a process is doing on one host you will not diagnose it across fifty, and it treats orchestration as a late stage, not a starting point. Kubernetes is an abstraction over Linux, networking and storage; learning it first means every failure is opaque.

Each stage states its prerequisites, the capability you should end up with, what to read, and how to tell you are ready to move on.

Stage 1 - Linux as an operating system, not a shell

Prerequisites. Basic command-line use: navigating the filesystem, editing files, reading a man page.

Outcome. Given a host that is “slow”, you can determine within a few minutes whether it is bound by CPU, memory, disk I/O or a lock, name the process responsible, and show the evidence, without installing anything.

What to read and do. Start with Linux systems here for orientation, then work at the level of processes, file descriptors and syscalls. The mental model to build: a process is a collection of kernel resources, and almost every mystery resolves into “which resource is it waiting on”.

Learn the tools by using them on a machine you have deliberately broken. Fill a disk and watch what fails first. Exhaust the file-descriptor limit and read the resulting error. Start a process that spins and one that blocks on I/O, and confirm you can tell them apart from top alone.

Learn systemd properly: unit files, dependency ordering, restart policies, and resource limits via cgroups. Almost everything you will later run in a container is constrained by the same cgroup mechanism, so the concepts are not wasted.

Ready to move on when. You can diagnose a saturated host to a named process and a named resource using only tools present on a default install, and you can write a systemd unit with correct dependency ordering and a restart policy that will not mask a crash loop.

Stage 2 - Networking, from the packet to TLS

Prerequisites. Stage 1.

Outcome. Given “the service is unreachable”, you can identify the failing hop (DNS, routing, firewall, listener, TLS handshake or application) with a specific command per hypothesis, and say which one produced which evidence.

What to read and do. Work bottom-up: addressing and routing, then TCP connection establishment and teardown, then DNS resolution order on the host, then TLS. Read the iproute2 documentation for the modern tooling (ip, ss) rather than the deprecated ifconfig and netstat.

Capture packets. tcpdump on a failing connection tells you in one command whether the SYN left, whether anything came back, and whether a reset arrived. That distinguishes a firewall drop (silence) from a closed port (RST) from a routing problem (no reply at all, and no route in ip route get).

For TLS, know what a certificate chain is, what the client validates, and the three failures you will actually see: expired leaf, missing intermediate (works in a browser that caches it, fails from a server that does not), and hostname mismatch. openssl s_client -connect host:443 -servername host shows all three.

Then connection state. Read ss -tan output and know what a large TIME_WAIT count means versus a full accept queue versus SYN_RECV accumulation.

Ready to move on when. You can debug a broken TLS connection to a specific cause from the command line, and explain what a health check on a load balancer is actually testing at each layer.

Stage 3 - Availability: load balancing and failover

Prerequisites. Stages 1-2.

Outcome. You can design a two-node service with automatic failover, state its failure modes including split brain, and explain why address failover and traffic distribution need separate components.

What to read and do. This is where the site has real depth. Read keepalived for VRRP: how a virtual IP moves between nodes, what the health-check script must verify, and why a check that only tests “is the process running” fails to protect you from a process that is running and broken.

Then HAProxy for TCP and HTTP load balancing with health checks that distinguish node roles. The PostgreSQL example there is worth reading even if you never run PostgreSQL, because it shows a health check that queries application state instead of liveness. That distinction is what makes automated failover safe.

The concept to internalise is split brain. Two nodes that cannot see each other but can both see clients will both claim the service address. VRRP alone does not prevent this; it needs either fencing or a quorum mechanism. Being able to state what your design does when the network partitions, and not merely when a node cleanly dies, is the difference between a high-availability setup and a high-availability diagram.

For the database side of the same problem, see PostgreSQL high availability and the Patroni reference.

Ready to move on when. You can build a keepalived pair, cause a failover by breaking the service and not the host, measure how long clients were affected, and describe what would happen under a network partition rather than a clean failure.

Stage 4 - Containers, and what they actually are

Prerequisites. Stages 1-3. The Linux stage matters here: a container is not a lightweight virtual machine, it is a process with namespaces and cgroups applied.

Outcome. You can explain a container’s isolation in terms of the specific kernel namespaces involved, build an image that is small and reproducible, and debug a container that exits immediately on start.

What to read and do. Read the OCI image specification for what an image really is (an ordered set of filesystem layers plus a config blob), then confirm it by unpacking one. Understanding layers is what makes build caching, image size and the “why is my secret still in the image after I deleted it in a later layer” problem obvious, not mysterious.

Build images deliberately: multi-stage builds so compilers do not ship to production, a non-root user, pinned base image digests instead of floating tags, and no secrets in build arguments. Then break things: a container that exits with code 0 immediately (the main process forked and returned), a container OOM-killed by its cgroup limit while the host has free memory, a volume mount that shadows the application directory.

Ready to move on when. You can explain what docker run does in terms of namespaces, cgroups and the layered filesystem, and you can get a shell into a running container’s namespaces to debug it without the image containing a shell.

Stage 5 - Orchestration with Kubernetes

Prerequisites. Stage 4, and stages 1-3 for the diagnosis skills. Also version control discipline (see Git and GitHub), because from here configuration lives in a repository.

Outcome. You can take a pod that is not running and follow the control-plane chain - scheduler, kubelet, container runtime, readiness probe, service endpoints - to the specific object or condition blocking it.

What to read and do. Read Kubernetes here, then the Kubernetes concepts documentation for the object model. Learn the reconciliation model first: you declare a desired state, controllers work continuously to make reality match, and nearly every question is “which controller is unhappy and what is it saying”. kubectl describe and kubectl get events --sort-by=.lastTimestamp answer that more often than logs do.

The failures worth practising, because they cover most of what you will see:

Get requests and limits right, because they are the most common source of self-inflicted production problems. A missing CPU request means the scheduler cannot place the pod sensibly; a CPU limit means throttling that looks like application slowness with idle nodes; a memory limit is a hard kill.

Then delivery: manifests in Git, templated with Helm or Kustomize, applied by a controller and not by a person with cluster credentials.

Ready to move on when. You can debug all five rows of the table above from kubectl alone, and you can describe what happens to in-flight requests during a rolling update of your own service.

Stage 6 - Observability and operating what you built

Prerequisites. Stages 1-5, and a service with real traffic.

Outcome. You can define service level objectives for a service you run, instrument it so those objectives are measurable, and alert on symptoms users experience, not on causes you happen to have a metric for.

What to read and do. The three signals (metrics, logs, traces) answer different questions. The common mistake is buying all three and instrumenting none of them deliberately. Metrics tell you something is wrong and roughly where; traces tell you which hop in a request; logs tell you why, once you already know where to look.

Read the Prometheus documentation for the pull model, labels and cardinality. An unbounded label such as a user ID or a request ID will take down your monitoring before it helps you. Read the OpenTelemetry documentation for vendor-neutral instrumentation, and Google’s Site Reliability Engineering book, which is freely available online, for SLOs, error budgets and the argument against cause-based alerting.

Alerting discipline: every alert should be actionable, have a written runbook, and correspond to something a user would notice. An alert that fires and is routinely ignored has trained your team to ignore alerts.

Note the gap: this site does not yet have reference pages for Prometheus, Grafana or OpenTelemetry. Use the primary sources above.

Ready to move on when. You can state a numeric SLO for a service you operate, show the dashboard that measures it, name the alerts that would fire before the budget is exhausted, and point at the runbook each one links to.

Coverage gaps