Systems and Infrastructure
The hardware and software stack beneath every application. Operating systems, processors, networking, and the distributed systems techniques that keep it all running reliably at scale. Click any topic to explore.
Deep Dives
Click any topic. Each includes architecture, key technologies, real-world deployment, and challenges.
Operating Systems
The layer that manages processes, memory, and hardware resources for everything else running on a machine.
Foundational
Operating Systems
The layer that manages processes, memory, and hardware resources for everything else running on a machine.
How It Works
An operating system provides the abstraction layer between hardware and applications. The kernel manages process scheduling (deciding which program runs on which CPU core), virtual memory (giving each process its own address space), file systems (organizing persistent storage), and device drivers (interfacing with hardware). System calls provide the API through which applications request OS services. Modern OS kernels also manage containerization primitives (namespaces, cgroups) used by Docker and Kubernetes.
Key Technologies
- Linux (dominant server OS, GPLv2)
- Windows (desktop dominant, WSL2 for Linux compatibility)
- macOS/Darwin (Apple, BSD-based Unix)
- FreeBSD (used by Netflix for CDN, Sony for PlayStation)
- RTOS (FreeRTOS, Zephyr for embedded/IoT)
Real-World Examples
Linux runs 96.3% of the top 1 million web servers (W3Techs). Windows has approximately 72% desktop market share. macOS powers development workflows at most tech companies. FreeBSD's networking stack is used by Netflix to serve terabytes of video per second. Android (Linux kernel) has 3+ billion active devices.
Challenges & Considerations
Kernel development requires deep expertise in C and hardware interfaces. Security vulnerabilities at the OS level affect every application running on it. Supporting the diversity of hardware configurations is a massive engineering effort. Real-time requirements for robotics and industrial control conflict with general-purpose OS design.
CPU Architecture
Instruction sets, pipelining, caches, and branch prediction determine how efficiently general-purpose code runs.
Foundational
CPU Architecture
Instruction sets, pipelining, caches, and branch prediction determine how efficiently general-purpose code runs.
How It Works
CPUs execute instructions from an Instruction Set Architecture (ISA). x86-64 (Intel/AMD) uses a complex instruction set (CISC) with hardware-decoded micro-ops. ARM uses a reduced instruction set (RISC) with fixed-length instructions, offering better power efficiency. RISC-V is an open ISA gaining traction. Modern CPUs use superscalar out-of-order execution, branch prediction, and multi-level caches (L1/L2/L3) to execute multiple instructions per cycle. Multi-core designs allow parallel execution of independent threads.
Key Technologies
- x86-64 (Intel Xeon, AMD EPYC for servers)
- ARM64/AArch64 (AWS Graviton, Apple M-series, Ampere Altra)
- RISC-V (open ISA, growing ecosystem)
- Branch prediction and speculative execution
- Cache hierarchy (L1: ~1ns, L2: ~4ns, L3: ~12ns, RAM: ~100ns)
Real-World Examples
AWS Graviton (ARM-based) processors offer up to 40% better price-performance than comparable x86 instances (AWS benchmarks). Apple's M-series chips (ARM) have transformed laptop computing with their performance-per-watt. AMD EPYC has taken significant server market share from Intel, reaching approximately 23% by 2024. RISC-V is used in billions of embedded devices and is being developed for server workloads by companies like SiFive and Ventana.
Challenges & Considerations
Speculative execution vulnerabilities (Spectre, Meltdown) remain an ongoing concern. The ARM/x86 duopoly means RISC-V adoption faces ecosystem challenges. Power consumption is a growing constraint as data centers consume 1-2% of global electricity. Compiler optimization for new architectures takes years to mature.
GPU Computing
Massively parallel processors originally built for graphics, now the primary engine for training and running neural networks.
Mainstream
GPU Computing
Massively parallel processors originally built for graphics, now the primary engine for training and running neural networks.
How It Works
GPUs contain thousands of small cores optimized for parallel execution of the same operation across many data elements (SIMT: Single Instruction, Multiple Threads). AI training involves billions of matrix multiplications, which map perfectly to GPU parallelism. NVIDIA's CUDA platform provides the software layer for programming GPUs. Tensor Cores (specialized matrix multiply units) accelerate the specific operations used in deep learning. High-bandwidth memory (HBM) provides the data throughput needed to keep thousands of cores fed.
Key Technologies
- NVIDIA H100/B200 (Hopper/Blackwell architecture)
- CUDA (NVIDIA's parallel computing platform)
- AMD MI300X (ROCm software stack, HIP for portability)
- Google TPU v5 (custom tensor processing ASIC)
- Apple Neural Engine (on-device inference)
Real-World Examples
NVIDIA reported data center revenue of $47.5 billion in fiscal year 2024 (NVIDIA earnings). A single H100 GPU costs approximately $25,000-$40,000. Training GPT-4-class models requires thousands of GPUs running for months. AMD's MI300X is gaining cloud adoption, available on Azure, Oracle Cloud, and others. Google uses TPUs internally for Search, YouTube, and Gemini model training.
Challenges & Considerations
NVIDIA's near-monopoly on training hardware creates supply constraints and high costs. US export controls restrict advanced GPU sales to certain countries. Power consumption of GPU clusters is substantial (a single H100 draws 700W). Programming GPUs efficiently requires specialized knowledge of memory hierarchies and parallel programming patterns.
Networking
Protocols and hardware that move bytes between machines reliably, from TCP/IP fundamentals to GPU cluster interconnects.
Foundational
Networking
Protocols and hardware that move bytes between machines reliably, from TCP/IP fundamentals to GPU cluster interconnects.
How It Works
Modern networking is built on the TCP/IP stack. TCP provides reliable, ordered byte streams between applications. HTTP/HTTPS runs on top of TCP for web traffic. DNS translates domain names to IP addresses. For data center networking, BGP routes traffic between autonomous systems. AI training clusters use specialized high-bandwidth interconnects (NVLink, InfiniBand, RoCE) to move gradients between GPUs at 400-800 Gbps. gRPC provides efficient RPC communication between microservices.
Key Technologies
- TCP/IP (foundational protocol suite)
- gRPC (Google, high-performance RPC)
- InfiniBand and RDMA (low-latency data center networking)
- NVLink and NVSwitch (NVIDIA GPU-to-GPU interconnect)
- BGP (Border Gateway Protocol for internet routing)
Real-World Examples
Cloudflare handles over 20% of global web traffic through its network. AWS, Azure, and GCP each operate private fiber networks spanning continents. NVIDIA's NVLink provides 900 GB/s bandwidth between GPUs in the same node. Modern AI training clusters use InfiniBand to connect thousands of GPUs with tail latencies under 2 microseconds.
Challenges & Considerations
Network latency is often the bottleneck in distributed AI training. The global routing system (BGP) is vulnerable to misconfigurations and hijacking. Transitioning to IPv6 remains incomplete after decades. High-bandwidth interconnects for AI clusters are expensive and vendor-locked.
Containers and Virtualization
Isolation mechanisms that let multiple workloads share hardware safely, from lightweight namespaces to full VMs.
Mainstream
Containers and Virtualization
Isolation mechanisms that let multiple workloads share hardware safely, from lightweight namespaces to full VMs.
How It Works
Virtualization creates isolated execution environments. Full VMs (KVM, Hyper-V) emulate complete hardware, running separate OS kernels with strong isolation. Containers (Docker, containerd) share the host kernel using namespaces (process, network, mount isolation) and cgroups (CPU, memory limits), making them lighter but with weaker isolation. MicroVMs (Firecracker, gVisor) provide a middle ground: VM-level isolation with container-like startup times.
Key Technologies
- KVM (Kernel-based Virtual Machine, Linux)
- Firecracker (AWS, powers Lambda and Fargate)
- gVisor (Google, application kernel for containers)
- containerd (industry-standard container runtime)
- QEMU (hardware emulation, cross-architecture)
Real-World Examples
AWS Lambda runs customer functions in Firecracker microVMs that boot in under 125 milliseconds. Google's gVisor provides an additional security layer for Google Cloud Run. KVM is the hypervisor behind most public cloud instances. VMware (now part of Broadcom) runs virtualization for most enterprise data centers.
Challenges & Considerations
Container isolation is weaker than VM isolation, which matters for multi-tenant security. Nested virtualization adds performance overhead. Container image sizes can grow large without careful management. GPU passthrough to VMs and containers requires specific hardware and driver support.
Distributed Systems
Techniques for coordinating many machines to behave like one reliable system: consensus, replication, and partial failure handling.
Mainstream
Distributed Systems
Techniques for coordinating many machines to behave like one reliable system: consensus, replication, and partial failure handling.
How It Works
Distributed systems coordinate multiple machines to provide availability, consistency, and partition tolerance (the CAP theorem states you can only guarantee two of three). Consensus protocols (Raft, Paxos) allow a cluster to agree on a value even if some nodes fail. Replication strategies (leader-follower, multi-leader, leaderless) trade off consistency, latency, and availability. Consistent hashing distributes data across nodes. Vector clocks and CRDTs handle conflict resolution in eventually consistent systems.
Key Technologies
- Raft (consensus protocol, used by etcd, CockroachDB)
- Paxos (foundational consensus, used by Google Spanner)
- CRDTs (Conflict-free Replicated Data Types)
- Consistent hashing (DynamoDB, Cassandra)
- etcd (distributed key-value store, Kubernetes backbone)
Real-World Examples
Every Kubernetes cluster relies on etcd (Raft consensus) for state. Google Spanner uses Paxos and TrueTime for globally consistent transactions. Amazon DynamoDB uses consistent hashing and vector clocks for its leaderless design. Apache Cassandra handles millions of writes per second across hundreds of nodes at Netflix and Apple.
Challenges & Considerations
The CAP theorem forces fundamental trade-offs with no perfect solution. Distributed debugging is extremely difficult (partial failures, network partitions, clock skew). Formal verification of distributed protocols is still an active research area. Latency increases with geographic distribution.
Observability
Metrics, logs, and traces instrumented across systems so operators can understand behavior and diagnose failures.
Growing
Observability
Metrics, logs, and traces instrumented across systems so operators can understand behavior and diagnose failures.
How It Works
Observability rests on three pillars: metrics (numeric measurements over time, e.g., request rate, error rate, latency), logs (structured event records), and traces (request paths across distributed services). Prometheus scrapes metrics from applications. OpenTelemetry provides a vendor-neutral SDK for instrumenting code. Grafana visualizes data from multiple sources. Alerting rules trigger notifications when metrics cross thresholds.
Key Technologies
- Prometheus (pull-based metrics, CNCF graduated)
- OpenTelemetry (unified instrumentation standard)
- Grafana (visualization, dashboards)
- Jaeger (distributed tracing, CNCF graduated)
- Loki (log aggregation, Grafana Labs)
Real-World Examples
Prometheus is the default monitoring system for Kubernetes clusters worldwide. OpenTelemetry has become the second-most active CNCF project (after Kubernetes itself). Grafana Labs raised $240M at a $6B valuation in 2022. Datadog, a commercial observability platform, has over 27,000 customers and $2.1B in annual revenue (2023).
Challenges & Considerations
The volume of observability data grows faster than the ability to store and query it. Alert fatigue from too many notifications degrades response times. Tracing across asynchronous and event-driven architectures is complex. The cost of commercial observability platforms scales linearly with data volume.
eBPF and Kernel Extensibility
Run sandboxed programs inside the Linux kernel for deep observability, networking, and security without kernel modules.
Growing
eBPF and Kernel Extensibility
Run sandboxed programs inside the Linux kernel for deep observability, networking, and security without kernel modules.
How It Works
eBPF (extended Berkeley Packet Filter) allows programs to run in a sandboxed virtual machine inside the Linux kernel. A verifier ensures eBPF programs cannot crash the kernel or access unauthorized memory. Programs attach to kernel hooks (system calls, network events, function entry/exit) and execute at kernel speed without context switches. This enables high-performance networking (bypassing the traditional network stack), deep observability (tracing every syscall), and security enforcement (blocking unauthorized operations), all without modifying kernel source code or loading traditional kernel modules.
Key Technologies
Real-World Examples
Meta uses eBPF for load balancing (Katran) across its entire infrastructure. Cloudflare uses eBPF for DDoS mitigation at the edge. Google uses Cilium for GKE Dataplane V2. Isovalent (Cilium creators) was acquired by Cisco in 2023. Netflix uses eBPF for performance analysis of its streaming infrastructure.
Challenges & Considerations
eBPF requires Linux 5.x+ kernels for full feature support. The eBPF verifier rejects programs it cannot prove safe, which limits what can be expressed. Debugging eBPF programs is harder than userspace code. The ecosystem is still maturing, with tooling gaps for complex use cases.
Storage Systems
Block storage, object storage, distributed file systems, and the I/O stack that determines data persistence and throughput.
Foundational
Storage Systems
Block storage, object storage, distributed file systems, and the I/O stack that determines data persistence and throughput.
How It Works
Storage systems sit between applications and physical media (SSDs, HDDs, NVMe drives). Block storage (EBS, Ceph RBD) provides raw disk volumes. Object storage (S3, MinIO) stores data as key-value blobs with metadata, optimized for scale and durability. File systems (ext4, XFS, ZFS, Btrfs) organize blocks into hierarchical directories. io_uring (Linux 5.1+) provides asynchronous I/O that dramatically reduces syscall overhead for high-IOPS workloads.
Key Technologies
Real-World Examples
Amazon S3 stores over 350 trillion objects (Amazon re:Invent 2024). Ceph provides the storage backend for many OpenStack deployments and is used by CERN for physics data. NVMe SSDs deliver 1M+ IOPS per drive, a 1000x improvement over spinning disks. io_uring enabled major performance improvements in databases like PostgreSQL and RocksDB.
Challenges & Considerations
Data durability requires replication, which multiplies storage costs. Storage tiering (hot, warm, cold) adds operational complexity. Distributed storage systems face the same CAP trade-offs as any distributed system. Encryption at rest and in transit adds computational overhead.