Apache Spark

Apache Spark is a distributed compute engine: it takes a query expressed over data that does not fit in one process, compiles it into a graph of stages, and executes it across a cluster of executors coordinated by a driver. The driver plans; executors hold data partitions in memory and on disk and run the tasks. That split - one planner, many workers - is the whole architecture.

Lazy evaluation and the DAG

DataFrame and SQL operations are lazy: they build a plan, and nothing executes until an action (count, write, collect, show) forces it. The plan compiles into a DAG of stages separated by shuffles - the points where data must move between partitions.

The rule that decides performance: narrow dependencies (map, filter) pipeline within a partition and cost nothing; wide dependencies (join, groupBy, repartition) shuffle data across the network, and the shuffle is where jobs spend their time and memory. A job with one giant shuffle will be slower than a job doing ten times the computation with none. explain() shows you the physical plan - the stages and the exchanges between them - and it is the first place to look when a job is slow.

Partitions and parallelism

Spark reads data into partitions - the unit of work a single task processes. Too few partitions leave executors idle; too many drown the scheduler in task overhead. The practical symptom map:

• Skew - one partition holds most of the data (one hot join key, one hot date). One task runs for hours while the rest finish in seconds. Salt the key, broadcast the small side, or split the skewed rows out.
• Small-file amplification. Thousands of tiny input files produce thousands of tiny partitions and a schedule dominated by task overhead. Compact upstream, or coalesce on read.
• Shuffle spill. A shuffle that does not fit in executor memory spills to disk, and a job that spills runs an order of magnitude slower. The fix is more partitions, more memory, or less data per key - not a bigger driver.

Structured Streaming

Structured Streaming runs the same DataFrame API over an unbounded source - Kafka, files, a socket - as a sequence of micro-batches (or continuously in the still-experimental continuous mode). Each micro-batch is a normal Spark job over new records; checkpointing tracks progress so a restarted job resumes where it stopped. State stores hold aggregates and joins across batches, and watermarking bounds how long late data is waited on - the two mechanisms that decide whether a streaming job keeps up or falls behind permanently.

Deployment and the ecosystem

Spark runs on its own standalone scheduler, YARN, Kubernetes, or as a managed service (Databricks, EMR, Dataproc, Synapse). Spark 4.0 pushed Spark Connect - a thin gRPC client that submits plans to a server rather than embedding the driver in your application - much further toward being the normal client: there is a Connect-enabled-by-default release tarball and a spark.api.mode switch that lets an existing application run over Connect, though the standard distribution still defaults to Classic. The practical consequence holds either way: the client-server split is what makes notebooks and thin clients practical against a shared cluster. The DataFrame API is the surface almost everyone should write against; RDDs remain for cases the optimiser cannot express, and pay for it by losing the optimiser entirely - and they are not available over Connect at all.

Spark versus the alternatives

• Spark is the generalist: batch ETL, SQL, ML, and micro-batch streaming in one engine, the broadest ecosystem, and the most operational surface to own.
• Apache Flink is the specialist for true event-at-a-time streaming: lower latency, richer event-time semantics, and a smaller operational community. If the workload is genuinely continuous, Flink is the more honest fit; if it is mostly batch with some streaming, Spark’s micro-batch model is usually enough.
• Trino / Presto is a query engine, not an ETL engine: fast interactive SQL over whatever it reads, but no durable execution state for long transformations - a query that dies restarts from zero. Use it to query, Spark to build.

Failure modes and diagnostics

• OOM on the driver from collect() or toPandas() on a large result - the driver is a single process and the classic way to kill it. Write or sample instead of collecting.
• Shuffle fetch failures and executor loss under memory pressure - the stage retries, the job gets slower each retry, and eventually fails. Look at spill metrics in the UI before adding hardware.
• Skew that looks like a hang. The UI shows one task at 99% for hours while the rest finished - that is skew, not a stuck cluster.
• Streaming jobs that fall behind. Input rate exceeds processing rate per micro-batch and the lag compounds - the trigger interval and the state store’s size are the levers, and a checkpointed job that cannot keep up needs repartitioning, not restarting.

What this is not for

Spark is overkill for data that fits in one process - a pandas or Polars job on one machine is simpler, faster to write, and cheaper to run up to a genuinely large threshold. It is not a database: it reads and writes storage, it does not own a table format or enforce consistency for you - that is what Iceberg and its peers are for. And it is not an orchestrator: scheduling, retries across pipelines, and dependencies between jobs belong to Airflow and its peers, not to Spark itself.