Apache Spark interview questions with answer outlines

Fifteen common Spark interview questions

1. Explain the Spark architecture: driver, executors, cluster manager.
2. What is lazy evaluation, and why does Spark use it?
3. Transformations versus actions?
4. Narrow versus wide transformations — what is a shuffle?
5. What are jobs, stages and tasks?
6. RDD versus DataFrame versus Dataset?
7. How does partitioning affect performance? repartition versus coalesce?
8. What join strategies does Spark use?
9. What is data skew, and how do you fix it?
10. cache versus persist — when should you cache?
11. What does the Catalyst optimiser do?
12. What is adaptive query execution?
13. How do you handle small files?
14. What is Structured Streaming?
15. A Spark job is slow. How do you investigate?

Execution model

• Architecture: the driver plans the job and coordinates; executors on worker nodes run tasks and hold cached data; a cluster manager (YARN, Kubernetes or a managed platform) allocates resources.
• Lazy evaluation: transformations build a plan; nothing runs until an action such as count, collect or write. This lets Spark optimise the whole plan.
• Narrow vs wide: narrow transformations (map, filter) work within a partition; wide ones (groupBy, join, distinct) need a shuffle that moves data across the network — the main cost in most jobs.
• Jobs, stages, tasks: each action creates a job; shuffles split jobs into stages; each stage runs one task per partition.

Joins, partitioning and skew

APIs and optimisation

DataFrames and Spark SQL are preferred over RDDs because the Catalyst optimiser can rearrange the logical plan — pushing filters down, pruning columns, choosing join strategies — and the Tungsten engine generates efficient code. Datasets add compile-time typing in Scala and Java. Adaptive query execution re-optimises at runtime using actual statistics: coalescing small shuffle partitions, switching join strategies and splitting skewed partitions. Cache only DataFrames reused several times, and unpersist when done.

Debugging a slow job

1. Open the Spark UI and find the slowest stage.
2. Check task duration distribution — a long tail means skew.
3. Look at shuffle read and write sizes and spill to disk.
4. Check the number of partitions: too few underuses the cluster; too many adds overhead.
5. Read the physical plan with explain() for unexpected shuffles or missing broadcast joins.
6. Check input: many small files or unpartitioned data scanned in full.

Small files and streaming

Thousands of tiny files slow both reading and metadata operations; compact them, write with fewer, larger files, and partition output by commonly filtered columns. Structured Streaming treats a stream as an unbounded table processed in micro-batches (or continuously), with checkpointing for fault tolerance and watermarks to bound state for late data. See data engineer interview questions and Kafka interview questions.

Practising

Reading answers is not the same as saying them. Pick ten of the questions above and answer them aloud against a timer; a free AI interview assistant like CoPilot Interview can ask Spark questions in a mock interview and flag answers that ran long or lacked an example. Prepare one story about a job you sped up, with the before-and-after time and the cause you found in the Spark UI.

Disclosure: CoPilot Interview is our product, so weigh what we say about it accordingly. Everything else on this page applies whichever tool you use, or none.

Frequently asked questions

What are the most common Spark interview questions?

Architecture, lazy evaluation, transformations and actions, shuffles, jobs and stages, RDDs versus DataFrames, partitioning, joins, skew, caching, Catalyst, adaptive query execution, small files, streaming and debugging slow jobs.

What is a shuffle in Spark?

Redistribution of data across partitions and executors, required by wide transformations such as joins and aggregations. It involves disk and network I/O and is usually the most expensive part of a job.

How do you fix data skew in Spark?

Salt hot keys to spread them across partitions, use adaptive query execution's skew-join handling, broadcast the smaller side where possible, or process hot keys separately.

repartition or coalesce?

Use repartition to increase partitions or rebalance data with a full shuffle; use coalesce to reduce partitions cheaply without a full shuffle, for example before writing output.