Home › Blog › PyTorch Interview Questions (2026): 30 Questions on Tensors, Autograd and Training Loops

PyTorch Interview Questions (2026): 30 Questions on Tensors, Autograd and Training Loops

PyTorch interview questions test whether you can write and debug a training loop: tensors and devices, autograd, nn.Module , train versus eval mode and data loading, then speed and memory with mixed precision, torch.compile and distributed training. PyPI listed torch 2.15.0 (October 28, 2026) as current on November 17, 2026.

PyTorch interview questions test whether you can write and debug a training loop: tensors and devices, autograd, nn.Module, train versus eval mode and data loading, then speed and memory with mixed precision, torch.compile and distributed training. PyPI listed torch 2.15.0 (October 28, 2026) as current on November 17, 2026. Two details from the 2.15 docs are worth knowing: torch.load uses weights_only=True by default (since 2.6), and torch.cuda.amp.autocast is deprecated in favour of torch.amp.autocast("cuda").

Below are 30 questions by level, five debugging scenarios and how ML interviews test this. Theory follow-ups are in deep learning interview questions; the role-level view is in machine learning engineer interview questions.

PyTorch interview questions: fundamentals (1–10)

  1. What is a tensor? An n-dimensional array, like a NumPy array, with a dtype, a device (CPU or GPU) and autograd support. Create one with torch.tensor, torch.zeros or torch.randn.
  2. How do tensors and NumPy arrays interact? torch.from_numpy(a) and t.numpy() share memory on the CPU. For GPU tensors or ones that require grad, use t.detach().cpu().numpy().
  3. How do you use a GPU? device = "cuda" if torch.cuda.is_available() else "cpu", then model.to(device) and move each batch with x.to(device). Every tensor in an operation must be on the same device.
  4. What is autograd? PyTorch records operations on tensors with requires_grad=True into a graph built as the code runs; loss.backward() computes gradients into the .grad of leaf tensors such as parameters.
  5. Why call optimizer.zero_grad()? Gradients accumulate across backward() calls by design. Without zeroing each step, every update uses the sum of all previous gradients.
  6. How do you define a model? Subclass nn.Module, create layers in __init__ and compute in forward. Call model(x), not model.forward(x), so hooks run. Submodules and nn.Parameter attributes register automatically.
  7. Write the core training loop. For each batch: optimizer.zero_grad(), out = model(x), loss = criterion(out, y), loss.backward(), optimizer.step(), plus a scheduler step where your schedule calls for it.
  8. model.train() versus model.eval()? They switch layers such as dropout and batch normalization between training and inference behaviour. eval() does not turn off gradient tracking.
  9. torch.no_grad() or torch.inference_mode()? Both stop graph recording during evaluation. inference_mode is stricter and slightly faster; tensors created inside it cannot be used in autograd later.
  10. What do Dataset and DataLoader do? A Dataset implements __len__ and __getitem__. A DataLoader batches, shuffles and loads in parallel with num_workers; pin_memory=True speeds host-to-GPU copies.

Intermediate PyTorch questions (11–20)

  1. What does CrossEntropyLoss expect? Raw logits and class indices (or class probabilities). It applies log-softmax internally, so adding a softmax first is a bug. For binary tasks, prefer BCEWithLogitsLoss on logits over BCELoss on probabilities.
  2. view, reshape or contiguous? view needs compatible memory and always shares data; reshape returns a view when it can and copies otherwise; after transpose or permute, call .contiguous() before view.
  3. What are detach() and item() for? detach() returns a tensor cut from the graph; item() returns a Python number. Log loss.item(), not loss, or you keep every step's graph alive.
  4. SGD, Adam or AdamW? SGD with momentum often generalizes well with tuning; Adam adapts per-parameter rates; AdamW decouples weight decay from the gradient update and is the usual choice for transformers.
  5. How do you initialize weights? Layers come with sensible defaults; override with torch.nn.init functions such as kaiming_normal_ or xavier_uniform_, applied through model.apply(fn).
  6. How should you save and load models? Save the state_dict, plus optimizer state and epoch for checkpoints: torch.save(model.state_dict(), path), then model.load_state_dict(torch.load(path)). Since 2.6, torch.load defaults to weights_only=True; pass False only for files you trust.
  7. When do you clip gradients? When gradients explode, common in RNNs and deep transformers: call torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) between backward() and step().
  8. How does gradient accumulation work? Divide the loss by N, call backward() on each micro-batch, and step and zero every N batches, which simulates an N-times-larger batch.
  9. Batch norm versus layer norm? Batch norm normalizes each channel across the batch and keeps running statistics, so it struggles with small batches. Layer norm normalizes each sample across its features, which is why transformers use it.
  10. What are buffers and hooks? register_buffer stores non-trainable state that moves with the model and is saved in the state_dict. Forward hooks capture activations; backward hooks inspect gradients.

Experienced and senior PyTorch questions (21–30)

  1. How do you use mixed precision? Wrap the forward pass in torch.autocast("cuda", dtype=torch.float16) and scale the loss with torch.amp.GradScaler("cuda") to avoid gradient underflow. With bfloat16 a scaler is usually unnecessary.
  2. What does torch.compile do? Introduced in PyTorch 2.0, it captures the model's Python code as graphs with TorchDynamo and compiles fused kernels, by default with the Inductor backend. Expect a slow first step, and watch for graph breaks and recompilations.
  3. DataParallel or DistributedDataParallel? DataParallel runs one process with threads and suffers from the GIL and an overloaded first GPU. DDP runs one process per GPU and all-reduces gradients; it is the recommended approach, launched with torchrun.
  4. What is FSDP? Fully Sharded Data Parallel shards parameters, gradients and optimizer state across GPUs, so you can train models that do not fit on one GPU, at the cost of more communication.
  5. How do you fix CUDA out-of-memory errors? Smaller batches with gradient accumulation, mixed precision, activation checkpointing (torch.utils.checkpoint), and freeing references to tensors that hold graphs. torch.cuda.memory_summary() shows usage; empty_cache() does not fix leaks.
  6. How do you make runs reproducible? Seed torch, NumPy and random, seed DataLoader workers, call torch.use_deterministic_algorithms(True) and disable cuDNN benchmarking. Results can still differ across hardware and versions.
  7. How do you write a custom autograd function? Subclass torch.autograd.Function with static forward(ctx, ...) and backward(ctx, grad_output), save tensors with ctx.save_for_backward, and verify with torch.autograd.gradcheck.
  8. Why can in-place operations break backward? Ops such as x.relu_() or += can overwrite values autograd saved for the backward pass, raising an error that a variable needed for gradient computation was modified in place. Use the out-of-place version.
  9. The GPU sits at 30 percent utilization. Why? Usually the input pipeline. Raise num_workers, enable pin_memory and persistent_workers, speed up decoding or move augmentation to the GPU, and confirm with torch.profiler.
  10. How do you export a model for serving? torch.export captures a whole-program graph ahead of time, torch.onnx.export produces ONNX for other runtimes, and a model server handles batching and versioning.

PyTorch scenario and troubleshooting questions

The loss turns NaN after a few hundred steps. Check the learning rate, exploding gradients (clip), log of zero or division by zero, and float16 overflow (use a GradScaler or bfloat16). torch.autograd.set_detect_anomaly(True) points to the operation.

Validation accuracy is lower than training suggests. Often model.eval() was not called, so dropout and batch norm stayed in training mode, or preprocessing differs between training and inference.

Memory grows every epoch. Losses are summed as tensors (total += loss), which keeps every graph alive. Use loss.item() and detach anything stored for later.

Training is slower than expected. Profile first. Typical wins: fix data loading, use mixed precision, raise the batch size, try torch.compile, and avoid frequent .item() or .cpu() calls that force synchronization.

A loaded model gives random predictions. Check what load_state_dict returns: strict=False can hide missing keys, and DDP checkpoints often carry a module. prefix. Also confirm the model is in eval mode.

How PyTorch interviews test this

FormatWhat it usually looks likeHow to prepare
Concept Q&AAutograd, zero_grad, train versus eval, losses on logitsExplain the training loop line by line
Live codingWrite a Dataset, a model and a training loop; fix a buggy loopPractice the five scenarios above as bug hunts
ML system designTrain and serve a model at scale: data, distributed training, servingDiscuss DDP versus FSDP, mixed precision and checkpoints
Research or applied roundImplement a layer or loss from a paperGet shapes right first and test with gradcheck

Array and shape fluency carries over from NumPy interview questions; for the other major framework, see TensorFlow interview questions.

Where CoPilot Interview fits

CoPilot Interview is an AI interview assistant. Its free AI mock interview builds PyTorch and machine learning questions from your résumé and the job description, reads them aloud and scores each spoken answer out of 10, which is a fast way to find the answers above that you know but cannot yet explain cleanly under pressure. The browser demo has 145 practice questions in 10 tracks, scores typed answers out of 10 and needs no signup. The free plan gives 15 minutes a day of live help in sessions of up to 15 minutes, with AI mock interviews included. Paid plans are Standard at $24.99 a month and Pro at $39.99 a month, in the desktop app for Windows 10/11 and macOS 12 or later.

Use it to practice. Online assessments, proctored tests, take-home tasks with rules and AI-led rounds forbid outside help, so prepare for those instead of bringing a tool. For any live interview, follow your employer's rules.

Disclosure: CoPilot Interview is our product, so weigh what we say about it accordingly.

Frequently asked questions

What are the most common PyTorch interview questions?

Tensors and devices, autograd and backward, why zero_grad is needed, nn.Module, the training loop, train versus eval mode, no_grad, DataLoader, CrossEntropyLoss on logits, saving state_dict, and DDP.

What is the latest PyTorch version?

PyPI listed torch 2.15.0, released October 28, 2026, when we checked on November 17, 2026.

What is the difference between model.eval() and torch.no_grad()?

model.eval() changes layer behaviour (dropout off, batch norm uses running statistics); torch.no_grad() stops gradient tracking. Use both for evaluation.

Is PyTorch or TensorFlow better for interviews?

Use whichever the job lists. Both test the same ideas: tensors, automatic differentiation, training loops, data pipelines and deployment.

How do I prepare for a PyTorch coding interview?

Write a full training loop from memory for a small dataset, then introduce and fix common bugs: missing zero_grad, softmax before CrossEntropyLoss, forgotten eval mode and summing loss tensors.

Practice this interview before the real one

The AI mock interview builds questions from your resume and the job description, reads them aloud and scores each spoken answer out of 10. Included on the free plan.

Start a mock interview