Deep learning interview questions with answer outlines

Fifteen common deep learning interview questions

1. Explain backpropagation.
2. What causes vanishing and exploding gradients, and how do you fix them?
3. Why is ReLU popular, and what are its problems?
4. What do batch normalisation and layer normalisation do?
5. Compare SGD with momentum and Adam.
6. How do you choose a learning rate? What are learning rate schedules?
7. What is dropout, and why does it reduce overfitting?
8. Why do residual connections help very deep networks?
9. How does a convolutional layer work, and why is it suited to images?
10. What limits RNNs, and how did LSTMs and transformers address it?
11. Why does weight initialisation matter?
12. What is transfer learning?
13. How would you debug a network whose loss isn't decreasing?
14. What is mixed-precision training?
15. How do you choose batch size?

Training fundamentals

• Backpropagation: applies the chain rule from the loss backwards through the network to compute each weight's gradient, which the optimiser uses to update the weights.
• Vanishing and exploding gradients: repeated multiplication through many layers shrinks or grows gradients exponentially. Fixes: ReLU-family activations, careful initialisation, normalisation layers, residual connections and gradient clipping.
• ReLU: cheap and doesn't saturate for positive inputs, but units can “die” if they always output zero; Leaky ReLU and GELU address that.
• Normalisation: batch norm normalises across the batch per feature and stabilises training of CNNs; layer norm normalises across features per example and suits transformers and variable-length sequences.

Optimisers and regularisation

Architectures

Convolutional layers slide small learned filters across an input, sharing weights so the same feature is detected anywhere — efficient and well suited to images. Recurrent networks process sequences step by step, which makes them slow to train and prone to forgetting long-range context; LSTMs and GRUs add gates to retain information longer. Transformers replace recurrence with self-attention, letting every position attend to every other in parallel, which scales far better and now dominates language and increasingly vision. Residual connections add a layer's input to its output so gradients flow through very deep networks.

Debugging a network that won't train

1. Overfit a tiny batch first; if the model can't, there's a bug.
2. Check data and labels are aligned and preprocessing matches between training and evaluation.
3. Lower the learning rate, or run a learning-rate range test.
4. Inspect gradient norms per layer for vanishing or exploding values.
5. Verify the loss function matches the output layer (for example, logits with cross-entropy).

Transfer learning and efficiency

Transfer learning starts from a model pre-trained on a large dataset and fine-tunes it on your task, needing far less data and compute. Freeze early layers when data is scarce; fine-tune more layers as data grows. Mixed-precision training uses 16-bit floating point for most operations with loss scaling to avoid underflow, roughly halving memory and speeding up training on modern GPUs. See computer vision interview questions and NLP interview questions.

Practising

The fastest way to find gaps is to be asked. Run a spoken mock round on deep learning topics — a free AI interview assistant such as CoPilot Interview builds one from your résumé and the job description and scores every answer — then study whatever you stumbled on. Be ready to sketch backpropagation for a two-layer network on a whiteboard — it's a common warm-up.

Disclosure: CoPilot Interview is our product, so weigh what we say about it accordingly. Everything else on this page applies whichever tool you use, or none.

Frequently asked questions

What are the most common deep learning interview questions?

Backpropagation, vanishing and exploding gradients, activation functions, normalisation, optimisers and learning rates, dropout and regularisation, residual connections, CNNs, RNNs and transformers, transfer learning and debugging training.

How do you fix vanishing gradients?

Use ReLU-family activations, good weight initialisation, normalisation layers and residual connections; use gated architectures such as LSTMs or transformers for long sequences.

What is the difference between batch norm and layer norm?

Batch normalisation normalises each feature across the batch; layer normalisation normalises across features within each example, which works better for transformers and small or variable batches.

Adam or SGD?

Adam converges quickly with little tuning and is the common default, especially for transformers. SGD with momentum and a good schedule can generalise better on some vision tasks.