How to calibrate interviewers

The four mechanisms

1. Write what each rating level looks like, in observable terms.
2. Shadow twice and be shadowed twice before interviewing alone.
3. Run periodic sessions scoring the same answer independently, then compare.
4. Review score distributions per interviewer and talk about the outliers.

Start with definitions, not training

Most calibration problems are definition problems. If the scorecard says “3 = meets expectations”, two reasonable people will disagree permanently, because the phrase contains no information. If it says “3 = broke the problem down unprompted, handled the main cases, needed a nudge on edge cases”, they will mostly agree, because the disagreement is now about what happened rather than about what the word means. Fix the rubric before running sessions.

Shadowing in both directions

• Shadow twice. The new interviewer observes, scores independently, and compares afterwards with the lead. The comparison conversation is the training.
• Be shadowed twice. They run the round while an experienced interviewer observes and scores. Debrief on the questions asked as much as the score given.
• Do not skip the comparison. Shadowing without an afterwards is just attendance.
• Keep it to four rounds total. Longer apprenticeships are rarely completed, and the marginal value after the fourth is small.

Calibration sessions that are worth the hour

Take one recorded answer or a role-played response, have everyone score it independently and silently, then reveal all scores at once. The useful part is not the average; it is the spread. Ask the highest and lowest scorer to explain, and you will usually find they weighted different parts of the answer or applied different unstated standards. Write down whatever you agree and add it to the rubric. Done twice a year with the same role-played answer, this also shows you whether the bar is drifting.

Watch the distributions

• The consistently generous interviewer passes almost everyone. Their scores carry no information, and the loop loses a stage.
• The consistently harsh interviewer is often respected for having a high bar, but may simply be assessing something the rubric does not contain.
• The all-3s interviewer is avoiding a decision. Usually a rubric problem — see whether their scale has a midpoint it should not.
• Compare on the same roles. Someone who only interviews seniors will look harsher than someone who only interviews juniors.

Calibrate the bar, not just the scale

Two interviewers can agree perfectly on what a 3 looks like and still disagree on whether a 3 is a hire. The scale is about observation; the bar is a business decision and belongs to the hiring manager. State it explicitly before a search begins — which attributes must be at least a 3, which may be a 2 if something else compensates. Teams that never do this re-litigate the bar in every debrief.

Drift is normal and needs a schedule

Bars drift with the market, with team pressure and with the last few candidates seen. A team that has interviewed six weak candidates will rate the seventh generously. The defence is periodic recalibration on a fixed answer rather than vigilance. Put it in the calendar twice a year; it will not happen otherwise.

Calibrating across levels

The hardest calibration problem is not between interviewers but between levels. A 3 for a graduate and a 3 for a staff engineer describe very different performances, and a rubric that does not say so will quietly hold juniors to a senior bar or the reverse. Write level expectations alongside the rating definitions: at junior, unprompted decomposition is a 4; at senior, it is the floor. Interviewers who move between levels in the same week need this written down, because recency makes the previous candidate the reference point.

What calibration cannot fix

• A rubric measuring the wrong thing. Perfect agreement on an irrelevant attribute is still irrelevant.
• Interviewers who do not ask follow-ups. Two people cannot agree on evidence that was never collected.
• A bar nobody has stated. Agreement on observation does not produce agreement on the hire/no-hire line.
• Shared-screen scoring. If interviewers can see each other's scores before submitting, you have consensus rather than calibration, and the two look identical in the data.

New interviewers need the brief, not just the rubric

Before each loop, tell every interviewer which attributes they own, which they do not, and what questions to ask. A calibrated interviewer who has not been briefed will still assess the wrong things. See how to design an interview loop and how to write an interview scorecard.

Where Interviewer Mode fits

Disclosure: we make CoPilot Interview, which includes Interviewer Mode for the hiring side and a real-time assistant used by candidates. We have an interest in you using the first, and our view of the second is that interviews should be designed to measure something an assistant cannot fake. Weigh the advice below accordingly.

Frequently asked questions

How do you calibrate interviewers?

Write observable definitions for each rating level, have new interviewers shadow twice and be shadowed twice with a comparison conversation each time, run periodic sessions where everyone scores the same answer independently and compares, and review each interviewer's score distribution against the group.

What is a calibration session?

A meeting where several interviewers independently score the same recorded or role-played answer, then reveal all scores at once. The value is in the spread rather than the average — asking the highest and lowest scorer to explain usually surfaces an unstated standard worth adding to the rubric.

Why do interviewers score the same candidate differently?

Usually because the rating definitions are vague, so “meets expectations” means different things to different people, or because they asked different follow-up questions and genuinely saw different evidence. The first is fixed by the rubric, the second by briefing and planned questions.

How often should we recalibrate?

Twice a year, scheduled, plus whenever the rubric changes or several new interviewers join. Bars drift with the market and with the last few candidates seen, and drift is not something vigilance catches — it needs a fixed comparison point.