Some thoughts on better interviewing
How would you design a question that selects the best match for the job?
Once upon a time, I had five candidates to interview. During the first interview, after a few boring competency questions and a few more bland technical questions — neither of which had given me any real insight about the candidate — I had an epiphany and suddenly asked this follow-up question:
“What is the most impactful piece of work that you’ve ever delivered, and why was it impactful?”
Listening to the candidate work through an answer was enlightening: they provided a walk-through of an interesting piece of work, displayed a huge sense of pride in it, and explained why they thought that piece of work had a lasting impact on their organisation.
After almost a hour of aimless questions, I realised that the key question to ask was the one I had just asked. We then asked the same question to each and every candidate in the process, and each and every time the insight given by their answer was great, and I believe it contributed to selecting the right candidate. I’m starting to ask myself if we could build the recruitment process around a question like that.
In a brilliant talk in which he tells the story of his 250 interviews (as an interviewer) for Google, Moishe Lettvin describes job interviews as
Noisy, Inaccurate, and Arbitrary
Watch the whole talk if you can as it’s very helpful to focus on what good and bad interviews are like. That is, assuming good interviews can exist.
Of course, the ideal way to recruit anyone for any role would be to getting that person to work in that role for a little while and see how they perform. Fundamentally, this is what’s behind the concept of probation and, of course, it doesn’t work as a way to select the right candidate because it doesn’t scale to more than one person.
There are different drives while recruiting. Getting the best candidate means checking that the skills required for the role are tested, and doing so fairly. Different organisations test candidates in different ways, but all seem to do it by creating structure around the interview process. That’s how we have the notoriously fuzzy competency-based interviews, darling of the Civil Service, and the 7-interviews-over-several-months for Silicon Valley tech roles, in which candidates are asked to solve problems requiring algorithms they’ll never see again in their professional life.
The issue I have with the idea of big structures in interviews is the same: they select candidates who are extremely good at passing the interview. Which is not the same thing as being the best match for the role they’re interviewing for. Having worked in both academia and the Civil Service, places who both have such a love of the competency system that they even build their recruitment portals around it, it seems evident to me that candidates who generally perform well are those who have, so to say, “trained” in answering competency questions.
There is a misconception that competency-based questions help the interview being fairer, as they force all candidates to build their answers in the same standardised way; but I would counter-argue that they advantage candidates who know how to bluff (yes, I can still probe them with follow-up questions, but it‘s inefficient and, in many cases, discouraged by the HR departments). There are much better solutions. For example, fairness can be much better addressed by having a diverse panel (this requires a blog post in itself: personal experience suggests me that candidates feel to be more fairly treated if they have a diverse panel, and they’re probably being effectively treated more fairly; as would the obsession over anonymous application for senior roles — where the candidate pool is much reduced, I have a strong suspicion that an anonymous application process won’t increase fairness and will end up privileging the already privileged).
Whereas for Google’s interview method (it’s not just Google, by the way), I’ve come to believe that all they select for is the… commitment to stick with Google’s interview method. Of course there would be nothing wrong with that, if the declared objective of the interview process was to hire employees so committed to learn all about algorithm they’ll never use and stick to exercising their knowledge for months, but that’s not the jobs they’re applying for nor the key skillset required to be successful in it. My critique is surely not an isolated one — and it doesn’t apply to tech roles alone, although it’s highlighted really well in a tech context — as there have been some famous, hilarious cases around this interview model, such as when Google rejected Homebrew’s creator for a software developer position:
Of course, a good objection is that technical interviews should capture technical skills while making sure these are genuinely and legitimately the candidate’s skills (i.e. they have no external ‘help’). But I believe there are many, many ways to do this that do not involve inverting a binary tree on a whiteboard: through a quasi real-life exercise, an assignment, a presentation. There might be no way to fully replicate a “work situation”, but few work situations will need the candidate to resolve such technical asks on the spot without researching, consulting with others, and — pardon me — Googling.
Going back to my initial observation, the question I asked was “tell me something you’re proud of”. Expanding on this concept, I’d love to be able to interview candidates with a single question sent to them before hand:
You have 30 minutes to prove to me that you’re the right person for this role. Use any method and content you feel like using in order to convince me.
Let me explain why I believe this is a good question. First of all, the candidate would come prepared to the interview with a clear view of themselves and their career, skills, expertise. There seems to be an unspoken drive to not revealing much to the candidate before they interview, lest they prepare too well. Can you imagine any other situation in which you’re trying to reduce the likelihood that someone will be rightly prepared for their task? Improvisation is not good, unless the job is that of the jazz pianist at Brasserie Zedel (and even then…)
A question like this would encourage the candidate to structure their discourse around how the requirement of the jobs align with their own. It would even give them a chance to address whenever their skills do not align perfectly to the requirement, and explain how they would plan to work towards a better alignment or how and why they believe that those requirements are less important looking at the big picture. If they fail to give a strong enough explanation, there would be a two-way understanding that they’re not the right candidate, or that’s not the right job (oh, I can see why recruiters will hate this).
Most importantly, the interview would not consist of questions that sound like “take the best answer from the hat”. I’ll repeat myself. Interviews always suffer from this problem: you ask a candidate something that, to some extent, will surprise them. I don’t see any reason why that should be the case. I want a candidate to be prepared to answer questions. And if they can do so by linking those questions to a more holistic presentation of why they are the best candidate, then I believe we’re in a win-win situation because my questions will be able to probe and dig much deeper into their knowledge and their story.
Of course, I can hear the criticism to this. This method advantages bullshitters, it’s not mindful of diversity, and is not measurable. I don’t entirely dismiss these critique, and I’m writing this blog post also because I want to hear people’s view on how to address them.
The first two issues are fundamentally centred around fairness. I don’t want to dismiss the issue of fairness, but as I’ve written above I’m not entirely sure that the current approaches do much in terms of fairness. A few conversations I’ve had at work with several staff members as we discussed the Black Lives Matter movement highlighted, for example, that one of the easiest, practical things we could do to make different groups feel welcome is to increase the diversity of interview panels. This single action, done visibly, would improve fairness and its perception because candidates would see someone like them on the panel — whether they’re black, white, gay, straight, neurodiverse.
Having a diverse panel will ensure different candidates will always be assessed by a variety of people, some of them similar, some of them different, which is another way of calling fairness. And this is just one simple action that can be taken to address fairness.
Measurability is more difficult to address, primarily because I’m not convinced that the current approaches do that right. There’s a lot of noise and useless metricism. “Educated to degree level or equivalent experience” — what does that even mean? And why is this important for the role? Why is this a key metric of success for the role? Is it a proxy for the right thing? I think there is a disconcerting level of arbitrariness in the current methods.
Of course, there are skills that can and should be measured, but also aspects that are quite difficult to capture and that should be down to the interview panel to assess — taking the plunge in trusting that the different angles covered by a diverse panel will be able to pick the best match for the role (yes, it’s about diversity once again). Ultimately, it can happen that multiple candidates will be equal matches for the role, and it’s down to the panel to decide which candidate to select. We accept that the panel can take that decision then, and that they have measured everything they could in the most effective and honest way. Sometimes obsessing over measurability won’t give any better result.
This is a half-formed idea, of course. I would love to hear what you think of it and how to address its flaws.
I send a weekly newsletter about all things data. You can subscribe here.
