Screening for Aptitude over Completeness

Over the many years that we've worked with each of our clients, we've learned a few things about growing teams. One of the biggest fallacies we've found in technical hiring is the idea that completeness is a meaningful way to assess ability in technical screenings.

Imagine the prompt of a technical screening is to write a program that tells you the fewest number of coins needed to make a certain amount of change (e.g. 2 quarters and 3 pennies is the fewest number of coins to make 53 cents).

The fallacy we've found in a lot of employers' screenings is that they think a developer who can complete this program in 45 minutes is fundamentally more hireable than a developer who cannot.

Completeness doesn't matter.

We strongly disagree with completeness-oriented approaches to technical screening. Here are a few reasons why:

• It's severely biased towards developers who have solved that exact problem before (or something very similar to it).
• Completing this challenge is, for many programmers, an achievement in overcoming anxiety rather than an achievement in demonstrating competence.
• A candidate who would complete the exercise if they were given twice the amount of time is likely no less competent than someone who only takes the allotted time.
• A candidate who would complete the exercise if they were not being watched is likely no less competent than someone who completes the exercise while being supervised.

The most common argument against these reasons is that the candidate will be talking through their solution, demonstrating aptitude as they solve the problem.

That's a fair response! We also like talking through the problems with candidates. Where we differ in philosophy is the idea that completeness matters at all.

Valuing completeness is inherently damaging to hiring culture.

When we suggest that "aptitude-oriented" is at odds with "completeness-oriented", our point is that aptitude-orientation aims to remove as many biases as possible from the screening process. "Completeness" is, in itself, a bias.

"Completeness" is also a lazy way to assess people. It's such a clear brightline that it removes the perception that a rubric is needed. So most companies will simply compress their feedback of a candidate to, "The candidate did/didn't complete the problem in the given time [+/- 30 minutes]." As much as the interviewer might want to draw more insights from their time with the candidate, the temptation to reduce the meaningful feedback to "complete/incomplete" is very strong.

The binary nature of "completeness" means that the interviewer doesn't actually have to pay much attention or think critically about the different ways that aptitude is demonstrated. Valuing completeness at all will pervade your team's hiring culture and actively fight the healthier goal of defining a specific rubric for qualities that you seek in your developers.

Is your hiring process aptitude-oriented?

It can be hard to know if your process is aptitude-oriented. Some questions that might help:

• Do you have a defined set of observable technical qualities which you train your interviewers to assess?
• What percentage of candidates who completed your technical screening challenge did you decide not to move further in the hiring process?
• (And the reverse question: How many candidates who don't complete the challenge do you move further?)
• Would someone who has solved your example technical challenge before have a higher chance to moving forward in the hiring process?

Moving forward

If you're looking to improve your technical screening process, we suggest two actions to start:

1. Define a set of observable technical qualities that you want in your developers.

Here are some examples:

• Creates meaningful abstractions to hide unnecessary business logic details.
• Writes meaningful unit tests.
• Senses and articulates code smells.

Note that none of these are specific to a particular challenge. These are simply qualities you might like to see in your developers.

2. Create a technical challenge that creates opportunities to demonstrate those qualities.

Once you have a challenge and the qualities you seek, the hard part can be mapping possible questions/actions/behaviors during the challenge to those qualities.

There's no way to do this comprehensively, but that doesn't mean you should forego that preparation entirely. Consider administering the assessment to devs that you have already hired: See how they approach the problem. Record the positive and negative patterns you see in their work to build a hypothetical profile of your hireable developer.

With a defined set of qualities, a meaningful technical challenge, and some documentation about how those two are connected, you should have enough to begin training your interviewers on how to assess technical aptitude.