Much buzz today on HN about developer hiring with Evan Carmi sharing his Google interview experience, which mostly consisted of language trivia and algorithmic tasks, and a devininterviews follow up arguing that this interviewing style leads to bad hires.
I don’t think there is one right developer interviewing technique. It is worth mastering a variety of interviewing techniques and then adjusting based on the job requirements, team composition, manager style and company culture.
Here is an initial (incomplete) list. I’ll update with new content from comments & conversations.
Programming language questions
Examples: what’s the order of C++ class data member initialization, how is the value of self determined in a Ruby block and what’s the difference between class and function decorators in Python.
Why: I want to know whether you know the tools of your trade.
Good for: quickly checking the box on language proficiency.
Useless for: determining whether you can code well in the language. I know developers who write kick-ass, elegant code using 50% of a modern language’s features. That’s awesome.
On-the-spot coding of algorithmic problems
Examples: write a binary search, write a Sudoku solution checker and write a substring search.
Why: I want to know which developer group you belong to: the folks who can solve algorithmic problems or the ones who could churn dozens of MVC screens but couldn’t build a fuzzy string matcher to save their lives, even with resources online.
Good for: vetting analytic thinking and coding precision under pressure as well as basic verbal communication & organization of thought. Bonus: easy way to see how you handle criticism.
Useless for: understanding how you’ll write practical code and handle criticism on a team. In the real world engineers solve problems not in front of a white board but at their desk, with Google, GitHub, their co-workers, hacker buddies, Twitter friends and the Stack Overflow community. If your development velocity is great, I most likely won’t care whether you do it by memorizing the algorithms textbook or submiting a dozen questions per week on Stack Overflow.
On-the-spot debugging
Examples: find the bugs in this listing or I have this project loaded in Eclipse and this bug ticket here — go at it.
Why: just as I want to understand your development workflow, I want to understand your debugging workflow.
Good for: getting a sense of how you wrap your head around other people’s code.
Useless for: being sure that’s how you’ll debug normally.
Take home programming tasks
Example: Here are two empty git repos. By 10am PT tomorrow build and deploy two applications on Heroku. The first exposes a REST API for logging session events with arbitrary JSON-based parameters and saves the events in an SQS queue. The second pulls events from SQS and saves them in a MongoDB database hosted on MongoHQ. You can use whatever gems you want.
Why: I want to get a sense of how you work in the real world. I want to look at the code you write but I care even more about what you consider “shippable”. Here are a few things I’d pay close attention to beyond the obvious code style/quality that you may not even know you’ll be evaluated on:
- How did you deal with fuzzy requirements?
- What was your git workflow: commit frequency/atomicity, batch size, branching strategies. How you use source control alone is a great indicator of how you’ll use source control on a team.
- Did you choose Rails or Sinatra? Why?
- Which AWS gems did you pick? Why? How did you evaluate the quality & fitness of a gem?
- Did you use TDD? I don’t mean did you deliver tests in the end. I’m interested in whether the tests hit the repository no later than the features.
- How extensive is your testing? Do you have integration tests? Acceptance tests? Did you set up dev & staging SQS queues?
- Did you think about securing the REST API? How?
- How do you manage sensitive info? Is your Amazon secret in the git repo or does it get set as an environment variable? What about the MongoHQ credentials? What about sanitizing the logs?
- How paranoid are you about failure? Did you pop() from SQS assuming a MongoDB write will succeed or did you peek() and only delete from SQS when you know the message contents made it into Mongo?
- Are you checking for duplicates in Mongo?
- How did you set up Heroku? Do you have staging apps as well as production apps? Did you set up deploy notification hooks? How about monitoring for the apps?
- Did you think about scaling? Did you write any code for it? Did you do any load testing? When does your current deployment break?
- Did you write any docs? Do you know your audience?
Good for: a litmus test of your ability to build, ship, operate and communicate. Getting a “fingerprint” for how you work.
Useless for: deeply understanding how productive you can be on a team. There is no teamwork component here and it’s hard to separate the various elements that contribute to the final product: your prior knowledge and experience, help you get from a buddy or online resources, luck. Plus, to really evaluate your performance will take a lot of work.
Take-home debugging
Examples: (actual problem I used to give) here is a simple linked list implementation in C++. Find the bugs.
Why: just as I want to understand your development workflow, I want to understand your debugging workflow. Some of the bugs are subtle. I hope you talk to friends about the problem.
Good for: getting a sense of how you wrap your head around other people’s code and how you work with others to solve problems.
Useless for: determining how much of the outcome was “pure you” vs. “you + others”
High-level questioning
Examples: from devin interviews:
- What’s the last project you worked on at your former employer?
- Tell me about some of your favorite projects.
- What projects are you working on in your spare time?
- What online hacker communities do you participate in?
- Tell me about some (programming/technical) issues that you feel passionately about.
Why: I want to get a sense of who you are as a developer and how you might work on a team.
Good for: knowing who to reject quickly and who might be a good cultural fit.
Useless for: deeply understanding who to say yes to. Bottom line: great interviewees can prep for and ace this because it’s too general.
Behavioral interviewing
Examples: tell me about a situation where you had to deal with a bug that only appeared under load in production + lots of specific follow-on questions
Why: two reasons: (1) other things being equal, past performance in similar situations is the best predictor of future success and (2) behavioral interviewing gets down into the nitty-gritty and breaks through BS effectively.
Good for: drilling into specific areas of interest
Useless for: covering a lot of ground quickly and useless if the interviewer is not experienced in the technique and does not apply it in a disciplined manner.
I know I’ve missed many techniques and the “high-level questions” group should be broken into several. I’d welcome ideas on how to do it.
Very insightful, Sim!
Impressed! Add this to your previous post where you formulated your requirements – and you’ve raised the bar too high 🙂 Perhaps beyond the addressable pool of employable talent and into the micro-clusters of independent mercenaries or startup founders…
Any luck meeting at least one candidate to tick all your boxes?
Tip for alternative approach: devote time to inhabit the hacker communities (the ‘darker’ ones, the better) – you’ll soon be able to identify valuable members there (they just stand out, even to the naked eye – not just with skill but lots of evidence of character). Show them (selectively) that you’re one of them (and perhaps a darker shade of black belt) and earn their respect. Then motivate them to join a conspiracy among black belts (not to ‘apply for a job’)…
I actually like a variation of the High Level Questioning where I ask the interviewee to describe a project from their resume and their role on it. Then I drill down and ask them to explain what exactly they did, how they did it, and to defend or critique their design choices. Since this is something they did directly, they should be knowledgeable about it and be able to discuss intelligently. It gives the candidate a chance to sell me on how he works and thinks about real world problems.
For programming questions, I always try to use real problems that we encountered as a test. This wouldn’t be something they could memorize from an online website, and gives insight into how they would consider a real world problem.
Aaron, this is a great strategy for getting a sense of a developer especially if you ask them to talk about why did they make the trade-offs they did and what they would do differently with the benefit of hindsight.
I’m curious: how many people have you hired in the 17 years since we last wrote code together? Can you think of any anti-patterns of interviewing: interviewer behaviors that you’ve learned more often than not lead to poor outcomes?
Most of my interviews have been done over the last 8 years or so at Microsoft. I’ve done about 30-40 of them over that period, and given “Hire” decisions about about 10-15.
I’ve never identified any real anti-patterns except impatience. There’s a temptation to hire “acceptable” rather than “great” just to fill a seat, especially after a long drawn-out hiring process. I’ve never seen settling like this end well. In almost every case, the hiree was gone within a year.
The other lesson I’ve learned is to trust my instincts when it comes to “group fit”. A candidate can seem like a great technical hire, but somehow not fit into the culture of the team. There were a couple people who nailed the technical aspects exceeding well, and we hired them, even though we felt maybe there was a bit of “off-ness” in the conversation. In the end, we regretted both decisions, because the hire never fit in, and could never connect with the rest of the team, hence diminishing their productivity.
True, the rush to make a hire because work is piling up is never a good thing. Rather than hiring a sub-optimal candidate, it’s better to intensify recruiting efforts.
Hi Sim,
I think these interviewing techniques might work for hiring college kids for summer internships but would be deeply insulting for senior devs. When I interview, the main skill I look for is the ability to use modularity to control complexity. Everything else you can look up in a book or on the web.
-Chris
Hey, Chris!
I don’t agree with you. “Using modularity to control complexity” is super-important but it’s just one aspect of being a great developer in an early stage starting. What does it say about how someone approaches customer development or dealing with uncertainty or knowing how to be a solid participant in an agile development process or being a good communicator or knowing how to test?
Also, if anyone whose track record doesn’t clearly speak to most of these feels offended by being asked to go into the weeds and engage around problem solving in some specific way, then I most definitely don’t want them on my team. Someone who is confident would take this as an opportunity to learn about me and my style as well as demonstrate how good they are. The ones who’d be offended either have a stellar track record or, more likely, are arrogant or lack confidence in their own abilities.
“..if anyone whose track record doesn’t clearly speak to most of these feels offended by being asked to go into the weeds and engage around problem solving in some specific way, then I most definitely don’t want them on my team..”
Exactly! Self-esteem and arrogance are not the same thing. Also – confidence is the exact opposite of primadonna. And even the most ‘stellar track record’ benefits form validation – there’s nothing offensive in that.
Good filter for undesirable divas, imho 🙂
Pingback: Hiring is a first class responsibility | The Ascending Iterator
I agree with Chris, how are some of these questions in any way beneficial? In the verbal part of the interview I’m not looking for technical knowledge “what’s the order of C++ class data member initialization,” …. thats ridiculous. I want to know how they communicate, what their background is, and if they can problem solve. What I do is put together a scope document, and allow them a setup of their preference (tools, DB, OS type, etc …) to complete a project in. I throw in a few curve balls to evaluate problem solving and critical thinking (which are the most important skills for any developer), and evaluate how they follow directions and how they code.
Todd, the “good for” sections above show when/why some of the questions can be beneficial. Are you suggesting that some of the questions/techniques are not suited to what they claim to be good for or are you making the point that other questions should also be asked? If it is the latter, then I wholeheartedly agree with you. If it is the former, please, share your thoughts more specifically.
As per your specific comment about you not looking for technical knowledge in the verbal part of the interview, that’s certainly one way of doing it. Personally, I like to fail quickly and cheaply. When I interview I don’t want to waste my time or the time of the interviewee and my teammates by delaying making a negative decision. Many projects can do fine with senior engineers who are not intimately familiar with the tools of their trade. The kinds of projects that I’m involved with typically cannot.
Back when I was asking candidates the question about C++ member initialization order I led the core team building the first Web application server. Java was for applets back then, .NET did not exist, PHP was in v1.0 and Ruby hadn’t crossed the Pacific. We had a large, high-performance, multi-threaded, cross-platform C++ code base where we managed everything from compiler bugs to sockets to database connections to threads to memory ourselves. We pretty quickly learned that anyone who did not understand the internals of how C++ worked was a liability.
Every interview technique needs to fit in the context of a particular team/project/company.