whom to interview?
[tags]hiring, resume,interviewing [/tags]
You’re growing your team. How do you sift through the pile of incoming resumes? For younger candidates, how important is a GPA? For more experienced candidates, how do you qualified relevant experience?Most job applications arrive electronically these days. Email is a wonderful thing and makes it extremely easy to receive applications without the overhead of taking phone calls. Unfortunately, this also means that many unqualified applicants can submit their resumes with relative ease. Applications will come from people with little or no correlation to those listed in the description of your job opening. Traditional wisdom would be to read all the resumes and sort them into a pile of ‘good’, ‘questionable’, and ‘no’. You schedule the ‘good’ pile for interviews first and if no worthy candidate presents them self then you proceed to the ‘questionable’ pile.
Identifying interview candidates this early falls short of my needs in third respects.
- I need to keep hiring rolling. I cannot wait three weeks for the pile of resumes to build, and then cutoff accepting new resumes to begin the process.
- In my experience, a resume proves tangentially helpful to assessing an candidate without additional information. Specifically, a resume cannot tell you whether a candidate possesses the skills they put on their resume or not. How do you get a good read when candidates only send a resume and cover letter?
- The ‘questionable’ pile contains too many candidates for me to interview.
Don’t read them. That’s right, ignore them entirely. Instead, send them a questionnaire in response. This brief questionnaire will ask the applicants to rate their skill level on a 1 through 10 scale in six to ten technologies/topics that were listed in the opening or relate directly to the opening. The responses to this questionnaire are not necessarily a good indication of skill. That means that someone who rates them self as a six might be just as good if not better than someone who rates an eight. However, if someone lists a zero, one, or two for some or all of the skills then you can be pretty sure they’re not a good match unless you’re planning on lots of training.
The biggest suprise with this tactic will be the honesty of the respondents. You will get a number of people who might have made it into a screen interview who will tell you they have absolutely no relevant skills or experience for your position in response to this questionnaire. This alone saves you from wasting lots of time talking to them and discovering that they are indeed beginners of their craft. Since we’ve skimmed these self-proclaimed unqualified applicants off the pile it’s time to figure out who actually knows their stuff.
For developers I hire, I have a list of the top 30 topics under four or five headings (such as Linux, PHP, MySQL, Design, and Process) that we discuss and use every day. These include everything from ‘vi’ to ‘Extreme Programming’. Choose topics that are quintessential to the job. It’s important that some require insight and some that can be answered with a simple query to your favorite search engine. It’s just as valuable for this list to be somewhat vague. This will increase the variety of answers you get some of which are “right” and some of which are just “off”. These are in quotes because only you will learn which answers are off base and which hit the mark. For example, when I ask people about security in PHP some people say things related to input validation and SQL injection, while others mention topics like protecting your SQL password or turning on safe_mode. What’s important is knowing which answers come from someone who’s worked with the given topic and the answers from someone who hasn’t.
Use your other resources to evaluate their answers. Compare their answers with their skill levels professed in the first questionnaire and their experience on their resume. Any conflicts raise questions–
- “how can your skill in PHP rate 9/10, but you cannot describe PEAR?”,
- “your resume indicates you spent 4 years working with XHTML/CSS but you rated yourself 2/10, why?”
Do realize, that with a long list of topics a very few number of candidates will know all the topics to the desired depth. I make no stipulations in the instructions for whether or not it’s acceptable to research the topics in order to provide an answer. Some people admit ignorance, others cite their sources, and some copy and paste. After reviewing a number of them it becomes more and more clear how to distinguish. Those who know their stuff will answer consistently with ample information to back up their claims and admit areas of lesser experience.After the second questionnaire you will notice a decrease in responses. Hopefully those people have either decided it’s too invasive to answer all these questions, or have decided that even though they lied in the first round their bluff has been called. The time has come to schedule interviews (or telephone interviews for those off site). Each respondent goes into the ‘interested’ or ‘not interested’ category based on the two piece of standard information (the two questionnaires) and the entirely nonstandard information–the resumes. The questionnaires will show some people apply knowledge and intelligence to their answers while others don’t. Look for answers that show experience or even preference. Using the example above, I get many responses to the ‘vi’ topic that say ‘I prefer emacs’, or ‘I use BBEdit’. These show to me that someone has heard of vi and have made their own judgment as to what they like better.
In evaluating the second set of responses I mentioned the value of comparing with experience and statements made on a resume. At this point, insight into the history and personality of a candidate emerge by reading the resume by itself to learn why and how people might have obtained the knowledge you saw in their questionaires. Do they have three degrees from prestigious universities? Do they scour the internet for interesting tidbits about technology? Or did they search for the answers on receipt of the questions? Thinks I like to see: diversity shows an ability to learn new things quickly, but means you should verify depth of knowledge in the interview.
In your decision of who and who not to interview start considering the fit with the team. Make sure to welcome candidates to the ‘interested’ pile who have skills or experience which complement those of your team. Perhaps they have expertise in an area which your team lacks. As you consider the candidates during the interviews this will be more crucial.
An interview will allow you to untangle all these indicators and weave a more complete picture of the candidate. I save the interview process itself for another time…
February 13th, 2006 at 7:12 pm
Hey David - nice post. Saw your comment on the digg’d article here:
http://www.oreillynet.com/pub/wlg/9175?wlg=yes
The questionaire tactic seems like a good weed out process.
Do you use a script you home-rolled, or something off the shelf?
February 13th, 2006 at 9:12 pm
We’ve created both of the mentioned questionnaires. Nothing fancy, just plain text emailed out. Respondants answer inline or some type answers up in their favorite word processor.
Example Questionnaires
February 22nd, 2006 at 9:29 pm
Great post, and I like your approach to the resume problem.
“Selection” (as HR people call it) is a very tricky beast. Studies show that most selection tools (resumes, interviews, etc.) have terrible R^2 values statistically.
People most commonly use an unstructured interview, which has an R^2 of about 6%. That is, you might as well throw a dart at a pile of resumes.
However, you can get the R^2 up to about 30-35% by using a combination of selection techniques, the most poweful being a structured interview and a work sample.
The structured interview is similar to David’s technique above, however extended to the interview. More than just a standard list of questions, have a predefined scale of answers. “I prefer emacs” might be a 3 out of 5. Meanwhile “vi much safer for editing system configuration files because the author typically changes one word or character at a time, thereby minimizing accidents” might be 5 of 5.
Now you can quantitatively compare your responses.
When you do call in a candidate for an interview, do a similar process face-to-face. Do not just see how well the candidate interacts with your team. Instead, work with your interview team (and it should be more than one person) to prepare standardized questions and measured responses.
Also use a work sample. This may be a code sample to be reviewed, but better yet make a 1-hr task. Perhaps a “spike solution” (in XP terms) so that domain knowledge is not necessarily a prerequisite.
That brings me to my final point: don’t focus entirely on domain knowledge. I’d rather have excellent thinkers and programmers than someone that knows all the tricks in the language used at this particular shop. While it is some advantage to have someone that can whip out PHP from day 1, in the long run you’d be better off having someone that knows how to communicate and design.
For example, I’d rather have a questionnaire talking about Patterns (”what is an iterator and why is it useful?”) than about specific technologies (”what php.conf settings are best for PHP security?”)
December 29th, 2006 at 6:47 am
I found your post and your questionnaires really interesting and even useful, but I wonder if some of your questions aren’t too open ended or elementary?
example:
* Facade Pattern
I think you’d get a lot of pretty generic answers about compliance standards they’d used to make functions at their last workplace wrap better.
What if someone comes back to you and talks about wrappers they made for some libraries their last company had for some database agnosticism? They go into how they made some Perl libraries for controlling some triggers they wrote, and then talk about the wrappers around some Java functions they wrote so they could use some standard API to talk to it. Finally they’d talk about the simplified interfaces they made to consume the Java objects underneath.
Still not what I want to see from an applicant. I want to know the process they went through to build it. Wouldn’t it be better to ask in the questionnaire for a specific instance where they used a Facade (and in my context, I’d ask them about wrappers and helper functions) and weed out the bad answers at the root instead of even providing Facade in the programming skills section? Harder to fake an actual task, or am I misunderstanding your question entirely?
The problem for me is too often to this question and your later question:
Explain the “facade” design pattern. Please give an example of
a “real world” case where one might use it.
I’d probably get the canned tiered design approach answer or some Microsoft garbage about web service interoperability. I don’t need tech speak, I need how they’re going to do it so I can see if they can adapt and be molded into the team I already have in place.
Some of the worst hires I’ve had were people who could quote me chapter and verse from a text book and give me short quick examples about this or that code, but were unable to adapt the text book example to the problems I wanted them to deal with.
I’d rephrase the question to be more specific:
Give me an example from your career or studies where you started with a complex system and needed to simplify it into a series of interfaces that were less complex and/or planned for multiple systems interfacing with the same layers. Follow this with some code examples for how you overcame individual problems. Keep this short, list between 3 and 5 solutions and give me an overarching view of your design/solution process. Draw me a picture if you need to. Have fun with this, tell me about how you struggled or dealt with the issues at hand. Solutions without code are a waste of time.
Or are you looking for something different?
I’d really appreciate your insights, I’m getting ready to start a hiring cycle now, and I’d love the feedback.
December 29th, 2006 at 8:20 pm
I’d argue that you’re missing a bit of the point. These questions are designed to screen interviewees and give a good idea of whether they’re qualified or not. You have to consider the other side of the coin: you don’t want to turn off potential candidates by making them answer too many questions without having ever spoken with them. Those with lots of experience on their resume will scoff at this and simply go elsewhere.
The other thing I think is important is that any answer about the n-tiered Microsoft approach is really missing the point of the facade design pattern. The questions you’re giving are better suited for an interview where the answers can be directed toward the level of detail and specificity you desire.
Finally, the important aspect here is simply to identify if they’re a smart individual, whether they have no idea what you’re talking about, or whether they are clutching their textbooks or certifications to give you vague 20,000 foot answers.
December 30th, 2006 at 7:13 am
That makes a lot of sense. I appreciate that.
I don’t know though… is there a happy medium where the question can get a little more specific? Smart is good, but experiences give me an idea about where they’ve worked before and how they worked there.