I love coding interviews because they are so much fun to do as a candidate. That said, stuff like this:
firmly believe they are the most objective way to evaluate candidates
This presumes candidates have experience like yours (or that experience like yours is what you’re trying to measure). I had a very senior coworker fail an interview once due to lack of knowledge of parsing strategies / parse trees. This person is not just “can code” but also can architect, lead, etc, but failed due to a question with the assumption that “anyone smart must ha e been exposed to parsing”.
That’s why I always prefer a portfolio-based interview. Then the problem they are explaining to you is one they know how to solve because they have already solved it! Of course, this doesn’t work well if you want to be able to hire career-long corporatists who have no portfolio. There is no one size fits all interview.
I’ve only done a couple of coding interviews and it turns out I have pretty bad performance anxiety and forget basic stuff or get so worked up I can’t think straight.
Apparently this would mean that I have faked the last 20+ years, dozens of open source projects, ~15 conference presentations, and several peer-reviewed papers.
And frankly if you think that’s the case then you should hire me because I’m obviously an amazing social engineer. :)
Repeated claims of “interviews are so hard, I get anxious, etc. etc.” in order to get an easier-to-pass process are also social engineering.
(I’m not saying you’re doing this…just you jogged my brain with your phrasing at the end there.)
My experience has been that as we’ve tried to be more accommodating as an industry, we’re getting more and more bad or ineffective actors into our systems. Given that my favored fair approach is too ruthless for most companies–hire anybody with a pulse who can fog a mirror and open an IDE, fire them after a week if they suck, repeat as needed–I think we’re definitely in for some trouble. This is compounded so much more than people wish to admit with the current implementation of DEI efforts, employee resource committees (or whatever your local version of officially-sanctioned factional support groups is), and so forth.
I don’t know what the right answer is, but I’m pretty sure dropping interviews ain’t it.
That’s not too brutal, HR doesn’t like it because it’s more work for them; management doesn’t like it because they’re afraid to fail unconventionally. Also then they’d need to work out a good onboarding process. Obviously you have to tell people you need them to do a week long paid interview.
I do like those, but I think they should be paid and I usually think they’re poorly designed/scoped. And when you’re doing a lot of interviewing at the same time, it can get annoying having that pile of stuff in the background.
Doesn’t that produce way too much overhead with training the new hires? Someone has to do the onboarding, especially with complex projects (and aren’t they all, nowadays?) or junior hires! Usually it takes a while (a week to a month) before someone is properly up and running anyway. If you have to do that with your shotgun approach to hiring, that would put a big cap on overall productivity while you’re in “hiring mode”.
This presumes candidates have experience like yours (or that experience like yours is what you’re trying to measure).
Is there any interviewing technique that doesn’t?
That’s why I always prefer a portfolio-based interview.
This assumes that the candidate has time to build a portfolio, or is comfortable talking about their previous job’s likely-proprietary code. It also biases towards candidates with interests like yours.
And it’s not uniform, which allows a huge amount of subjectiveness to creep in.
There is no one size fits all interview.
And yet, giving different types of interviews to different candidates is a non-starter for hopefully obvious reasons.
Yes. @singpolyma is proposing a portfolio review, which encourages the candidate to teach the interviewers about a subject they are unlikely to be as familiar with as the candidate. That’s a very relevant skill that code challenges can’t demonstrate.
It also biases towards candidates with interests like yours.
I once had a candidate walk us through her ClojureScript project. We didn’t know ClojureScript or have any particular interest in it. But it was the only non-proprietary code she could show us. She was able to demonstrate both her skill in writing software and, just as importantly, an ability to explain her decision making and values in the process. We had to swallow our pride and ask some questions about syntax, but it was one of the best interviews I’ve ever conducted.
Yes, there was a lot of subjectivity in that interview. That’s life. But interviewers will practice an open or closed mind regardless of format. It’s the code challenge that’s the greater refuge for closed mindedness.
I would care to bet that part of the intention behind ‘with interests like yours’ is an interest in having programming projects at home. I know this is not a universal opinion, but I think that having personal projects is irrelevant to the performance of a senior engineer.
I guess that means you either get very few applicants or that you’re so selective that almost all get filtered out and your search ends as soon as you find one that passes.
On multiple occasions, I’ve found myself in the intermediate situation where multiple good candidates showed up at the same time, but I still had to choose one. In that case, even if you allow people to demonstrate themselves with some freedom, you also need a basis in order to carry out the unfortunate task of comparing them.
If you have 2 candidates who have impressed you, why does one have to be better or worse, and what makes you think you have the process or information to determine who is better? Just pick a winner at random. Or pick the one who used a sans serif font on their resume, or some other arbitrary factor.
Ironically, as strange as that sounds, both your suggestion of hiring them both and @fly’s suggestion of flipping a coin would’ve worked quite well in my case in hind sight…
What I think a lot of people miss is that coding interviews, like all interviews, need to be carefully researched and designed. They can’t be done off the cuff or based on your intuition. That leads to things like Schottky’s approach to interviewing (stress them and see if they can think on their feet) which dominated engineering for decades. Despite the fact that Schottky was such a terrible manager that his entire engineering team quit and started a competing company.
First, most people don’t know what they’re trying to find out from an interview. You’re not going to find out much, so you had better be clear on what you need to know. What are the skills that you cannot afford to train someone in? It’s probably a lot fewer than you think. Being able to express simple solutions as structured programs is the bar I generally aim for.
An interviewee is stressed. That means that their ability to work through corner cases or do free associating thinking is compromised. That means that I start with something trivial, something with no gotchas or corner cases, ideally nothing more than “if input A, return B, if input B, return C.” I’ve even started with “write a function that returns the square of a number.” Ideally I link it into the sequence of progressive development of problems that I am using, but it doesn’t have to be. This helps get people over being frozen.
I use a sequence of problems that can degrade gracefully. Sometimes an interviewee is capable of solving the full problem without the stressful environment, but is stuck. If I can give them a simpler version they can often solve it and then go right on to solve the previous, more difficult version of the problem. Conversely, you want to have a sequence that continues past where anyone is likely to get.
I also intentionally choose problems that don’t have lots of corner cases. Corner cases cost lots of time in the interview when you could be learning something, and stress makes interviewees terrible at handling them. I think that binary search is a terrible interview problem because it requires dealing with lots of edge cases to get it right, and it also doesn’t degrade well.
You should be using the same interview every time, and you should have given it to people whose abilities you know. Ideally whole teams are using the same interview and shadowing each other so they have some kind of shared standard.
The other thing I like to do is throw in something that’s not straight coding, but related. Often I do something with layout in memory, such as if you have four possible values for a type, how many bits do you need to represent it? How many bits if you use one character for representing the type instead? If you have a billion of them, how much savings do you get from using the compact one instead of the character? People get really thrown by this, which is a problem. I blame the emphasis on leetcode for interview prep. But I find that I can usually take a couple minutes and talk them out of their corner. Sometimes it turns out they have no clue. Sometimes they have no confidence but give the right answer easily once you unstick them.
But the key point in all of this is that interviews are a experimental technique from the social sciences. You have to treat them as such, with all the confounding and reproducibility problems that implies.
First, most people don’t know what they’re trying to find out from an interview. You’re not going to find out much, so you had better be clear on what you need to know. What are the skills that you cannot afford to train someone in? It’s probably a lot fewer than you think. Being able to express simple solutions as structured programs is the bar I generally aim for.
This is really important. In assessment theory, people talk about VRIP:
Validity: is the thing that we’re measuring actually part of some self-consistent construct?
Reliability: what is the margin of error in our assessment? If we measure the same cohort twice (e.g. with two different years’ exams for the same subject) then will we get the same results?
Impact: how well does the construct and the subset of it that we measure reflect what people who are using these examination results actually care about? For example, is an exam a good predictor of university or vocational performance?
Practicality: can we actually measure the thing that we want to measure within a budget that people are willing to spend on assessment?
I’ve rarely seen anyone think about anything other than the last of these in the context of interviews. If you want to hire a good software engineer but you can’t even define what characteristics a good software engineer has, how do you expect to be able to measure them in an interview?
I am OK with CS 101 DS and algo type of questions. The ones I am afraid the most are with additional “real-world” knowledge: design an efficient shuffling deck of cards, OOP design of parking garage, meeting time scheduling. I’m not familiar with playing card games. I don’t drive a car and have no idea wtf how a parking garage operates.
Fizzbuzz is pretty good. The only issue I’ve encountered is not knowing or not remembering what modulo is. I certainly didn’t know the first time I was asked to implement it.
When I was 17, I did very badly in a Cambridge undergraduate admissions interview. The interviewer asked me a simple question with an obvious answer. Cambridge interviews are notoriously hard and so I assumed that I was missing something important and stalled for a long time while I tried to work out what the trick part of the question was. Eventually they gave up and told me the answer. It was precisely the one that I’d thought of immediately but since I’d been primed to expect a difficult question, I couldn’t believe that this was really what they were asking. I think I used somewhere between a quarter and a half of the total interview time failing to answer a question that they were expecting would take 30 seconds. I did not get an offer.
Questions like Fizzbuzz worry me because I’d expect them to trigger the same response from a lot of people. Fizzbuzz, in particular, has a very large solution space. A trivial implementation might look something like this:
for (int i=1 ; i<max ; i++)
{
if ((i % 3) == 0)
printf("fizz\n");
else if ((i % 5) == 0)
printf("buzz\n");
else if (((i %3) == 0) && ((i %5) == 0)
printf("fizzbuzz\n");
else
printf("%d\n", i);
}
You could remove one of the branches by allowing fall-through into the buzz case even after you’ve done fizz, but then you need to worry about atomicity of the writes. Is that something the interview is expecting the candidate to think about? Are they expecting the candidate to think about the problem a bit more and realise that the pattern repeats 15 times (because 3 and 5 are both prime and 15 is 3x5 and therefore the lowest common multiple)? If so, the solution should look something like this, which avoids any modulo arithmetic:
for (int i=1 ; i<max ; i+=15)
{
printf("%d\n%d\nfizz\n%d\nbuzz\nfizz\n%d\n%d\nfizz\nbuzz\n%d\nfizz\n%d\n%d\nfizzbuzz\n"
i, i+1, i+3, i+6, i+7, i+10, i+12, 1+13);
}
With some special-case logic at the end for an incomplete sequence. Of course, printf is a very expensive call. You could optimise this by computing it once for each digit length and then just doing decimal arithmetic in the string.
And so on. There is a lot more optimisation that you can do with fizzbuzz. Knowing what the interviewer expects from the first attempt may not be obvious.
Except it’s not necessary… it’s not particularly hard to do the arithmetic to figure divisibility or keep a couple extra loop counters and solve it without knowing/using the operator while still not introducing any new concepts to the solution. You might have to give a candidate doing that a bit more time, perhaps. But if anything, seeing that solution and then offering an explanation of modulo w/ a chance to revise is likely to be an even better assessment.
The status quo being defended is akin to grading a student 100% on a final exam. I realize that’s still common in some countries, but it favors people who can cram (or who can pay for test prep) over actually relevant competency, skill, and curiosity.
Arguments that portfolio and take-home exercise reviews are too time consuming don’t wash. Most FAANGs and their imitators conduct batteries of multi-hour interviews, sometimes over several days. Arguments that they’re too subjective don’t wash, either. If the job is to use a language that has hundreds of built-in methods for manipulating ADTs to solve problems in another domain, what’s so objective about picking your favorite one and making someone re-implement it on the spot while you watch?
I don’t actually encounter many cheaters amongst the applicants I’ve interviewed over the years, but then again, I’ve never worked for a FAANG. I’m sure it can be overwhelming. But I know a lot of people who do work at FAANGs, and uncharitable though it may be for me to say, some of them have only ever learned enough to look competent and are disappointingly uncurious. You can actually catch cheaters and shallow learners if you stop cross examining your applicants. Open ended questions don’t leave much cover for them to hide behind.
I love take-homes. They require a lot of time, but that can be at night or on the weekend when I don’t have many other responsibilities. I get a taste of what types of problems the company actually wants to solve instead of fictional algorithms questions. And best of all – lots of people won’t even attempt them, which gives me an advantage.
The problem with cheating is real, though. I don’t know how to solve that.
can be at night or on the weekend when I don’t have many other responsibilities
Selecting for candidates that don’t have “many other responsibilities” has a ton of biases employers probably don’t (or at least shouldn’t) want to take on - biases which are probably great for you! For example, it selects for younger people from wealthy, high-class families.
know a lot of people who do work at FAANGs, and uncharitable though it may be for me to say, some of them have only ever learned enough to look competent and are disappointingly uncurious.
I would say that FAANG selects for this, since they want docile, compliant workers who won’t complain about the inflexible and slow work environment.
First of all, candidates don’t like these - since they may take a long time to finish,
My experience has been the opposite and the people I’ve talked to prefer them.
It’s more convenient than asking a candidate to take PTO to do an entire day’s (5+ hours) of interviews, or spend days preparing on LeetCode. Being quick on your feet is also a skill, so it also evens the playing field because a lot of people just don’t interview well. It also gives you the opportunity to have you current programmers take the test to see what sort of solutions to expect, and learn the sort of things they might want to ask in the interview.
Candidates can work on their own time and at their own leisure, with whatever suite of tools are most comfortable to them. If someone doesn’t have a portfolio to show it provides you the opportunity to see what type of code they write.
“Hi, here’s a prompt of a little project to do, should take at most a couple of hours, you have a week to work on it at your leisure, send it back when your done and let us know if you have any questions.”
Second, these assignments are a burden on the hiring team as well, since they take a long time to prepare and evaluate.
the plight of SWEs who have to spend lots of time on interviews
these assignments are very vulnerable to cheating
Our coding test takes maybe 2 hours, and I take a half an hour to maybe an hour to go through it. I’d rather spend that time on my own going through someone’s code than asking someone on-the-spot questions. It also gives me the basis of questions to ask the interviewee (why’d you do it this way? What tradeoff did you make here? Why didn’t you use this language feature here, etc.), which I give to them a few days before the interview so they can do their own research, like you would normally on a project. Since they have time to prepare, you don’t need to shy from asking detailed questions.
You copy the returned projects into a folder for analysis against other solutions for plagiarism. When I start asking detailed questions from the code review, it becomes apparent who knows what they’re talking about, since especially when given the questions beforehand, people can see opportunities to delve deep and show off what they know.
It’s more convenient than asking a candidate to take PTO to do an entire day’s (5+ hours) of interviews
I definitely agree with this. Having to take a PTO to do a virtual on-site interview is exhausting and the preparation is (LeetCode etc.) takes a lot of time after work and on the weekends to do.
You copy the returned projects into a folder for analysis against other solutions for plagiarism.
There has been cases of false positives with automated code plagiarism services [1], but asking them questions about their code should out the ones who cheated and like you said shows if they know what they’re talking about.
My current employer required an actual work sample take home test and I really enjoyed it instead of the typical LeetCode style questions.There was a few solutions on Google, but it seemed they randomized some things to catch cheating and I still had to do an interview with the hiring manager. MIT[2], Caltech[3], and Stanford[4] had mini courses for preparation for these type of interviews.
Red Balloon Security (RBS) had an interesting interview process. They first had a screening challenge which answers could be found via a Google Search, but the second one required extracting a secret Bitcoin wallet private key from a hard drive with a rootkit installed. The time limit for the hard drive challenge was a week, but the Bitcoin wallet had $3k worth of Bitcoin at the time. An additional interview was required before an offer I think and a few people on glassdoor complained about being ghosted after completing it, but it seemed like an interesting interview challenge and offered an incentive to complete it.
these assignments are a burden on the hiring team as well
I suppose that’s an issue, but for the RBS one it was more of a CTF challenge instead of writing up a detailed report etc. So it seemed as long as you found the flag you would move on the next round (I didn’t complete the hard drive challenge for RBS). For my current employer I had to write a detailed write up, but was told to only spend an hour max. I imagine that it took someone maybe 15-20 minutes to review my 5 page write up PDF, but I could be wrong.
It feels like a big assertion to say that there are tons of people faking coding (but could otherwise pass a software interview). Are there any actual numbers that back that up?
I only have one quibble with this. I really dislike the coding interviews that dig into low-level coding of data structures when I’m applying for a senior web developer position. I don’t know how to reverse each word of a string in place. I’m so used to doing ruby’s Array#each or other languages’ equivalents,that I need to look up the syntax for a C for loop.
Note, my ruby version of reverse the words in a string would be str.split(' ').map!(&:reverse).join(' ') and this would be faster than any pure ruby implementation.
I think it’s easy to do a basic verbal interview first to see if there is a possible fit and if it seems so to have them do actual work work with the team. That way both sides actually find out whether they are a good fit or not. Of course that can also be done remotely. If one things it requires more time one can can consider to do it in some form of trial time.
I think in IT we are lucky to be able to do that. It’s really low risk and low effort to do that. A lot of other fields don’t really have that possibility.
But no matter what one does for interviews it should be something that a person actually does at the company. Don’t make people solve university style algorithm questions, if their job description isn’t “solves university style algorithm questions all day”. You need people being good at algorithms? Well, great, let them solve that problem. I am sure you have a task in the backlog that requires that, since you feel the need to hire such a person.
I think it’s bad for the company and rude to the person being interviewed to give completely wrong impressions of the company and their work. Creating a contract based on that seems like setting things up for failure. I was lucky, because I never really had to deal with that, but when I hear such stories I always imagine myself in such a situation and politely pointing out that based on the interviewing process (which after all is usually the only information I can judge from) I don’t think there will be a match and that therefor I don’t want to waste their time.
So basically what I am trying to say is that the structure doesn’t seem to matter that much. And also that pure algorithmic knowledge of an employee also isn’t really the thing one wants to/should focus on. There’s also other factors that aren’t so easy to measure that make an applicant a fit or not. Nobody wants to sign a contract with a person that is on the human side disliked by the team or simply will hate their job. In such a situation the company will lose value and the applicant gets set up for burnout.
Coding interviews are necessary because there are too many non-coders out there pretending to be coders. You need to make the candidates write code.
Having said that, 1-hour coding interviews in the style of “show me that you can fetch and show some data from a public API” is probably the right size, as it shows enough of the day to day practices. Anything beyond that (especially take-home exercises) is stretching it.
Coding interviews are necessary because there are too many non-coders out there pretending to be coders.
I have run many, many interviews at multiple companies. I have yet to encounter the mythical “non-coder” trying to trick the company into hiring them. As far as I can tell, the whole idea is based on a literal vicious cycle where anyone who fails an interview is deemed to be utterly unable to do any coding, and that’s used as justification for ratcheting up the difficulty level, which results in more people failing and being deemed unable to code, which justifies ratcheting up the difficulty…
As you can see, roughly 25% of interviewees are consistent in their performance, but the rest are all over the place. And over a third of people with a high mean (>=3) technical performance bombed at least one interview.
In the interviews they didn’t do well in, they probably were labeled as unqualified, incapable, etc. – but in the rest of their interviewers they were top performers.
I have run many, many interviews at multiple companies. I have yet to encounter the mythical “non-coder” trying to trick the company into hiring them.
Lucky you. I haven’t run that many interviews, and so far I can remember a few cases:
Junior finishing degree from degree mill, trying to pass a 5 million banking system as his own. Upon further investigation, his contribution to the code base was around 20 changes to README files.
HR woman wanting to join because HR is so easy she expects to pick programming by just working with us.
Applicant-provided code looks good. But the code in the on-site interview is very different (way lower quality). Upon further investigation, turns out the provided sample was written by her boss, not actally by the applicant.
20 years of experience on CV. Can only do what the IDE offers as buttons/dropdowns. Unable to debug the SOAP authentication in their company API because there is no button for it.
Sure, some of these can work as programmers if you are willing to lower enough the requirements. But without a technical interview where they get to write some code, or explain some code they wrote in the past, you wouln’t find out.
And yes, I also have hired people without degrees that were able to demo me something they built. I had to teach them later working in teams, agile, git, github and other stuff, but they still did good nodejs and mongo.
It’s very hard to completely bomb an interview if you can write some code. You can write wrong abstractions, bad syntax, no tests, and that is still so much better than not being able to open an IDE, or not being able to open a terminal and type ls.
have hired people without degrees that were able to demo me something they built. I had to teach them later working in teams, agile, git, github and other stuff,
This doesn’t seem degree related, since degree work won’t teach you working in teams, agile, git, or github
I have run many, many interviews at multiple companies. I have yet to encounter the mythical “non-coder” trying to trick the company into hiring them.
I’ve seen it happen; I’ve worked with someone who managed to work at a company for several years, even though it was an open secret that they couldn’t code without pairing (he was tolerated because he was likeable, and took care of a lot of the non-programming related chores)
I’ve also seen people strut into an interview with great confidence, only to write code that didn’t remotely resemble the syntax of the language they were supposedly proficient in. If this was due to anxiety, they certainly didn’t show it.*)
I don’t think it’s common enough to warrant all the anxiety about “fake programmers”, but it’s not completely imaginary.
*) I live in a country where software shops are among the few white-collar employers that’ll happily accept people who don’t speak the native language. That might mean we get more applicants for whom software development wasn’t exactly their life’s calling.
I don’t know how the situation arose, but by the time I got there, he was already known as the guy who did the “chores”, and avoided pairing to - presumably - reduce the chance of being found out. This all changed when new management came in, which implemented some - very overdue - changes.
Maybe the takeaway should be that there might be room for “non-coders” in development teams, but the situation as it was can’t have been easy on anybody.
Coding interviews are necessary because there are too many non-coders out there pretending to be coders. You need to make the candidates write code.
I can’t speak for all jurisdictions, but this feels a little overblown to me, at least in the Australian market.
If someone manages to sneak through and it turns out they have no experience and can’t do the job, you can dismiss them within the 6 month probation period.
Yes, you would have likely wasted some time and money, but it shouldn’t be significant if this is a disproportionately small number of candidates.
The thing is the culture in the US is quite different- some companies will fire people willy nilly, which is bad for morale, and other companies are very hesitant to fire anyone for incompetence, because by firing someone you leave them and their family without access to health care (and it’s bad for morale). Either way, you do actually want to make good decisions most of the time when hiring.
coding assignments during an interview (eg 15min to code an algorithm for strings, graphs, dynamic programming, etc.) – are always bad. Bad for the employer, for candidate and for the industry as a whole.
Mostly because the employer is assuming to hire people to write ‘novels’, often mutli-volume novels (if I may use that analogy),
but, instead, they test the candidates with ‘jeopardy-style’ questions.
take home coding is better,
because ( a) it reduces the effects of already-wrong filtering,
( b ) it gives a an opportunity to the employer to see how candidate structures and expresses their thought, after
they had time to think. Which is closer to the type of environment they will be in
What is still bad about take home interviews are the following:
does not elicit the candidate’s ability to structure a larger system (if you are hiring a person with 10+ years of experience, most likely you are looking for it)
does not elicit candidate’s interest areas and the progress they made on their own in those areas (eg self-study competencies)
does not allow the candidate to demonstrate their particular strengths (that may be well outside the particular coding experiment)
I get that all those additional qualities can be somehow learned by other means, but the coding question is always
used as ‘gateway’. In my view if a take-home coding question is still used, it should be up to the candidate to
decide at which stage of the interview to do it.
As I posted in some other replies in the similar topics. There is an ample space for a technology+social-sciences startup that could make hiring process much better (and not just for software-dev industry).
What we have today in this space is really really bad. It slows down progress in the industry by suffocating creativity and long-term vision in software products, by narrowing down employment to folks that can prepare well for tests, and that have good/quick thinking ability.
Good part about it, that the companies that practice the ‘code-within-X-minute’ style of exercises, are filtering out really great candidates that become available to other employers with much better interview hygiene.
Anyone run into a situation where you had a candidate just have someone completely do all their interviewing for them? With zoom interviews, it seems like in a corporate setting where maybe the people interviewing don’t end up working with the candidate, this may actually be happening?
Slight tangent but I honestly have so much fun studying for CS interviews – not the leetcode grind, but the brushing up on theory, data structures / algorithms, and just thinking about things at a more conceptual / academic level than I get to during my job.
Also, as an interviewee, I found that the type of interview questions / process a company is a useful way to gauge whether I’d like working there, since different types of processes select for different types of people, and are somewhat indicative of how well fleshed out the role I’m applying for actually is.
I love coding interviews because they are so much fun to do as a candidate. That said, stuff like this:
This presumes candidates have experience like yours (or that experience like yours is what you’re trying to measure). I had a very senior coworker fail an interview once due to lack of knowledge of parsing strategies / parse trees. This person is not just “can code” but also can architect, lead, etc, but failed due to a question with the assumption that “anyone smart must ha e been exposed to parsing”.
That’s why I always prefer a portfolio-based interview. Then the problem they are explaining to you is one they know how to solve because they have already solved it! Of course, this doesn’t work well if you want to be able to hire career-long corporatists who have no portfolio. There is no one size fits all interview.
I’ve only done a couple of coding interviews and it turns out I have pretty bad performance anxiety and forget basic stuff or get so worked up I can’t think straight.
Apparently this would mean that I have faked the last 20+ years, dozens of open source projects, ~15 conference presentations, and several peer-reviewed papers.
And frankly if you think that’s the case then you should hire me because I’m obviously an amazing social engineer. :)
Repeated claims of “interviews are so hard, I get anxious, etc. etc.” in order to get an easier-to-pass process are also social engineering.
(I’m not saying you’re doing this…just you jogged my brain with your phrasing at the end there.)
My experience has been that as we’ve tried to be more accommodating as an industry, we’re getting more and more bad or ineffective actors into our systems. Given that my favored fair approach is too ruthless for most companies–hire anybody with a pulse who can fog a mirror and open an IDE, fire them after a week if they suck, repeat as needed–I think we’re definitely in for some trouble. This is compounded so much more than people wish to admit with the current implementation of DEI efforts, employee resource committees (or whatever your local version of officially-sanctioned factional support groups is), and so forth.
I don’t know what the right answer is, but I’m pretty sure dropping interviews ain’t it.
That’s not too brutal, HR doesn’t like it because it’s more work for them; management doesn’t like it because they’re afraid to fail unconventionally. Also then they’d need to work out a good onboarding process. Obviously you have to tell people you need them to do a week long paid interview.
For me, instead of a live coding interview, I very much prefer a 1-2 hour take home project. I knock those out of the park pretty well.
It would be nice to have the option.
I do like those, but I think they should be paid and I usually think they’re poorly designed/scoped. And when you’re doing a lot of interviewing at the same time, it can get annoying having that pile of stuff in the background.
Doesn’t that produce way too much overhead with training the new hires? Someone has to do the onboarding, especially with complex projects (and aren’t they all, nowadays?) or junior hires! Usually it takes a while (a week to a month) before someone is properly up and running anyway. If you have to do that with your shotgun approach to hiring, that would put a big cap on overall productivity while you’re in “hiring mode”.
Is there any interviewing technique that doesn’t?
This assumes that the candidate has time to build a portfolio, or is comfortable talking about their previous job’s likely-proprietary code. It also biases towards candidates with interests like yours.
And it’s not uniform, which allows a huge amount of subjectiveness to creep in.
And yet, giving different types of interviews to different candidates is a non-starter for hopefully obvious reasons.
Yes. @singpolyma is proposing a portfolio review, which encourages the candidate to teach the interviewers about a subject they are unlikely to be as familiar with as the candidate. That’s a very relevant skill that code challenges can’t demonstrate.
I once had a candidate walk us through her ClojureScript project. We didn’t know ClojureScript or have any particular interest in it. But it was the only non-proprietary code she could show us. She was able to demonstrate both her skill in writing software and, just as importantly, an ability to explain her decision making and values in the process. We had to swallow our pride and ask some questions about syntax, but it was one of the best interviews I’ve ever conducted.
Yes, there was a lot of subjectivity in that interview. That’s life. But interviewers will practice an open or closed mind regardless of format. It’s the code challenge that’s the greater refuge for closed mindedness.
I would care to bet that part of the intention behind ‘with interests like yours’ is an interest in having programming projects at home. I know this is not a universal opinion, but I think that having personal projects is irrelevant to the performance of a senior engineer.
Not obvious at all. Why not allow candidates to choose between (eg) coding interview and portfolio review?
Presumably because comparing candidates who went through radically different interviews would be tremendously difficult.
Hmm. I guess I’ve never been a low demand high supply enough situation to be “comparing candidates”, at least for full time.
I guess that means you either get very few applicants or that you’re so selective that almost all get filtered out and your search ends as soon as you find one that passes.
On multiple occasions, I’ve found myself in the intermediate situation where multiple good candidates showed up at the same time, but I still had to choose one. In that case, even if you allow people to demonstrate themselves with some freedom, you also need a basis in order to carry out the unfortunate task of comparing them.
You could just flip a coin.
If you have 2 candidates who have impressed you, why does one have to be better or worse, and what makes you think you have the process or information to determine who is better? Just pick a winner at random. Or pick the one who used a sans serif font on their resume, or some other arbitrary factor.
Exactly. Either just hire them both (are you really not gonna need another any time soon?) Or else it doesn’t really matter which you pick.
Ironically, as strange as that sounds, both your suggestion of hiring them both and @fly’s suggestion of flipping a coin would’ve worked quite well in my case in hind sight…
I don’t think that’s ironic. I think that’s the point.
What I think a lot of people miss is that coding interviews, like all interviews, need to be carefully researched and designed. They can’t be done off the cuff or based on your intuition. That leads to things like Schottky’s approach to interviewing (stress them and see if they can think on their feet) which dominated engineering for decades. Despite the fact that Schottky was such a terrible manager that his entire engineering team quit and started a competing company.
First, most people don’t know what they’re trying to find out from an interview. You’re not going to find out much, so you had better be clear on what you need to know. What are the skills that you cannot afford to train someone in? It’s probably a lot fewer than you think. Being able to express simple solutions as structured programs is the bar I generally aim for.
An interviewee is stressed. That means that their ability to work through corner cases or do free associating thinking is compromised. That means that I start with something trivial, something with no gotchas or corner cases, ideally nothing more than “if input A, return B, if input B, return C.” I’ve even started with “write a function that returns the square of a number.” Ideally I link it into the sequence of progressive development of problems that I am using, but it doesn’t have to be. This helps get people over being frozen.
I use a sequence of problems that can degrade gracefully. Sometimes an interviewee is capable of solving the full problem without the stressful environment, but is stuck. If I can give them a simpler version they can often solve it and then go right on to solve the previous, more difficult version of the problem. Conversely, you want to have a sequence that continues past where anyone is likely to get.
I also intentionally choose problems that don’t have lots of corner cases. Corner cases cost lots of time in the interview when you could be learning something, and stress makes interviewees terrible at handling them. I think that binary search is a terrible interview problem because it requires dealing with lots of edge cases to get it right, and it also doesn’t degrade well.
You should be using the same interview every time, and you should have given it to people whose abilities you know. Ideally whole teams are using the same interview and shadowing each other so they have some kind of shared standard.
The other thing I like to do is throw in something that’s not straight coding, but related. Often I do something with layout in memory, such as if you have four possible values for a type, how many bits do you need to represent it? How many bits if you use one character for representing the type instead? If you have a billion of them, how much savings do you get from using the compact one instead of the character? People get really thrown by this, which is a problem. I blame the emphasis on leetcode for interview prep. But I find that I can usually take a couple minutes and talk them out of their corner. Sometimes it turns out they have no clue. Sometimes they have no confidence but give the right answer easily once you unstick them.
But the key point in all of this is that interviews are a experimental technique from the social sciences. You have to treat them as such, with all the confounding and reproducibility problems that implies.
This is really important. In assessment theory, people talk about VRIP:
I’ve rarely seen anyone think about anything other than the last of these in the context of interviews. If you want to hire a good software engineer but you can’t even define what characteristics a good software engineer has, how do you expect to be able to measure them in an interview?
I am OK with CS 101 DS and algo type of questions. The ones I am afraid the most are with additional “real-world” knowledge: design an efficient shuffling deck of cards, OOP design of parking garage, meeting time scheduling. I’m not familiar with playing card games. I don’t drive a car and have no idea wtf how a parking garage operates.
Make them do a fizzbuzz. That’s like algorithms 101, yet it reveals so much on the programmer:
Fizzbuzz is pretty good. The only issue I’ve encountered is not knowing or not remembering what modulo is. I certainly didn’t know the first time I was asked to implement it.
When I was 17, I did very badly in a Cambridge undergraduate admissions interview. The interviewer asked me a simple question with an obvious answer. Cambridge interviews are notoriously hard and so I assumed that I was missing something important and stalled for a long time while I tried to work out what the trick part of the question was. Eventually they gave up and told me the answer. It was precisely the one that I’d thought of immediately but since I’d been primed to expect a difficult question, I couldn’t believe that this was really what they were asking. I think I used somewhere between a quarter and a half of the total interview time failing to answer a question that they were expecting would take 30 seconds. I did not get an offer.
Questions like Fizzbuzz worry me because I’d expect them to trigger the same response from a lot of people. Fizzbuzz, in particular, has a very large solution space. A trivial implementation might look something like this:
You could remove one of the branches by allowing fall-through into the buzz case even after you’ve done fizz, but then you need to worry about atomicity of the writes. Is that something the interview is expecting the candidate to think about? Are they expecting the candidate to think about the problem a bit more and realise that the pattern repeats 15 times (because 3 and 5 are both prime and 15 is 3x5 and therefore the lowest common multiple)? If so, the solution should look something like this, which avoids any modulo arithmetic:
With some special-case logic at the end for an incomplete sequence. Of course,
printfis a very expensive call. You could optimise this by computing it once for each digit length and then just doing decimal arithmetic in the string.And so on. There is a lot more optimisation that you can do with fizzbuzz. Knowing what the interviewer expects from the first attempt may not be obvious.
Except it’s not necessary… it’s not particularly hard to do the arithmetic to figure divisibility or keep a couple extra loop counters and solve it without knowing/using the operator while still not introducing any new concepts to the solution. You might have to give a candidate doing that a bit more time, perhaps. But if anything, seeing that solution and then offering an explanation of modulo w/ a chance to revise is likely to be an even better assessment.
The status quo being defended is akin to grading a student 100% on a final exam. I realize that’s still common in some countries, but it favors people who can cram (or who can pay for test prep) over actually relevant competency, skill, and curiosity.
Arguments that portfolio and take-home exercise reviews are too time consuming don’t wash. Most FAANGs and their imitators conduct batteries of multi-hour interviews, sometimes over several days. Arguments that they’re too subjective don’t wash, either. If the job is to use a language that has hundreds of built-in methods for manipulating ADTs to solve problems in another domain, what’s so objective about picking your favorite one and making someone re-implement it on the spot while you watch?
I don’t actually encounter many cheaters amongst the applicants I’ve interviewed over the years, but then again, I’ve never worked for a FAANG. I’m sure it can be overwhelming. But I know a lot of people who do work at FAANGs, and uncharitable though it may be for me to say, some of them have only ever learned enough to look competent and are disappointingly uncurious. You can actually catch cheaters and shallow learners if you stop cross examining your applicants. Open ended questions don’t leave much cover for them to hide behind.
Plenty of senior engineers, myself included, are really selective about takehomes because the time commitment is excessive.
The only time I’ve seen a take home done well it came with unit tests and harness code provided, and a bug to fix.
I love take-homes. They require a lot of time, but that can be at night or on the weekend when I don’t have many other responsibilities. I get a taste of what types of problems the company actually wants to solve instead of fictional algorithms questions. And best of all – lots of people won’t even attempt them, which gives me an advantage.
The problem with cheating is real, though. I don’t know how to solve that.
Selecting for candidates that don’t have “many other responsibilities” has a ton of biases employers probably don’t (or at least shouldn’t) want to take on - biases which are probably great for you! For example, it selects for younger people from wealthy, high-class families.
You also have to do a small coding exercise or have them talk you through their solution.
I would say that FAANG selects for this, since they want docile, compliant workers who won’t complain about the inflexible and slow work environment.
My experience has been the opposite and the people I’ve talked to prefer them.
It’s more convenient than asking a candidate to take PTO to do an entire day’s (5+ hours) of interviews, or spend days preparing on LeetCode. Being quick on your feet is also a skill, so it also evens the playing field because a lot of people just don’t interview well. It also gives you the opportunity to have you current programmers take the test to see what sort of solutions to expect, and learn the sort of things they might want to ask in the interview.
Candidates can work on their own time and at their own leisure, with whatever suite of tools are most comfortable to them. If someone doesn’t have a portfolio to show it provides you the opportunity to see what type of code they write.
“Hi, here’s a prompt of a little project to do, should take at most a couple of hours, you have a week to work on it at your leisure, send it back when your done and let us know if you have any questions.”
Our coding test takes maybe 2 hours, and I take a half an hour to maybe an hour to go through it. I’d rather spend that time on my own going through someone’s code than asking someone on-the-spot questions. It also gives me the basis of questions to ask the interviewee (why’d you do it this way? What tradeoff did you make here? Why didn’t you use this language feature here, etc.), which I give to them a few days before the interview so they can do their own research, like you would normally on a project. Since they have time to prepare, you don’t need to shy from asking detailed questions.
You copy the returned projects into a folder for analysis against other solutions for plagiarism. When I start asking detailed questions from the code review, it becomes apparent who knows what they’re talking about, since especially when given the questions beforehand, people can see opportunities to delve deep and show off what they know.
I definitely agree with this. Having to take a PTO to do a virtual on-site interview is exhausting and the preparation is (LeetCode etc.) takes a lot of time after work and on the weekends to do.
There has been cases of false positives with automated code plagiarism services [1], but asking them questions about their code should out the ones who cheated and like you said shows if they know what they’re talking about.
My current employer required an actual work sample take home test and I really enjoyed it instead of the typical LeetCode style questions.There was a few solutions on Google, but it seemed they randomized some things to catch cheating and I still had to do an interview with the hiring manager. MIT[2], Caltech[3], and Stanford[4] had mini courses for preparation for these type of interviews.
Red Balloon Security (RBS) had an interesting interview process. They first had a screening challenge which answers could be found via a Google Search, but the second one required extracting a secret Bitcoin wallet private key from a hard drive with a rootkit installed. The time limit for the hard drive challenge was a week, but the Bitcoin wallet had $3k worth of Bitcoin at the time. An additional interview was required before an offer I think and a few people on glassdoor complained about being ghosted after completing it, but it seemed like an interesting interview challenge and offered an incentive to complete it.
I suppose that’s an issue, but for the RBS one it was more of a CTF challenge instead of writing up a detailed report etc. So it seemed as long as you found the flag you would move on the next round (I didn’t complete the hard drive challenge for RBS). For my current employer I had to write a detailed write up, but was told to only spend an hour max. I imagine that it took someone maybe 15-20 minutes to review my 5 page write up PDF, but I could be wrong.
It feels like a big assertion to say that there are tons of people faking coding (but could otherwise pass a software interview). Are there any actual numbers that back that up?
I only have one quibble with this. I really dislike the coding interviews that dig into low-level coding of data structures when I’m applying for a senior web developer position. I don’t know how to reverse each word of a string in place. I’m so used to doing ruby’s
Array#eachor other languages’ equivalents,that I need to look up the syntax for a Cforloop.Note, my ruby version of reverse the words in a string would be
str.split(' ').map!(&:reverse).join(' ')and this would be faster than any pure ruby implementation.I think it’s easy to do a basic verbal interview first to see if there is a possible fit and if it seems so to have them do actual work work with the team. That way both sides actually find out whether they are a good fit or not. Of course that can also be done remotely. If one things it requires more time one can can consider to do it in some form of trial time.
I think in IT we are lucky to be able to do that. It’s really low risk and low effort to do that. A lot of other fields don’t really have that possibility.
But no matter what one does for interviews it should be something that a person actually does at the company. Don’t make people solve university style algorithm questions, if their job description isn’t “solves university style algorithm questions all day”. You need people being good at algorithms? Well, great, let them solve that problem. I am sure you have a task in the backlog that requires that, since you feel the need to hire such a person.
I think it’s bad for the company and rude to the person being interviewed to give completely wrong impressions of the company and their work. Creating a contract based on that seems like setting things up for failure. I was lucky, because I never really had to deal with that, but when I hear such stories I always imagine myself in such a situation and politely pointing out that based on the interviewing process (which after all is usually the only information I can judge from) I don’t think there will be a match and that therefor I don’t want to waste their time.
So basically what I am trying to say is that the structure doesn’t seem to matter that much. And also that pure algorithmic knowledge of an employee also isn’t really the thing one wants to/should focus on. There’s also other factors that aren’t so easy to measure that make an applicant a fit or not. Nobody wants to sign a contract with a person that is on the human side disliked by the team or simply will hate their job. In such a situation the company will lose value and the applicant gets set up for burnout.
Coding interviews are necessary because there are too many non-coders out there pretending to be coders. You need to make the candidates write code.
Having said that, 1-hour coding interviews in the style of “show me that you can fetch and show some data from a public API” is probably the right size, as it shows enough of the day to day practices. Anything beyond that (especially take-home exercises) is stretching it.
I have run many, many interviews at multiple companies. I have yet to encounter the mythical “non-coder” trying to trick the company into hiring them. As far as I can tell, the whole idea is based on a literal vicious cycle where anyone who fails an interview is deemed to be utterly unable to do any coding, and that’s used as justification for ratcheting up the difficulty level, which results in more people failing and being deemed unable to code, which justifies ratcheting up the difficulty…
And that’s not just my personal opinion/anecdote: this online interviewing platform has a decent sample size and says:
In the interviews they didn’t do well in, they probably were labeled as unqualified, incapable, etc. – but in the rest of their interviewers they were top performers.
Lucky you. I haven’t run that many interviews, and so far I can remember a few cases:
Sure, some of these can work as programmers if you are willing to lower enough the requirements. But without a technical interview where they get to write some code, or explain some code they wrote in the past, you wouln’t find out.
And yes, I also have hired people without degrees that were able to demo me something they built. I had to teach them later working in teams, agile, git, github and other stuff, but they still did good nodejs and mongo.
It’s very hard to completely bomb an interview if you can write some code. You can write wrong abstractions, bad syntax, no tests, and that is still so much better than not being able to open an IDE, or not being able to open a terminal and type
ls.This doesn’t seem degree related, since degree work won’t teach you working in teams, agile, git, or github
I’ve seen it happen; I’ve worked with someone who managed to work at a company for several years, even though it was an open secret that they couldn’t code without pairing (he was tolerated because he was likeable, and took care of a lot of the non-programming related chores)
I’ve also seen people strut into an interview with great confidence, only to write code that didn’t remotely resemble the syntax of the language they were supposedly proficient in. If this was due to anxiety, they certainly didn’t show it.*)
I don’t think it’s common enough to warrant all the anxiety about “fake programmers”, but it’s not completely imaginary.
*) I live in a country where software shops are among the few white-collar employers that’ll happily accept people who don’t speak the native language. That might mean we get more applicants for whom software development wasn’t exactly their life’s calling.
I would hope that after years of pairing they started being able to code… Otherwise I blame their pairing partners.
I don’t know how the situation arose, but by the time I got there, he was already known as the guy who did the “chores”, and avoided pairing to - presumably - reduce the chance of being found out. This all changed when new management came in, which implemented some - very overdue - changes.
Maybe the takeaway should be that there might be room for “non-coders” in development teams, but the situation as it was can’t have been easy on anybody.
I can’t speak for all jurisdictions, but this feels a little overblown to me, at least in the Australian market.
If someone manages to sneak through and it turns out they have no experience and can’t do the job, you can dismiss them within the 6 month probation period.
Yes, you would have likely wasted some time and money, but it shouldn’t be significant if this is a disproportionately small number of candidates.
The thing is the culture in the US is quite different- some companies will fire people willy nilly, which is bad for morale, and other companies are very hesitant to fire anyone for incompetence, because by firing someone you leave them and their family without access to health care (and it’s bad for morale). Either way, you do actually want to make good decisions most of the time when hiring.
Mostly because the employer is assuming to hire people to write ‘novels’, often mutli-volume novels (if I may use that analogy), but, instead, they test the candidates with ‘jeopardy-style’ questions.
take home coding is better, because ( a) it reduces the effects of already-wrong filtering, ( b ) it gives a an opportunity to the employer to see how candidate structures and expresses their thought, after they had time to think. Which is closer to the type of environment they will be in
What is still bad about take home interviews are the following:
does not elicit the candidate’s ability to structure a larger system (if you are hiring a person with 10+ years of experience, most likely you are looking for it)
does not elicit candidate’s interest areas and the progress they made on their own in those areas (eg self-study competencies)
does not allow the candidate to demonstrate their particular strengths (that may be well outside the particular coding experiment)
I get that all those additional qualities can be somehow learned by other means, but the coding question is always used as ‘gateway’. In my view if a take-home coding question is still used, it should be up to the candidate to decide at which stage of the interview to do it.
As I posted in some other replies in the similar topics. There is an ample space for a technology+social-sciences startup that could make hiring process much better (and not just for software-dev industry).
What we have today in this space is really really bad. It slows down progress in the industry by suffocating creativity and long-term vision in software products, by narrowing down employment to folks that can prepare well for tests, and that have good/quick thinking ability.
Good part about it, that the companies that practice the ‘code-within-X-minute’ style of exercises, are filtering out really great candidates that become available to other employers with much better interview hygiene.
Anyone run into a situation where you had a candidate just have someone completely do all their interviewing for them? With zoom interviews, it seems like in a corporate setting where maybe the people interviewing don’t end up working with the candidate, this may actually be happening?
Slight tangent but I honestly have so much fun studying for CS interviews – not the leetcode grind, but the brushing up on theory, data structures / algorithms, and just thinking about things at a more conceptual / academic level than I get to during my job.
Also, as an interviewee, I found that the type of interview questions / process a company is a useful way to gauge whether I’d like working there, since different types of processes select for different types of people, and are somewhat indicative of how well fleshed out the role I’m applying for actually is.