They are also very game-able, in that you don’t necessarily need to understand the why. If you do enough problems, it’s very likely that you will see similar problems in interviews.
If this is your only signal in an interview, what can you really hope to understand about the candidate’s performance on the job? Did they whip up an optimal answer on the fly, or are they just good at lying?
To be fair: I’ve done a couple of leet-code-style interviews, and just getting the right answer would never be enough. The whole point is to hear you talk through and explain, both the problem and why your solution works. Just knowing the solution would not be good enough. In some cases, what they really want is to hear how you explore and learn something you don’t initially understand, so getting the solution quickly isn’t the slam-dunk you might think. In fact, it could even work against you, if the interviewer suspects that you already knew the answer and didn’t disclose that fact up-front. (Of course, whether or not the way people deal with learning in a job interview environment is relevant to how they do it in a real-world environment is another question entirely….)
I didn’t accept the offer, but a while ago (at a FAANG) I got a question that I had seen in the past, but it had been long enough that I had to fumble a bit to the answer, which I knew.
One of the smarter people I know interviewed at a household-name “Big Tech” company a few years back and was given a graph problem to solve. Turned out to be one that has two standard algorithms, depending on what you’re optimizing for. The candidate produced one of those standard algorithms, but the interviewer — who was a software engineer at the company — had memorized the other one as the sole answer, and auto-flunked the candidate for not coming up with it, without even bothering to go through the code.
So it’s nice that you’ve have a couple experiences where people expected you to memorize a convincing patter on top of memorizing the standard answer to their standard question, but that does not make this good.
To be very clear: I explicitly did not say that this was good. I am merely commenting on the idea that this would commonly be “your only signal in an interview”.
(And just to be clear, my “couple experiences” are actually a consistent pattern observed over decades, on both sides of the interview table. The situation you describe in your anecdote would have been quite common 25 years ago, but I would say it has become less and less widespread in the industry since then, particularly in the last 12 years or so. Of course my experiences are only my own; I make no claim to have done rigorous studies.)
Honestly, my experience has gone the other way: interviews, and finding work in general, were much more pleasant in the early-mid 2000s and have just gotten nastier and nastier ever since. Some of this is probably due to the increase in people applying to programming jobs, and attempts to standardize and avoid some of the most obvious liability issues, but there seems to have been a sort of reinforcing vicious cycle that kicked in at some point where people took pride in having obnoxious interviews that flunked huge numbers of even qualified candidates, and then started competing with each other for the prestige of having, and passing, the nastiest interviews.
Ironically, a few years ago I interviewed at Netflix, which used to be known for a brutal technical interview, and found it wasn’t, because at least for the team I interviewed on they were using a much simpler and refreshingly practical approach (take-home exercise around a scaled-down version of a thing they actually had to do, on-site mostly revolved around talking about how to scale it back up to the real thing). So maybe the tide turned at least a little bit, and will start percolating outward from there, though from all I hear the other big FAANG/whatever-we-call-them-now places are still doing brutal algorithm challenge stuff.
Recently interviewed at the fruit company and was sorely disappointed but not surprised to get a dynamic programming question in the interview. I was of course unable to solve in the given timeframe because I couldn’t make the mental leap to find the relation between the smaller problem solution and the overall solution.
I really like the code review approach given in this article, but personally I couldn’t ever gain buy in at companies I’ve worked at to implement this approach.
However, at Segment we did recognize the ineffectiveness of leetcode style questions so we instead asked the candidates to build a Chat server instead. It’s a moderately difficult but solvable problem in 90 min, given constraints (which is another signal we’re looking for, scoping down the problem).
It certainly wasn’t a silver bullet, but I personally felt like we got a lot more signal from that question than any other I’ve seen proposed for interview questions.
But is that 90 minute problem a take home problem? If it is, I think from an interviewee perspective, I use the interview to determine information about the panel (and the company) as much as they are getting information about me, so long take homes can be asymmetrical. To be clear, I’m suggesting leet coding is the answer either.
Me too. Otherwise, I would like to discuss my public projects. Some interviewers have asked for that, I think it’s the best option if you have some publicly available code. It also makes it obvious how good or bad the candidate is through their coding idioms, design and engineering techniques.
For me, coding interviews are a big burden because they require a lot of prior preparation and they don’t mimic real working conditions, where pressure is certainly smaller. They actually bring back memories of all crazy exams at college, which were like Leet Code.
We were given a computer and we had to implement a new feature on top of our course project with some insane time pressure, usually 60 min. The new feature was usually designed to destroy your design, to find bugs or to discover poor algorithmic performance.
For example, the data structures course gave us a 10 MB XML file which described a gigantic bus and train transport network, with all sorts of edge cases. You could see all students panicking because their parser crashed or it had some inefficiency which made it dog slow. So they had to fix that and implement some new functionality. Impossible.
They discriminate against people who have higher stress level / stage fright.
For many jobs, this is a feature, not a bug. If a short challenge with a friendly interviewer leaves you frozen, you likely won’t do well when something goes wrong in production and people are counting on you to fix it, or depending on you to deliver something.
The big assumption implicit in your comment is that the person’s stress response will be triggered by the production failure in the same way as the interview. Often, this is not the case. For some people, being judged by folks that they don’t know, in a high-stakes environment (whether or not you get a job offer depends on the outcome), will trigger a lot more stress than trying to solve the kind of problem that they’re trained to solve and have practiced solving in prior jobs. Worse, for sociopaths, this is the exact opposite way around: they won’t be very stressed by the interview because they don’t value the opinions of their interviewers. This kind of process is likely to select for confidence, which is often inversely proportional to competence. If you’re hiring for a sales position or something similar where confidence is a key attribute for a successful individual then that’s great.
Use this if you want to hire developers to code urgent things in extremely tight deadlines (< 1h) in front of an audience.
But generally speaking, depending on the size and culture of the company, not every single developer you’re going to hire will even (need to) have production access.
Also, there will also be people who are a bit shy initially, or unfamiliar with the question’s particular algorithm or such. This doesn’t mean that after some onboarding period they wouldn’t be fully confident manipulating and debugging the code they wrote themselves, even under high pressure.
At my employer we don’t interview to solve problems. We interview to have a conversation about a problem. We expect some code to be written but don’t like the idea of “you have 60-90 mins to solve this tricky puzzle”. It isn’t a real world situation. So instead our process is not about solving anything but vetting if the person can walk and talk. Solving the problem is just a way to show-off.
Everyone will crap their pants on a timed puzzle and then you only end up hiring people who show off. I’d rather hire someone willing to learn hands-on and demonstrate said learning and understanding.
We interview to have a conversation about a problem.
Similar to what I prefer. Ideally we can have a civilized chat about various programming languages, and talk a bit about what makes them special and their pros and cons.
If I hire for, say, a role which happens to be mostly Python, then it’s a very bad signal if you only ever seen python, and if you can’t say anything bad about it. I guess I’m mostly trying to figure out how much context you have, and how much interest you have in programming in general.
Absolutely. I can tell how involved you are and your experience level from your discussion of the language. Half the “experienced” devs I interview can’t tell me what an Array.prototype.reduce does or write a Promise wrapper. Yet they love Javascript and have been working in it for X years?
I do a lot of work that articles about hiring claim nobody ever does. The most recent project I shipped at work involved writing a highly domain optimized dynamic programming algorithm, along with a few data structures that would probably be considered at least somewhat “leetcode” like (domain optimized variants of suffix trees and BK trees, and a couple of linked list variants designed to exploit caching better for our access patterns).
Even so, I’m not a fan of leetcode questions and asking these sorts of things during interviews. In a world before leetcode, I wouldn’t expect an average developer to go into a question like this well prepared even if they were strong with CS fundamentals, because so much of this naturally arises from spending time with the problem. CS is big enough that if you’ve been working in a different class of problem for a while, you’re just not going to immediately make the right connections and insights. At best you end up testing how similar the most recent work someone was doing is to the question you asked- rather than understanding their general competence.
Now that we have leetcode the problem is different: the “just grind leetcode” culture removes whatever small signal there might have been in talking through CS problems and replaces them with rote memorization in a form that most people will fail to generalize to real world problems.
I think CS is important for developers to know (whether they have a CS degree or not), but leetcode just isn’t the way to evaluate that.
Partly it’s that the sort of algorithm-challenge stuff involved in typical interviews is genuinely really rare to need to do as a constant day-to-day thing, so making that the default skill set to test for is not great. If you do need to do it regularly, OK, but that doesn’t make it true for the rest of us.
Partly it’s that even places like Google have long acknowledged that doing well on these types of timed/competitive algorithm challenges negatively correlates with on-the-job performance, so it’s not even providing a useful signal.
But mostly it’s that the conditions under which such things are posed are horrific and unrealistic. When you need to roll out a dynamic programming solution, for example, I highly doubt that your employer locks you in a room with no access to a proper editor/IDE, documentation, or other resources, and demands you do it syntactically perfectly, on the first try, on a whiteboard, within 20 minutes, while explaining every step of it to a hostile interlocutor who keeps butting in with questions, or else you’ll be fired on the spot.
But that’s the way the interview is run. Can you at least grant that maybe this is not good?
(and to be clear, that’s without getting into the other massive issue, which is whether these algorithm challenges really are synonymous with or usefully test “CS” or “CS knowledge” or “CS fundamentals” of any sort)
Just to be clear, I agree with you. I was saying that I do a lot of this kind of algorithmic development in my day to day work and I still think leetcode is a bad idea. I think general CS knowledge is good, but quizzing people on implementing algorithms in an interview isn’t. I’d much rather talk through concepts at a high level and learn about their interests and how they’ve been able to apply ideas to their work, and maybe have them teach me about something they think is interesting or useful.
I’m wary of “general CS knowledge” or anything really that emphasizes “CS”, simply because a lot of the current horror started with FizzBuzz, which was invented because of how many people had strong “CS” but couldn’t actually write practical code.
To some extent that’s the holy war about CS versus software engineering and whether to just teach a bunch of theory or try to do more applied stuff and practical techniques. But also as someone who came late to any sort of proper “CS” knowledge — mostly because I knew someone who was TA’ing university courses and became their study buddy to make sure they knew the material — I have to say that across a long career I can’t really think of a time when that knowledge and only that knowledge helped or would have helped. It’s a bit like the people who insist that you have to take a bunch of math and generally certain very specific courses like linear algebra, because what it boils down to is that math was how they learned abstract reasoning, so they decide it’s the only way anyone can ever learn abstract reasoning.
I recently interviewed for a new role and I really liked how they structured their technical interview: 2 hours for a task, with a short introduction over a video call, then independent work with another check-in halfway through. Finally, half an hour’s call to talk through your solution.
It was a good balance of having space to think about the problem without someone breathing down your neck, while still having an interviewer available for questions and discussion.
And I’m surprised that despite Computer Modern being associated with TeX, whose goal is to produce great typography, this blog makes the typographical error of using justified text without hyphenation. This creates distractingly large spaces between words on small screens – I had to read the page in Reader View.
They are also very game-able, in that you don’t necessarily need to understand the why. If you do enough problems, it’s very likely that you will see similar problems in interviews.
If this is your only signal in an interview, what can you really hope to understand about the candidate’s performance on the job? Did they whip up an optimal answer on the fly, or are they just good at lying?
To be fair: I’ve done a couple of leet-code-style interviews, and just getting the right answer would never be enough. The whole point is to hear you talk through and explain, both the problem and why your solution works. Just knowing the solution would not be good enough. In some cases, what they really want is to hear how you explore and learn something you don’t initially understand, so getting the solution quickly isn’t the slam-dunk you might think. In fact, it could even work against you, if the interviewer suspects that you already knew the answer and didn’t disclose that fact up-front. (Of course, whether or not the way people deal with learning in a job interview environment is relevant to how they do it in a real-world environment is another question entirely….)
I didn’t accept the offer, but a while ago (at a FAANG) I got a question that I had seen in the past, but it had been long enough that I had to fumble a bit to the answer, which I knew.
One of the smarter people I know interviewed at a household-name “Big Tech” company a few years back and was given a graph problem to solve. Turned out to be one that has two standard algorithms, depending on what you’re optimizing for. The candidate produced one of those standard algorithms, but the interviewer — who was a software engineer at the company — had memorized the other one as the sole answer, and auto-flunked the candidate for not coming up with it, without even bothering to go through the code.
So it’s nice that you’ve have a couple experiences where people expected you to memorize a convincing patter on top of memorizing the standard answer to their standard question, but that does not make this good.
To be very clear: I explicitly did not say that this was good. I am merely commenting on the idea that this would commonly be “your only signal in an interview”.
(And just to be clear, my “couple experiences” are actually a consistent pattern observed over decades, on both sides of the interview table. The situation you describe in your anecdote would have been quite common 25 years ago, but I would say it has become less and less widespread in the industry since then, particularly in the last 12 years or so. Of course my experiences are only my own; I make no claim to have done rigorous studies.)
Honestly, my experience has gone the other way: interviews, and finding work in general, were much more pleasant in the early-mid 2000s and have just gotten nastier and nastier ever since. Some of this is probably due to the increase in people applying to programming jobs, and attempts to standardize and avoid some of the most obvious liability issues, but there seems to have been a sort of reinforcing vicious cycle that kicked in at some point where people took pride in having obnoxious interviews that flunked huge numbers of even qualified candidates, and then started competing with each other for the prestige of having, and passing, the nastiest interviews.
Ironically, a few years ago I interviewed at Netflix, which used to be known for a brutal technical interview, and found it wasn’t, because at least for the team I interviewed on they were using a much simpler and refreshingly practical approach (take-home exercise around a scaled-down version of a thing they actually had to do, on-site mostly revolved around talking about how to scale it back up to the real thing). So maybe the tide turned at least a little bit, and will start percolating outward from there, though from all I hear the other big FAANG/whatever-we-call-them-now places are still doing brutal algorithm challenge stuff.
Recently interviewed at the fruit company and was sorely disappointed but not surprised to get a dynamic programming question in the interview. I was of course unable to solve in the given timeframe because I couldn’t make the mental leap to find the relation between the smaller problem solution and the overall solution.
I really like the code review approach given in this article, but personally I couldn’t ever gain buy in at companies I’ve worked at to implement this approach.
However, at Segment we did recognize the ineffectiveness of leetcode style questions so we instead asked the candidates to build a Chat server instead. It’s a moderately difficult but solvable problem in 90 min, given constraints (which is another signal we’re looking for, scoping down the problem).
It certainly wasn’t a silver bullet, but I personally felt like we got a lot more signal from that question than any other I’ve seen proposed for interview questions.
But is that 90 minute problem a take home problem? If it is, I think from an interviewee perspective, I use the interview to determine information about the panel (and the company) as much as they are getting information about me, so long take homes can be asymmetrical. To be clear, I’m suggesting leet coding is the answer either.
It wasn’t a take home problem, no. We did offer lots of time for them to get to know the interviewers and ask questions though.
I actually like take home’s as a candidate. As long as it’s not egregiously long, but then again I can just not do it.
Me too. Otherwise, I would like to discuss my public projects. Some interviewers have asked for that, I think it’s the best option if you have some publicly available code. It also makes it obvious how good or bad the candidate is through their coding idioms, design and engineering techniques.
For me, coding interviews are a big burden because they require a lot of prior preparation and they don’t mimic real working conditions, where pressure is certainly smaller. They actually bring back memories of all crazy exams at college, which were like Leet Code.
We were given a computer and we had to implement a new feature on top of our course project with some insane time pressure, usually 60 min. The new feature was usually designed to destroy your design, to find bugs or to discover poor algorithmic performance.
For example, the data structures course gave us a 10 MB XML file which described a gigantic bus and train transport network, with all sorts of edge cases. You could see all students panicking because their parser crashed or it had some inefficiency which made it dog slow. So they had to fix that and implement some new functionality. Impossible.
For many jobs, this is a feature, not a bug. If a short challenge with a friendly interviewer leaves you frozen, you likely won’t do well when something goes wrong in production and people are counting on you to fix it, or depending on you to deliver something.
The big assumption implicit in your comment is that the person’s stress response will be triggered by the production failure in the same way as the interview. Often, this is not the case. For some people, being judged by folks that they don’t know, in a high-stakes environment (whether or not you get a job offer depends on the outcome), will trigger a lot more stress than trying to solve the kind of problem that they’re trained to solve and have practiced solving in prior jobs. Worse, for sociopaths, this is the exact opposite way around: they won’t be very stressed by the interview because they don’t value the opinions of their interviewers. This kind of process is likely to select for confidence, which is often inversely proportional to competence. If you’re hiring for a sales position or something similar where confidence is a key attribute for a successful individual then that’s great.
I have a bit of social anxiety, so doing anything (job interviews, parties) around new people is hard.
On the other hand, I’m really good at dealing with production fires or other things which produce a positive but manageable stress response.
The OP is saying that though:
But generally speaking, depending on the size and culture of the company, not every single developer you’re going to hire will even (need to) have production access.
Also, there will also be people who are a bit shy initially, or unfamiliar with the question’s particular algorithm or such. This doesn’t mean that after some onboarding period they wouldn’t be fully confident manipulating and debugging the code they wrote themselves, even under high pressure.
At my employer we don’t interview to solve problems. We interview to have a conversation about a problem. We expect some code to be written but don’t like the idea of “you have 60-90 mins to solve this tricky puzzle”. It isn’t a real world situation. So instead our process is not about solving anything but vetting if the person can walk and talk. Solving the problem is just a way to show-off.
Everyone will crap their pants on a timed puzzle and then you only end up hiring people who show off. I’d rather hire someone willing to learn hands-on and demonstrate said learning and understanding.
Similar to what I prefer. Ideally we can have a civilized chat about various programming languages, and talk a bit about what makes them special and their pros and cons. If I hire for, say, a role which happens to be mostly Python, then it’s a very bad signal if you only ever seen python, and if you can’t say anything bad about it. I guess I’m mostly trying to figure out how much context you have, and how much interest you have in programming in general.
Absolutely. I can tell how involved you are and your experience level from your discussion of the language. Half the “experienced” devs I interview can’t tell me what an Array.prototype.reduce does or write a Promise wrapper. Yet they love Javascript and have been working in it for X years?
I do a lot of work that articles about hiring claim nobody ever does. The most recent project I shipped at work involved writing a highly domain optimized dynamic programming algorithm, along with a few data structures that would probably be considered at least somewhat “leetcode” like (domain optimized variants of suffix trees and BK trees, and a couple of linked list variants designed to exploit caching better for our access patterns).
Even so, I’m not a fan of leetcode questions and asking these sorts of things during interviews. In a world before leetcode, I wouldn’t expect an average developer to go into a question like this well prepared even if they were strong with CS fundamentals, because so much of this naturally arises from spending time with the problem. CS is big enough that if you’ve been working in a different class of problem for a while, you’re just not going to immediately make the right connections and insights. At best you end up testing how similar the most recent work someone was doing is to the question you asked- rather than understanding their general competence.
Now that we have leetcode the problem is different: the “just grind leetcode” culture removes whatever small signal there might have been in talking through CS problems and replaces them with rote memorization in a form that most people will fail to generalize to real world problems.
I think CS is important for developers to know (whether they have a CS degree or not), but leetcode just isn’t the way to evaluate that.
Partly it’s that the sort of algorithm-challenge stuff involved in typical interviews is genuinely really rare to need to do as a constant day-to-day thing, so making that the default skill set to test for is not great. If you do need to do it regularly, OK, but that doesn’t make it true for the rest of us.
Partly it’s that even places like Google have long acknowledged that doing well on these types of timed/competitive algorithm challenges negatively correlates with on-the-job performance, so it’s not even providing a useful signal.
But mostly it’s that the conditions under which such things are posed are horrific and unrealistic. When you need to roll out a dynamic programming solution, for example, I highly doubt that your employer locks you in a room with no access to a proper editor/IDE, documentation, or other resources, and demands you do it syntactically perfectly, on the first try, on a whiteboard, within 20 minutes, while explaining every step of it to a hostile interlocutor who keeps butting in with questions, or else you’ll be fired on the spot.
But that’s the way the interview is run. Can you at least grant that maybe this is not good?
(and to be clear, that’s without getting into the other massive issue, which is whether these algorithm challenges really are synonymous with or usefully test “CS” or “CS knowledge” or “CS fundamentals” of any sort)
Just to be clear, I agree with you. I was saying that I do a lot of this kind of algorithmic development in my day to day work and I still think leetcode is a bad idea. I think general CS knowledge is good, but quizzing people on implementing algorithms in an interview isn’t. I’d much rather talk through concepts at a high level and learn about their interests and how they’ve been able to apply ideas to their work, and maybe have them teach me about something they think is interesting or useful.
I’m wary of “general CS knowledge” or anything really that emphasizes “CS”, simply because a lot of the current horror started with FizzBuzz, which was invented because of how many people had strong “CS” but couldn’t actually write practical code.
To some extent that’s the holy war about CS versus software engineering and whether to just teach a bunch of theory or try to do more applied stuff and practical techniques. But also as someone who came late to any sort of proper “CS” knowledge — mostly because I knew someone who was TA’ing university courses and became their study buddy to make sure they knew the material — I have to say that across a long career I can’t really think of a time when that knowledge and only that knowledge helped or would have helped. It’s a bit like the people who insist that you have to take a bunch of math and generally certain very specific courses like linear algebra, because what it boils down to is that math was how they learned abstract reasoning, so they decide it’s the only way anyone can ever learn abstract reasoning.
I recently interviewed for a new role and I really liked how they structured their technical interview: 2 hours for a task, with a short introduction over a video call, then independent work with another check-in halfway through. Finally, half an hour’s call to talk through your solution.
It was a good balance of having space to think about the problem without someone breathing down your neck, while still having an interviewer available for questions and discussion.
Off-topic, but I can’t believe someone liked the computer modern fonts enough to make their blog use them.
And I’m surprised that despite Computer Modern being associated with TeX, whose goal is to produce great typography, this blog makes the typographical error of using justified text without hyphenation. This creates distractingly large spaces between words on small screens – I had to read the page in Reader View.
The simplest explanation is simply that believing TeX design is “good” is cargo culting to begin with. :)