1. 26

  2. 19

    Meh. I’m all for ideas but I’m sceptical. First, I think there are lots of developers for whom PRs actually aren’t a big part (or a part at all) of their job, and judging people on something they never got in the habit of doing is unlikely to bring out the best of them.

    Second, what’s a good/bad practice, or good/bad style, etc. are pretty much religion at this point (unless the focus were strictly on pedantic language mastery). I’ve seen wildly divergent cultures and best practices. Do you lose points for having the wrong religion? Do you lose points if the interviewer feels like you’re nagging & bikeshedding? What if you come from a culture where nagging & bikeshedding is avoided and devs prefer not to step on each others’ toes too much? Even things like pointing out the lack of tests is a matter of culture. And how persuasive do you want to be? Do you have to be? It is context sensitive. And there will be bias one way or another due to the language you use. People lose points because they didn’t know the secret rules of the game. Which isn’t much better than not having roughly memorized the algo they were gonna whiteboard.

    “Haha, you didn’t know what what answers we wanted. Better luck next time!”

    Honestly I think a “debug and fix this, and explain the problem & solution in a PR” kind of interview problem would be much nicer. It tests the hard skills you need to write and maintain software, without so much emphasis on religion (unless they have prejudice about how you do the work).

    1. 2

      Strong agreement. When I onboard someone, I try to give them a rough first PR to inculculate the local culture.

      It’s an innovative idea for an interview though, to be honest. But probably unbelievably biased in general.

      1. 2

        I try to give them a rough first PR to inculculate the local culture.

        I would personally just think “this guy is such an a***ole. Instead of having clear guidelines on what’s desired out of my contributions, having someone nitpicking every details just to show “how we do it here” is very negative to me.

        1. 1

          Part a - The problem is culture isn’t something you can encode in law/rules/guidelines. Culture is much broader, it occurs at the joints of definitions, shadows the institution of policies, and is broader. We can have style guides, but culture dominates. So it’s also a

          Part b - I am totally up front: “Welcome to the company, I’m going to nit at each code deviation from the local style. We both know that its just the local argot, but you need to know this” - Note that it’s not just style, but how we interact, and getting bathed initially in the local Way eases the experience later on. It’s not a exercise in snottery.

    2. 8

      I think there’s a lot that’s appealing about this approach, and I might even think it’s better than a traditional interview. However, here are some limitations I see:

      1. It’s true that PRs are a big part of an engineer’s job, but so is writing code, and you’re giving that up. Obvious, but worth pointing out, imho. Of course there’s some correlation, but that’s one that runs in both directions. Perhaps it’s true that giving a good code review is a better indicator of code writing ability than vice versa, but I’m not sure.
      2. Validity: while you make good points about objectivity/rubrics, you’re still viewing a limited slice of the candidate’s ability. I’d argue it might be more limited than a code test. Code review culture differs a lot by organization, and writing code might transfer more. I think it’s easier to tell someone “your implementation should/should not be optimized for performance/pay special attention to edge cases” in a code test than to tell them about the assumed code review culture.
      3. This prioritizes non-verbal communication over verbal. It also selects for people who do well at composing everything ahead of time, rather than reading their audience/responding to feedback, etc.
      1. 1

        I think the “make an addition” is meant to imply that the interviewee writes code that will work within the code base.

        1. 2

          When I read the post, I was thinking that sounded like a very small change, but I don’t know if that’s actually implied.

          1. 1

            You can also ask the person to make a PR to the codebase, and ask the team to add comments and see how the person reacts. That would also help to see how some people react to PR comments.

        2. 6

          I know some people who have actually been trying almost exactly this for the past 6 months. It does work, although it requires the evaluators to be trained carefully.

          1. 4

            At what stage in the interview process do you have this in mind? If it’s late-ish, seems plausible. If it’s early-ish, seems like a lot to ask up front from a candidate to spend a day grokking a codebase when they’re still at the point where they might be summarily rejected after 10 minutes’ review. You do mention that some people might not have the time, but even for those who do, is it a good use of their time?

            The academic-job version of this is a university wanting your initial application for a faculty position to come with a custom-made syllabus for one of their new courses, or a review of their degree program with suggested revisions, or something of that kind. (Distinct from asking you to send in an example syllabus of a course you might teach or have taught in the past, which isn’t custom, employer-specific work.) I usually pass on applying to those. I am happy to prep custom material if I made it to the shortlist and the potential employer shows they’re serious enough about me as a candidate to fly me out for an on-site interview, though. Then it seems fair and more likely to be time not wasted.

            1. 4

              I figured this would replace the in person technical interviews. So for a company, the recruiting flow might be

              1. Initial Phone Interview
              2. Maybe a fast phone technical filter interview
              3. Give them the project, tell them to make a change (and that they’ll be reviewing a couple of PRs for the onsite)
              4. On site: culture fit, discuss their change to the codebase, have them do the code review
              5. Hire/no hire decision.
              1. 1

                For me, this would be a red flag - 2/3 interviews (including the absolute worst kind, the phone interview) and a day long coding test? Maybe if I was applying to be CTO of a Fortune 500 company. For a lowly developer, this is too much.

                1. 4

                  You’ve definitely been luckier with interviews than I have. Every company I’ve ever interviewed with had at least three rounds!

                  1. 1

                    My last two interviews were just one short onsite session each. Both led to an offer. I turned down companies with more involved & time consuming process.

            2. 2

              I personally prefer asking for code examples they’ve written (can be totally out of context) so that I can see how they write, comment, name variables, etc. After getting those, talking them out. Why did you do X, why not try Y? Additionally, asking them about their specific experience listed on their resume seems to draw out interesting details (or shows that they’re BSing).

              I don’t find the tests themselves very helpful. There have been zero times in my career where someone has said “do this task in an hour and I’m going to sit here with you asking you about what you’re doing as you do it.” So why do I want to hire for that trait? It doesn’t translate to day-to-day. You’re going to hit lulls in work - how do you handle that? How do you motivate yourself to do work that sucks? Those are the things I want to talk about and learn.

              1. 2

                I think it’s odd to list “interviewer time” as an issue with the current system and then offload a day of work onto the interviewee as if the current process isn’t expensive enough from that angle.

                1. 2

                  This adds nothing over using rubrics for any exercise.

                  Really there are two parts to this proposal:

                  1. Use rubrics and other standardization mechanisms;
                  2. Use a task that is more like a work task.

                  Both of these are good. It is in no way responsive to discovering all the information about how it would be to work with a candidate, any more than white boarding does. Code production tasks test aspects of code production; code review tasks test communication and understanding skills.

                  Accordingly, I think adding such a task in place of one of the produce code assessments is a good idea, but there’s nothing uniquely standardized about a code review task.

                  1. 4

                    There is one big difference: with a whiteboarding exercise, a lot of the hire/no hire decision is based on the interviewer’s memory and notes. That’s stuff like “how much prodding did the candidate need” or “how much did they communicate their thought process”. With a code review exercise, all of the decision information is available to everyone.

                    1. 4

                      That’s not inherent to whiteboarding. Structured grading and video recording can be part of the standardized assessment.

                      Your proposal mixes together using some best practices with using a task that makes it easier to implement those best practices, because it occurs in a durable medium, but that also collects almost completely different information.

                      The actual task is also only suitable for experienced engineers - being good at code reviews is an important part of the senior role for many reasons, and is something inexperienced engineers will never be good at.

                      Whiteboard coding tests code production, which happens at all levels, but more so at the junior level. Neither are actually complete assessments of engineering skill, but to replace one with the other is to throw the baby out with the bath water.