1. 14

  2. 5

    Shouldn’t an experiment have a “control” group?

    3 hour test is a big investment just to find out I might not like working there.

    1. 4

      Author of the post here. Yeah, we’ve worried that 3 hours upfront might turn away some people. But when you think about it, when you sign up for a phone interview, you’re blocking off an hour of time on your schedule anyway and you probably won’t know for sure that you want to work somewhere until you go onsite. Plus, reading the programming challenge instructions takes about 2 minutes and you can decide right away if you don’t want to do it. A lot of people who’ve done the challenge have told us that they wanted to do it because it’s just an interesting problem.

      A control group is difficult when you’ve got limited bandwidth. What we’re comparing against is our own past experiences interviewing and being interviewed. You’re right, it’s not an experiment in the scientific sense; it’s just a thing we’re trying out to see if it goes better, and so far, it’s been going very well.

      1. 2

        Well, thanks for sharing what you’re company is doing. Hopefully, it keeps working for you.

        How much time did you spend developing the test?

        1. 1

          We spent about 5-6 hours writing it up, doing the task ourselves, and then updating it. It didn’t take too long to come up with the idea, because we adapted a cool problem we came across developing the site.

    2. 4

      Sorry, no. I don’t do 24-hour homework assignments on the chance that I might get an interview.

      I used to send companies like the above a quote for my services when asked to perform such things (much to their chagrine), but sadly I have more important things to do these days.

      1. 2

        The fallacy here is thinking that all jobs are the same. The reason you want to devote a fair bit of time (e.g. 3 days) to figuring out whether you should work for a given company is that if you aren’t working the best job possible (which very well may be your own thing) you are mis-investing your time. So, while you should only do something like a 24-hour interview with a company that you’re already excited about, to veto such a processes a-priori is only to deny yourself information that is very relevant to the decision. That you happen to give them information about you during that process is incidental (from your perspective). Furthermore, should you be working for a company that does not put adequate effort into deciding who they hire?

      2. [Comment removed by author]

        1. 1

          Over-aggressive filtering is often counter-productive

          Absolutely agree on that point. We think having someone write real code is a better filter than asking them to write it on a whiteboard. I’ve had the displeasure of turning away candidates who I thought were awesome at previous jobs due to their inability to figure out some esoteric algorithmic trick to get the function they had 30 minutes to write down to O(n) time, and I’d like to avoid that in the future.

          This is why it pays to have a personal github account

          It absolutely does. In fact, we love people who make substantial contributions to open source, because it’s such a strong indicator of programming ability. It’s a very strong factor in our decision process, and for people who have many years of experience, only more so.

          tell me why I should work for you?

          1. Interesting technology: we need to develop great semantic analysis libraries for different programming languages (including dynamic ones), and a backend that indexes the entire universe of open source code at a semantic level (think function calls, type references, module imports, etc., not just regexes).
          2. Cool problem that affects your daily life: We want to use that information to help programmers learn how to choose between and use new libraries more quickly.
          3. The chance to do a lot of your work open source. Our core semantic analysis libraries are open source and we’ve aggressively open sourced a lot of other libraries as well: https://github.com/sourcegraph. And the opportunity to talk about all this at your favorite conferences.
          4. Small and dedicated team that’s incredibly passionate about working on something very near and dear to our hearts.
        2. 3

          “For prospective candidates, three hours might seem like a lot of time for an initial “interview”, but in reality, it saves time for everyone.”

          I may be a bit jaded here, but it seems that this tactic saves time mostly for the employer who no longer has to sift through the ‘chaff’, and makes the interviewee assume a large portion of the risk up front. As a potential candidate, I would find this worrying, and barring anything else that made the company truly stand out, I wouldn’t bother with the test. I am sure it is not the interviewers intent, but the technique comes across as being in bad faith.

          Also, for anyone with responsibilities (deadlines, teams to manage, children, etc), a “24 hour window” to work on a 3+ hour project is going to be a bit unrealistic.

          1. 1

            Thanks for the feedback. Regarding the 24 hour window, so far most of the candidates we’ve interviewed are in college or recent graduates. We’d definitely consider expanding the window for anyone who has a more demanding schedule. Our intent is to be more flexible than phone interviews by allowing someone to work around their schedule, instead of setting a specific time block.

            And we’d definitely hop on the phone with a candidate to tell them more about Sourcegraph up front. All the people who’ve done the challenge so far we met previously at a job fair or through a mutual contact. The post doesn’t do a good job of mentioning this, but we’re definitely not saying, “Here, before we even talk to you, you must do this 3 hour task.”

            I hope I can convince you that this experiment is not in bad faith. Having gone through a bjillion of these ourselves and sunk many hours into interviews that didn’t pan out, we wanted to try to make things better for all parties involved.