1. 25

Does it matter when they occur in the hiring process (i.e., whether they’re being used as a first-pass technical screen or whether they’re only being presented to candidates who’ve already passed a couple rounds of interviews)? Does it vary with the seniority of the position?

  1. 48

    IMHO, it absolutely matters when in the hiring process. It should be after screening. It should be for people on the short list, and it should be paid. Our project was done by in house people in about 4 hours. We send that home with candidates as the 2nd to final step of the process. When they arrive with the code, we give them a cashiers check for $500 and then we do a group code review. Worst case they made $500 and are made to feel foolish in a code review. Best case they have $500 and a new job. Additionally after the code review we give them a exact time for hire / no-hire decision call.

    1. 8

      For the curious, it turns out that this is just below the IRS’s $600 threshold which would require you to file a 1099 form for the payee. (They are still supposed to report the income, though.)

    2. 15

      Had an interview recently where the take-home was the first step after applying. I was given a week to finish what was meant to be a 4-8 hours small project. There was an optional step in the project description to add whatever extra features you might want. All of this was unpaid, but that wasn’t even the main issue. After successfully doing the take-home and going through 2 phone calls with both the CEO and CTO they decided they can’t hire any more junior folks.

      That whole process took a bit less than a month. Please don’t structure your interview like that. If you can’t hire me because I am a junior dev, don’t make me jump through so many hoops. Lesson learned.

      1. 10

        Avoid places that use assignments for screening. It’s very lopsided, they literally contribute zero effort in the interview process but might still make you jump through the hoops even if uninterested.

      2. 13

        I’ve known developers who simply won’t do interview processes if they are asked to start out with a take home technical screen.

        1. 6

          I used to accept take-home coding challenges but after my last experience I now flat-out refuse them. I have a CS degree, have been developing Ruby applications for 8 years now (all of this is documented in my LinkedIn profile) and I also have a couple of Github repos so you can see how my code looks like and I also have a job. If you still need a coding challenge to figure out if I’m a good fit for your company then this means that you have doubts about my skills so it’s better to say “no” from the get-go.

          During my last interview I was given a coding challenge right after the company approached me. Requirements were vague (they even specified that in the test) and it took me roughly 8 hours to finish it. They were unhappy with my implementation for various reasons that depended a lot on how you understood the vague requirements and I ended up being rejected.

          At the end of the day, if you hire me as a contractor, I’ll still have to work 30 days for you before I ever see a paycheck. You’ll have plenty of time to figure out my coding skills in that timeframe.

          1. 0

            It me! <3

          2. 13

            I have a simple answer: anywhere from 1 to 3 hours.

            Sure, some people think that it’s unfair and discriminates against busy people, etc. But for me by the time I’m interested in a company and a position, I’m willing to spend the evening on a small exercise where I can show, in a way that they can compare to other people, that I can write decent well tested code given a description of a problem. Frankly I’m usually more than happy to do a test like this because I know (as someone who’s been on both sides of the table) that the average applicant (after we’ve screened resumes and talked on the phone) will submit code that doesn’t compile and/or is racy and/or crashes while running it.

            I’ve done a couple of these tests for senior-level positions and I can say a code sample seems to be what all the teams I respect have done.

            1. [Comment removed by author]

              1. 11

                You don’t think your time and expertise is worth money?

                My time and expertise are worth whatever someone else is willing to pay. Sometimes that “someone else” is me. I spend oodles of my free time working on code that I give away for free. I probably spend more time working on code that I give away, per week, than I do on code that I’m paid to write.

                When I’ve done coding assignments for interviews, I’ve always enjoyed them, even if they took a few hours to do. I’d never expect monetary compensation. My compensation is the ability to move forward in that particular interview. If it’s worth it to me, then I’ll do the coding assignment, because that’s the requirement they’ve established. I can either decide “no, it’s not worth it and move on” or I can decide “yes, it is worth it.” And only I get to make that decision. Not you. Take your shaming somewhere else.

                You either do some throwaway code which never gets used and is a waste

                That’s bullshit. I’ve voluntarily written code that gets thrown away. Sometimes I do it because it’s a fun little task. Sometimes I do it because I want to learn something. Sometimes I do it because I don’t actually know that I’ll be throwing it away; my method just turns out to be wrong. What does it matter the reason? Whether it’s for fun or an interview, throwing away code is a perfectly fine thing to do.

                1. 5

                  You don’t think your time and expertise is worth money?

                  Sure, my time and expertise are worth money. But I get something worth much more than my hourly rate X 3 hours. I get to work at a place where I know that basic engineering skills are valued and my peers have a certain level of competence.

                  I will happily trade 3 hours of my life for the knowledge that the people I spend 40 hours/week with for the next several years can walk the walk.

                  I literally feel uncomfortable working for a company that doesn’t do a small coding test. Why? Because I am a decent bullshitter, who happens to also be a decent engineer. If they hire me based on what I’ve told them, I know there is a good chance they hire bullshitters. I’m in a minority these days, but I think that writing some software given some requirements under a deadline is actually a pretty reasonable facsimile of some aspects of day to day software development. As is explaining a concept to your peers. Possibly on gasp a whiteboard.

                  But let me back up a sec. I think it’s 100% OK to ask for money for doing the test. Personally, I don’t because frankly I take a job search really seriously and spend many many nights and weekends researching companies and my ratio of phone screens to job offers is pretty close to 100%, so I’m more than willing to invest 3 hours of my life in the process of figuring out if this company is going to be the place I’m going to work at for the next several years.

              2. 8

                It kinda depends on what you’re evaluating. Things to consider:

                • Are they allowed outside resources on the exam? If they aren’t, what are you going to do if they cheat?
                • Are they being evaluated on the “best” way to implement a solution? If they are, are you providing a testing/benchmarking framework to give them feedback as they work?
                • Are they being evaluated on relevant things? Doing linked-list manipulation in C for a JS frontend position is maybe not a great predictor of success (and yes, I’ve been on both ends of this.)
                • Are they allowed to ask for clarification as they go, after a brief reading period, or not at all?
                • Are they being giving “tricky” questions? If so, why?

                I’m not against take-home exams (though, these days I expect to be paid for my time when taking them or working on small test projects), but it’s really easy to gimp a candidate for no good reason.

                As an example, one of the best questions I ever got was a line of APL (this was for a Java-heavy job). I answered as best I could–which was “I don’t know this, but after looking here and here, here’s what I think it does”–and the person hiring me just laughed and said “I include that because I know nobody has seen it before…I want to see how people handle it.” My answer, being honest about my ignorance but still trying to solve it, seemed to be the correct one.

                1. 19

                  Are they allowed outside resources on the exam? If they aren’t, what are you going to do if they cheat?

                  The moment you start defining anything that’s reasonably within the day-to-day job you’re hiring for as “cheating,” you have failed to define a good test. As an applicant, I’ve seen take-homes that you’re required to complete with a program running on your computer that blocks all internet access for a few hours to make sure you don’t “cheat”, or “web IDEs” that disable copy/paste so that they can monitor your keystrokes and make sure you’re the one who typed it.

                  All of this is nonsense; from my (albeit relatively short) time hiring I’ve found that (a) cheating is not a significant issue and (b) the couple of times I’ve caught someone cheating, it was immediately, blindingly obvious when I talked to them about their solution for a few minutes. By making someone give up their normal tools (stack overflow, vim/emacs, whatever) you put them on their back foot, and they are going to perform much worse than they would if they could use their normal toolkit. Worse still, you turn away good applicants who know they don’t have to deal with that BS to find a job.

                  I know this isn’t exactly what OP was asking about: to that, I’d say that I’ve gotten relatively few complaints doing a 3-hour-ish coding problem in which you’re expected to write about 100 lines of code as a gate between phone screens and having them come to the office for an interview; you should already have a good feeling about the person if you’re going to ask them to do a bunch of work. I’ve never worked for a place that paid candidates to complete the challenge, but I think a couple would have been open to it if a candidate had asked, especially if they were a senior engineer.

                  1. 3

                    So, I didn’t mean for that to be interpreted as negatively as it seems to have.

                    It’s more of just being honest with yourself about the predictive power of the test. For example, if it’s a certain type of position that’s well-paying and you are using certain channels to get people (mass mailings, job boards, whatever) then you need to expect that some of your applicants are going to use external resources (“cheating”) to pass your tests.

                    That being the case, it’s important to understand both a) how important is it, really, if somebody uses outside resources and b) how would you tell if they did?

                    If, for some reason, it’s super important that they not “cheat” on the take-home, then maybe you should be considering what it is about your daily developer’s job that makes it unthinkable to use (“cheat” with) Stack Overflow, Wikipedia, and so forth.

                    If it’s super important that you be able to tell if they cheat–say, you are going to have them be a sales engineer in front of clients and they need to not look clueless after successfully gaming your system–then you need to maybe change your testing environment to instead be interactive over the phone, or to include a discussion section where you pair and review their code…if they have not done the work, that usually is pretty obvious.

                    Anyways, at the end of the day, focusing too much on “cheating” on a test puts you into a suspicious headspace and also may signal a candidate that you are going to be scrutinizing their every move. Better (in my opinion) to make it clear what external resources are allowed and to be clear with yourself about what you hope to accomplish–if it’s just trying to catch people in an interview being dishonest, well, that’s rather besides the point of seeing candidates!

                  2. 4

                    these days I expect to be paid for my time when taking them or working on small test projects

                    Have you been successful in asking for this at companies which didn’t offer to pay up-front? Or do you just withdraw from the process if they seem to expect you to do it unpaid?

                    One of the reasons I posted this question is because lately some of the test projects I’ve seen people ask for have been getting complex enough that I’d be reluctant to do that much work for free.

                    1. 4

                      I’m not sure that this will be much help, but the last folks I did this with I offered to do a small project for–a bit of technical writing and system diagramming and basically sitting down with their software lead and doing knowledge transfer. I pitched it as a “Well, I’m not available right now, but here’s something I’ve done with other teams in the past, would that be useful? Here’s what a day or two of that would run you.”

                      After doing that, they liked the work enough and wanted to see if I’d come out proper and do a small project, and agreed to do that at my normal rate, and if things went well it’d turn into an offer. Things went well and they were wonderful folks to work with even though I ended up accepting another position.

                      The next time I get a chance to do a project for an interview, I’ll happily explain my rates and we can figure something out. If they want you enough to do a project, they’re not going to balk at a few hours of contract time. Plus, it’ll give you a feel for how they act when money is involved–those folks earlier were impeccable and professional.

                      If you’re fresh out of school and hard-up for work, maybe consider a take-home test as something you don’t expect compensation for. But any project more complicated than a long lunch break (especially if it’s solving one of their business domain problems!) should never be done freely.

                  3. 7

                    It depends on the market. When the economy is rough I might well do a cage match to get a janitor position, but if you can afford to be picky, screw that.

                    What I see over the decades is gradual escalation of the interview process. What used to be a check of references/grade sheet and shop talk has degraded through IQ tests into whiteboard coding and then take away assignments. Sure I understand the undercurrents of market reality behind that, lots of unqualified people and posers. But being a part of collateral damage I just can’t be sympathetic.

                    If you apply some really interesting place and the company makes you do some homework, could still be worth to grind your teeth and do that, but you should still ask if your hours will be compensated. OTOH if approached by poachers and they expect some Eye of The Tiger performance it should be immediate ‘no’.

                    1. 7

                      We ask the final-stage candidates to do a <= 2hour coding task. (Devs on the team including myself complete it in 1-1.5 hours). I’d expect to hire approx 1-in-2 or 1-in-3 of the people we ask to do the task.

                      Previous interview stages have involved a brief phone screen (one person our end), followed by a longer (40min) phone interview (two people our end). We follow up the coding task with a face to face interview (1-2h), where discussing the code they have written is a significant part of the agenda.

                      I think it absolutely does matter where it comes in the process. Using it as an early filter is I think very disrespectful to the candidates time. But I find it does have value. It is possible to replace it with a review of their published work (e.g. github) but I find the fixed assessment has the following benefits:

                      • it is easier to compare across candidates
                      • it is easier to work out which code was “all their own work” on a project
                      • although it does take some time to do, I think it actually benefits people with less free time (since they don’t have a large portfolio of open source to refer to)

                      The coding task is on their own time, using their own tools and resources, writing code to speak to a demo HTTP service.

                      1. 7

                        Good developers will usually have lots of options for employers. So unless your company really stands out in terms of compensation or how interesting the work is, then good developers will just look for work elsewhere.

                        It’s my opinion that junior developers generally have more free time and are more interested in having their skills validated so they are happier to spend their time taking a test.

                        The worst possible challenge question: Solve this real problem that we have in our product. We will be using your solution to improve our product and we will not be paying you for it. Since we don’t know the solution to the problem ourselves, we have no idea what the scope of work required is. If you complete this challenge successfully, we will request another interview to discuss which of our open positions suit you best.

                        1. [Comment removed by author]

                          1. [Comment removed by author]

                            1. 0

                              Apparently their programmers showed up after work to back you up: you’re at 13 up now. :)

                          2. 9

                            it depends on how much they are going to pay me divided by what I consider to be a fair hourly rate for my contracting work. spec work should be shunned and shamed in every industry.

                            1. 4

                              Serious questions for everyone with opinions on this subject:

                              • What other industry is this a thing in?
                              • What makes this industry think that this is a reasonable approach to take when interviewing people?
                              • What protections do people have when doing interviews like this? (how do you know the company isn’t going to take the stuff you built, turn around and sell it, and not hire you?)
                              • What insights do people think they gain from the results of these take-home challenges?
                              1. 5

                                What protections do people have when doing interviews like this? (how do you know the company isn’t going to take the stuff you built, turn around and sell it, and not hire you?)

                                We use a toy model of a problem we had (and solved) a couple of years ago. Since it’s already part of our product and covered with layer after layer of complex business logic, there’s no way we’d be able to profit off their 1 hour solution.

                                What insights do people think they gain from the results of these take-home challenges?

                                • We can have a standardized question and rubric, so responses can be judged more objectively than technical phone interviews.
                                • We get some signals: did they read the instructions and spec? Does their solution match the sample output we provided? Did they include the tests we asked for?
                                • A lot of people say that it’s hard to interview and write code on the spot, so this allows people to work at their own pace. The project is scoped to about 1-1.5 hours, but they spend as long or as little on it as they want.
                                • Interviewees get insight into the technical challenges we have.
                                1. 3

                                  What other industry is this a thing in?

                                  Academia. Nevermind you getting paid to take a test; you have to pay to take the test. Similarly for actuarial sciences.

                                  Even if nobody else did it (or even if you think my examples are invalid for $reasons), does that mean nobody should ever do it? What room is there to experiment with different interview techniques?

                                  What makes this industry think that this is a reasonable approach to take when interviewing people?

                                  I don’t know how to ascribe opinions to an entire industry. But it seems reasonable as a way of doing a sanity check on a candidate’s coding skills if there is no other way.

                                  What protections do people have when doing interviews like this? (how do you know the company isn’t going to take the stuff you built, turn around and sell it, and not hire you?)

                                  I’m not aware of any protections other than the ability of the individual to decline the test.

                                  What insights do people think they gain from the results of these take-home challenges?

                                  • Depth of thought/analytical thinking/problem solving.
                                  • Coding style.
                                  • Ability to write an implementation that matches a specification.
                                  • Testing habits.
                                  • Craftsmanship.
                                  • If someone hands me some code, I can answer a very concrete question about it after review: Is this code that I would be happy to maintain?

                                  What insights do people think they are missing from not giving take-home challenges?

                                2. 4

                                  My previous employer used a code test that was very open-ended. This test was sent after a screening call. The minimum was the production of a very simple library to model a basic blog.

                                  1. A blog has posts with a title and labels.
                                  2. Posts have comments with a name and a title.

                                  If a candidate sent us something that looked like this, or the minimum equivalent in the language of their choice (we preferred to interview in Java and Scala but could find interviewers versed Python, Ruby, and C):

                                  class Blog(val posts: SortedSet[Post])
                                  class Post(val body: String, 
                                             val comments: SortedSet[Comment], 
                                             val labels: Set[String])
                                  class Comment(val name: String, 
                                                val body: String)

                                  Then they’d technically passed the test. Cheeky, but sufficient.

                                  However, we encouraged more: build system, tests, more intentional design decisions using mixins, use of inheritance, use of encapsulation, use of higher-order functions, etc. If they sent us the solution in the body of the email or as a Gist, we didn’t even try to compile it. Note that the above won’t compile, but it demonstrates intent: the Posts and Comments should be sorted somehow. Those and labels should be unique.

                                  We told takers explicitly not to spend more than 90 minutes on the exercise. Most of us who did it could get the basics done in 30 minutes with a build system and tests taking 30 minutes each (provided the taker didn’t do TDD).

                                  I’m in a new role now and largely intend to keep this basic filter in place. It’s a short enough exercise that the bare minimum to get in the door can be done in a lunch break, on the train, or banged out in an email body looking busy. People spend more time arguing on Facebook.

                                  However, I won’t send it out if I can find evidence of the candidate’s competency through Github, etc.

                                  1. 4

                                    My current employer (Heroku) has a pretty unique way of doing this. After interviews, which can include showing code samples and answering technical questions in a non-trivia fashion, we offer to bring in a candidate and have them work with our team and codebase for a day or two. We call it the “starter project” and it is very useful for both us and the candidate. They get to see our processes (we’re a distributed team but some developers work at HQ), get to know the team members and see our code quality.

                                    At the end of their project, they are asked to give a 30-45 minute presentation on what they worked on, what challenges they faced, and what they learned. We don’t expect them to ship a feature or even have things fully working. This is purely an evaluation of how they work and approach problems. I’ve done take home work, coding challenges, and whiteboarding in the past. This method has been by-far my favorite.

                                    It isn’t without its problems since it involves people taking off a day or two of work and they sometimes have to be brought on-site (all expenses paid and we’re flexible if they can’t travel). Thankfully, it all occurs during the work day and candidates can use their own machine with a familiar development environment.

                                    1. 4

                                      I have been asked to do a >3 hour test before, and I agreed to it out of curiosity.

                                      Had no idea what language or problem I would be solving, so as to not allow me to cheat. The company uses multiple languages and frameworks (they are an agency)

                                      Enter a small room with a Mac Mini and a set of instructions, could have been a challenge for The Crystal Maze, honestly. No ethernet and disabled wifi to stop me from cheating (?), VS Code and a PDF with some kind of ECMAScript 5 documentation as a reference.

                                      The challenge consisted of writing a small app to convert ‘english string of a single number’ to an integer.

                                      I spent the entire 3 hours scoping out the problems, and the structure of what would be my classes and methods. I was pretty annoyed when they barged in and told me to stop. I spent some time after the interview writing up the solution in my own way just as the problem was lingering in my head and I needed to get rid of it.

                                      By the way, I was also supposed to package up the app in their bespoke framework, write a frontend and make it work on their iPhone.

                                      • Who would be able to write a working solution, something they are happy within this amount of time?

                                      • Why don’t these people let you use the Internet? Why not just make logs and review them with a grain of salt?

                                      • Are grads studying for this exact kind of test nowadays? I feel like I am missing something

                                      Anyway, I didn’t get the job and I am pretty relieved.

                                      1. 7

                                        First off, I hate these, but this is an industry where everything gets worse each year (cramped open-plan offices, “Agile” micromanagement, a field saturated with low-skill engineers) and this seems to be no exception.

                                        There’s no upside. You never get a better position based on acing one. No one is going to say, “wow, he really did a great job and we’re going to make him Principal Engineer” based on a code test. So what’s the point, if the best-case scenario is getting the same job you otherwise would have gotten, and if there’s a nonzero chance of being rejected for reasons that have nothing to do with the code (e.g. “we don’t like that language” syndrome or “not enough unit tests”– who unit-tests a throwaway script?)

                                        Two hours is my hard limit.

                                        1. 2

                                          Maybe I’ll believe it’s about merit when the CEO’s/CTO’s start getting periodic tests of their management and IT skills by skeptical, HR people. I haven’t seen that happen despite the pay and criticality of their role. ;)

                                        2. 3

                                          I am a proponent of coding tests, mainly because I have interviewed a rather large number of people and seen a lot of coding tests that appeared to reflect a complete lack of comprehension and skill. As unpleasant as they are (as a candidate), isn’t it a lot better to take a little time and have some level of confidence that the company is at least attempting to hire competent people? CS degrees mean nothing; certifications mean nothing; self-stated skills and abilities certainly mean nothing; and firing someone can be a long and difficult process.

                                          The alternative is contract-to-hire, which I hate a lot more.

                                          That said, if a company is giving you more than a 4 hour test, it’s asking too much.

                                          1. 1

                                            Why do you hate contract to hire a lot more?

                                            1. 1

                                              It requires a lot more investment by the candidate. You can have a secure full time job, decide to leave, and not know whether you’ll have a job in a month. Although take-home exercises and whiteboard problems can be a pain in the rear, it’s a lot more reasonable to me that you know you’re going to be hired full-time (or not) after the interview.

                                          2. 3

                                            The company I work for has a long history of doing unpaid “as much time as you can give” starter projects, in which you work closely with a member of the team on a task you’d actually be doing if you were hired. When I did mine, 3 years ago, I gave up two days, which is basically foolish, but showed my skills, and learned a lot more about the team, company and other things than I would have otherwise. This was valuable signal for both parties.

                                            My team decided to drop the starter project when we started to hire again, and settled on a coding task that mimics the system the candidate would be supporting at a much smaller scale, but large enough that brute force solutions don’t work well. We give the candidate 4 hours (it took me 15 minutes, and some colleagues about 1.5 hours) to complete this (we haven’t had a single person refuse), and basically make a decision after this. In addition, the problem spec includes the discussion questions that we’ll chat about during the technical debriefing of the coding task.

                                            We’ve had about 8 candidates go through this, and not a single one has complained about the length of time commitment, or stated the problem was too tricky, or anything but fair. However, we’ve had a success rate of 2 out of 8. It doesn’t test data structures, or algorithms, and there are very liberal bounds on acceptable runtime length. It’s fundamentally, sum up fields in a file grouping by ‘foo’, ‘bar’. The part that has tripped people up is almost always related to the fact you can’t store everything in memory. And, the guy with a 64GB machine, didn’t understand how to use Python’s dictionary type…

                                            I do wish we paid the person for their time, but I’ve been quite happy with the results of this recent experiment. Lots of candidates that looked good on paper that were just… not… very great.

                                            1. 1

                                              Working closely to the new team is great also for the employer, because you can understand the culture and the philosophy of the new team. At on of my previous job, there was some problems (narcisism, lack of competence, micromanagement, lack of empathy, etc.) that you can recognize easily working closely with the developers.