In all seriousness, I’m not sure how I’d react to this. I’ve played Factorio a little, but I don’t think I’d react well to someone judging my abilities over it. When I played it, I was always comfortable with just small-scale quarry operations. While some people get really far in it, it seems way too much like work for me to bother with that kind of thing.
That, and the giant bugs in your codebase aren’t trying to eat you.
This is how I feel when a company has picked a very specific library and judges me in my entirety over my skill in that one library. I think using Factorio as a benchmark of how to play Factorio is opposite of hiring for aptitude but also lazy analysis of the hiring company.
We need someone who is good at Linux
I maintain Ubuntu
*points to printed job description* It says Linux
Companies making Factorio an arbitrary waterline would be surprising if I ran into it but the overall idea is not surprising.
Yeah. This seems especially bad in Ruby shops. It’s super frustrating applying for roles that are supposedly language agnostic but then
(after like 7 interviews and technicals) being required to do a substantial take-home project in Rails. I don’t think these companies realize how magical Rails is and how you’re immediately knee-capping someone who doesn’t have any experience with that particular ecosystem.
In all seriousness, I’m not sure how I’d react to this.
Being in the process of multiple interviews and looking for a job actually, I can easily imagine myself in this situation. I would laugh, do it seriously and get along with the rest of my day. It is not worse and better than the technical interview, the ones when I have to explained my CV in length or state my defaults or focus on this specific software when I stated that I used of the alternative on my CV or any roleplay.
Feeling insulted because of a job interview was what drove me mad in the beginning of the the process and I had to work a lot on myself to not take everything personal. I ask pseudo-naive questions to get a hold of the mentality of the company and try not to judge them on their processes.
I kinda-sorta get the point the author’s trying to make, but I don’t think Factorio (or something similar) is a better way of finding out how someone works on software. You could take this and say “build a shed with someone” and then move on to “we’re scaling up to building hundreds of sheds” - they’re both impractical and both have nothing to do with scaling software or determining anything about how someone solves programming/infrastructure/software problems. My dad is great at problem solving with mechanical things but doesn’t know the first thing about software or how/why you’d need to do things to make it work correctly or scale. The skills aren’t 1:1 transferable.
All that to say I disagree with the conclusion that Factorio would be a better way to assess skills (even if it were practical). I do a lot of technical interviewing and I like to tell people that the solution correctness is not really what I’m looking for, it’s the way they think about and break down a problem that I care about (in a software sense, not in .. how to build a shed).
Fun article! I think a lot of people missed the very last paragraph though, where the author says:
Collectively, the software industry simply has no idea how to hire software developers. Factorio is probably the best technical interview we have right now, and that’s embarassing.
Collectively, the software industry simply has no idea how to hire software developers.
I’ve been exposed to bad interviews and good ones. I don’t think the author sufficiently justifies this point.
Factorio is probably the best technical interview we have right now
Again, insufficient justification is given for this extraordinary claim. The whole article hand waves about how there’s similarities between code and playing Factorio, like for example you have to fix bugs in Factorio (“woa bugs, we have those in code too! :o”), but the article’s conclusion does not follow from its body.
If Factorio were multiplayer, and the other players just randomly rewrote the rules of the game for no well-explained reason, it would be more appropriate in evaluating how good a person was at software development.
It is still too narrow of an environment to properly model the complexity that you can enconter while developing software. For example, the bugs that arise in Factorio arise from, essentially, a mismatch in the supply and capacity of the flow of materials. Its “complexity” arises from simply adding more of these pipelines. Software is usually not so simple.
I’m an ex developer of factorio, I have to ask: have you actually used this on a real candidate? Not sure how I would respond :D
I specifically remember great anger over my building a left hand drive train system during play testing (I’m Irish, we use left hand drive, most other devs are from mainland Europe, so use right hand drive).
I’m French, and my train systems are also left hand drive, for two reasons:
France’s trains also use left hand drive (even though the cars use right hand drive).
It places the signals in the insides of the tracks.
The first reason is pedantically subjective, but I believe placing signals inside of the tracks has real, technical value in some cases. And it’s more elegant. In any case, I still screw it up sometimes and forget that my own system is left hand drive…
The thing every commenter seems to be missing here is that the author concludes that Factorio is both unacceptable and terrible to use for interviewing (they note that doing it in less than 8 hours is noteworthy), yet it is still better than any other method popularly used, which is meant as an indictment of the popular methods.
If I showed up to an interview and the interviewer said “let’s play factorio” I would want to leave. I would be too polite to actually do it, but you can be sure it would make the rounds in the whisper network that showing up is a waste of time.
It’s interesting that the vast majority of the comments, both here and on the blog itself are about the idea of Factorio as a technical interview technique instead of the broader point of the article. I assume that’s because of how detailed the author takes this thought experiment, and the obvious shortcomings of gaming as an interview substitute versus the not so obvious solution to the technical interview problem. Maybe we’ve stumbled onto something. As someone who interviews developers regularly, it seems getting developers to simply read this blog post could be a suitable interview technique.
Imagine walking into an interview, the interviewer hands you this post and says “We’ve been experimenting with different technical interview techniques lately. Read this and let me know if you want to play Factorio as a part of this interview process.” Then proceeds to leave the room while the dev reads it over. How would you expect candidates of different experience levels to respond? Anyone who seriously wants to play the game is probably not the best fit, however:
I would expect Junior Devs to say they wouldn’t want to play, and reword the basic conclusion from the article.
“Intermediate” Devs would probably say the same, and provide some personal anecdotes from bad technical interviews they’ve had in the past.
Finally, I would expect Senior Devs to take the opportunity to talk about the failings of traditional technical interviewing, and provide examples of how to develop a better technical interviewing solution that we could work through during the interview.
I’ve played the tutorial at some point: it was hard, even though I was experienced. The analogy with development is only felt during regular open play, over several hours. Me, I’ve learned to reassess my relationship with technical debt: I tend to want the perfect thing, and I can write (or design) that perfect thing. But that takes time, which could be used instead to produce more things.
Lesson learned: even code that’s not good enough can be good enough for now. It should be replaced when it ceases to be, and no sooner. (Note: that’s my lesson. Tactical tornadoes should instead learn to replace code as soon as it ceases to be good enough. In general, it’s hard to assess when the time is right.)
The date says it’s a few days old but I have a serious case of Déjà vu.
In any case, taking this just a bit seriously, I don’t really think it maps to programming knowledge and ability in a general sense.
Some people play these games because they enjoy them, some because they resemble logic and programming, and others hate them because they are like programming.
I 100% fall into the last group. Doing something in one of these games feels like programming and thinking while completely drunk and having my hands tied behind my back while typing with my toes. Or handing me a language I’ve never used, no docs, no editor except Word (so “ will have to typed 3x to not be the writing ones) and so on. I don’t really think you could assess anything useful besides “this person seems to have written some code in the past”.
How much of these results come down to overfitting? (Please check my math on this because I’m not a statistician, merely someone who’s halfway through Algorithms to Live By.)
Many if not most tech job openings receive a large (sometimes huge) number of applications. The majority (if not vast majority) of these applications constitute obvious noise, e.g. unqualified people submitting scattershot in order to maintain their unemployment pay. Therefore even a literally random process would filter out far more false positives than false negatives.
But how much noise continues on to the technical interview filter? If only slightly more than half of the remaining applicants fail out of a fair technical interview, that means a coin flip would have filtered out more false positives than false negatives on average. Given how few true positive data are produced by the hiring process and the subjectivity of such evaluations, I strongly suspect that most interview processes could be proven statistically indistinguishable from random selection.
My hypothesis: The tech industry might derive more value for its dollar by paying under-qualified people to stay out of the hiring pipeline than to passively filter them out of it.
My hypothesis: The tech industry might derive more value for its dollar by paying under-qualified people to stay out of the hiring pipeline than to passively filter them out of it.
This would massively incentivize people who are under-qualified to do programming jobs to make themselves legible to whatever system would be put in place to actually pay this money.
For some reason this reminds me of reading in Wired many many years ago that Joi Ito (OG 90s VC) predicted that being part of a WoW guild was the next step in networking.
I’m not very interested in engaging with the practicalities of using Factorio as an interview question because it’s a bit too silly, but otherwise this article is a great explanation of why Factorio is such a nice programmer toy.
The Debugging section is interesting. IME Factorio isn’t chaotic enough for you to see much weird behaviour. “Material ends up in incorrect place” rarely come up. It is a great system inside which to get to experience debugging throughput problems, as the article discusses. I think this is kind of neat and maybe an educational experience because it’s otherwise kind of hard to get to see and interact with systems that are operating at the limits of throughput and experiencing backlogs in funny places.
God why is this still a thing every week? Stop giving potential hires bullshit meaningless “business problems” to solve with whiteboard pseudocode or video games. Give them 1 page of code you know and they’ve never seen before. Ask them questions about it. In 5 minutes, with 100% accuracy, EVERY TIME, you’ll know whether or not they can actually code. If you can’t, you’ve accidentally been promoted above where you should be.
There’s an achievement for doing it in under 8 hours (there are several finish-time achievements but the 8-hour achieve “there is no spoon” is the hardest), but realistically a noon will take a LOT longer than that for their first time. I’d guess 100+ hours, although that’s just playing the game for fun and without specifically rushing to rocket-launch.
It really depends on how hard you rush towards launching the rocket - the “game over” condition (quotes there because you can play beyond launching a rocket).
You could do it in a few hours if you rush and have an idea what you need to do. First time with no guides will probably be longer. Speedrunners can do it in about an hour - edit: hour and a half.
Personally I have played maybe 3 or 4 times and I always gave up before launching the rocket, after quite a few hours. I’m definitely not good at Factorio but this game is both quite difficult and truly great.
I kinda see what the author is trying to get at, in the sense that self-organisation, debugging are all really valuable skills to have. But I think they’ve forgotten just how much overhead there is in learning a new programming language (or even to play something like factorio) and internalising it’s worldview. Never mind trying to do that and solve a real problem at the same time.
Ideally interviewing folks would prefer to give folks the space to shine, but to do that, you need to get out of the interviewee’s way. I tend to feel that this kind of difficulty is more multiplicative than additative, so forcing someone to use something that’s radically unfamiliar (as Factorio is to most) is not going to show folks at their best.
Unless you’re looking for folks to play Factorio with, that is. But that’d be … weird.
It is also wildly impractical, taking over 20 hours in an initial multiplayer playthrough, or 8 hours if you have a lot of people and know what you’re doing.
I used factorio to explain automation, standardization, inter-dependencies and complex systems to adult students. I totally agree that Factorio (or other automation games) can totally work for this and I’m planning to expand on this educational side. Factorio can do more to explain computational thinking than Scratch or those other toy programming languages will ever do, exactly because it doesn’t care about looking like programming. It’s completely detached from any technological restraint on computational thinking and this makes it perfect.
Don’t give aphyr any ideas….
In all seriousness, I’m not sure how I’d react to this. I’ve played Factorio a little, but I don’t think I’d react well to someone judging my abilities over it. When I played it, I was always comfortable with just small-scale quarry operations. While some people get really far in it, it seems way too much like work for me to bother with that kind of thing.
That, and the giant bugs in your codebase aren’t trying to eat you.
This is how I feel when a company has picked a very specific library and judges me in my entirety over my skill in that one library. I think using Factorio as a benchmark of how to play Factorio is opposite of hiring for aptitude but also lazy analysis of the hiring company.
Companies making Factorio an arbitrary waterline would be surprising if I ran into it but the overall idea is not surprising.
Yeah. This seems especially bad in Ruby shops. It’s super frustrating applying for roles that are supposedly language agnostic but then (after like 7 interviews and technicals) being required to do a substantial take-home project in Rails. I don’t think these companies realize how magical Rails is and how you’re immediately knee-capping someone who doesn’t have any experience with that particular ecosystem.
Being in the process of multiple interviews and looking for a job actually, I can easily imagine myself in this situation. I would laugh, do it seriously and get along with the rest of my day. It is not worse and better than the technical interview, the ones when I have to explained my CV in length or state my defaults or focus on this specific software when I stated that I used of the alternative on my CV or any roleplay.
Feeling insulted because of a job interview was what drove me mad in the beginning of the the process and I had to work a lot on myself to not take everything personal. I ask pseudo-naive questions to get a hold of the mentality of the company and try not to judge them on their processes.
I’d probably not work for this company. I would think that I won’t fit in culturally.
I’d feel very, very much insulted.
100% - Hell, I’ve played Factorio and still would be taken aback by the suggestion.
I’d have to pause a bit to decide whether to just stop the interview and leave, or cause a scene first.
I’d simply state that I feel deeply insulted by the process and leave. No need for a scene.
Factorio was the first game I ever noticed occupied the new (but on the grow) niche that I like to call “not actually a game, but a job”
Check out Banished
After putting several days into it, I read a review of the game that likened it to balancing a spreadsheet
I kinda-sorta get the point the author’s trying to make, but I don’t think Factorio (or something similar) is a better way of finding out how someone works on software. You could take this and say “build a shed with someone” and then move on to “we’re scaling up to building hundreds of sheds” - they’re both impractical and both have nothing to do with scaling software or determining anything about how someone solves programming/infrastructure/software problems. My dad is great at problem solving with mechanical things but doesn’t know the first thing about software or how/why you’d need to do things to make it work correctly or scale. The skills aren’t 1:1 transferable.
All that to say I disagree with the conclusion that Factorio would be a better way to assess skills (even if it were practical). I do a lot of technical interviewing and I like to tell people that the solution correctness is not really what I’m looking for, it’s the way they think about and break down a problem that I care about (in a software sense, not in .. how to build a shed).
Fun article! I think a lot of people missed the very last paragraph though, where the author says:
… and everyone is discussing how they’d react to Factorio used as an interviewing method. 😔
I can’t blame people for noping out before the author got to the point, though.
I’ve been exposed to bad interviews and good ones. I don’t think the author sufficiently justifies this point.
Again, insufficient justification is given for this extraordinary claim. The whole article hand waves about how there’s similarities between code and playing Factorio, like for example you have to fix bugs in Factorio (“woa bugs, we have those in code too! :o”), but the article’s conclusion does not follow from its body.
If Factorio were multiplayer, and the other players just randomly rewrote the rules of the game for no well-explained reason, it would be more appropriate in evaluating how good a person was at software development.
It is still too narrow of an environment to properly model the complexity that you can enconter while developing software. For example, the bugs that arise in Factorio arise from, essentially, a mismatch in the supply and capacity of the flow of materials. Its “complexity” arises from simply adding more of these pipelines. Software is usually not so simple.
I’m an ex developer of factorio, I have to ask: have you actually used this on a real candidate? Not sure how I would respond :D
I specifically remember great anger over my building a left hand drive train system during play testing (I’m Irish, we use left hand drive, most other devs are from mainland Europe, so use right hand drive).
(OT but…) Thank you (and the rest of Wube) for a really fantastic game! I’ve and so many people I know have got so much value out of Factorio.
Trains and subways (but not trams) still drive on the left in Sweden (Sweden went over to RHS road traffic in 1967).
As in France, because IIRC, Britain built the train system there.
I’m French, and my train systems are also left hand drive, for two reasons:
The first reason is pedantically subjective, but I believe placing signals inside of the tracks has real, technical value in some cases. And it’s more elegant. In any case, I still screw it up sometimes and forget that my own system is left hand drive…
The thing every commenter seems to be missing here is that the author concludes that Factorio is both unacceptable and terrible to use for interviewing (they note that doing it in less than 8 hours is noteworthy), yet it is still better than any other method popularly used, which is meant as an indictment of the popular methods.
If I showed up to an interview and the interviewer said “let’s play factorio” I would want to leave. I would be too polite to actually do it, but you can be sure it would make the rounds in the whisper network that showing up is a waste of time.
It’s interesting that the vast majority of the comments, both here and on the blog itself are about the idea of Factorio as a technical interview technique instead of the broader point of the article. I assume that’s because of how detailed the author takes this thought experiment, and the obvious shortcomings of gaming as an interview substitute versus the not so obvious solution to the technical interview problem. Maybe we’ve stumbled onto something. As someone who interviews developers regularly, it seems getting developers to simply read this blog post could be a suitable interview technique.
Imagine walking into an interview, the interviewer hands you this post and says “We’ve been experimenting with different technical interview techniques lately. Read this and let me know if you want to play Factorio as a part of this interview process.” Then proceeds to leave the room while the dev reads it over. How would you expect candidates of different experience levels to respond? Anyone who seriously wants to play the game is probably not the best fit, however:
I would expect Junior Devs to say they wouldn’t want to play, and reword the basic conclusion from the article.
“Intermediate” Devs would probably say the same, and provide some personal anecdotes from bad technical interviews they’ve had in the past.
Finally, I would expect Senior Devs to take the opportunity to talk about the failings of traditional technical interviewing, and provide examples of how to develop a better technical interviewing solution that we could work through during the interview.
I downloaded the demo and never managed to beat the tutorial level. 😞
I’m really not a bad developer though.
So we won’t get “Ten essays on Factorio” from you then?
I’ve played the tutorial at some point: it was hard, even though I was experienced. The analogy with development is only felt during regular open play, over several hours. Me, I’ve learned to reassess my relationship with technical debt: I tend to want the perfect thing, and I can write (or design) that perfect thing. But that takes time, which could be used instead to produce more things.
Lesson learned: even code that’s not good enough can be good enough for now. It should be replaced when it ceases to be, and no sooner. (Note: that’s my lesson. Tactical tornadoes should instead learn to replace code as soon as it ceases to be good enough. In general, it’s hard to assess when the time is right.)
This is all well and good, but if experience has taught me anything, I expect my next interview to require me to live program FizzBuzz using Factorio.
The date says it’s a few days old but I have a serious case of Déjà vu.
In any case, taking this just a bit seriously, I don’t really think it maps to programming knowledge and ability in a general sense.
Some people play these games because they enjoy them, some because they resemble logic and programming, and others hate them because they are like programming.
I 100% fall into the last group. Doing something in one of these games feels like programming and thinking while completely drunk and having my hands tied behind my back while typing with my toes. Or handing me a language I’ve never used, no docs, no editor except Word (so “ will have to typed 3x to not be the writing ones) and so on. I don’t really think you could assess anything useful besides “this person seems to have written some code in the past”.
How much of these results come down to overfitting? (Please check my math on this because I’m not a statistician, merely someone who’s halfway through Algorithms to Live By.)
Many if not most tech job openings receive a large (sometimes huge) number of applications. The majority (if not vast majority) of these applications constitute obvious noise, e.g. unqualified people submitting scattershot in order to maintain their unemployment pay. Therefore even a literally random process would filter out far more false positives than false negatives.
But how much noise continues on to the technical interview filter? If only slightly more than half of the remaining applicants fail out of a fair technical interview, that means a coin flip would have filtered out more false positives than false negatives on average. Given how few true positive data are produced by the hiring process and the subjectivity of such evaluations, I strongly suspect that most interview processes could be proven statistically indistinguishable from random selection.
My hypothesis: The tech industry might derive more value for its dollar by paying under-qualified people to stay out of the hiring pipeline than to passively filter them out of it.
This would massively incentivize people who are under-qualified to do programming jobs to make themselves legible to whatever system would be put in place to actually pay this money.
This is a culture screen, not a tech screen. Your testing for someone’s affinity for a certain sort of game. I would simply walk out.
It also discriminates against people with disabilities who can’t play the game.
For some reason this reminds me of reading in Wired many many years ago that Joi Ito (OG 90s VC) predicted that being part of a WoW guild was the next step in networking.
Well, the CEO of Shopify is a big Starcraft fan, so it might not be too far off!
I’m not very interested in engaging with the practicalities of using Factorio as an interview question because it’s a bit too silly, but otherwise this article is a great explanation of why Factorio is such a nice programmer toy.
The Debugging section is interesting. IME Factorio isn’t chaotic enough for you to see much weird behaviour. “Material ends up in incorrect place” rarely come up. It is a great system inside which to get to experience debugging throughput problems, as the article discusses. I think this is kind of neat and maybe an educational experience because it’s otherwise kind of hard to get to see and interact with systems that are operating at the limits of throughput and experiencing backlogs in funny places.
God why is this still a thing every week? Stop giving potential hires bullshit meaningless “business problems” to solve with whiteboard pseudocode or video games. Give them 1 page of code you know and they’ve never seen before. Ask them questions about it. In 5 minutes, with 100% accuracy, EVERY TIME, you’ll know whether or not they can actually code. If you can’t, you’ve accidentally been promoted above where you should be.
I’ve not played Factorio yet but how long does a full run take on average? Or is it just time boxed regardless? Lol.
There’s an achievement for doing it in under 8 hours (there are several finish-time achievements but the 8-hour achieve “there is no spoon” is the hardest), but realistically a noon will take a LOT longer than that for their first time. I’d guess 100+ hours, although that’s just playing the game for fun and without specifically rushing to rocket-launch.
My first run was 100+ hours; speedrunners finish it in a little over 2.5 hours on default settings.
It really depends on how hard you rush towards launching the rocket - the “game over” condition (quotes there because you can play beyond launching a rocket).
You could do it in a few hours if you rush and have an idea what you need to do. First time with no guides will probably be longer. Speedrunners can do it in about an
hour- edit: hour and a half.A reasonably knowledgable pair of players, going in with a good premade plan, can launch a rocket in 3 hours. Default settings.
My first play through was 40+ hours
Personally I have played maybe 3 or 4 times and I always gave up before launching the rocket, after quite a few hours. I’m definitely not good at Factorio but this game is both quite difficult and truly great.
I’ve dabbled in it for years, when 1.0 came out I finally did a full play through and it took around 30 hours.
I would instantly walk out of the interview.
Judge me by my technical ability and aptitude, by what I have built, not my interest or disinterest in a computer game.
I kinda see what the author is trying to get at, in the sense that self-organisation, debugging are all really valuable skills to have. But I think they’ve forgotten just how much overhead there is in learning a new programming language (or even to play something like factorio) and internalising it’s worldview. Never mind trying to do that and solve a real problem at the same time.
Ideally interviewing folks would prefer to give folks the space to shine, but to do that, you need to get out of the interviewee’s way. I tend to feel that this kind of difficulty is more multiplicative than additative, so forcing someone to use something that’s radically unfamiliar (as Factorio is to most) is not going to show folks at their best.
Unless you’re looking for folks to play Factorio with, that is. But that’d be … weird.
I used factorio to explain automation, standardization, inter-dependencies and complex systems to adult students. I totally agree that Factorio (or other automation games) can totally work for this and I’m planning to expand on this educational side. Factorio can do more to explain computational thinking than Scratch or those other toy programming languages will ever do, exactly because it doesn’t care about looking like programming. It’s completely detached from any technological restraint on computational thinking and this makes it perfect.