There are a few points that make sites that rely on “distributed expertise” fail with time:
Stagnant population. You see this a lot with moribund wikis. The original wiki fell victim to this. There aren’t enough newly-interested people to keep investing their time in the site.
Lack of expertise. As people become more experienced and knowledgeable, they also tend to get more demanding jobs, hobbies, and families, and thus are less willing to spend time answering questions with no reward.
Rules lawyering. Gamification (a term that seems to have died out as people realized its flaws, maybe) means that a site is a game. Games get exploited: people become experts on the rules and develop strategies to maximize performance, or they become so strict that it’s no longer fun for casual players.
StackOverflow has all of these problems. Its devotees are a stagnant, self-selecting subgroup who earned a lot of points back when it was easy to earn points answering questions like “What is the difference between git pull and git fetch?” (8931 upvotes) or “How to redirect to another webpage?” (7262 upvotes). The ones who are still participating are primarily experts in StackOverflow, not in technologies. Any new blood is likely to be immediately put off by having their question closed because it’s a “duplicate” of an old question that’s just different enough to not answer their question, and that results in an opportunity cost: all of their potential future value is lost to the site.
These structural problems were obvious at the beginning of the site, but were dismissed by most participants and moderators at that time. It’ll be interesting to see if they manage to find a business model after the recent layoffs.
You start by rewarding an easy to measure outcome that is mostly indicative for the hard to measure outcome you really want, then you stand back and watch how those smart upright apes find ways to reach the former while skirting the latter.
I have thought about “decaying” karma, so that each point has a half-life. The biggest issue with that combined with StackOverflow as it currently stands is that it would lead, over time, to karma deflation. After all, you can’t very well ask “How to redirect to another webpage?” again, so the roughly 36k karma earned by that question and the 125k+ earned by its answers couldn’t be replaced—just as nobody starting on the site today can ever hope to catch people who earned those cheap points when the site was young.
nobody starting on the site today can ever hope to catch people who earned those cheap points when the site was young.
I wonder if some sort of redistributive scheme could work: on a monthly basis, remove 1% of everyone’s points & reällocate to all those who’ve been active over the past month.
Fair enough, Stack Overflow is no longer the thriving community it once was (or promised to become), partly because all the low hanging fruit (easy but common questions with clear answers) has been picked.
But it’s still the go-to site for most developers. There’s no community as wide or as big or with a strong a catalog of existing answers out there that can replace it. When you search the web for a problem or question, SO is still commonly the top result, and often answers the question with a high quality, structured response.
I started using StackOverflow in the beta days and have a hojillion points now. I’m still passively getting points from things I posted at the start.
However, points aren’t meant to say “you’re a good developer.” They’re meant to say “you contribute the kind of content that StackOverflow users want and therefore we trust you to keep doing so.” Editing privileges, etc, flow from that understanding.
It’s probably impossible for someone starting today to ever catch up to my score. But I’d argue it doesn’t matter, if they can get enough points to get privileges and keep curating the site. (If not, the site will suffer.)
I’m not sure if my success there is due more to the fact that I know how to ask questions on StackOverflow or to the fact of my high score. But for me at least, the demise of StackOverflow is greatly exaggerated, and has been for years.
I’m not sure if my success there is due more to the fact that I know how to ask questions on StackOverflow or to the fact of my high score.
That’s a pretty simple experiment. Make a new account and ask some questions and see how they are treated. I suspect you’ll have so much less success that it’ll surprise you.
I think it’s harder to gain reputation today than it was a few years ago. I program in C# and all of the basic questions, things like : “What is an abstract class” have already been asked. These questions have garnered huge amount of reputation to people (see https://stackoverflow.com/questions/761194/interface-vs-abstract-class-general-oo) for what is basically basic knowledge of the programming language.
Today most questions in established languages like C# are either: 1- a duplicate, 2- very specific so as not to be interesting to most people.
If someone wants to gain reputation in any meaningful way, you have to start following a new language/framework/library, hope that it gets traction (and thus a lot of traffic on Stack) and answer the early easy questions about it.
Stack Overflow seems to try and solve two problems, and I think a single solution for both of them is causing some of the issue here. These two problems are:
Be a place for developers to ask questions and get some help.
Be a go to source of documentation for solutions to specific problems or explanations of concepts (kind of two separate things here too, but I digress).
As some other comments here have stated, the fact that all the “low-hanging fruit” has been claimed means that beginners that come to the site with some of these basic questions are downvoted and closed out due to the mod practices outlined in the article, even though SO is often promoted to newer devs as a great place to get help (maybe not so much anymore though, it seems).
The problem is one group (new users, non-expert devs) see the site as a “get help” tool, while another group are vehemently trying to curate the site as a source of documentation that has no place for “invalid” content.
StackOverflow’s documentation experiment seemed like a good way to optimize for the second use case, but clearly there were some issues there. Overall though, it seems like the functionality of the site needs to provide more streamlined UX for both of the use cases, instead of mashing it all together in one. Maybe that’s not the vision intended for Stack Overflow, but I see it as a cause of some of this friction at least.
Stack overflow peeves me no end….. Google often takes me there as the number one hit for stuff I really really want to know the answer to…. And the Question has been closed by the moderator as “unhelpful”. Yes asshat moderator, it is indeed unhelpful now thanks to you.
I find it quite strange to see many comments (here, now; but also elsewhere when this comes up) about how (a) gamification detracts from the site’s main purpose (being a knowledge base) and (b) it’s hard to get points now, well-thought-out questions and answers often don’t get many points, other people got lots of points without much effort.
Surely (a) implies that the points are meaningless, and hence (b) is a non-issue? This seems especially true when it comes to questions: the few times I’ve asked a question on SO or another stackexchange site, it’s because I want to know the answer; not because I want more meaningless internet points.
The other common complaint is overly-strict closing, which certainly does sound like a delicate issue which could put off many people from participating further.
Actions are gated behind points. You can’t answer “protected” questions or post links until you have 10; can’t comment until 50. Then, too, there’s a network effect where people are, perversely, more likely to upvote answers by high-reputation users. Thus, the points do matter; the ones that matter the most are the hardest to get; and there’s an entrenched group of high-rep users who accrue reputation just by virtue of having high reputation already.
Actions are gated behind points. You can’t answer “protected” questions or post links until you have 10; can’t comment until 50.
Ah yes, I forgot about this.
The failure mode I was mostly thinking about was considering SO reputation when comparing prospective new hires, which would give these ‘internet points’ a real economic value, and cause all sorts of distortion.
Thus, the points do matter; the ones that matter the most are the hardest to get;
Do you mean those initial 10 or 50 points? I see how this is bad, but also why something like that is necessary to prevent spam (lobste.rs uses invites for a similar reason). I don’t know enough to comment on the particular tradeoffs (e.g. how SO spam correlates with reputation requirements).
Then, too, there’s a network effect where people are, perversely, more likely to upvote answers by high-reputation users… and there’s an entrenched group of high-rep users who accrue reputation just by virtue of having high reputation already.
I personnally wasn’t complaining on the gameification of the site but the difficulty to get reputation nowdays (which you do mention).
I do agree with your points and your second paragraph made me rethink my arguments.
To be frank, my main motivation for getting reputation (which is probably a bad one) is if I ever want to find a job using StackOverflow Jobs or if a potential employer decides to look up my profile on StackOverflow. But I’m probably stressing over nothing.
This is a really thoughtful article. it highlights a bunch of lessons anyone interested in community building should take careful note of.
I wonder if allowing upvoting / downvoting for bad behavior could help? e.g. if you make a shitty comment on a new user’s question, folks could downvote you for being a jerk, thus earning some small amount of karma for themselves, and if someone gets enough downvotes, their rep takes a hit.
I hope there are places like SO, where questions can be answered with a minimum of discussion, and especially a tight reign on debate. For example, if I’m configuring Linux to do something and I run into a snag, I want a place to ask about it without someone dropping into the thread and saying how I’m a horrible person for using a terrible copyleft license and how we must all talk about licensing politics now.
Because that kind of thing works: If someone drops that kind of grenade into the discussion, everything is now about the grenade, not the original question, which is left to molder way up at the top of the page.
My point is, strict moderation by a moderation team has a place. Maybe SO got enough of the details wrong it’s doomed at this point, but that doesn’t condemn the whole concept of a factual and focused Q&A site.
I read the article and the comments here, and I can’t help but think of USENET. An any arbitrary group, people would ask questions, others would answer. Enough time would pass and one person (or a few) would basically create a curated list of questions and answers (“Frequently Asked Questions” aka FAQ) that would be posted periodically (usually monthly). I am seriously surprised something like this hasn’t popped up on Stack Overflow.
The value of the FAQ is significant: it essentially archives a set of questions and moves them off the discussion table. This lets experienced users move on to something new, while still providing an answer to new users with those questions.
This would be a great SO feature. I think the suggested question list you get when writing a question tries to accomplish this.
There are a few points that make sites that rely on “distributed expertise” fail with time:
StackOverflow has all of these problems. Its devotees are a stagnant, self-selecting subgroup who earned a lot of points back when it was easy to earn points answering questions like “What is the difference between
git pullandgit fetch?” (8931 upvotes) or “How to redirect to another webpage?” (7262 upvotes). The ones who are still participating are primarily experts in StackOverflow, not in technologies. Any new blood is likely to be immediately put off by having their question closed because it’s a “duplicate” of an old question that’s just different enough to not answer their question, and that results in an opportunity cost: all of their potential future value is lost to the site.These structural problems were obvious at the beginning of the site, but were dismissed by most participants and moderators at that time. It’ll be interesting to see if they manage to find a business model after the recent layoffs.
Better explained by the concept of https://en.wikipedia.org/wiki/Perverse_incentive
You start by rewarding an easy to measure outcome that is mostly indicative for the hard to measure outcome you really want, then you stand back and watch how those smart upright apes find ways to reach the former while skirting the latter.
I followed Atwood and Spolsky during the startup of SO, and I think they’d thought through the implications as best as they could.
We “know” gamification is bad now, partly because SO, through its undeniable success, proved it.
The site and concept may be moribund now, but for a while they were successful. We should all be so lucky with the things we create.
I’m wondering if halving points once a year or so would somehow help with some of those issues.
I have thought about “decaying” karma, so that each point has a half-life. The biggest issue with that combined with StackOverflow as it currently stands is that it would lead, over time, to karma deflation. After all, you can’t very well ask “How to redirect to another webpage?” again, so the roughly 36k karma earned by that question and the 125k+ earned by its answers couldn’t be replaced—just as nobody starting on the site today can ever hope to catch people who earned those cheap points when the site was young.
I wonder if some sort of redistributive scheme could work: on a monthly basis, remove 1% of everyone’s points & reällocate to all those who’ve been active over the past month.
If karma is no longer working, why not axe it?
If it drives out those who are merely responding to gamification, perhaps that is for the better.
Fair enough, Stack Overflow is no longer the thriving community it once was (or promised to become), partly because all the low hanging fruit (easy but common questions with clear answers) has been picked.
But it’s still the go-to site for most developers. There’s no community as wide or as big or with a strong a catalog of existing answers out there that can replace it. When you search the web for a problem or question, SO is still commonly the top result, and often answers the question with a high quality, structured response.
Is there a better place out there?
I started using StackOverflow in the beta days and have a hojillion points now. I’m still passively getting points from things I posted at the start.
However, points aren’t meant to say “you’re a good developer.” They’re meant to say “you contribute the kind of content that StackOverflow users want and therefore we trust you to keep doing so.” Editing privileges, etc, flow from that understanding.
It’s probably impossible for someone starting today to ever catch up to my score. But I’d argue it doesn’t matter, if they can get enough points to get privileges and keep curating the site. (If not, the site will suffer.)
I’ve seen complaints about “rules lawyering” and such for years, and yes, it does happen. But personally I’m still able to get good answers quickly. Eg, I posted https://stackoverflow.com/questions/47016737/xpath-for-text-matches-one-of-the-following-strings a month ago and nearly immediately got great help.
I’m not sure if my success there is due more to the fact that I know how to ask questions on StackOverflow or to the fact of my high score. But for me at least, the demise of StackOverflow is greatly exaggerated, and has been for years.
That’s a pretty simple experiment. Make a new account and ask some questions and see how they are treated. I suspect you’ll have so much less success that it’ll surprise you.
I think it’s harder to gain reputation today than it was a few years ago. I program in C# and all of the basic questions, things like : “What is an abstract class” have already been asked. These questions have garnered huge amount of reputation to people (see https://stackoverflow.com/questions/761194/interface-vs-abstract-class-general-oo) for what is basically basic knowledge of the programming language.
Today most questions in established languages like C# are either: 1- a duplicate, 2- very specific so as not to be interesting to most people.
This means that the question will not get many views and votes making significant reputation gains almost impossible. The most votes I got was on a very simple syntax question for Rust. Mostly because the language was new and these simple questions hadn’t been asked yet (https://stackoverflow.com/questions/26665471/semicolons-are-optional-in-rust/26665514#26665514).
If someone wants to gain reputation in any meaningful way, you have to start following a new language/framework/library, hope that it gets traction (and thus a lot of traffic on Stack) and answer the early easy questions about it.
Stack Overflow seems to try and solve two problems, and I think a single solution for both of them is causing some of the issue here. These two problems are:
As some other comments here have stated, the fact that all the “low-hanging fruit” has been claimed means that beginners that come to the site with some of these basic questions are downvoted and closed out due to the mod practices outlined in the article, even though SO is often promoted to newer devs as a great place to get help (maybe not so much anymore though, it seems).
The problem is one group (new users, non-expert devs) see the site as a “get help” tool, while another group are vehemently trying to curate the site as a source of documentation that has no place for “invalid” content.
StackOverflow’s documentation experiment seemed like a good way to optimize for the second use case, but clearly there were some issues there. Overall though, it seems like the functionality of the site needs to provide more streamlined UX for both of the use cases, instead of mashing it all together in one. Maybe that’s not the vision intended for Stack Overflow, but I see it as a cause of some of this friction at least.
Stack overflow peeves me no end….. Google often takes me there as the number one hit for stuff I really really want to know the answer to…. And the Question has been closed by the moderator as “unhelpful”. Yes asshat moderator, it is indeed unhelpful now thanks to you.
I find it quite strange to see many comments (here, now; but also elsewhere when this comes up) about how (a) gamification detracts from the site’s main purpose (being a knowledge base) and (b) it’s hard to get points now, well-thought-out questions and answers often don’t get many points, other people got lots of points without much effort.
Surely (a) implies that the points are meaningless, and hence (b) is a non-issue? This seems especially true when it comes to questions: the few times I’ve asked a question on SO or another stackexchange site, it’s because I want to know the answer; not because I want more meaningless internet points.
The other common complaint is overly-strict closing, which certainly does sound like a delicate issue which could put off many people from participating further.
Actions are gated behind points. You can’t answer “protected” questions or post links until you have 10; can’t comment until 50. Then, too, there’s a network effect where people are, perversely, more likely to upvote answers by high-reputation users. Thus, the points do matter; the ones that matter the most are the hardest to get; and there’s an entrenched group of high-rep users who accrue reputation just by virtue of having high reputation already.
Ah yes, I forgot about this.
The failure mode I was mostly thinking about was considering SO reputation when comparing prospective new hires, which would give these ‘internet points’ a real economic value, and cause all sorts of distortion.
Do you mean those initial 10 or 50 points? I see how this is bad, but also why something like that is necessary to prevent spam (lobste.rs uses invites for a similar reason). I don’t know enough to comment on the particular tradeoffs (e.g. how SO spam correlates with reputation requirements).
I put this all in my “so what?” bucket :)
I personnally wasn’t complaining on the gameification of the site but the difficulty to get reputation nowdays (which you do mention).
I do agree with your points and your second paragraph made me rethink my arguments.
To be frank, my main motivation for getting reputation (which is probably a bad one) is if I ever want to find a job using StackOverflow Jobs or if a potential employer decides to look up my profile on StackOverflow. But I’m probably stressing over nothing.
This is a really thoughtful article. it highlights a bunch of lessons anyone interested in community building should take careful note of.
I wonder if allowing upvoting / downvoting for bad behavior could help? e.g. if you make a shitty comment on a new user’s question, folks could downvote you for being a jerk, thus earning some small amount of karma for themselves, and if someone gets enough downvotes, their rep takes a hit.
I hope there are places like SO, where questions can be answered with a minimum of discussion, and especially a tight reign on debate. For example, if I’m configuring Linux to do something and I run into a snag, I want a place to ask about it without someone dropping into the thread and saying how I’m a horrible person for using a terrible copyleft license and how we must all talk about licensing politics now.
Because that kind of thing works: If someone drops that kind of grenade into the discussion, everything is now about the grenade, not the original question, which is left to molder way up at the top of the page.
My point is, strict moderation by a moderation team has a place. Maybe SO got enough of the details wrong it’s doomed at this point, but that doesn’t condemn the whole concept of a factual and focused Q&A site.
I read the article and the comments here, and I can’t help but think of USENET. An any arbitrary group, people would ask questions, others would answer. Enough time would pass and one person (or a few) would basically create a curated list of questions and answers (“Frequently Asked Questions” aka FAQ) that would be posted periodically (usually monthly). I am seriously surprised something like this hasn’t popped up on Stack Overflow.
The value of the FAQ is significant: it essentially archives a set of questions and moves them off the discussion table. This lets experienced users move on to something new, while still providing an answer to new users with those questions.
This would be a great SO feature. I think the suggested question list you get when writing a question tries to accomplish this.
Wasn’t it the reason to create Stackoverflow Documentation? (I am not sure, I did not follow closely that part)