1. 26
  1.  

  2. 27

    I want to give some feedback on the survey. (background: I did co-design the first and second Rust survey)

    The demographics section feels very odd. I pull out the questions so that people here don’t have to scroll:

    • What is your gender?

      • Male
      • Female
      • Non-binary
    • Do you identify as transgender?

      • Yes
      • No

    There’s a couple of issues with this: if you open up the floor for diverse genders, you should go all in. There’s people that are none of the three, most notably agender, but the list goes on. Everything else is clumsy. Second, the question on “how do you identify as transgender” is contested territory, many trans folk I know refuse that question because they are trans, not identify as.

    Another issue is “Which country do you live in?”, it doesn’t poll for nationality. This might be fine, I’m just not sure what the goal of the question is, then.

    I’d like to challenge them on a different territory though: they are the only identity questions asked beyond the usual (age, highest education, etc). This means that other things (e.g. disability) are ignored and thus (implicitly) communicated as uninteresting. Which begs the question: why is gender and trans so important that it is polled? What actions can be taken based on those numbers? This is a representation issue, for a good overview of the question “should we count or not”, I can highly recommend: https://media.ccc.de/v/camp2015-6876-data_and_discrimination_representing_marginalised_communities_in_data

    For that reason, we have completely bypassed that question from our survey and replaced it by a pair of questions:

    • “Do you consider yourself a member of an underrepresented demographic in technology?”
      • I won’t list them all here, but it lists 14 characteristics that we are aware of, such as gender identity, race, but also language skill + a free form field
    • “Do you feel your situation makes it difficult for you to participate in the Rust community?”
      • Yes
      • No
      • Maybe

    Those questions were reviewed from at least one person from each of the groups we list, if possible more. We’ve still run into issues in the first version, where we asked “Are you member of an underrepresented demographic?”. One of the feedback we got was black people in pre-dominantly countries being confused by the question. For that reason, we moved to the more lax question above. The second question makes the thing actionable: if someone is in a group and feels the Rust community is inaccessible because of that, we have something to follow up.

    This yields a ton of better results (at the cost of some people trolling the question).

    I am happy that more and more communities are doing those surveys. I’d be very interested in an experience exchange, or potentially even some standardisation, allowing us to compare results. There’s almost nothing comparative around.

    I’d be super interested in how anonymization works for you. Because of the number of open question we have in our survey, we generally don’t publish the results. To our reading and research into data anonymization techniques, it’s pretty easy to still identify individuals from the data set, especially from countries that are less well populated. We do instead commit to keeping data private. Given that you do ask for things like transness, this may keep people from answering.

    I also just remembered that feedback on missing demographics was given in 2017 to the first survey: https://lobste.rs/s/f5vsl9/first_annual_haskell_users_survey#c_o9l1zk, even with an offer to talk about how to design those. I wonder what was missing from our side for you to take up that offer. I found the issue for the survey, was there a review step of some sort? https://github.com/haskellweekly/haskellweekly/issues/57

    1. 18

      Thank you for this thoughtful and thorough comment. I was clearly made in good faith and it’s helpful, yet I can’t help but feel demoralized by it. It feels impossible to get this right! There is no “survey team” – it’s just me. I don’t really consume the results of the survey, so I’m trying my best to create a set of questions that I think are broadly useful. It’s very time consuming to write the survey, collect results, react to feedback, and produce an analysis. This is on top of my day job, making the newsletter, recording the podcast, and otherwise trying to live life.

      All that to say: I don’t really have the mental bandwidth to deal with this right now. I’ll reach out to you later about it.

      1. 14

        There is no “survey team” – it’s just me.

        Which, to be clear, is exactly the reason why I reach out and reached out back then: We can give you a lot of acquired knowledge, experience and practice up for grab. You don’t have to do this alone. The overload many feel in those areas is because we’re not reaching out to people that give experience, potentially from other communities - which is the reason I highlighted that I am interested to know what blocked you from reaching out, I want to fix that.

        If you have the feeling that you are left alone in running this survey, I can also give some advice there.

        All that to say: I don’t really have the mental bandwidth to deal with this right now. I’ll reach out to you later about it.

        Fully understood and given that the survey is started, I would also recommend to not fix it this year. I’m available for any help.

        1. 6

          Thanks for doing this!

          1. 6

            yet I can’t help but feel demoralized by it. It feels impossible to get this right!

            Don’t let any of this get you down. Step away and don’t sweat it for a while.

            Real talk, this reads like more is going on right now than this survey. Take some time and take care of yourself. Haskell and lobster.rs will be here when you’re ready.

          2. 8

            Sincere question: wouldn’t it be better to just leave out identity questions in general (other then age and level of education) and ask whether you “feel welcome” in the community (with add-on why question)?

            I mean, some people find it interesting to talk about these kind of things (gender identity, socioeconomic status, race), I personally would consider these things irrelevant within the context of Haskell usage/adoption. Don’t get me wrong, they’re interesting on a personal level, imo just not relevant to “this context”

            1. 11

              A couple of reasons: These things are super relevant to adoption, because the way we phrase the questions is to find blockers to adoption. People do gravitate towards communities they feel safe and accepted in and are willing to bring new people with them. It’s a natural function of having to fight less fights and having more time for code.

              Case in point: the JavaScript and Python communities have a very effective outreach program towards all the groups mentioned in our survey, and we know where they stand in the language world.

              Also, polling for these questions isn’t magic. There’s standard methods and on the level we operate on, any will do and especially a carbon copy from a project with more experience - I would recommend getting in touch though, to ask for the thinking behind the questions. There’s a tendency in the FOSS world to reinvent the wheel though and it’s a growing frustration point for me that while we share code, we don’t share organisational practice and learning. I am not alone in this opinion, for example the recent Roadworks Ahead PDF version study highlights the issue of uniform skill sets and devaluing of skills like surveying, managing and outreach in FOSS projects. The reason why I haven’t posted this study here is by the way: I will immediately get an “off-topic” flag (as on my feedback above).

              But more to the point: “feeling welcome” is even more of a simple barometer, you can’t get a lot of actionable things out of it. FOSS communities are very uniform (that comes mainly from the fact that the access path is uniform), so it’s not uncommon that a lot of the in-group feels happy. Also, it’s hard to figure out what numbers are good and what are bad. Also, a certain base number of people will always feel not welcome - it’s a statistical process. You get in touch with a community and your first 3 interactions are bad. You check “unwelcome”.

              The polling methodology above is effective because it’s simple, easily understood by readers (by asking for a feeling) and gives a quick way of feedback. The cross-referencing gives us the ability to find out that “non-English-speaking Rustaceans don’t feel welcome”. This isn’t a randomly picked variable, a common form of exclusion is grading arguments of non-English speakers by the level of people who went to debate club at school. A survey is the prime location to find out issues like this.

              Also, you don’t want to run this as a separate survey. While structured, targeted polling towards communities can be useful, the general yearly survey is the right moment to answer one or two questions to find things that catch your eye. This is the one everyone answers. Some people might fall in a dimension that they hadn’t considered to every raise. Others never felt empowered to represent themselves. This might go both ways: sometimes, you find groups are unexpectedly high in representation. What are we doing right and shouldn’t break? Or, other way put: if there’s a substantially worse representation of a group in a certain locale, are we looking at a missing stair that we don’t know of?

              We heavily use our survey to inform future project policy and follow up with trends and information from all of the questions very closely. None are in at random.

              1. 3

                (WoW, thanks for your in-depth and insightful comment, pdf is on reading list for tonight)

                I’ve never run a FOSS community nor do I identify as an active participant (I’ve had my first PR ever merged into IHP this week, and that was docs only), so I’m really blue on these kind of topics. Your comment makes perfect sense and - at least to me - indicate that more nuance is required in these kind of things.

                While you indicate that this “is not rocket science”, I do think (and agree with OP) that this is really hard to get right. So optimizing for “don’t insult too much of my target audience” SOUNDS like a good strategy.

                Thanks!

                1. 4

                  I’ve never run a FOSS community nor do I identify as an active participant (I’ve had my first PR ever merged into IHP this week, and that was docs only), so I’m really blue on these kind of topics. Your comment makes perfect sense and - at least to me - indicate that more nuance is required in these kind of things.

                  Note: Surveys are a effective tool to investigate passive participants. People who never spend more time on you then filling out the survey.

                  While you indicate that this “is not rocket science”, I do think (and agree with OP) that this is really hard to get right. So optimizing for “don’t insult too much of my target audience” SOUNDS like a good strategy.

                  Thanks for highlighting that, that may have come out wrong. What I’m lamenting - and it’s frustrating to me - is two factors at play:

                  • The uphill battle of connecting FOSS communities to organisational practice beyond producing the artifact. I specifically say “organisational”, as FOSS projects are not businesses and need to be managed differently. Still, many tools are the same.
                  • A tendency (and this is where my “rocket science” stab comes from) to work in silos and not ask or copy from other projects. Even if you are not fully happy with a standard, go and take one and then iterate. This is how accessibility documentation spread in conferences in the Ruby space. There’s multiple reasons why I think that this is, but phew, other post.

                  I really don’t want to judge people doing this in their free time, my frustration comes from the amount of pain I think we could save ourselves from. Pro-active feedback for example is always better than retroactive (or missing) feedback. The retroactive may not be fixed in the moment, the missing may just leave things broken forever.

                  The second point can be seen again and again, especially part of those communities that are often considered “secondary skills”. My example here is conference organisation: I have to date budgeted and run 15 community conferences. My first one (eurucamp 2012) famously went bankrupt. This is terrible: you have volunteer people running a conference for 200 international people, and in the end, they are liable. This one worked out by another, older conference letting us tap into their disaster pool, looking at our budget and telling us what was our mistake. It was a classic beginner budgeting mistake. I learned 3 things here: the importance of sound budgeting, that there’s a network of conferences that support each other and that there’s a lot of solidarity going around. Had we talked to them earlier, we would have had a lot less stress.

                  Since then, I’m a heavy budget optimiser, particularly around volunteer budgets (easy to create, easy to come up with, making sure you are not liable yourself) and offer my experience for free to FOSS events. Yet, even in the Rust space, people are timid at asking for that 30 minute call that might save them 4 month of panic over budget. I usually help those events shortly before the event, when they already missed all deadlines for venue cancellation. I’d rather give my experience early.

                  The fact that our movement let people who go into liability over 40k$ and more and they are not aware that there’s an experience - and even cash pool to tap into is mind-boggling to me. There’s a number of really simple programs implemented around that on really small budgets (e.g. the Ruby Central conference grant or the Python and Wordpress Foundations small event support). But it’s not known enough in communities that this is a place people can turn to.

                  1. 1

                    Do you have a blog post about this? I’ve done conference organizing, but never heard of this!

                    1. 2

                      I have a half-written one here on budgeting: https://github.com/rust-community/events-team/blob/budget-guide/guides/CONFERENCE_BUDGETING.md (I need to run another pass through it, there’s glaring typos in there)

                      There’s good general books out there, but the budgeting component is often lacking. I tried to write down a bit of it, but while budgeting techniques are general, but also super local. So I prefer 30 minute coaching sessions with people.

                      https://www.quirksmode.org/coh/ has a conference organisers handbook that is good. It still postulates you can’t run a sizable conference under 200 EUR, had I known that before, eurucamp and RustFest wouldn’t have ended up 3-4 days affairs that cost ~100 and pay their speakers ;). The reason why I don’t write about those stunts is because I need to execute them with a team that knows about the how very well and how to trust your fundraising process.

                      There’s another really good PDF book by the people behind UIConf Berlin, but I can’t find it right now.

                      That eurucamp went bankrupt is not something we ever made very public, but all the ex-organisers speak openly about it.

              2. 3

                I partially agree, there are factors that are probably orthogonal or at best indirect to usage (gender, nationality, country or residence), but others are probably more interesting (sex, age, languages, level of education), especially in the context of Haskell, a language with a lot of academic bagage that are intereting when asking questions on how the community is developing.

                1. 4

                  Especially nationality, country and residence can be super direct: the availability of an active meetup scene, conferences and speaker exchanges can be a huge booster. National scenes may form a (formal or informal) organisation, with organisers knowing each other and providing access to their networks. The other way around, I have also seen national scenes running into problems because of bad blood between some of the members. The latter is super hard to fix, but being aware of it can be interesting. But it isn’t uncommon that once you open a conversation channel, give people mental backing and skills to deal with situations, things improve. A lot of people connect that to “kicking people out”, which it almost always isn’t - but for example making people aware that they are having fights on the backs of others often makes them draw reasonable conclusions.

                  But I agree with you in general: Surveys of global scope give you a global overview. They at best give you pointers, assessment of what’s actually around can be take up to years. But “hey, we’ve seen this odd thing, got an idea?” is a good conversation starter.

                  BTW, I did not pick “disability” at random above, as disability is a frequent exclusion factor in programming communities. Which is odd, given that our tools of trade are rather accessible. Interesting side fact: Online conferences have btw. lead to less accessibility for blind people, as many of the platforms used are not accessible. Side subject, though.

                  1. 2

                    The access to a meetup scene is more related to the region you live in, as do you live in a village in the middle of Siberia or St. Petersburg. Nationality should be irrelevant too, just because someone has a German citizenship, doesn’t have to mean that they live there or even have lived there.

                    1. 2

                      The access to a meetup scene is more related to the region you live in, as do you live in a village in the middle of Siberia or St. Petersburg.

                      But this is very focused on a the datapoint itself.

                      The point I tried to work out is that it is particularly not that, aside from your extreme example. Meetups also grow and may create ways to include more people from futher away (such as running a small hackday, where people would drive to from further away). Again: organisers in a country (or adjecant ones, though it’s notable how visible borders still are!) tend to form a group, leading to a boosting effect and creating their own procedures and dynamics. The German Rust/Ruby/JavaScript organiser scene exists and is a rather mappable thing.

                      Nationality should be irrelevant too, just because someone has a German citizenship, doesn’t have to mean that they live there or even have lived there.

                      This depends heavily on the location you are in. There’s places where foreign nationals culturally have less access or more access.

                      Still, we may follow a red herring here:

                      A direct approach around the question of meetups is to actually ask for the info wanted:

                      • “Do you attend meetups?”
                        • “If no? Why not?” (Distance/Topics/Other)
                      1. 2

                        I agree that the last question would be a better way to solve this specific issue, but I think that demographic questions also serve to better understand issues you didn’t know to explicity state before creating the survey.

                        And just to clarify, my “extreme example” intends to show that proximity to a city or a urban area is a better indicator to whether or not you interact with any community on a face-to-face basis.

                        1. 1

                          The point @zge is making is that driving from Anchorage to Seattle isn’t quite the same thing as driving from Miami to New York. Driving from Tijuana to San Francisco as a Mexican national isn’t quite like either of those trips.

                          This doesn’t invalidate your points. My point is that answers to “If not, why” need to be quite granular to be useful.

                2. -2

                  There’s people that are none of the three, most notably agender

                  By definition, everything that is not male or female is non-binary.

                  “how do you identify as transgender” is contested territory, many trans folk I know refuse that question because they are trans, not identify as.

                  Interesting that you call these folks trans, only to say that they do not identify as such. Why are you invalidating their identity?

                  1. 3

                    Interesting that you call these folks trans, only to say that they do not identify as such. Why are you invalidating their identity?

                    They are not invalidating. Re-read.