1. 48
  1.  

  2. 15

    Duplicating comment from Reddit:

    I would add another challenge: building an integrated tooling ecosystem. In the long ran, ability to write programs to semantically manipulate gigantic codebases seems pretty important to me, and we are quite far from that. We have rustfmt, and rust-analyzer, and Miri, but they all are black boxes. We don’t have a quasi-stable data model of the language, and we don’t have a library implementing that.

    Partially this is due to the language itself: rust’s source code is somewhat disconnected from its semantics, which makes building a natural editable model hard. But this itself i think is partially due to priorities. It seems that historically Rust emphasized tooling-friendliness to much lesser degree than, eg, Dart, Go, or Carbon.

    1. 13

      The issues with library (“crate”) organization are already apparent, and unless something is done about it relatively soon I think we’ll see a fracturing of the Rust ecosystem within 5 years. IMO the fundamental problem is that crates.io is a flat namespace (similar to Hackage or PyPI).

      For example, the other day I needed to create+manipulate CPIO files from within a Rust tool. The library at https://crates.io/crates/cpio has no documentation and limited support for various CPIO flavors, but it still gets the top spot on crates.io just due to the name. There’s also https://crates.io/crates/cpio-archive, which is slightly better (some docs, supports the “portable ASCII” flavor) but it’s more difficult to find and the longer name makes it seem less “official”.

      If I wanted to write my own CPIO library for Rust, it wouldn’t be possible to publish it on crates.io as cpio. I would face the difficult choice between (1) giving it an obscure and opaque codename (like hound the WAV codec, or bunt the ANSI color-capable text formatter) or (2) publishing it the C/C++ way as downloadable tarball[0] on my website or GitHub or whatever.

      Go has a much better story here, because libraries are identified by URL(-ish). I couldn’t publish the library cpio, it would be john-millikin.com/go/cpio or github.com/jmillikin/go-cpio/cpio or something like that. The tooling allows dependencies to be identified in a namespace controlled by the publisher. Maven has something similar (with Java-style package names anchored by a DNS domain). Even NPM provides limited namespacing via the @org/ syntax.

      [0] By the way, from what I can tell Cargo doesn’t support downloading tarballs from specified URLs at all. It allows dependencies to come from a crates.io-style package registry, or from Git, but you can’t say “fetch https://my-lib.dev/archive/my-lib-1.0.0.tar.gz”. So using this option limits the userbase to less common build tools such as Bazel.

      1. 3

        If I wanted to write my own CPIO library for Rust, it wouldn’t be possible to publish it on crates.io as cpio

        library name doesn’t have to match package name. You can publish jmillikin_cpio which people could use as use cpio.

        1. 8

          Yes but I think the point was that if I, as someone who doesn’t know anything about rust ecosystem, was looking for a cpio package, i probably would not go beyond the “official” cpio.

          1. 7

            If you had to choose between jcreekmore/cpio and indygreg/cpio, you still wouldn’t know which is the better one.

            1. 12

              That’s the point though I think: it makes it more obvious that you actually need to answer that question, because both look equally (not) “official”/“authoritative”.

              1. 4

                I think you’re really stretching this example, because short of at_and_t_bell_labs/cpio there can’t be any official/authoritative cpio package. There may be a popular one, or ideally you should get a quality one. So to me this boils down to just search and ranking. crates-io has merely text-based search without any quality metrics, so it brings up keyword spam instead of good packages.

                The voting/vouching for packages that @mtset suggests would be better implemented without publishing forks under other namespaces as votes. It could be an upvote/star button.

                1. 3

                  If crates.io had organization namespaces, then an “official” CPIO library might have the package name @rust/cpio.

                  This would indicate a CPIO package published by the Rust developers, which would be as close to “official” as putting it into the standard library.

                  1. 2

                    That would be good for officialness, but I think it’s nether realistic nor useful.

                    We are approaching 100K crates. Rust-lang org already has more work than it can handle, and can’t be expected to maintain more than a drop in a bucket of the ecosystem. See what’s available on GitHub under rust-lang-nursery and rust-lang-deprecated.

                    And official doesn’t mean it’s a good choice. You’d have rust-lang/rustc-serialize inferior to dtolnay/serde. And rust-lang/mpsc that is slower and less flexible than taiki-e/crossbeam-channel. And rust-lang/tempdir instead of stebalien/tempfile, and rust-lang/lazy_static instead of matklad/once_cell.

                    1. 1

                      We are approaching 100K crates. Rust-lang org already has more work than it can handle, and can’t be expected to maintain more than a drop in a bucket of the ecosystem.

                      Yep! That’s true! In a healthy ecosystem, the number of official packages is extremely small as a percentage. Look at C++, C#, Java, Go – there might be a few dozen (at most) packages maintained by the developers of the language, compared to hundreds of thousands of third-party packages.

                      And official doesn’t mean it’s a good choice. You’d have rust-lang/rustc-serialize inferior to dtolnay/serde. And rust-lang/mpsc that is slower and less flexible than taiki-e/crossbeam-channel. And rust-lang/tempdir instead of stebalien/tempfile, and rust-lang/lazy_static instead of matklad/once_cell.

                      Also yep! And also (IMO) totally normal and healthy. The definition of “good choice” will vary between users. Just because a package is maintained by the language team doesn’t mean it will be appropriate for all use cases. That’s why Go’s flags package can co-exist with third-party libraries like github.com/jessevdk/go-flags or github.com/spf13/pflag.

                    2. 1

                      I wish you had put that at the top of your original post. :)

                      1. 2

                        I think it’s a minor point, to be honest. Even if crates.io never gets organizational namespaces, just being able to upload to a per-user namespace would be a sea-change improvement over current state.

                2. 1

                  Personally, I think this is a great use case for a social-web system; we’ve already seen this with metalibraries like stdx and stdcli, though none have stood the test of time. I think a namespacing system with organizational reexports could really shine; I’d publish cpio (sticking with the same example) as mtset/cpio, and then it could be included in collections as stdy/cpio or embedded/cpio or whatever. Reviews and graph data would help in decisionmaking, too.

              2. 6

                There’s some issues with that approach.

                First, I do want the package name to match the library name, or at least be ${namespace}${library_name} where ${namespace} is something clearly namespace-ish. If I did not have this requirement then I would name crates.io packages with a UUID. And to be honest, I don’t think anyone would do that remapping – people would type use jmillikin_cpio::whatever and grumble about the arrogance of someone who uses their own name in library identifiers.

                Second, a namespace provides access control. I’m the only person who can create Go libraries under the namespaces john-millikin.com/ or github.com/jmillikin/, but anyone in the world can create crates.io packages starting with jmillikin_. It’s just a prefix, it has no semantics other than implying something (ownership) to human viewers that may or not be true.

                1. 4

                  And to be honest, I don’t think anyone would do that remapping – people would type use jmillikin_cpio::whatever and grumble about the arrogance of someone who uses their own name in library identifiers.

                  To clarify, it’s the author of the library who sets its default name. You can have the following in Cargo.toml of your cpio library:

                  [package]
                  name = “jmillikin_cpio”
                  
                  [lib]
                  name = “cpio”
                  

                  Users would then use jmillikin_cpio in their Cargo.tomls, but in the code the name would be just cpio.

                  This doesn’t solve the problem of access control, but it does solve the problem of names running out.

                  1. 3

                    Yes, per my post I’m aware that’s possible, I just think it would be bad. What you propose would be semantically equivalent to using a UUID, since the package name and library name would no longer have any meaningful relationship.

                    In other words, I think your code example is semantically the same as this:

                    [package]
                    name = "c3f0eea3-72ab-4e79-a487-8b162153cfd1"
                      
                    [lib]
                    name = "cpio"
                    

                    Which I dislike, because I think that it should be possible to compute the library name from a package name mechanically (as is the idiom in Go).

                2. 4

                  Using a prefix doesn’t provide access control which is an important feature of namespacing. If there’s no access control, you don’t really have a namespace.

                  For example, I might publish all my packages as leigh_<package> to avoid collisions with other people, but there’s nothing stopping someone else from publishing a package with a leigh_ prefix.

                  This is a real problem, especially with the prevalent squatting going on on crates.io.

                  For example, recently I was using a prefix for a series of crates at work, and a squatter published one of the crates just before I did. So now I have a series of crates with a prefix that attempts to act as a namespace, and yet one of the crates is spam.

                  Most other ecosystems have proven that namespacing is an effective tool.

              3. 9

                On Rust’s “diversity numbers being terrible” and worse than the industry – is it worse than open source? For example, I think the last time anyone checked, there were many fewer women open source programmers vs. paid career programmers.

                That is, I think all open source projects have this issue (and lobste.rs and HN seem to as well :-/ )

                1. 4

                  My impression is that Rust’s gender ratio number is in fact worse than open source. One possible reason is that compiler is a field gender-coded to be masculine. That is, I think Rust-the-project’s number is worse than Rust ecosystem (e.g. bodil, the maintainer of im crate, is a woman) but similar to, say, LLVM.

                  1. [Comment removed by moderator pushcx: Phrasing abuse as hypotheticals doesn't make it acceptable to post.]

                    1. 22

                      To expand on @madhadron’s reply, it might be helpful to look at explicit examples of how women were pushed out of early computer science. Here’s an article by Mar Hicks which gives some, focusing on the UK:

                      These examples also often fail to reckon with the fact that women were long favored for early computing jobs because they were seen as less intelligent, less valuable workers.[4] As computing work professionalized and rose in status in the late 20th century, women workers did not rise with the tide.[5] Instead, women became viewed as no longer being a good fit for the job.[6]

                      The UK, our closest historical cousin, provides a compelling and cautionary tale in the history of computing. In 1959, a woman computer programmer trained two men with no computing experience to take over as managers in the computing installation in the British government that she was in charge of in all but name. After she had trained them and they stepped into management roles, she was demoted to an assistantship below them. Behind the scenes, her supervisors discussed her talents candidly, and also the fact that as a woman she was not suitable for higher-level positions that might involve more responsibility or—heaven forbid—managing men employees. Her supervisors wrote to each other about when she might likely leave to get married, hoping turnover through marriage would take care of the problem of what to do with her.[7]

                      These patterns of discrimination may just seem like ancient history, but there is a lot of cultural inertia that I think we ought to pay attention to.

                      On the matter of innate differences between men and women, e.g. whether maybe women just aren’t biologically wired to like analytical activities like programming, I can recommend the book Inferior: How Science Got Women Wrong and the New Research That’s Rewriting the Story by Angela Saini. It investigates whether some common stereotypes about differences between men and women have any basis in science. I think the book is particularly valuable because the author tried to be impartial by interviewing scientists on both sides of the barricade.

                      1. 13

                        it might be helpful to look at explicit examples of how women were pushed out of early computer science

                        Another example is NASA and American business where “computers” were replaced with male programmers instead of training that existing workforce.

                      2. 20

                        In US 18% of computer science undergrads are women, in India it’s 42%. https://cacm.acm.org/magazines/2015/5/186026-decoding-femininity-in-computer-science-in-india/fulltext

                        1. 3

                          That implies nothing about what the number of female science undergrads in the US would be in ideal circumstances.

                          Data from the most ‘feminist’ countries shows that the more freedom and equality women have, the more likely they are to not be interested in certain professions.

                        2. 18

                          Many of these were created after 2000, which makes it difficult to argue this is still just a historical artifact that was grandfathered in

                          You should try asking some women what their experiences in tech fields is like. I’ve personally heard stories of blatant discrimination against women in STEM fields long after 2000, so suggesting that sexism has been “solved” and women are encouraged to enter these fields just as much as men seems optimistic, at best.

                          1. 30

                            The only obvious conclusion is that dismantling explicitly constructed systems of privilege is hard. The US and South Africa are still living with lingering apartheid (though calling the US system that is anachronistic, since the later South African system was based on the US one). In computing, the British threw away their early advantages in computing to force it to be a male dominated field. Microcomputing was explicitly aimed at males. MIT for decades explicitly discriminated to keep women out of engineering and computing. This continues to the well documented abuses and discrimination that women regularly suffer in computing. This is within the larger context of the lingering system of treating women as chattel that, again, we are still dealing with the lingering aftereffects of.

                            a misguided belief in a gender balance that will never arrive? Would that really be so bad?

                            You would never consider uttering that about “skin color balance.” None of the current knowledge of sex differences in cognition suggest any particular advantage in either direction for programming, so it is an equivalent statement.

                            1. 5

                              You would never consider uttering that about “skin color balance.” None of the current knowledge of sex differences in cognition suggest any particular advantage in either direction for programming, so it is an equivalent statement.

                              Someone who uttered “a misguided belief in a gender balance” just might.

                            2. 21

                              never been more accessible

                              And yet there are still assholes like you making comments like these, so we haven’t come as far as we’d like to think. In all my years I’ve never seen any evidence that women have a biological disadvantage compared to men in our field. But I’ve seen countless examples of people like you turning women away, making them feel excluded, bitter, and jaded.

                              I’m disgusted your comment got so many upvotes on Lobsters.

                              1. 6

                                I appreciate your post. I think it makes good sense. I know you are getting a lot of people disagreeing, but I want to let you know that there are people out here who agree with your point of view. I think many of those replying are not as familiar with the subject as they think. I was originally going to PM you this, but I think it’s important to disagree with this sort of mass “consensus” publicly.

                                1. 7

                                  I guess with this familiarity you claim you must have some pretty good references on the matter to share with all of us?

                                  1. 3

                                    The claims may or may not be true, but I think they are exactly right that it’s borderline banned to even consider that they might be true or to discus it.

                                  2. 2

                                    FYI your comment was just publicly reported to the mods.

                                    (I do agree with the reporter that these kinds of discussions or submissions do not belong in Lobste.rs as there are better places for them.)

                              2. 3

                                One thing I’m worried about is having common datatypes established. I guess this falls under “Navigating the crate ecosystem”.

                                I don’t want every crate to bring in its own Vector2D, Vector3D, LinkedList, Color etc. I don’t use Rust enough to have a good understanding of how painful this is currently, maybe somebody could enlighten me, but every time I run into something like this in C# it’s quite painful. I want the community to agree over time on some infrastructural base for datatypes.

                                C# minimizes this by having a huge ass standard library. I don’t know what Rust’s approach will be, if any.

                                1. 4

                                  I think this is less of a problem with Rust, or at least I would redefine the problem as needing a foundational set of traits that everyone implements for their datastructures. Rust traits are a lot more powerful than C# interfaces (particularly with a subset of GATs being stabilized now), and I think they provide a better way forward than requiring the standard library to carry everything.

                                  Rust already has things like indexing, iteration, and hashing baked in to the standard library, but extending that to cover things like: a richer form of key-based access (e.g. the Entry pattern seen in HashMap/BTreeMap), the general class of structure (map/graph/tree/etc.); intrusiveness; whether a data structure is purely functional or not (i.e. changes to the structure create a new structure, rather than mutate the old). This would allow libraries to implement functionality on top of any data structure that implements a requisite set of traits, rather than needing to implement/provide their own.

                                  That kind of thing is much better for the ecosystem in my opinion, but is a hard problem in its own right, because the traits need to be general, but not so general that they are useless because of abstraction overhead. I’m not even sure what the right set of traits are, but I do think there are some missing.

                                2. 2

                                  An important aspect of inclusivity is being able to accommodate a diversity of opinions. If we can only get along when everyone agrees, then we cannot be diverse or inclusive. While our preference for consensus has served us well in some areas, it has also caused problems. Our culture of avoiding conflict rather than resolving it is unhealthy and has led to dysfunctional governance.

                                  Could anyone from the Rust community point to the specific problems this has caused? Rust has always had the reputation of being more inclusive compared to other communities; this report directly contradicts that, so I’m naturally curious.

                                  1. 1

                                    I would add one more thing that I worry about from time to time: Now that rust isn’t (as much) financially supported by Mozilla, is the language too complex to manage? Async/await isn’t even done yet but it added a huge new surface area. I think it would be good to focus on simplifying/unifying some parts of the language to let more of it fit into people’s heads at once. For example, making async-vs-sync or macros more transparent (I have no idea how) would go a long way.