1. 22

  2. 14

    The possibility to test the compiler on almost all public Rust code is amazing, and it’s possible thanks to Cargo being the standard build system. Can you imagine the effort to build and run tests on 30000 random C projects?

    1. 17

      Perl has been doing this for over more than a decade, building and testing everything on CPAN. http://stats.cpantesters.org/

      1. 5

        I gave a talk recently about how Perl’s monthly releases happen which covered this and more.

        1. 2

          That’s amazing, I had no idea.

        2. 3

          I actually was imagining that recently for benchmarking verification and testing tools. I found that they’re hosted in many places, use a bunch of different build systems, preprocessor magic that hurts reproducibility, and (see Coverity paper) most seem to pick a different dialect of C which may or may not be analyzable.

          Just getting them all pulled and buildable for first benchmark would be a nightmare.

          1. 1

            There are a lot of languages I think could theoretically do this. D for example could do with dub I would think.

            1. 1

              Sure, all you need is a registry. But we’re doing it since… Forever.

            2. 1

              Julia does something similar, I think

            3. 2

              Crater is really heavy on resources: at the moment I think we have 24 cores and 48GB of RAM, 4 terabytes of disk space, so it’s not something where you can throw out some small virtual machine and get meaningful results out of it. running cargo test on the entire ecosystem takes a week

              Wow, I hadn’t realized how expensive Crater is. I wonder how much of that is due to the scope of the project and how much is because of the slow compile times in Rust.

              I’ve seen this affect other parts of Rust too, I forget where but I think I remember Centril mentioning that rollups are a really unpleasant solution to how long the CI tests take to run (> hour).

              Niko mentioned using Cranelift for faster compile times a while back (https://internals.rust-lang.org/t/possible-alternative-compiler-backend-cretonne/4275), which I’d really like to see, I think it would help the ecosystem a lot.