1. 54
  1.  

  2. 32

    I wasn’t implying. I was stating a fact.

    And he’s wrong about that.

    https://github.com/uutils/coreutils is a rewrite of a large chunk of coreutils in Rust. POSIX-compatible.

    1. 12

      So on OpenBSD amd64 (the only arch rust runs on… there are at least 9 others, 8 7 or 6 of which rust doesn’t even support! )… this fails to build:

      error: aborting due to 19 previous errors
      
      error: Could not compile `nix`.
      warning: build failed, waiting for other jobs to finish...
      error: build failed
      
      1. 8

        Yep. The nix crate only supports FreeBSD currently.

        https://github.com/nix-rust/nix#supported-platforms

      2. 8

        The openbsd guys are stubborn of course, though they might have a point. tbh somebody could just fork a BSD OS to make this happen. rutsybsd or whatever you want to call it.

        edit: just tried to build what you linked, does cargo pin versions and verify the downloads? fetching so many dependencies at build time makes me super nervous. Are all those dependencies BSD licensed? It didn’t even compile on my machine, maybe the nixos version of rust is too old - i don’t know if the rust ecosystem is stable enough to base an OS on yet without constantly fixing broken builds.

        1. 10

          just tried to build what you linked, does cargo pin versions and verify the downloads?

          Cargo pins versions in Cargo.lock, and coreutils has one https://github.com/uutils/coreutils/blob/master/Cargo.lock.

          Cargo checks download integrity against the registry.

          For offline builds, you can vendor the dependencies: https://github.com/alexcrichton/cargo-vendor, downloading them all and working from them.

          Are all those dependencies BSD licensed?

          Yes. Using: https://github.com/onur/cargo-license

          Apache-2.0/MIT (50): bit-set, bit-vec, bitflags, bitflags, block-buffer, byte-tools, cc, cfg-if, chrono, cmake, digest, either, fake-simd, filetime, fnv, getopts, glob, half, itertools, lazy_static, libc, md5, nodrop, num, num-integer, num-iter, num-traits, num_cpus, pkg-config, quick-error, rand, regex, regex-syntax, remove_dir_all, semver, semver-parser, sha2, sha3, tempdir, tempfile, thread_local, time, typenum, unicode-width, unindent, unix_socket, unreachable, vec_map, walker, xattr

          BSD-3-Clause (3): fuchsia-zircon, fuchsia-zircon-sys, sha1

          MIT (21): advapi32-sys, ansi_term, atty, clap, data-encoding, generic-array, kernel32-sys, nix, onig, onig_sys, pretty-bytes, redox_syscall, redox_termios, strsim, term_grid, termion, termsize, textwrap, void, winapi, winapi-build

          MIT OR Apache-2.0 (2): hex, ioctl-sys

          MIT/Unlicense (7): aho-corasick, byteorder, memchr, same-file, utf8-ranges, walkdir, walkdir

          It didn’t even compile on my machine, maybe the nixos version of rust is too old - i don’t know if the rust ecosystem is stable enough to base an OS on yet without constantly fixing broken builds.

          This is one of my frequent outstanding annoyances with Rust currently: I don’t have a problem with people using the newest version of the language as long as their software is not being shipped on something with constraints, but at least they should document and test the minimum version of rustc they use.

          coreutils just checks against “stable”, which moves every 6 weeks: https://github.com/uutils/coreutils/blob/master/.travis.yml

          Can you give me rustc --version?

          Still, “commitment to stability” is a function of adoption. If, say, Ubuntu start shipping a Rust version in an LTS release, more and more people will try to stay backward compatible to that.

          1. 2

            rustc 1.17.0 cargo 0.18.0

            1. 11

              You’re probably hitting https://github.com/uutils/coreutils/issues/1064 then.

              Also, looking at it, it is indeed that they use combinatior functionality that became available in Rust 1.19.0. std::cmp::Reverse can be easily dropped and replaced by other code if 1.17.0-support would be needed.

              Thanks, I filed https://github.com/uutils/coreutils/issues/1100, asking for better docs.

              1. 1

                thanks for doing that, great community outreach :P

          2. 5

            Rust is “stable” in the sense that it is backwards compatible. However it is evolving rapidly so new crates or updates to crates may require the latest compiler. This won’t mean you’ll have to constantly fix broken builds; just that pulling in new crates may require you to update to the latest compiler.

            1. 4

              Yes, Cargo writes a Cargo.lock file with versions and hashes. Application developers are encouraged to commit it into version control.

              Dependencies are mostly MIT/Apache in the Rust world. You can use cargo-license to quickly look at the licenses of all dependencies.

              Redox OS is fully based on Rust :)

            2. 4

              Although you’re right to point out that project, one of Theo’s arguments had to do with compilation speeds:

              By the way, this is how long it takes to compile our grep:

              0m00.62s real 0m00.63s user 0m00.53s system

              … which is currently quite undoable for any Rust project, I believe. Cannot say if he’s exaggerating how important this is, though.

              1. 10

                Now, at least for GNU coreutils, ./configure runs a good chunk of what rust coreutils needs to compile. (2mins for a full release build, vs 1m20.399 just for configure). Also, the build is faster (coreutils takes a minute).

                Sure, this is comparing apples and oranges a little. Different software, different development states, different support. The rust compiler uses 4 cores during all that (especially due to cargo running parallel builds), GNU coreutils doesn’t do that by default (-j4 only takes 17s). On the other hand: all the crates that cargo builds can be shared. That means, on a build farm, you have nice small pieces that you know you can cache - obviously just once per rustc/crate pairing.

                Also, obviously, build farms will pull all kinds of stunts to accelerate things and the Rust community still has to grow a lot of that tooling, but I don’t perceive the problem as fundamental.

                EDIT: heh, forgot --release. And that for me. Adjusted the wording and the times.

                1. 5

                  OpenBSD doesn’t use GNU coreutils, either; they have their own implementation of the base utils in their tree (here’s the implementation of ls, for example). As I understand it, there’s lots of reasons they don’t use GNU coreutils, but complexity (of the code, the tooling, and the utils themselves) is near the top of the list.

                  1. 6

                    Probably because most(all?) the openBSD versions of the coreutils existed before GNU did, let alone GNU coreutils. OpenBSD is a direct descendant of Berkeley’s BSD. Not to mention the licensing problem. GNU is all about the GPL. OpenBSD is all about the BSD(and it’s friends) license. Not that your reason isn’t also probably true.

                  2. 2

                    That means, on a build farm, you have nice small pieces that you know you can cache - obviously just once per rustc/crate pairing.

                    FWIW sccache does this I think

                  3. 7

                    I think it would be more fair to look at how long it takes the average developer to knock out code-level safety issues + compiles on a modern machine. I think Rust might be faster per module of code. From there, incremental builds and caching will help a lot. This is another strawman excuse, though, since the Wirth-like languages could’ve been easily modified to output C, input C, turn safety off when needed, and so on. They compile faster than C on about any CPU. They’re safe-by-default. The runtime code is acceptable with it improving even better if outputting C to leverage their compilers.

                    Many defenses of not using safe languages is that easy to discount. And OpenBSD is special because someone will point out that porting a Wirth-like compiler is a bit of work. It’s not even a fraction of the work and expertise required for their C-based mitigations. Even those might have been easier to do in a less-messy language. They’re motivated more by their culture and preferences than any technical argument about a language.

                    1. 3

                      It’s a show stopper.

                      Slow compile times are a massive problem for C++, honestly I would say it’s one of the biggest problems with the language, and rustc is 1-2 orders of magnitude slower still.

                      1. 12

                        It’s a show stopper.

                        Hm, yet, last time I checked, C++ was relatively popular, Java (also not the fastest in compilation) is doing fine and scalac is still around. There’s people working on alternatives, but show stopper?

                        Sure, it’s an huge annoyance for “build-the-world”-approaches, but well…

                        Slow compile times are a massive problem for C++, honestly I would say it’s one of the biggest problems with the language, and rustc is 1-2 orders of magnitude slower still.

                        This heavily depends on the workload. rustc is quite fast when talking about rather non-generic code. The advantage of Rust over C++ is that coding in mostly non-generic Rust is a viable C alternative (and the language is built with that in mind), while a lot of C++ just isn’t very useful over C if you don’t rely on templates very much.

                        Also, rustc stable is a little over 2 years old vs. C/C++ compilers had ample headstart there.

                        I’m not saying the problem isn’t there, it has to be seen in context.

                        1. 9

                          C++ was relatively popular, Java (also not the fastest in compilation) is doing fine and scalac is still around.

                          Indeed, outside of gamedev most people place zero value in fast iteration times. (which unfortunately also implies they place zero value in product quality)

                          rustc is quite fast when talking about rather non-generic code.

                          That’s not even remotely true.

                          I don’t have specific benchmarks because I haven’t used rust for years, but see this post from 6 months ago that says it takes 15 seconds to build 8k lines of code. The sqlite amalgamated build is 200k lines of code and has to compile on a single core because it’s one compilation unit, and still only takes a few seconds. My C++ game engine is something like 80k if you include all the libraries and builds in like 4 seconds with almost no effort spent making it compile fast.

                          edit: from your coreutils example above, rustc takes 2 minutes to build 43k LOC, gcc takes 17 seconds to build 270k, which makes rustc 44x slower…

                          The last company I worked at had C++ builds that took many hours and to my knowledge that’s pretty standard. Even if you (very) conservatively say rustc is only 10x slower, they would be looking at compile times measured in days.

                          while a lot of C++ just isn’t very useful over C if you don’t rely on templates very much.

                          That’s also not true at all. Only small parts of a C++ codebase need templates, and you can easily make those templates simple enough that it has little to no effect on compile times.

                          Also, rustc stable is a little over 2 years old vs. C/C++ compilers had ample headstart there.

                          gcc has gotten slower over the years…

                          1. 6

                            Even if you (very) conservatively say rustc is only 10x slower,

                            Rustc isn’t slower to compile than C++. Depends on the amount of generics you use, but the same argument goes for C++ and templates. Rust does lend itself to more usage of generics which leads to more compact but slower-compiling code, which does mean that your time-per-LOC is higher for Rust, but that’s not a very useful metric. Dividing LOCs is not going to get you a useful measure of how fast the compiler is. I say this as someone who has worked on both a huge Rust and a huge C++ codebase and know what the compile times are like. Perhaps slightly worse for Rust but not like a 2x+ factor.

                            The main compilation speed problem of Rust vs C++ is that it’s harder to parallelize Rust compilations (large compilation units) which kind of leads to bottleneck crates. Incremental compilation helps here, and codegen-units already works.

                            Rust vs C is a whole other ball game though. The same ball game as C++ vs C.

                            1. 2

                              That post, this post, my experience, lines, seconds… very scientific :) Hardware can be wildly different, lines of code can be wildly different (especially in the amount of generics used), and the amount of lines necessary to do something can be a lot smaller in Rust, especially vs. plain C.

                              To add another unscientific comparison :) Servo release build from scratch on my machine (Ryzen 7 1700 @ 3.9GHz, SATA SSD) takes about 30 minutes. Firefox release build takes a bit more. Chromium… even more, closer to an hour. These are all different codebases, but they all implement a web browser, and the compile times are all in the same ballpark. So rustc is certainly not that much slower than clang++.

                              Only small parts of a C++ codebase need templates

                              Maybe you write templates rarely, but typical modern C++ uses them all over the place. As in, every STL container/smart pointer/algorithm/whatever is a template.

                              1. 2

                                To add another unscientific comparison :) Servo release build from scratch on my machine (Ryzen 7 1700 @ 3.9GHz, SATA SSD) takes about 30 minutes. Firefox release build takes a bit more. Chromium… even more, closer to an hour. These are all different codebases, but they all implement a web browser, and the compile times are all in the same ballpark. So rustc is certainly not that much slower than clang++.

                                • Firefox 35.9M lines of code
                                • Chromium 18.1M lines of code
                                • Servo 2.25M lines of code

                                You’re saying compiling 2.25M lines of code for a not feature complete browser that takes 30 minutes is comparable to compiling 18-35M lines of code in ‘a bit more’?

                                1. 4

                                  Line counters like this one are entirely wrong.

                                  This thing only counted https://github.com/servo/servo. Servo code is actually split among many many repositories.

                                  HTML parser, CSS parser, URL parser, WebRender, animation, font sanitizer, IPC, sandbox, SpiderMonkey JS engine (C++), Firefox’s media playback (C++), Firefox’s canvas thingy with Skia (C++), HarfBuzz text shaping (C++) and more other stuff — all of this is included in the 30 minutes!

                                  plus,

                                  the amount of lines necessary to do something can be a lot smaller in Rust

                                  1. 2

                                    Agreed, it grossly underestimates how much code Chromium contains. You are aware of the horrible depot_tools and the amount of stuff they pull in?

                                    My point was, you are comparing a feature incomplete browser that is a smaller code base at least in one order of magnitude but takes 30 minutes compared to “closer to an hour” of Chromium. If think your argument doesn’t hold - you are free to provide data to prove me wrong.

                                  2. 3

                                    Servo’s not a monolithic codebase. Firefox is monolithic. It’s a bad comparison.

                                    Chromium is also mostly monolithic IIRC.

                          2. 2

                            Free- and OpenBSD can compile userland from source:

                            So decent compile times are of essence, especially if you are targeting multiple architectures.

                          3. 6

                            Well, ls is listed as only semi done, so he’s only semi wrong. :)

                            1. 11

                              The magic words being “There has been no attempt”. With that, especially by saying “attempt”, he’s completely wrong. There have been attempts. At everything he lists. (he lists more here: https://www.youtube.com/watch?v=fYgG0ds2_UQ&feature=youtu.be&t=2112 all of what Theo mentions has been written, in Rust, some even have multiple projects, and very serious ones at that)

                              For a more direct approach at BSD utils, there’s the redox core utils, which are BSD-util based. https://github.com/redox-os/coreutils

                              1. 2

                                Other magic words are “POSIX compatible”. Neither redox-os nor the uutils linked by @Manishearth seem to care particularly about this. I haven’t looked all that closely, but picking some random utils shows that none of them is fully compliant. It’s not even close, so surely they can’t be considered valid replacements of the C originals.

                                For example (assuming that I read the source code correctly) both implementations of cat lack the only POSIX-required option -u and the implementations of pwd lack both -L and -P. These are very simple tools and are considered done at least by uutils…

                                So, Theo may be wrong by saying that no attempts have been made, but I believe a whole lot of rather hard work still needs to be done before he will acknowledge serious efforts.

                                1. 5

                                  This rapidly will devolve into a no true scotsman argument.

                                  https://github.com/uutils/coreutils#run-busybox-tests

                                  uutils is running the busybox tests. Which admittedly test for something other than POSIX compliance, but neither the GNU or BSD coreutils are POSIX-compliant anyway.

                                  uutils is based on the GNU coreutils, redox’s ones are based on the BSD ones, which is certainly a step in the right direction and can certainly be counted as an attempt.

                                  For example (assuming that I read the source code correctly) both implementations of cat lack the only POSIX-required option -u and the implementations of pwd lack both -L and -P.

                                  Nobody said they were complete.

                                  All we’re talking about is Theo’s rather strong point that “there has been no attempt”. There has.

                            2. 1

                              I’m curious about this statement in TdR in the linked email

                              For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.

                              Is this true?

                              1. 15

                                As always with these complaints, I can’t find any reference to exact issues. What’s true is that LLVM uses quite a bit of memory to compile and rustc builds tend not to be the smallest themselves. But not that big. Also, recent improvements have definitely worked here

                                I do regularly build the full chain on a ACER c720p, with FreeBSD, which has a celeron and 2 GB of RAM, I have to shut down the X server and everything before, but it works.

                                As usual, this is probably an issue of the kind “please report actual problems, and we work fixing that”. “We want to provide a build environment for OpenBSD and X Y Z is missing” is something we’d be happy support, some fuzzy notion of “this doesn’t fulfill our (somewhat fuzzy) criteria” isn’t actionable.

                                Rust for Haiku does ship Rust with i386 binaries and bootstrapping compilers (stage0): http://rust-on-haiku.com/downloads

                                1. 10

                                  As always with these complaints, I can’t find any reference to exact issues.

                                  Only because it’s a thread on the OpenBSD mailing lists, people reading that list have the full context of the recent issues with Firefox and Rust.

                                  I’ll assume you just don’t follow the list so here is the relevant thread lang/rust: update to 1.22.1

                                  • For this release, I had lot of problem for updating i386 to 1.22.1 (too much memory pressure when compiling 1.22 with 1.21 version). So the bootstrap was initially regenerated by crosscompiling it from amd64, and next I regenerate a proper 1.22 bootstrap from i386. Build 1.22 with 1.22 seems to fit in memory.

                                  As I do all this work with a dedicated host, it is possible that ENOMEM will come back in bulk.

                                  And if the required memory still grows, rustc will be marked BROKEN on i386 (and firefox will not be available anymore on i386)

                                  1. 7

                                    Only because it’s a thread on the OpenBSD mailing lists, people reading that list have the full context of the recent issues with Firefox and Rust.

                                    Sure, but has this:

                                    And if the required memory still grows, rustc will be marked BROKEN on i386 (and firefox will not be available anymore on i386).

                                    Reached the Rust maintainers? (thread on the internals mailing list, issue on rust-lang/rust?)

                                    I’m happy to be corrected.

                                    1. 7

                                      Reached the Rust maintainers? (thread on the internals mailing list, issue on rust-lang/rust?)

                                      I don’t know. I don’t follow rust development, however the author of that email is a rust contributor like I mentioned to you in the past so I assume that it’s known to people working on the project. Perhaps you should check on that internals mailing list, I checked rust-lang/rust on github but didn’t find anything relevant :)

                                      1. 7

                                        I checked IRLO (https://internals.rust-lang.org/) and also nothing. (“internals” by the way referring to the “compiler internals”, we have no closed mailing list). The problem on projects of that scale seems to be that information travel is a huge issue and that leads to aggrevation. The reason I’m asking is not that I want to disprove you, I just want to ensure that I don’t open a discussion that’s already happening somewhere just because something is going through social media currently.

                                        Thanks for pointing that out, I will ensure there’s some discussion.

                                        Reading the linked post, it seems to mostly be a regression when doing the jump between 1.21 to 1.22, so that should probably be a thing to keep an eye out for.

                                      2. 2

                                        Here’s a current Rust bug that makes life hard for people trying to work on newer platforms.

                                  2. 2

                                    I’m skeptical; this has certainly worked for me in the past.

                                    I used 32 bit lab machines as a place to delegate builds to back when I was a student.

                                    1. 4

                                      Note that different operating systems will have different address space layout policies and limits. Your effective space can vary from possibly more than 3GB to possibly less than 2GB.

                                2. 17

                                  In OpenBSD there is a strict requirement that base builds base.

                                  This is actually a pretty reasonable requirement.
                                  I’ve never thought about it before, but probably to challenge C on real operating systems development a safe programming language should be cheap and fast to compile. To me, LLVM is a huge downside of Rust (and many other languages).

                                  1. 9

                                    openbsd base is built with clang now, which uses llvm. But yeah, llvm is not fast.

                                    1. 4

                                      openbsd base is built with clang now…

                                      I believe that is only currently for the x86 & amd64 platforms.

                                      1. 2

                                        Is (modern) gcc THAT much faster to compile itself?

                                        1. 5

                                          I’ve been playing a little with rumprun-netbsd with a self-compiled, for which I needed to build gcc and some build tooling, on top of that I had to compile rustc. Yes, compiling gcc is definitely faster.

                                      2. 7

                                        To me, LLVM is a huge downside of Rust (and many other languages).

                                        We are aware of that and there’s plans to make LLVM just one of the backends at some point.

                                        https://github.com/stoklund/cretonne is a beginning of what could be a fast code-generator. Obviously, proper code generation and optimisation is a huge task.

                                        It’s annoying currently, but, TBQH, mostly constrained by the number of hands available to work on the problem.

                                        1. 6

                                          Hehe, if they want speed try qbe https://c9x.me/compile/ . :P

                                          1. 3

                                            I wouldn’t be surprised if something like that is used, but Rust needs some optimisations very much, so all these things need to be checked.

                                            Also, I should add that Rust does encode a lot of information beneficial to the code generator (such as full aliasing information), so having something that doesn’t assume the input language to be c-like has a huge benefit. LLVM for example suffers from that problem.

                                            Also, an interface for that must be written first :).

                                            1. 1

                                              I was going to suggest qbe myself… ;-)

                                          2. 4

                                            but probably to challenge C on real operating systems development a safe programming language should be cheap and fast to compile.

                                            The Wirth-style systems languages always followed that pattern. The compiler wasn’t acceptable until it could compile itself lightening-fast. The early versions of Pascal got ported to over 70 architectures back when machines were really diverse. I haven’t seen any indication Theo et al tried to adopt that stuff. So, I’m calling the compile-time argument bogus since they wouldn’t switch anyway.

                                            1. 10

                                              So instead of rewrite everything in rust it should have been rewrite everything in pascal? When does it end? FWIW, Theo at point had very high praise for the plan9 C compiler precisely because it was so fast, but there used to be licensing issues and I’m not sure how well it would work with today’s source tree.

                                              1. 4

                                                You’d have asked the question, “So, we should keep fighting with these same code-level issues with C for everything we write? When does it end?” Then, you would’ve just started writing Pascal with the other stuff ported later over time. Alternatively, a C-like language without its problems plus some benefits of alternatives. You wouldn’t have to rewrite anything since you’d still be benefiting from it. If the compilers didn’t catch up, you’d make it output C. You’d still be writing that stuff now with the mitigations and such just being extras because your code prevents most problems. Then, you’d be looking to add benefits of tech like Eiffel Method (esp Design-by-Contract), SPARK/Frama-C, or recently Rust’s temporal/concurrency safety. Just like Ada and Eiffel camps did over time upgrading their languages for improved safety with backward compatibility.

                                                Or did you think the Ada and Eiffel people have to rewrite their stuff in a new language every 5-10 years to avoid common vulnerabilities or problems maintaining large systems that seem to be C-specific?

                                              2. 1

                                                Edit OK, I now see you’re continuing on your comment here:

                                                https://lobste.rs/s/4cf21p/re_integrating_safe_languages_into#c_4ch7ug

                                                The threading on this site takes a bit getting used too…

                                                Previous version for transparency:

                                                OpenBSD supports 12 hardware platforms. Or am I missing something from your statement below?

                                                The early versions of Pascal got ported to over 70 architectures back when machines were really diverse. I haven’t seen any indication Theo et al tried to adopt that stuff.

                                                1. 3

                                                  Portability requiring C was one of the old counters. In this case, Rust was dismissed on portability. I always point out the system languages more easily ported than C or already pretty portable got no adoption from them. Plus, they can use one that compiles to C if it’s that important.

                                                  Yet, even if Rust does compile to those architectures, he’ll just gripe about something else instead of using it.

                                                  Note: Ill respond to Ted’s bigger comments later today or tonighy when Im off work. This one was quick.

                                            2. 29

                                              I think there’s a bigger point missing here with the focus on Rust. Neither Theo or most of the OpenBSD developers want to leave C. There’s been system languages that are safer, compile faster, allow unsafe when necessary, and either had a langX-to-C compiler or could be modified to. The team had no interest in switching despite massive reduction in bugs they offer with almost no effort on developers part as language does it by default. Theo’s recent post cites compile times, compatibility with other architectures, and building self. Wirth-style languages that did that got no adoption from him. If Rust does it, he likewise won’t use it since that’s not what’s going on.

                                              What’s going on is Theo and OpenBSD developers like C (maybe love it), want to use C, and already have a lot of C. Given all that together, they’ll keep using C. Don’t let that last part fool you as a justification for keeping on C for avoiding costly rewrites: OpenBSD people rewrite stuff all the time. They love doing that shit usually to improve security or maintainability. Recently, they added a new randomization that took all kinds of work and discussion for a security argument so probabilistic they can’t tell me precisely what attacks it will stop that old OS wouldn’t. They put massive amounts of effort into those kinds of projects. Rewriting some core utils in a safe language that provably eliminates entire classes of errors at root causes? And one they can learn in days (eg Wirth’s) to months (eg Rust)? Now that is too much work. Someone else will have to do all that, make the compiler meet arbitrary criteria, etc before OpenBSD folks will even begin to consider it.

                                              These double standards on where they put in effort show they do what they do because it’s their culture and preference, not for purely technical reasons. They’re no different than most groups building stuff in that sense. At least they build high-quality stuff they give out for free. Can’t knock them too much given how rare that is (quality part that is). That said, I encourage people to address Theo’s complaints even if they have nothing to do with whether they’ll rewrite OpenBSD in Rust or whatever else. That’s because compilation time, self-hosting, and OpenBSD utilities in safe languages are genuinely-important issues for those of us who might actually use that stuff. The end result of putting that work in is faster, safer tooling with a bunch of utilities written in it. Proponents of safe, systems languages win by dealing with the very things Theo is griping about regardless of whether OpenBSD adopts the tools.

                                              And don’t forget there’s other security-focused OS’s with a lot less labor and maturity that can use the effort. If he turns the stuff down, go to Genode, Redox, Minix 3 (availability-focused) or whoever. They’ll improve over time with people needing practical OS’s for embedded or server appliances having more choices over time. As Redox showed, the end result can happen quite quickly if done in a safe language since you spend less time fixing code-level vulnerabilities. Those with fast compile times would also iterate new features faster. So, have at it I say!

                                              1. 21

                                                I think you’re ignoring the costs of having a system written in multiple languages. There already is one significant component written in a not C language: pkg_add, written in perl. And it has exactly one developer who does all the work and zero other devs touch it.

                                                Something which probably isn’t visible from the outside is how much code gets shared and migrates around within the source tree. I write a function to do something fancy in this utility. It gets copied to that utility. It gets modified. It evolves. It lands in libc. More programs use it. That’s not impossible with rust, since you can write C bindings, but it’s the kind of thing that doesn’t happen. Switching to rust, for all the good that might do for new code, would kill ongoing development of the existing C code because new techniques and patterns would not be backported.

                                                1. 3

                                                  “I think you’re ignoring the costs of having a system written in multiple languages. There already is one significant component written in a not C language: pkg_add, written in perl. And it has exactly one developer who does all the work and zero other devs touch it.”

                                                  I’m saying you should pay it while minimizing it. My claim is one could have improved a safe language of the 90’s or earlier that was close to C. Wirth stayed doing that. Hansen did. Some others did. I keep weaking the requirements to fit your constraints. My comments here include a type/memory-safe superset of C that’s easier to parse, uses C’s types/interfaces, seemlessly includes C libraries, and optionally produces C. It’s hard for me to see how that’s a big cost. It also in no way compares to Perl.

                                                  Whereas, you’re ignoring the costs of using C for your project in this comment. The C language was cobbled together from BCPL to implement a tiny, unsafe OS on PDP-11 hardware constraints. The OpenBSD project is trying to use it to build a large, safe, and secure piece of software on platforms a decade or so more modern. It’s such a bad fit that you all have created your own libraries, coding style, and even OS to make up for all its failures. In essence, there’s your type of C and the regular: almost like two, similar languages already. On top of it, all the details a high-level language will get right automatically have to be manually handled by your people. It sounds like your superset and modifications to C are more labor intensive while leaving all of its problems there with mitigations for some of them. That’s a bigger cost to accept than mine.

                                                  “Something which probably isn’t visible from the outside is how much code gets shared and migrates around within the source tree.”

                                                  Which would’ve happened if you all picked a safer language early on. Now, it might require the compiler/translator to generate human-readable C that conforms to your standards for what’s acceptable. The cool thing is a superset with macros and/or advanced types might let one reuse those patterns with less cognitive overhead plus more machine checks.

                                                  “would kill ongoing development of the existing C code because new techniques and patterns would not be backported.”

                                                  That sounds like a cultural problem. The web folks manage to keep their monoliths working with multiple, programming languages. I’m talking the good developers where it actually works. Then, you say selectively introducing something new would kill all the C activity? They must want to kill it. They could keep it both going if they wanted with the rewrites incremental. With a no-friction proposal like mine, it would be even easier after they build the initial tooling. Regardless, I’d start with the filesystem, the networking stack, and server software that operate on top of them. Those seem to have been the most critical components for OpenBSD for a while now. At the least, a significant subset of OpenBSD representing much of its real-world use might end up memory-safe with all sorts of validation.

                                                2. 3

                                                  Of everything I have seen, checked-c was the only thing i could imagine openbsd adopting, and even then, only if it had a way to translate back to normal C so the code can be distributed. e.g. sshd

                                                  1. 3

                                                    I think that is a little disingenuous, when you consider that they currently support 12 architectures, and generally I can run the same code on my sparc64, amd64, i386, macppc and armv7 machines - where as a lot of these system languages to do not run on all those architectures.

                                                    1. 1

                                                      A safer C or similar language compiling to C would run on same architectures with less effort due to fail fast, HLL features, and (if Wirth-style) fast compilation. The Oberon language was ported to PPC and ARM IIRC with minimal effort (i.e one or two students’ project). Im not sure what each GCC backend took. If harder, then it would be further support.

                                                      Again, point is the portability argument doesnt counter my recomendation since the compile-to-C variant gives you at least the same ISA support as C compilers.

                                                      1. 6

                                                        The Oberon language was ported to PPC and ARM IIRC with minimal effort (i.e one or two students’ project). Im not sure what each GCC backend took. If harder, then it would be further support.

                                                        The effort of building a compiler backend depends much more on the assembly quality you want than on the compiler. To build a minimalistic backend for a new architecture in GCC is probably not much harder than in any other compiler. The biggest necessary part is instruction selection (translating IR operations into target architecture instructions). It might be a little bit more work if GCC has more possible IR operations due to fancy optimizations, but overall the process is tedious leg work. Register allocation and scheduling are the other two necessary parts, but they can usually be mostly reused across backends unless you have something fancy. For popular backends GCC (and LLVM and …) also have a ton of (optional) optimizations in the backend. Building those is a neverending story where we only approximate some illusive optimum. You can write arbitrary amounts of code here.

                                                        1. 1

                                                          Appreciate the info on that. Then, probably scratch that part of my comment.

                                                  2. 20

                                                    I feel like a lot of people are missing some very key points. Especially given there have been links to posix compliant coreutil replacements - Sure TDR might have been wrong about their existence, but he wasn’t wrong in the sense that those tools are likely not going to ever be able to build on things like macppc or alpha:

                                                    1. Rust only builds on amd64 on OpenBSD (i386 runs out of memory) (amd64 and i386 for GHC).
                                                    2. LLVM is only used on OpenBSD in amd64 and i386 while OpenBSD has to src-build on at least: alpha, amd64, arm64, armv7, hppa, i386, loongson, macppc, octeon, sgi, sparc64. Even if you had a core utils replacement - it’s not a comprehensive replacement, you would have to maintain two sets of code, one for “safe” and one for the rest of architectures. I don’t care how safe your language is - doubling down on code base size isn’t a good way to prevent bugs!
                                                    3. For a lang like rust to be any part of the OS - #1 and #2 would have to be resolved.
                                                    4. Now you can write the code to replace tools like ls.. or the kernel.. or what ever.

                                                    Given skade‘s comment on not having enough hands available to make a new backend work, I am not holding my breath for new architectures being supported on an OS that isn’t even officially supported!

                                                    1. 9

                                                      Given skade‘s comment on not having enough hands available to make a new backend work, I am not holding my breath for new architectures being supported on an OS that isn’t even officially supported!

                                                      To be clear: All I’m saying is that we need hands, we can grow them at any time. We are very well set up for contributions (we have stellar merge and discussion times), but we need to focus the resources we can freely allocate.

                                                      If the problem grinds someones gears, we’d be very happy to have and support them, but we also understand very much that people have other things to do. The thing is that we don’t want a half-arsed solution now, we’d prefer a better later.

                                                      I totally see how rustc is just not for you currently, especially given #1/#2.

                                                      1. 7

                                                        There might be understated cultural component IMHO - language package managers probably don’t mesh with an OS that has its own, and I doubt that they want to have to use cargo/cabal/etc in the base system, especially remote packages.

                                                        1. 3

                                                          especially remote packages

                                                          Yeah, great point - everything must be local.

                                                          1. 10

                                                            This, on the other hand, is an issue actively being worked on. debian has similar issues.

                                                            Issue: https://github.com/rust-lang/rust-roadmap/issues/12

                                                            Relevant RFC: https://github.com/aturon/rfcs/blob/build-systems/text/0000-build-systems.md

                                                            This talks about integrating with bazel and similar mostly, but the scope includes OS builds and the issues are similar.

                                                            Mostly, the plan is to allow to use cargo tooling just for dependency resolution and then build according to that plan with the build system of your choice.

                                                            Note that this is an eRFC, which will end up with an experimental feature intended for testing. This isn’t the final cut, there needs to be a proper RFC after that. We need to get that into the hands of users, though.

                                                      2. 9

                                                        I’ve been using Rust based commandline tools for a while now, ripgrep to replace grep and fd-find to replace find.

                                                        These are not commandline-preserving compatible with the old gnu versions, for me that’s a bonus: they feel modern, are fast and are written in a memory-safe language. In my experience you don’t usually write replacements of old utilities in a 100% compatible way, but just switch to using the new tools and change your assumptions.

                                                        100% drop-in replacement rewrites of old C software in Rust or in other memory-safe languages is an impossible goal and it’s not something that is worth trying. Reimagining how old utilities would look like in a modern environment, however is probably a worthy goal.

                                                        1. 7

                                                          In my experience you don’t usually write replacements of old utilities in a 100% compatible way, but just switch to using the new tools and change your assumptions.

                                                          Basic tools like grep, wc, cut etc are all over build chain of Unix-based systems. That’s by “POSIX-compliant” is so important, you need to know a new utility can handle an edge case in a Makefile somewhere.

                                                          1. 7

                                                            Make is a really bad example here, given how much software needs gmake or bsd make and doesn’t support both in practice.

                                                            (Also, by the way, one of the reasons why I think it is good that rustc is not built with make anymore)

                                                            1. 2

                                                              I’m not that knowledgeable about the differences between different flavors of make , but surely all support running POSIX utilities for whatever reason (searching for installed binaries, checking whether a file exists, etc)?

                                                              1. 5

                                                                The differences start appearing as soon as you try to do something as basic as wildcards.

                                                          2. 2

                                                            they feel modern, are fast and are written in a memory-safe language

                                                            They “feel modern” is subjective, so I’ll pass on that point. Saying they’re fast implies that the current implementation is slow, and is “written in a memory-safe language” really an advantage of a program being written in a memory safe language? Seems a little circular.

                                                            I’m by no means an expert so my opinion doesn’t count for shit and so I take no sides in this, but it seems like the OP made a bunch of really good points that need addressing, and this really doesn’t.

                                                            1. 22

                                                              For a tool like ripgrep, that it is written in a memory safe language isn’t a terribly direct benefit to an end user. In particular, folks don’t usually use the tool in a context where a vulnerability would be that damaging.

                                                              With that said, memory safety has other benefits, like (IMO) lower development time. I’ve been maintaining ripgrep for over a year now, and I literally don’t spend time debugging memory related bugs. I don’t just mean “it doesn’t happen that often,” I mean, “it’s never happened, not once.” That’s a lot more time I can put into fixing other types of bugs or adding features. This is good for users, albeit indirectly. Granted, this is probably a pretty lame argument, but I’m just relaying my own experience here as I see it.

                                                              This of course varies from programmer to programmer. A better C or C++ programmer than myself, for example, might not spend any time debugging memory related bugs either. But I’m not that good.

                                                              One might also make an argument in favor of using another memory safe language that has a GC, but the onus is on them to show that the tool can actually compete performance wise. I do suspect it is possible, although it hasn’t been done. I’d expect D to be capable, and probably also Go. (There are some ripgrep-like tools written in Go, but they break down once you push them a little harder than simple literal searches. I cannot conclude that this is a breakdown in Go itself because there is missing implementation work to make it run faster. Particularly in the regex engine.)

                                                              Saying they’re fast implies that the current implementation is slow

                                                              GNU grep isn’t slow in the vast majority of cases. On a single file search, ripgrep might edge it out. As you increase the complexity of the regex (say, by using lots of Unicode features without any literals), then ripgrep starts to get a lot faster (~order of magnitude). In the simpler cases, ripgrep tends to edge out GNU grep by using a mildly smarter heuristic that lets it spend more time in an optimized routine like memchr in common cases. But GNU grep could easily adopt the latter. It would be harder for them to fix their Unicode problems, and probably not worth their time. (Because once you stick a literal into the pattern, it speeds back up again.)

                                                              However, if you say “compare default usages of these tools, including recursive search,” then ripgrep is going to toast GNU grep just because it will use parallelism by default. Which makes this a mostly uninteresting comparison if you’re curious about the details of performance difference, but that doesn’t stop it from being a very interesting UX comparison. That’s typically the source of claims like “ripgrep is so much faster than grep,” which are unfortunately also easily interpreted as a really interesting claim which provokes the question, “oh, so what is the interesting implementation choice that makes it faster?” When folks find out that it’s just parallelism or ignoring certain files (that are in your .gitignore), they rightfully feel cheated. :-) Another less common source of this that people use “grep” to refer to both BSD and GNU grep, and BSD grep has markedly different performance characteristics than GNU grep.

                                                              Performance is complicated, so if you really want to the dirty details, just skip straight to my analysis: http://blog.burntsushi.net/ripgrep/

                                                              1. 11

                                                                I should throw out a complement for your work on ripgrep. It’s impressive. I think ripgrep lands on the wrong side of XY problem style arguments (or more like gets stuck in the middle) because everybody wants to talk about what’s best without first defining requirements. :(

                                                                1. 3

                                                                  Hah, yes, indeed. And thanks. :-) I try to keep tabs on it as best I can and keep it in check, but it’s hard!

                                                                2. 3

                                                                  Although, putting my security hat on, memory-safety is one of things where you just never know how a certain program will get used. I mean, imagemagick is just one of the obvious cases, but what about more subtle stuff such as running something from CI? Untrusted data has a tendency to show up in unexpected places.

                                                                  1. 3

                                                                    Yeah, that is a good point. And I’ve been slowly splitting ripgrep into smaller crates. Other CLI utilities (like fd) are benefiting from that, but that definitely increases the odds of the code showing up in even more unexpected places. The core search routines haven’t been split out yet, but it’s on the list!

                                                                  2. 2

                                                                    “I don’t just mean “it doesn’t happen that often,” I mean, “it’s never happened, not once.” That’s a lot more time I can put into fixing other types of bugs or adding features. This is good for users, albeit indirectly.”

                                                                    That’s not a lame argument: it’s the very argument that safer languages such as Ada, Java, and C# sold businesses on. Knocking out problems that take up lots of debugging time lets you improve your existing product more or build new ones. If you have competition, then you can potentially improve at a faster rate than them. Outside business, there’s quite a few FOSS projects each doing something similar that compete with each other on features and performance even if not security so much. So, being able to rapidly add or fix features could help one get ahead.

                                                                    The champions of that in what few studies exist were the Common LISP and Smalltalk languages. However, if wanting low-level performance, one will have to go with something else. The prior studies put Ada ahead of C with half the defects plus easier maintenance. Rust is most comparable to it in focus areas or strengths. So, choosing Rust for low-defect development without a GC has some empirical support indirectly. I think an interpreted variant with REPL and great debugging might be great, though, for closing some of the gap between it and those dynamic languages.

                                                                    1. 4

                                                                      Sorry, I didn’t mean “lame” as in “bad” or “incorrect,” but rather, “tired” or “old.” i.e., Everyone has heard it before.

                                                                      1. 1

                                                                        Oh, that makes sense.

                                                                        1. 2

                                                                          But yeah otherwise totally agree with everything you said. Even your top-level comment too.

                                                                    2. 2

                                                                      I would add, writing grep in any language with a complicated runtime probably wouldn’t be nice for openbsd. Pledging() relies on knowing what the program/runtime is going to do at any given point, I have had Go problems get killed by random syscalls that the Go runtime decided to make late into program execution.

                                                                      1. 2

                                                                        Thankfully, Rust isn’t one of those languages :) Its “runtime” does uhh… stack backtraces… and possibly some other little things I guess. It’s similar to C++.

                                                                        killed by random syscalls

                                                                        That, IMO, is the worst part of pledge. Capsicum gracefully denies syscalls, letting the program handle the error instead of blowing up :P Sure blowing up lets you quickly debug a core dump of a simple C program (like OpenBSD’s base utilities), but with more complex software written in different languages, I’d much rather handle the damn error.

                                                                    3. 3

                                                                      Ripgrep is a lot faster than grep for general use; not necessarily because the search algorithm is faster, but because it defaults to ignoring .gitignored files and BLOBs, which usually is what you want.

                                                                      1. 2

                                                                        They “feel modern” is subjective, so I’ll pass on that point. Saying they’re fast implies that the current implementation is slow, and is “written in a memory-safe language” really an advantage of a program being written in a memory safe language? Seems a little circular.

                                                                        The parent said “they feel modern, are fast”, not “they feel modern, are fast_er_”. The point is that they are on par, not that the current state is “slow”.

                                                                        I’m by no means an expert so my opinion doesn’t count for shit and so I take no sides in this, but it seems like the OP made a bunch of really good points that need addressing, and this really doesn’t.

                                                                        Given that around here, there’s a discussion on how people can’t find out what the OP exactly meant, I find this statement a bit confusing.

                                                                      2. 1

                                                                        ripgrep does some terribly silly things. One is not having an option to write output as:

                                                                        file1:line MATCH1
                                                                        file1:line MATCH2
                                                                        file2:line MATCH1
                                                                        

                                                                        Which would be obvious to somebody familiar with awk or cut. ‘modern’ often means forgotten knowledge.

                                                                        1. 20

                                                                          ripgrep does exactly that when you use it in a pipeline. When you print to a tty, it uses a “prettier” format. If you don’t like the prettier format, the --no-heading will force the more traditional format. Put it in an alias and then forget about it.

                                                                          ‘modern’ often means forgotten knowledge

                                                                          And is also often a good thing. Compare and contrast POSIX grep’s and ripgrep’s support for Unicode, for example. ripgrep “forgets” a lot of stuff and just assumes UTF-8 everywhere. This works well enough in practice that I’ve never once received a complaint. Well, that’s not true. I did receive one complaint: that ripgrep can’t search UTF-16 files, which are not uncommon on Windows. But I fixed that by adding a bit of code to ripgrep that transcodes the corpus from UTF-16 to UTF-8 when a UTF-16 BOM is detected, and hey, it works! But if I didn’t “forget” about POSIX, then that might not have been possible at all.

                                                                          This comment should not be interpreted as an argument against POSIX, OpenBSD’s (or any other OS) use of it. Instead, I’d like you to interpret it as a criticism of the myth that “the good ol’ days” ever existed at all. My bottom line is that history has a lot to teach us, and just because we don’t copy it exactly as it was doesn’t mean it was forgotten. Sometimes mistakes are made, or perhaps even more frequently, technology simply changes and evolves that makes repeating historical choices a hard pill to swallow in some contexts.

                                                                          So how about instead of focusing on “forgotten” knowledge (and, to the same extent on the other end of the spectrum, avoid the new and shiny—but I’m speaking to a certain audience here so I’ve omitted that) we just focus on the problem we’re trying to solve instead? If there’s some ancient (or new, I don’t care) wisdom that I don’t know about that could make ripgrep a better tool, then I’m all ears.

                                                                          Also, I’ve never once marketed ripgrep as “modern.” ;-)

                                                                          1. 2

                                                                            Most of what I said wrong was based on a mistake that wasted a bunch of my time (which annoyed me), So take the comments written in a bad mood with a grain of salt. And I do like how it respects .ignore files, a tool was definitely needed to deal with that.

                                                                            I do wonder why that “prettier” mode is needed in the first place though. It just seems like a waste of effort and I can’t see the gain. I think AG might do the same, so maybe you were just copying.

                                                                            1. 8

                                                                              ag does do the same, and ag in turn copied it from ack. That particular format has been around since ack has been around, which certainly isn’t as long as grep, but ack has been around for a while by now.

                                                                              In my experience, for every complaint about ripgrep’s defaults, there’s probably N other people that actually like the defaults. For example, I’ve heard from folks that think respecting .ignore by default is a terrible or silly decision. But the defaults don’t need to be some universal statement on what’s right; it’s just my approximation of what most people want based on my perception of the world. It also continues a long held tradition started by ack, and ripgrep definitely descends from ack.

                                                                              While I don’t know for sure, if I didn’t use the pretty format by default, I’m pretty sure I’d have legions of former ag/ack users lining up to complain about it. You might think that’s silly, but I’ve heard of people who use ag over ripgrep because ag is easier to type than rg!

                                                                              1. 1

                                                                                use ag over ripgrep because ag is easier to type than rg

                                                                                haha. If they care about easy typing, they should use shell aliases. I have aliased ‘sr’ (for ‘search’) to mean rg||ag||pt||ack depending on what’s available.

                                                                          2. 1

                                                                            Are you sure? I’m not at the computer right now, but IIRC it outputs exactly like that when it’s writing to a pipe, not a terminal. Try piping into less.

                                                                            1. 1

                                                                              It seems you are right, modern just means violating least surprise. :) At least the output was colourised - I suppose that is to remind me I am not colour blind (or rub the fact in if I am).

                                                                              1. 4

                                                                                Even good old ls does that though, to be fair. Send it to a pipe, and it outputs a newline separated list of file/directory names, send it to a tty, and it outputs a nicely formatted table, probably with ANSI color escape codes.

                                                                                1. 2

                                                                                  It’s only GNU ls that outputs colors.

                                                                                  1. 3

                                                                                    It’s definitely not only GNU ls that outputs colors, see FreeBSD ls -G.

                                                                                  2. 1

                                                                                    I absolutely hate that about ls too, I guess I’m wrong, but I don’t feel to bad about being wrong in this case.

                                                                                  3. 3

                                                                                    I’m not surprised at applications providing nicely formatted output to terminals, checking whether stdout is a tty is an old trick :)

                                                                                    1. 1

                                                                                      If by “nicely formatted” you mean “colored”, then sure. Changing the first-level textual content of the output is a huge and highly undesirable surprise to me, though. I would like to be able to predict what the input to the next piece of pipeline will look like without running it through cat first just to trick isatty.

                                                                            2. 12

                                                                              Fun fact: wall time of building rust is longer than building the other 17,000+ packages in pkgsrc put together.

                                                                              1. 2

                                                                                Source for that?

                                                                                I’d be surprised because aren’t a lot of those packages GNU packages which use the (slow) autoconf?

                                                                                Or does pkgsrc somehow skip the autoconf?

                                                                                If they each take 1 second, that’s 17,000 seconds, or 283 minutes, or nearly 5 hours. Rust takes longer than that, or the average package takes less than 1 second (impressive if true.)

                                                                                1. 6

                                                                                  My SmartOS bulk builds, e.g. https://twitter.com/jperkin/status/644209661516906496. The last rust build (1.21) took 24317 seconds by itself. Note that we distribute our builds across 48 or so concurrent builds, and I concede that “put together” wasn’t the best wording, hence the apparent confusion.

                                                                                  1. 2

                                                                                    Oh that is interesting. Do you have any details on this setup? How do you deal with dependencies and so forth?

                                                                                    I didn’t know that Manta was being used for builds. Do you work for Joyent? I actually went to a talk at the Joyent office in SF about Manta around the time of launch (I think 3 or 4 years ago).

                                                                                    It seems like a really cool technology. As far as I remember, it’s basically a “polyglot” MapReduce with tasks run in jails. I’ve worked with MapReduce quite a bit, and I agree it needs the polyglot/shell treatment (and so do its successors like Flume and Google Cloud Dataflow). It runs in containers at Google, but it’s still at its heart a C++ framework.

                                                                                    Eventually I would like fold this paradigm in my shell [1], but that is still a long ways off.

                                                                                    As for the build part, I’d like to do a similar thing for Alpine Linux in the nearer term. I wrote about experimenting with it here [2]. Alpine still runs their builds on a handful machines and they get backed up a lot! The non-x86 builds appear to be in poor shape in particular.

                                                                                    I would probably run it on a single machine, but do some tricks to avoid reinstalling dependencies for every build, like using bind mounts.


                                                                                    So you’re saying that using 48 machines to build the entire SmartOS repo is faster than building Rust on a single machine? Yes it’s a little misleading, although not exactly unfair if the Rust build doesn’t parallelize. I imagine it must parallelize at least a little bit though. Although I think Rust does do more global compilation than C/C++ do because it doesn’t have the notion of translation units.

                                                                                    [1] http://www.oilshell.org/blog/2017/01/19.html

                                                                                    [2] http://www.oilshell.org/blog/2017/10/06.html

                                                                                    1. 2

                                                                                      Manta is only used for storing the results. Obviously we love Manta but it’s not really suited for this type of work. The builds themselves are standard pkgsrc bulk builds using pbulk. I wrote up a bit about it here: https://www.perkin.org.uk/posts/building-packages-at-scale.html

                                                                                      1. 1

                                                                                        Great writeup! Just read the whole thing. A few thoughts:

                                                                                        1. I’m jealous of Dtrace! I quite don’t understand the ptime and then dtrace -n invocations in the post. Is the dtrace -n limited to whatever is a child of ptime? How does that work?

                                                                                        2. I’m surprised that so much win was due to shell inefficiency, but that’s good to know I would have expected more of the win to be due to scheduling. But that’s good, because it supports my thesis that “shell is a problem”, hence the reason I’m working on a shell!

                                                                                        How does your scheduling work? You start jobs on VMs with a fixed number of chroots and a fixed package list? Or is it more dynamic than that? That is, if you have 48 nodes, is it one job per package build (17K jobs total), or one job per node (48 jobs total), etc.? I couldn’t quite tell from the blog post. I’m not familiar with Joyent’s cloud or pkgsrc and its parameters. Is the number of VMs fixed or dynamic?

                                                                                        I would have expected a lot of the work to go into scheduling (i.e June 2013 section), but it looks like a lot of it was optimizing shell stuff in pkgsrc.

                                                                                        I know that Debian’s build system is very inefficient with regard to shell too. One of the reasons I was interested in Alpine Linux is that the build scripts were more sane. But I have yet to measure the performance there. Everything seems a lot faster in general.

                                                                                        But yes I would like a Linux distro where building every package under the sun in 5 hours is possible! That is pretty amazing! I’m not sure any Linux distro can do that, but I’d love to be corrected.

                                                                              2. 6

                                                                                I like the idea to rewrite basic tools in Rust. Not just because it’s a more secure language than C, but because it’s so much more expressive. I think many complicated programs would benefit from this. Many of the basic tools are full of hackery and obscure stuff you can do in C.

                                                                                However, I agree with the assessment that this isn’t reliable, as long as there isn’t a small self contained Rust compiler written in C which can bootstrap itself. Rustc is a beast to compile as are bigger programs written in Rust.

                                                                                1. 3

                                                                                  I don’t understand the Rust advocacy that wants everything to get rewritten in Rust. Technical issues aside, nobody in their right mind is going to toss out millions of lines of debugged code to rewrite in a new language simply because of hand wavy claims that it’s theoretically safer.

                                                                                  It’s most bizarre that they’re wanting OpenBSD to do so, because the OpenBSD devs are known for being some of the best C developers around. They’re already expert at avoiding and fixing and working around the issues Rust is supposed to solve, so there’s little effort on their part to keep using C safely, but a massive effort to switch to Rust.

                                                                                  1. 6

                                                                                    Rust was born for rewriting a major system, and its major selling point is “C, but safe” (I disagree that it’s handwavy, for what it’s worth, but the selling point is the important bit, not whether it’s valid). It seems pretty natural to me that rust developers see big C code and feel an urge to rewrite it in rust.

                                                                                    And it’s not a worthless effort, either. Rust rewrites have exposed bugs in practice, and frequently result in patches to the older C code.

                                                                                    As for why OpenBSD, I don’t think that there’s any organizational effort to target it. It seems more likely to me that developers who are interested in safety might be drawn to both rust and OpenBSD for their reputations for safety. From that standpoint, it doesn’t seem bizarre at all.

                                                                                    1. 2

                                                                                      Rust was born for rewriting a major system, and its major selling point is “C, but safe” (I disagree that it’s handwavy, for what it’s worth, but the selling point is the important bit, not whether it’s valid). It seems pretty natural to me that rust developers see big C code and feel an urge to rewrite it in rust.

                                                                                      If Rust developers want to rewrite stuff in Rust, then they should just go do it. What bothers me is that they’re nagging everybody else to do it instead. Don’t waste everybody’s time trying to convince us. Just go write the code and show us how much better it is. If it’s really better then people will notice and switch to Rust on their own.

                                                                                      I also take issue with the attitude that any project written in C or C++ is automatically full of memory bugs and security problems. I work on a large-ish C++ project, and have first hand experience that it’s not the case. To my knowledge the product we ship has never had a production bug due to a memory access violation, buffer overflow, or anything like that.

                                                                                      Any C or C++ developer worth hiring is very aware of those problems, and knows to avoid them. And we have language tools like references and shared pointers, and analysis tools like Coverity and Valgrind that catch most problems before we ship. Whether it’s the compiler or another build tool finding the bugs doesn’t matter as long as the bugs are found.

                                                                                      I’m not naive enough to say that we don’t have any memory bugs hiding somewhere, but it’s hardly worth the trouble of teaching 120 people a new language and rewriting a half million lines of code.

                                                                                      And it’s not a worthless effort, either. Rust rewrites have exposed bugs in practice, and frequently result in patches to the older C code.

                                                                                      But rewriting in any language will have that effect. A person can’t rewrite something without understanding what it’s doing, and the process of understanding and scrutinizing it is sure to find bugs. No doubt a C++ rewrite of a Rust program would find bugs in the Rust program, too.

                                                                                      1. 4

                                                                                        What bothers me is that they’re nagging everybody else to do it instead.

                                                                                        From the earlier mailing list post:

                                                                                        what if someone actually bothered? Under what conditions would you consider replacing one of the current C implementations with an implementation written in another, “safer” language?

                                                                                        That doesn’t sound like nagging anyone else to do work, it sounds like a reasonable question: If I did the work, would that actually be enough? It strikes me as perfectly acceptable to ask a project whether contributions would be accepted before actually doing significant development.

                                                                                        Theo’s answer was a resounding “no”.

                                                                                        Maybe you’ve had other, personal experiences with people asking you to rewrite things in a new language, but that is clearly not the case here.

                                                                                        1. 0

                                                                                          I haven’t personally been asked to rewrite anything, but I see it all the time on sites like this one.

                                                                                          Theo’s answer is only a resounding “no” when it’s assumed the person asking the question isn’t going to do the work that needs to be done to include the Rust code. In other words, they’re not going to special case a handful of utilities on one or two platforms, and before Rust can be used on the base OpenBSD system it has to follow the rules and support all of the platforms OpenBSD supports.

                                                                                          1. 4

                                                                                            I see it all the time on sites like this one

                                                                                            I see this claim all the time. I see people unironically requesting a rewrite in Rust… actually, I don’t think I’ve ever seen that.

                                                                                            If you want your claim to be taken seriously, you must at the very least link some examples.

                                                                                    2. 4

                                                                                      I don’t understand the Rust advocacy that wants everything to get rewritten in Rust. […] It’s most bizarre that they’re wanting OpenBSD to do so

                                                                                      I wouldn’t understand it either. Sounds like wishful thinking at best, and downright rude at worst. Can you show me the people that are telling the OpenBSD folks that they want them to rewrite everything in Rust? I’d be happy to send them a polite request that they knock it off, assuming I know how to reach them.

                                                                                      I did read the OP and I didn’t see any such advocacy, but perhaps you’re aware of other posts on the OpenBSD mailing list advocating that they rewrite everything in Rust?

                                                                                      1. 6

                                                                                        I think there’s a between the lines implication which causes people to perceive things differently. I mean, if somebody wants ripgrep, for example, on openbsd, it’s only a pkg_add ripgrep away. So what is meant when somebody starts talking about rust on an openbsd list? A very strange way to ask how do I install ripgrep.

                                                                                        Put another way, it’d be like I visit your house, where you have the walls covered in Monet paintings. And then I start explaining how Picasso is so much better than Monet. You get upset, and then I say “oh, I’m just saying in general, why would you think I’m referring to your decor?”

                                                                                        1. 6

                                                                                          Sure… But, I mean, the original sender didn’t even mention Rust. They talked about memory safe languages and brought up Haskell. I don’t know if they were trying to be coy (and maybe they were, based on the video they linked), but ya know, as much as folks are sick of “Rust advocacy,” I’m personally sick of people overblowing it. I’m obviously deeply entrenched in Rust things so I probably take it a bit more personally, but we do try very hard to be practical about these things and understand that memory safety isn’t the be-all-end-all of programming (I agree with many of your counterpoints to nickpsecurity, for example). But that is never ever going to stop people from spouting stupid stuff like “using a memory unsafe language in this day and age is just irresponsible.” I can’t stand that crap, it’s the same kind of shaming people use against lots of things, e.g., see pretty much anything that the PL people say about Go.

                                                                                          I get the other perspective here. I maintain lots of projects and I get my fair share of drive by questions that don’t really make any sense or unwittingly imply that the amount of work or maintenance involved shouldn’t be part of the calculus. And yeah, they are super frustrating. But it’s just what people do.

                                                                                          1. 4

                                                                                            Ah, good point. On the bright side, if people hear safe and immediately think of rust, that might be considered a sign of success.

                                                                                      2. 4

                                                                                        nobody in their right mind is going to toss out millions of lines of debugged code to rewrite in a new language simply because of hand wavy claims that it’s theoretically safer

                                                                                        Mozilla is doing exactly that with Firefox (36MLoC according to OpenHub). You could say they went even further and invented a new language first.

                                                                                      3. 1

                                                                                        Personally, I think Ada is not ready to replace C in a BSD base system. It doesn’t have mature competing implementations under friendly license. As far as I know, the only free Ada implementation is a GNU one.

                                                                                        Rust is even less ready. It doesn’t even have a standard, so even if I wanted to start a new Rust implementation, that would be a show stopper. At the present time, creating anything new that has to be durable in Rust is like writing in Python 2.

                                                                                        Changes in an operating system require work, obviously. If Rust came with the workforce ready to rewrite a base (like in BSDs) from C to Rust, then that might be the first argument in favor of doing it. Claimed language safety is not it, especially not since all the static and dynamic analysis work that has been done over the last decade.