1. 37
  1.  

  2. 20

    1000 times yes. If Google wants their own libc that’s their business. But LLVM should not be part of their “manifest destiny”. The corporatization of OSS is a scary prospect, and should be called out loud and clear like this every time it’s attempted.

    Otherwise more software infrastructure will go the way of the current near monoculture in web browsers.

    1. 9

      The corporatization of OSS is a scary prospect

      The tool we have to prevent it is copyleft. Notice that Google are talking about making a whole new libc, not enclosing glibc. That’s a feature of copyleft.

      Licences with copyleft terms still permit corporate uses and enable business cases: Google, FB, Amazon, and others all use copyleft software; Red Hat, Canonical, MuseScore, and others base their whole business model on it. But they prevent corporate enclosure, in that a business cannot control the software to satisfy their interests and ignore others. NeXT/Apple did not get to “direct” GCC for everyone in a way that made it work for them, and not for others. In fact, they eventually took their ball elsewhere.

      1. 1

        NeXT/Apple did not get to “direct” GCC for everyone in a way that made it work for them, and not for others. In fact, they eventually took their ball elsewhere.

        Praise the Sun. LLVM is much better platform than GCC for compiler development. Much more open and free.

      2. 4

        The corporatization of OSS is a scary prospect

        OSS is a corporatised term, you’ve been got from the get go.

        1. 1

          Isn’t LLVM already a google project?

          1. 3

            Apple was the first major corporate sponsor. Chris Lattner and Vikram Adve created its as students and later Lattner joined Apple to create a working group there.

            1. 1

              I am aware of that, but I was under the impression that Google input already strongly drives optimization decisions and that Google engineers are prominent in LLVM development.

              1. 1

                Yes. It’d possibly be a fair point, though I still see other companies and organizations involved, I’m not sure to what extent this has an impact on priorities. I’d be curious to hear about recent trade-offs the project has made.

        2. 18

          I’d really like to know why they seem to list requirements that seem squarely in musl libc’s core design goals yet post this like it’s a novel suggestion. Perhaps they have reasons for skipping on musl but it seems lazy or contemptuous to not at least mention why they would prefer to avoid existing glibc alternatives.

          1. 10

            The only thing that comes to mind is that google doesn’t own/control musl, so google’s proposed changes may not be accepted by musl. With their own libc, google can introduce things that other libc implementation would never merge.

            1. 7

              This is easy to say about any project but I found this post originally via twitter from the musl author: https://twitter.com/RichFelker/status/1143292587576635402

              There has likely been no discussion of what might be accepted. If the merge problem is really the case, it probably doesn’t belong in LLVM either. Good riddance throw-it-over-the-fence style OSS if you ask me. Google can keep it to themselves if they’re incapable of this kind of conversation as a corporation (not trying to take offense to developers who may be stuck between to hard places as employees).

              1. 6

                This is easy to say about any project

                Well, yeah, because it’s true a lot of the time. Happens all the time and it’s totally understandable. It really is not even remotely a stretch to imagine that the goals of MUSL wouldn’t align with the goals of Google.

                I once wondered whether I should try to contribute a faster version of memchr to MUSL, but just looking at the tickets on that project made me immediately reconsider. Which isn’t to say MUSL is bad, but it’s to say that MUSL clearly has a specific set of goals in mind, and they do not always line up with everyone else’s goals.

                1. 7

                  There is actually a interesting mailing list thread with googlers on the musl mailing list from a few years back where they considered including musl in chromium, which failed at the end.

                  TL;DR: Lawyers/Legal team had a problem with some files/headers that are in “public domain” and requested a re-license of those files.

                  https://www.openwall.com/lists/musl/2016/03/15/1

                  1. 6

                    Interesting. I’ve been on the bad end of a bunch of Googlers and their licensing concerns too. Not a pleasant experience.

                2. 3

                  This is easy to say about any project

                  Well, yea, but it’s not every day a major company decides to go off and do their own implementation (or fork) of (insert thing here with a some widely-available OSS implementations), and google has a history of doing this (BoringSSL immediately comes to mind).

              2. 7

                Rich Felker (of musl) posted a follow-up in the thread, taking the viewpoint that: 1) LLVM shouldn’t build its own from-scratch libc, and preferably 2) shouldn’t ship a libc at all, whether a new one or musl or otherwise.

                1. 3

                  isn’t musl linux-only?

                  1. 2

                    Not technically but it seems to have been designed with Linux in mind & using it with others kernels can require a lot of effort.

                2. 8

                  Very well written. Technical and moral arguments. Let’s hope the googs have enough sense not to ignore them. Business/corporate arguments would also be welcome.

                  For fun let’s dive into predictions of what will happen over a few years of a theoretical Google LibC (googslic?) lifecycle:

                  • Multiple incompatible major releases that island people & products using it (and/or abandonment).
                  • Arbitrary integration with fleeting G products. “Why is there a GetPinkyFingerprintByOAuthShardShovelPepsi in the libc?”
                  • Kitchen sinking. Parts of a web browser make it into the libc.
                  • Fear-driven adoption. Developers told or hinted that they will not have an option to use other things at a later point (like OpenGL support rumoured to be dropped in Win many years back)
                  • Temporary (artificial) performance, publicity, ranking or otherwise endorsements given to projects that use it (ala AMP).
                  • Counter-responses to community problems being written by PR rather than devs.
                  1. 3

                    There already is Google libc: Bionic. I am interested in how much of these predictions came true in last 10 years.

                    1. 2

                      Don’t forget the always popular “on by default phone-home analytics, so we can see how many people are using it”.

                    2. 8

                      Are there even any non-Google voices in that discussion besides this one?

                      It was a bit of a terrifying revelation to me when I realized that this response, which reads very much like a neutral third-party trying to moderate the discussion, presumably some influential non-Google LLVM contributor, was actually written by the Google-lead of the whole project: https://news.ycombinator.com/item?id=20282198

                      1. 2

                        Wow, nice catch! I also thought that message was from a neutral “moderator”.

                        I guess this is a cut-and-dried thing anyway; no way Google is going to reconsider anything here. It’s just a matter of will the LLVM project let itself be pushed around by Google.

                        1. 2

                          I find it interesting asking to keep things technical instead of using terms like “botch” when the original post says:

                          Within Google, we have a growing range of needs that existing libc implementations don’t quite address.

                          I would love to hear what ‘needs’ they are talking about.

                        2. 6

                          Rich Felker (musl developer) just replied with some pretty good points against Google doing this: https://lists.llvm.org/pipermail/llvm-dev/2019-June/133308.html

                          1. 9

                            What triggered me was the very first sentence in the quote:

                            Within Google, we have a growing range of needs that existing libc implementations don’t quite address.

                            What on Earth could be so specific about those needs that can’t be addressed by any existing implementation of the runtime used in myriads of projects big and small? Sounds like a post-rationalization of someone’s NIH syndrome.

                            1. 10

                              I interpreted that “growing range of needs” as “growing number of engineers at grade 8 who need new projects for the next performance summary cycle”.

                              1. 6

                                Given the initial target proposed is x86_64, this very likely has to do with various kernel interface changes Google has tried. Some that they’ve talked about in years past are quite interesting but to take advantage of changes without expensive changes all software, it seems like they need a libc that is easier to customize. So the technical impetus is there (IMO of course, I don’t work there).

                                Instrumentation of code and improvements around LLVM sanitizers and related fuzzing tools could also go hand-in-hand. I’m not sure what technically prevents this today outside of convenience and possibly the ability to build tools that are simpler because there are cheap assumptions that can be made.

                                While I’m not sure where I fall with Rich’s third point, I can see the reasoning. Perhaps the real problem isn’t something to be solved as part of the LLVM project. While the Clang project is already a large umbrella and includes a C++ standard library implementation but that still relies on a system layer like libc below, so I’m not sure that some of the other dismissals really answer the issue.

                                1. 1

                                  Sounds like a post-rationalization of someone’s NIH syndrome.

                                  Sounds like the entire history of Google to me.

                                2. 5

                                  Avoiding monoculture preserves the motivation for consensus-based standards processes rather than single-party control (see also: Chrome and what it’s done to the web) and the motivation for people writing software to write to the standards rather than to a particular implementation.

                                  Sometimes I wonder if anyone with power over at Google has the capacity (or will) to realize the damage Google does with every pie they stick their fingers into.

                                  1. 4

                                    Google has a lot of really bright people, which makes them feel really confident in their decisions. I like that they’re at least soliciting feedback here. I like to think the engineer here really wants to help and will take the feedback and contribute to an existing project in a light-touch kind of way. My fear is that they will either end up google-stomping it or be google-stomped as an employee and end up looking at their creation going “WHAT HAVE I DONE?!”

                                    1. 13

                                      Google has a lot of really bright people, which makes them feel really confident in their decisions. I like that they’re at least soliciting feedback here.

                                      For me the problem of this thread on the LLVM list is: what is there to give feedback on?

                                      From the initial e-mail:

                                      Within Google, we have a growing range of needs that existing libc implementations don’t quite address.

                                      But there is no description of the shortcomings that should be addressed. So, the whole thread feels like “we are going to do this anyway, so you can cooperate or be left out”, rather than having a genuine discussion of the problems of existing libc implementations and how they could be addressed.

                                    2. 1

                                      The very problem is that they don’t have to, they can do as they please.

                                    3. 5

                                      Tangentially, this debates makes me wish we wish we had better libc versus POSIX syscall library separation. There’s no reason the library that holds strlcpy needs to be Linux/amd64 specific bar any optional optimized code paths. If we can make libcs drop-in replacements like they are for C++ STL (bar ABI issues), that’d be cool.

                                      1. 8

                                        I work at Google but have no idea about any of these LLVM libc plans. I speak only for myself. But I have worked a little on the musl port / fork in Fuchsia as part of trying to port interesting C programs. The lack of separation between:

                                        • support functions like strlcpy, etc
                                        • low level io functions like open, read, write, etc
                                        • high level io functions like fopen, fprintf, etc
                                        • posix functions like getpwnam
                                        • linux specific functions (wrapping Linux syscalls)
                                        • glibc specific functions limits various interesting use cases.

                                        Imagine container images with libcs that only contain the subset of functionality they use. Imagine being able to statically link the libc with link time optimization, inlining, etc.

                                        Imagine being able to easily share the printf implementation between a libc on Linux, OpenBSD and the bare-metal OS you want to hack on at night. Imagine if you could configure the printf to only include the subset of functionality you want. Imagine if the compiler was aware of the printf subset you’ve configured.

                                        I don’t know if that’s what you’ll be do with this proposed LLVM libc but all those things are hard to impossible with existing libcs and one might imagine that a libc built with LLVM’s as-a-library philosophy would make many of these easy or at least possible.

                                        1. 1

                                          I am rather sorry to bother you, but since you worked on Fuchsia’s musl fork: Do you have any idea what Rich Felker is referring to when he writes:

                                          … corporate development teams are uniquely qualified to utterly botch a libc … Google has done it once already with their fork of musl in Fuchsia … making random useless and broken changes in their fork.

                                          1. 6

                                            Not really. Our use case is quite different from the normal musl use-case. Fuchsia uses a microkernel so while a Linux libc is almost exclusively an interface between application code and the kernel via high-level syscalls, on Fuchsia the libc is largely an interface between application code and various other processes providing system services like networking, filesystems, etc. The kinds of changes we make would no sense to merge into a Linux libc and might look “useless and broken” to someone looking at libc from that perspective. On the other hand I’m sure we’ve done some dumb stuff. Some of our team members have 30 years of experience developing and maintaining an open source libc, but I for example do not.

                                      2. 3

                                        Their latest reply also feels rather Google:

                                        If there can be a C++ standard library and runtime implementation as part of the LLVM project, I do not see a reason why there cannot be a libc implementation as part of the LLVM project.

                                        Of course there can be an LLVM libc, just as there can be an LLVM anything. The question is should, which needs to be justified with sound whys, none of which have been given so far.

                                        1. 2

                                          It’s worth noting that GCC does have a “GCC libc” in the form of newlib.

                                          1. 2

                                            He presents compelling arguments for why LLVM shouldn’t attempt to build a C or C++ compiler, but less for why not to build a libc.

                                            1. 1

                                              Within Google, we have a growing range of needs…

                                              Something smells fishy. And Fuchsia.

                                              1. 6

                                                There’s a reply by someone on the Fuchsia team in the email thread - doesn’t look like it’s created with Fuchsia in mind so far.

                                                1. 2

                                                  No kidding, since they aren’t even planning to support aarch64 in the initial implementation.

                                              2. 1

                                                To be very clear: we don’t expect our needs to exactly match everyone else’s – part of our impetus is to simplify things wherever we can, and that may not quite match what others want in a libc. That said, we do believe that the effort will still be directly beneficial and usable for the broader LLVM community, and may serve as a starting point for others in the community to flesh out an increasingly complete set of libc functionality.

                                                Google? Being humble & honest? I bet this can’t last for an entire mailing list post…

                                                1. 1

                                                  Meta: this should probably be merged with the other post it’s essentially a dupe of. I’m not sure we need multiple separate stories continuing the same discussion of the same mailing list thread.

                                                  1. 1

                                                    You can message a mod (I pinged @pushcx yesterday for a similar request) to get a thread merged!

                                                    There’s an open issue for a feature for this:

                                                    https://github.com/lobsters/lobsters/issues/519

                                                    1. 1

                                                      This happened. I’m not convinced the side-effect of this discussion being merged into and hidden on page 2 was good, though.

                                                      1. 1

                                                        Lobste.rs is still low-traffic enough that I can keep up with all comments by browse the /comments page.

                                                        Merging threads is good, especially for linked content like this…

                                                        1. 1

                                                          When I made my comment, both stories were on the front page (one near the top and one near the bottom), which is part of why it seemed excessive. I guess it’s a bit of a judgment call when enough time has passed to warrant a separate discussion vs. continuing the existing one.