1. 9

    I’ve been using Rust based commandline tools for a while now, ripgrep to replace grep and fd-find to replace find.

    These are not commandline-preserving compatible with the old gnu versions, for me that’s a bonus: they feel modern, are fast and are written in a memory-safe language. In my experience you don’t usually write replacements of old utilities in a 100% compatible way, but just switch to using the new tools and change your assumptions.

    100% drop-in replacement rewrites of old C software in Rust or in other memory-safe languages is an impossible goal and it’s not something that is worth trying. Reimagining how old utilities would look like in a modern environment, however is probably a worthy goal.

    1. 7

      In my experience you don’t usually write replacements of old utilities in a 100% compatible way, but just switch to using the new tools and change your assumptions.

      Basic tools like grep, wc, cut etc are all over build chain of Unix-based systems. That’s by “POSIX-compliant” is so important, you need to know a new utility can handle an edge case in a Makefile somewhere.

      1. 7

        Make is a really bad example here, given how much software needs gmake or bsd make and doesn’t support both in practice.

        (Also, by the way, one of the reasons why I think it is good that rustc is not built with make anymore)

        1. 2

          I’m not that knowledgeable about the differences between different flavors of make , but surely all support running POSIX utilities for whatever reason (searching for installed binaries, checking whether a file exists, etc)?

          1. 5

            The differences start appearing as soon as you try to do something as basic as wildcards.

      2. 2

        they feel modern, are fast and are written in a memory-safe language

        They “feel modern” is subjective, so I’ll pass on that point. Saying they’re fast implies that the current implementation is slow, and is “written in a memory-safe language” really an advantage of a program being written in a memory safe language? Seems a little circular.

        I’m by no means an expert so my opinion doesn’t count for shit and so I take no sides in this, but it seems like the OP made a bunch of really good points that need addressing, and this really doesn’t.

        1. 21

          For a tool like ripgrep, that it is written in a memory safe language isn’t a terribly direct benefit to an end user. In particular, folks don’t usually use the tool in a context where a vulnerability would be that damaging.

          With that said, memory safety has other benefits, like (IMO) lower development time. I’ve been maintaining ripgrep for over a year now, and I literally don’t spend time debugging memory related bugs. I don’t just mean “it doesn’t happen that often,” I mean, “it’s never happened, not once.” That’s a lot more time I can put into fixing other types of bugs or adding features. This is good for users, albeit indirectly. Granted, this is probably a pretty lame argument, but I’m just relaying my own experience here as I see it.

          This of course varies from programmer to programmer. A better C or C++ programmer than myself, for example, might not spend any time debugging memory related bugs either. But I’m not that good.

          One might also make an argument in favor of using another memory safe language that has a GC, but the onus is on them to show that the tool can actually compete performance wise. I do suspect it is possible, although it hasn’t been done. I’d expect D to be capable, and probably also Go. (There are some ripgrep-like tools written in Go, but they break down once you push them a little harder than simple literal searches. I cannot conclude that this is a breakdown in Go itself because there is missing implementation work to make it run faster. Particularly in the regex engine.)

          Saying they’re fast implies that the current implementation is slow

          GNU grep isn’t slow in the vast majority of cases. On a single file search, ripgrep might edge it out. As you increase the complexity of the regex (say, by using lots of Unicode features without any literals), then ripgrep starts to get a lot faster (~order of magnitude). In the simpler cases, ripgrep tends to edge out GNU grep by using a mildly smarter heuristic that lets it spend more time in an optimized routine like memchr in common cases. But GNU grep could easily adopt the latter. It would be harder for them to fix their Unicode problems, and probably not worth their time. (Because once you stick a literal into the pattern, it speeds back up again.)

          However, if you say “compare default usages of these tools, including recursive search,” then ripgrep is going to toast GNU grep just because it will use parallelism by default. Which makes this a mostly uninteresting comparison if you’re curious about the details of performance difference, but that doesn’t stop it from being a very interesting UX comparison. That’s typically the source of claims like “ripgrep is so much faster than grep,” which are unfortunately also easily interpreted as a really interesting claim which provokes the question, “oh, so what is the interesting implementation choice that makes it faster?” When folks find out that it’s just parallelism or ignoring certain files (that are in your .gitignore), they rightfully feel cheated. :-) Another less common source of this that people use “grep” to refer to both BSD and GNU grep, and BSD grep has markedly different performance characteristics than GNU grep.

          Performance is complicated, so if you really want to the dirty details, just skip straight to my analysis: http://blog.burntsushi.net/ripgrep/

          1. 11

            I should throw out a complement for your work on ripgrep. It’s impressive. I think ripgrep lands on the wrong side of XY problem style arguments (or more like gets stuck in the middle) because everybody wants to talk about what’s best without first defining requirements. :(

            1. 3

              Hah, yes, indeed. And thanks. :-) I try to keep tabs on it as best I can and keep it in check, but it’s hard!

            2. 3

              Although, putting my security hat on, memory-safety is one of things where you just never know how a certain program will get used. I mean, imagemagick is just one of the obvious cases, but what about more subtle stuff such as running something from CI? Untrusted data has a tendency to show up in unexpected places.

              1. 3

                Yeah, that is a good point. And I’ve been slowly splitting ripgrep into smaller crates. Other CLI utilities (like fd) are benefiting from that, but that definitely increases the odds of the code showing up in even more unexpected places. The core search routines haven’t been split out yet, but it’s on the list!

              2. 2

                “I don’t just mean “it doesn’t happen that often,” I mean, “it’s never happened, not once.” That’s a lot more time I can put into fixing other types of bugs or adding features. This is good for users, albeit indirectly.”

                That’s not a lame argument: it’s the very argument that safer languages such as Ada, Java, and C# sold businesses on. Knocking out problems that take up lots of debugging time lets you improve your existing product more or build new ones. If you have competition, then you can potentially improve at a faster rate than them. Outside business, there’s quite a few FOSS projects each doing something similar that compete with each other on features and performance even if not security so much. So, being able to rapidly add or fix features could help one get ahead.

                The champions of that in what few studies exist were the Common LISP and Smalltalk languages. However, if wanting low-level performance, one will have to go with something else. The prior studies put Ada ahead of C with half the defects plus easier maintenance. Rust is most comparable to it in focus areas or strengths. So, choosing Rust for low-defect development without a GC has some empirical support indirectly. I think an interpreted variant with REPL and great debugging might be great, though, for closing some of the gap between it and those dynamic languages.

                1. 4

                  Sorry, I didn’t mean “lame” as in “bad” or “incorrect,” but rather, “tired” or “old.” i.e., Everyone has heard it before.

                  1. 1

                    Oh, that makes sense.

                    1. 2

                      But yeah otherwise totally agree with everything you said. Even your top-level comment too.

                2. 2

                  I would add, writing grep in any language with a complicated runtime probably wouldn’t be nice for openbsd. Pledging() relies on knowing what the program/runtime is going to do at any given point, I have had Go problems get killed by random syscalls that the Go runtime decided to make late into program execution.

                  1. 2

                    Thankfully, Rust isn’t one of those languages :) Its “runtime” does uhh… stack backtraces… and possibly some other little things I guess. It’s similar to C++.

                    killed by random syscalls

                    That, IMO, is the worst part of pledge. Capsicum gracefully denies syscalls, letting the program handle the error instead of blowing up :P Sure blowing up lets you quickly debug a core dump of a simple C program (like OpenBSD’s base utilities), but with more complex software written in different languages, I’d much rather handle the damn error.

                3. 3

                  Ripgrep is a lot faster than grep for general use; not necessarily because the search algorithm is faster, but because it defaults to ignoring .gitignored files and BLOBs, which usually is what you want.

                  1. 2

                    They “feel modern” is subjective, so I’ll pass on that point. Saying they’re fast implies that the current implementation is slow, and is “written in a memory-safe language” really an advantage of a program being written in a memory safe language? Seems a little circular.

                    The parent said “they feel modern, are fast”, not “they feel modern, are fast_er_”. The point is that they are on par, not that the current state is “slow”.

                    I’m by no means an expert so my opinion doesn’t count for shit and so I take no sides in this, but it seems like the OP made a bunch of really good points that need addressing, and this really doesn’t.

                    Given that around here, there’s a discussion on how people can’t find out what the OP exactly meant, I find this statement a bit confusing.

                  2. 1

                    ripgrep does some terribly silly things. One is not having an option to write output as:

                    file1:line MATCH1
                    file1:line MATCH2
                    file2:line MATCH1
                    

                    Which would be obvious to somebody familiar with awk or cut. ‘modern’ often means forgotten knowledge.

                    1. 19

                      ripgrep does exactly that when you use it in a pipeline. When you print to a tty, it uses a “prettier” format. If you don’t like the prettier format, the --no-heading will force the more traditional format. Put it in an alias and then forget about it.

                      ‘modern’ often means forgotten knowledge

                      And is also often a good thing. Compare and contrast POSIX grep’s and ripgrep’s support for Unicode, for example. ripgrep “forgets” a lot of stuff and just assumes UTF-8 everywhere. This works well enough in practice that I’ve never once received a complaint. Well, that’s not true. I did receive one complaint: that ripgrep can’t search UTF-16 files, which are not uncommon on Windows. But I fixed that by adding a bit of code to ripgrep that transcodes the corpus from UTF-16 to UTF-8 when a UTF-16 BOM is detected, and hey, it works! But if I didn’t “forget” about POSIX, then that might not have been possible at all.

                      This comment should not be interpreted as an argument against POSIX, OpenBSD’s (or any other OS) use of it. Instead, I’d like you to interpret it as a criticism of the myth that “the good ol’ days” ever existed at all. My bottom line is that history has a lot to teach us, and just because we don’t copy it exactly as it was doesn’t mean it was forgotten. Sometimes mistakes are made, or perhaps even more frequently, technology simply changes and evolves that makes repeating historical choices a hard pill to swallow in some contexts.

                      So how about instead of focusing on “forgotten” knowledge (and, to the same extent on the other end of the spectrum, avoid the new and shiny—but I’m speaking to a certain audience here so I’ve omitted that) we just focus on the problem we’re trying to solve instead? If there’s some ancient (or new, I don’t care) wisdom that I don’t know about that could make ripgrep a better tool, then I’m all ears.

                      Also, I’ve never once marketed ripgrep as “modern.” ;-)

                      1. 2

                        Most of what I said wrong was based on a mistake that wasted a bunch of my time (which annoyed me), So take the comments written in a bad mood with a grain of salt. And I do like how it respects .ignore files, a tool was definitely needed to deal with that.

                        I do wonder why that “prettier” mode is needed in the first place though. It just seems like a waste of effort and I can’t see the gain. I think AG might do the same, so maybe you were just copying.

                        1. 8

                          ag does do the same, and ag in turn copied it from ack. That particular format has been around since ack has been around, which certainly isn’t as long as grep, but ack has been around for a while by now.

                          In my experience, for every complaint about ripgrep’s defaults, there’s probably N other people that actually like the defaults. For example, I’ve heard from folks that think respecting .ignore by default is a terrible or silly decision. But the defaults don’t need to be some universal statement on what’s right; it’s just my approximation of what most people want based on my perception of the world. It also continues a long held tradition started by ack, and ripgrep definitely descends from ack.

                          While I don’t know for sure, if I didn’t use the pretty format by default, I’m pretty sure I’d have legions of former ag/ack users lining up to complain about it. You might think that’s silly, but I’ve heard of people who use ag over ripgrep because ag is easier to type than rg!

                          1. 1

                            use ag over ripgrep because ag is easier to type than rg

                            haha. If they care about easy typing, they should use shell aliases. I have aliased ‘sr’ (for ‘search’) to mean rg||ag||pt||ack depending on what’s available.

                      2. 1

                        Are you sure? I’m not at the computer right now, but IIRC it outputs exactly like that when it’s writing to a pipe, not a terminal. Try piping into less.

                        1. 1

                          It seems you are right, modern just means violating least surprise. :) At least the output was colourised - I suppose that is to remind me I am not colour blind (or rub the fact in if I am).

                          1. 4

                            Even good old ls does that though, to be fair. Send it to a pipe, and it outputs a newline separated list of file/directory names, send it to a tty, and it outputs a nicely formatted table, probably with ANSI color escape codes.

                            1. 2

                              It’s only GNU ls that outputs colors.

                              1. 3

                                It’s definitely not only GNU ls that outputs colors, see FreeBSD ls -G.

                              2. 1

                                I absolutely hate that about ls too, I guess I’m wrong, but I don’t feel to bad about being wrong in this case.

                              3. 3

                                I’m not surprised at applications providing nicely formatted output to terminals, checking whether stdout is a tty is an old trick :)

                                1. 1

                                  If by “nicely formatted” you mean “colored”, then sure. Changing the first-level textual content of the output is a huge and highly undesirable surprise to me, though. I would like to be able to predict what the input to the next piece of pipeline will look like without running it through cat first just to trick isatty.

                        1. 9

                          Hah, I was actually curious whether AST will make a move. Good to see he did.

                          Still, it’s sad that he doesn’t seem to care about ME.

                          1. 7

                            Whether he cares about ME is irrelevant here. By releasing the software under most (all?) free software and open source licenses, you forfeit the right to object even if the code is being used to trigger a WMD - with non-copyleft licenses you agree not to even see the changes to the code. That’s the beauty of liberal software licenses :^)

                            All that he had asked for is a bit of courtesy.

                            1. 4

                              AFAIK, this courtesy is actually required by BSD license, so it’s even worse, as Intel loses here on legal ground as well.

                              1. 5

                                No, it is not - hence the open letter. You are most likely confused by the original BSD License which contained the so called, advertising clause.

                                1. 5

                                  Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

                                  http://git.minix3.org/index.cgi?p=minix.git;a=blob;f=LICENSE;h=a119efa5f44dc93086bc34e7c95f10ed55b6401f;hb=HEAD

                                  1. 9

                                    Correct. The license requires Intel to reproduce what’s mentioned in the parent comment. The distribution of Minix as part of the IME is a “redistribution in binary form” (i.e., compiled code). Intel could have placed the parts mentioned in the license into those small paper booklets that usually accompany hardware, but as far as I can see, they haven’t done so. That is, Intel is breaching the BSD license Minix is distributed under.

                                    There’s no clause in the BSD license to inform Mr. Tanenbaum about the use of the software, though. That’s something he may complain about as lack of courtesy, but it’s not a legal requirement.

                                    What’s the consequence of the license breach? I can only speak for German law, but the BSD license does not include an auto-termination clause like the GPL does, so the license grant remains in place for the moment. The copyright holder (according to the link above, this is Vrije Universiteit, Amsterdam) may demand compensation or acknowledgment (i.e. fulfillment of the contract). Given the scale of the breach (it’s used in countless units of Intel’s hardware, distributed all over the globe by now), he might even be able to revoke the license grant, effectively stopping Intel from selling any processor containing the then unlicensed Minix. So, if you ever felt like the IME should be removed from this world, talk to the Amsterdam University and convince them to sue Intel over BSD license breach.

                                    That’s just my understanding of the things, but I’m pretty confident it’s correct (I’m a law student).

                                    1. 3

                                      It takes special skill to break a BSD license, congrats Intel.

                                      1. 5

                                        Actually, they may have a secret contract with the University of Amsterdam that has different conditions. But that we don’t know.

                                        1. 2

                                          Judging from the text, doesn’t seem AST is aware of it.

                                          1. 2

                                            University of Amsterdam (UvA) is not the Vrije University Amsterdam (VU). AST is a professor at VU.

                                      2. 1

                                        I’ve read the license - thanks! :^)

                                        The software’s on their chip and they distribute the hardware so I’m not sure that actually applies - I’m not a lawyer, though.

                                        1. 5

                                          Are you saying that if you ship the product in hardware form, you don’t distribute software that it runs? I wonder why all those PC vendors were paying fees to Microsoft for so long.

                                          1. 2

                                            For the license - not the software

                                            1. 3

                                              Yes, software is licensed. It doesn’t mean that if you sell hardware running software, you can violate that software’s license.

                                          2. 3

                                            So, they distribute a binary form of the OS.

                                            1. 4

                                              This is the “tivoization” situation that the GPLv3 was specifically created to address (and the BSD licence was not specifically updated to address).

                                              1. 2

                                                No, it was created to address not being able to modify the version they ship. Hardware vendors shipping GPLv2 software still have to follow the license terms and release source code. It’s right in the article you linked to.

                                                BSD license says that binary distribution requires mentioning copyright license terms in the documentation, so Intel should follow it.

                                                1. 3

                                                  Documentation or other materials. Does including a CREDITS file in the firmware count? (For that matter, Intel only sells the chipset to other vendors, not end users, so maybe it’s in the manufacturer docs? Maybe they’re to blame for not providing notice?)

                                                  1. 3

                                                    You have a point with the manufacturers being in-between Intel and the end users that I didn’t see in my above comment, but the outcome is similar. Intel redistributes Minix to the manufacturers, which then redistribute it to the end-users. Assuming Intel properly acknowledges things in the manufacturer’s docs, it’d then be the manufacturers that were in breach of the BSD license. Makes suing more work because you need to sue all the manufacturers, but it’s still illegal to not include the acknowledgements the BSD license demands.

                                                    Edit:

                                                    Does including a CREDITS file in the firmware count?

                                                    No. “Acknowledging” is something that needs to be done in a way the person that receives the software can actually take notice of.

                                                    1. 2

                                                      The minix license doesn’t use the word “acknowledging” so that’s not relevant.

                                                      1. 2

                                                        You’re correct, my bad. But “reproduce the above copyright notice” etc. aims at the same. Any sensible interpretation of the BSD license’s wording has to come to the result that the receivers of the source code must be able to view those parts of the license text mentioned, because otherwise the clause would be worthless.

                                              2. 1

                                                If they don’t distribute that copyright notice (I can’t remember last seeing any documentation coming directly from Intel as I always buy pre-assembled hardware) and your reasoning is correct, then they ought to fix it and include it somewhere.

                                                However, the sub-thread started by @pkubaj is about being courteous, i.e. informing the original author about the fact that you are using their software - MINIX’s license does not have that requirement.

                                    2. 7

                                      I think he is just happy he has a large company using minix.

                                      1. 5

                                        Still, it’s sad that he doesn’t seem to care about ME.

                                        Or just refrains from fighting a losing battle? It’s not like governments would give up on spying on and controlling us all.

                                        1. 6

                                          Do you have a cohesive argument behind that or are you just being negative?

                                          First off, governments aren’t using IME for dragnet surveillance. They (almost certainly) have some 0days, but they aren’t going to burn them on low-value targets like you or me. They pose a giant risk to us because they’ll eventually be used in general-purpose malware, but the government wouldn’t actually fight much (or maybe at all, publicly) to keep IME.

                                          Second off, security engineering is a sub-branch of economics. Arguments of the form “the government can hack anyone, just give up” are worthless. Defenders currently have the opportunity to make attacking orders of magnitude more expensive, for very little cost. We’re not even close to any diminishing returns falloff when it comes to security expenditures. While it’s technically true that the government (or any other well-funded attacker) could probably own any given consumer device that exists right now, it might cost them millions of dollars to do it (and then they have only a few days/weeks to keep using the exploit).

                                          By just getting everyday people do adopt marginally better security practices, we can make dragnet surveillance infeasibly expensive and reduce damage from non-governmental sources. This is the primary goal for now. An important part of “marginally better security” is getting people to stop buying things that are intentionally backdoored.

                                          1. 2

                                            Do you have a cohesive argument behind that or are you just being negative?

                                            Behind what? The idea that governments won’t give up on spying on us? Well, it’s quite simple. Police states have happened all throughout history, governments really really want absolute power over us, and they’re free to work towards it in any way they can.. so they will.

                                            They (almost certainly) have some 0days, but they aren’t going to burn them on low-value targets like you or me.

                                            Sure, but do they even need 0days if they have everyone ME’d?

                                            They pose a giant risk to us because they’ll eventually be used in general-purpose malware

                                            Yeah, that’s a problem too!

                                            Defenders currently have the opportunity to make attacking orders of magnitude more expensive, for very little cost. [..] An important part of “marginally better security” is getting people to stop buying things that are intentionally backdoored

                                            If you mean using completely “libre” hardware and software, that’s just not feasible for anyone who wants to get shit done in the real world. You need the best tools for your job, and you need things to Just Work.

                                            By just getting everyday people do adopt marginally better security practices, we can make dragnet surveillance infeasibly expensive and reduce damage from non-governmental sources.

                                            “Just”? :) I’m not saying we should all give up, but it’s an uphill battle.

                                            For example, the blind masses are eagerly adopting Face ID, and pretty soon you won’t be able to get a high-end mobile phone without something like it.

                                            People are still happily adopting Google Fiber, without thinking about why a company like Google might want to enter the ISP business.

                                            And maybe most disgustingly and bafflingly of all, vast hordes of Useful Idiots are working hard to prevent the truth from spreading - either as a fun little hobby, or a full-time job.

                                          2. 4

                                            It reads to me like he just doesn’t want to admit that he’s wrong about the BSD license “providing the maximum amount of freedom to potential users”. Having a secret un-auditable, un-modifiable OS running at a deeper level than the OS you actually choose to run is the opposite of user freedom; it’s delusional to think this is a good thing from the perspective of the users.

                                            1. 2

                                              And the BSD code supported that by making their secret box more reliable and cheaper to develop.

                                            2. 3

                                              Oh, it’s still not lost. ME_cleaner is getting better, Google is getting into it with NERF, Coreboot works pretty well on many newish boards and on top of that, there’s Talos.

                                            3. 2

                                              He posted an update in which he says he doesn’t like IME.

                                            1. 23

                                              Seems like a good argument against using BSD licenses.

                                              1. 7

                                                Why? I have more faith in management engine knowing it is minix than some shit that intel wrote themselves.

                                                1. 20

                                                  I suppose the section “Powerful, Reliable Software Can Be Bad” of https://www.gnu.org/philosophy/open-source-misses-the-point.en.html is relevant here :)

                                                  1. 10

                                                    If anything, we’d be better off if we found that Intel’s ME was total garbage. It lets an alternative supplier differentiate on more secure software to get some sales. Then, Intel will either try to get people to ignore them with their other advantages, improve the security of their software, or buy the competitor to get their solution. Currently, as license allows, Intel just freeloaded off a bunch of work taxpayers in Europe paid for with some free labor by Tannenbaum et al to solve their problem. The ME stack is still garbage per recent threads.

                                                    Alternatively, they could’ve just paid a RTOS vendor for a stack. The going rate for those targeting robustness with networking and filesystems was $50,000 OEM last I checked. After they acquired Wind River, they’d have access to highly-reliable OS that’s been used in all kinds of things. Also, a separation kernel (VxWorks MILS) with carefully-crafted networking plus NSA pentesting. So, they do have both paid and free alternatives that are better than Minix 3 if they didn’t prefer freeloading off others’ work to save fifty grand or so on a project that nets them billions. I’m starting to lean back toward GPLing or AGPLing everything with dual-licensing to reduce this. They can pay to remove the copyleft.

                                                    Edited to change “ripping off” to “freeloading off” as dxtr noted.

                                                    1. 4

                                                      If I create something and then give it to you - no strings attached - are you then ripping me off?

                                                      1. 5

                                                        Not really. I should’ve said freeloading like parasites. I wonder, though, about what motivates people to freely work for companies under a license that insures mainly the companies benefit versus one where they contribute something back. I originally liked the BSD licenses to increase the amount of high-quality code the companies might be using to make stuff better in general. I’m not so sure we should do that now seeing how (a) that creates bad incentives for the companies to constantly freeload versus GPL/APGL projects and (b) they keep modifying that stuff into insecure or seemingly-malicious software like Intel did.

                                                        The folks aren’t doing anything great by giving them the code. They’re just helping monopolists and oligopolists further ensure the status quo that damages users, developers, and hobbyists while minimizing their operational costs for benefit of owners or shareholders. They also use their fortunes to pay lobbyists to reduce our rights in areas such as copyright and patent law. That phrasing depicts what actually goes on versus the public good people sold me on long ago with BSD/MIT licenses. I wonder how many BSD/MIT contributors that wanted corporate uptake would stick to it if they saw that as the ultimate goal of their contributions. Also, were told the companies often change the code to defeat its flexibility, reliability, or security benefits.

                                                        I’m sure plenty would stay in the game but I am curious how many would switch licenses. Also, which would they prefer switching to for balancing widespread uptake and maximizing contributions.

                                                        1. 4

                                                          People use BSD-alikes because their goal isn’t to coerce people into opening their sources, their goal is to make using their software as easy a possible. They’re not working for rewards from future would-be customers, they’re working because they feel some software which does not exist, should.

                                                          1. 2

                                                            “they’re working because they feel some software which does not exist, should.”

                                                            I imagine most building open-source software fit that category. It can be done with copyleft licenses, though, with little impact to most users.

                                                            1. 3

                                                              Sure, and a subset of those people are interested in keeping their work from people who don’t “deserve it”, but not everybody is - and those who aren’t, usually choose a non-viral license because they want more people using their stuff.

                                                              1. 1

                                                                That’s true. A good point to make.

                                                      2. 2

                                                        If anything, we’d be better off if we found that Intel’s ME was total garbage.

                                                        Are you implying it’s not?

                                                        Don’t know about you, but I don’t need an unmodifiable, unremovable, totally compromised operating system running an HTTP server inside my CPU.

                                                        Never asked for this, wasn’t told by Apple that they were selling me this, and have no plans to buy another computer with it.

                                                        1. 2

                                                          Good luck finding one without it.

                                                          1. 1

                                                            Possibly can but will be performance hit:

                                                            https://news.ycombinator.com/item?id=15646175

                                                          2. 1

                                                            It’s definitely garbage. I’m setting up something broader than just Intel where I want them to show what their proprietary stuff is worth, users to find out, and a better alternative to potentially show up. Those can be vetted proprietary (eg shared-source) or FOSS.

                                                            I could be really wrong but I think AMD is missing a golden opportunity to differentiate on security or trustworthiness of CPU’s like Blackberry and then Apple tried to do in smartphones. Two lines of products, one without management and one with enterprise-controllable version, might push those losses back a little bit esp from foreign sales. They could let third parties of different jurisdictions inspect the management code or its loader since high-performance, legacy-compatible x86 is a patent minefield for competitors anyway. My hypothetical alternatives would have to make some kind of sacrifice in performance, cost, or both. AMD could charge right in.

                                                            1. 1

                                                              I could be really wrong but I think AMD is missing a golden opportunity to differentiate on security or trustworthiness of CPU

                                                              I doubt AMD has a choice in the matter. It really doesn’t make sense for Intel to have it in all their CPUs; in the consumer CPUs where no user will ever use the management engine, it’s just a bunch of extra hardware on the die, wasting space and increasing complexity and cost. The only reason I can think of would be that someone forced their hand, and I can imagine the NSA wouldn’t hate having a backdoor into every single Intel (or AMD) CPU in the world with ring -3 access.

                                                              1. 1

                                                                They have several, possible benefits to having that enterprise technology in their chips:

                                                                1. The functionality for providing security enhancements is the same in each. Enterprise and repair shops also wanted management benefits.

                                                                2. The DRM capabilities the entertainment industry wanted and might have paid for.

                                                                3. The backdoors the NSA might have demanded or paid for.

                                                                4. The common technique for saving on mask costs (millions) by merging I.P. from several use cases into fewer mask layers.

                                                                Ok. The original release on Intel’s side was vPro which had all kinds of benefits for enterprises, esp security. The Trusted Computing Group, of which Intel was part, also wanted to use that stuff for DRM for movies and MP3’s. They probably had financial incentives which might likewise be used to make them go more private again. The NSA is an unknown here where they might have promised them something for money or defense contracts. I know the ME’s weren’t mandatory because not all chip vendors that were in the U.S. were building management engines into their CPU’s. They could possibly put their foot down saying they’d take money to 0-day the firmware instead which would let us put in better firmware but NSA still hits most targets.

                                                                The last thing on my list is an industry practice to get development costs down. The best example was the hard disks which showed different amounts of storage but had same platters with same amount of space. The platters and components for writing them had a fixed cost. So, they used firmware deception to tier the pricing. Another example in an ASIC from a friend in hardware was him discovering a cellular radio in an embedded peripheral that wasn’t supposed to connect to anything. He said it wasn’t malicious: the company just reused a mobile SoC they sell for a different purpose with different packaging to squeeze more ROI out of existing chip. Aside from these oddities, the main form of reuse is just doing pre-proven blocks of hardware in a certain process node on new projects. Once they wire the first CPU instance to a ME, it was possibly cheaper to just reuse that on each iteration of that instance esp given ME’s were originally small (ARC cores).

                                                                So, there’s the overall analysis of what parties and concerns are involved. The amounts they’re currently losing are much bigger than anything Hollywood or NSA paid them. Highest payout I saw for NSA was around $100 million per telecom for access to their national networks. That was something they could use constantly whereas this they’d have to use sparingly. Couldn’t be much more. The trick is, like with Raptor Workstation, how many people would actually pay for a computer without the backdoor, how much extra, and what total revenues to project for AMD? I’m less confident in demand side than I am in supply side.

                                                        2. 2

                                                          Technically, we don’t really know what is in it, since the final result is closed source. Maybe they added a bunch of “shit that Intel wrote themselves”.

                                                          1. 1

                                                            Just from a personal point of view. I don’t want my software to be used to spy on users without me even being asked about it.

                                                          2. 2

                                                            Then they would have just used a different OS. MacOS has slowly been ripping all the GPLv3 code out of their OS. That’s why they use an ancient version of GPLv2 bash and manually backport all the security fixes.

                                                            1. 1

                                                              On the contrary - it shows that anyone can use such software without all the bull$%^& which surrounds, i.e. the GPL. All that he is asking for is simple: Hi, We’re using your software. Cheers, Bye!

                                                              1. 11

                                                                He spends 1/3rd of the letter asking talking about the fact that someone benefitted from his hard work and he didn’t get any acknowledgement of it. Then he goes and says something like: “I don’t mind, of course, and was not expecting any kind of payment since that is not required.” The whole thing feels and reads regretful to me. I don’t know AST, so don’t really know his personality, or anything, but if I spent 1/3rd of the letter talking like that, I know it’d be because I felt I missed a big opportunity and I’m trying to convince myself that it was fine.

                                                                1. 1

                                                                  If there’s anything that AST might regret is the fact that MINIX hasn’t been released under a permissive license earlier and the fact that Linux and the *BSDs got themselves firmly established.

                                                                  Him regretting not getting anything back out of it after fighting with the publisher to get the code released under a permissive license? Seriously? ;^)

                                                                  The way I read the letter is him setting the scene before mentioning that letting him know would have been a polite thing to have done - mentioning that without said background information would have looked a bit weird.

                                                                  Anyway, if I were the author of said code, I’d merely like to know.

                                                                2. 1

                                                                  Yes, and that’s what I wouldn’t want to happen to my software.

                                                              1. 2

                                                                Does any one here remember the bug in Solaris 8 (I think the initial releases– it was certainly patched later), where rm did not leave out . and ..? As you would expect, rm -rf .* on any directory had amusing consequences!

                                                                ps: Having been bitten by it a few times, while testing our product, I know it existed, but I cant find the information on the bug any more. If any one remembers, I would be really glad!.

                                                                1. 1

                                                                  Shouldn’t it be the shell’s responsibility to expand the glob pattern .*?

                                                                  1. 2

                                                                    The shell expands it to “..”. It’s rm’s responsibility to skip “..”.

                                                                    That said, while there have been rumors that this or that unix would delete “..”, the earliest sources I have available to me check for that case and ignore it. Other than systemd, I’m not aware of a system that actually had this bug. Solaris 8 would seem to be at least a decade too late.

                                                                    Here’s a rather old copy of rm. As you can see, it checks for “..” and won’t remove it.

                                                                    https://github.com/dspinellis/unix-history-repo/blob/Research-V7-Snapshot-Development/usr/src/cmd/rm.c

                                                                    1. 1

                                                                      I did make this mistake recently with, I think chmod. I wanted to make root’s dot files and directories world inaccessible and proceeded to make the entire system starting with /root/.. more secure.

                                                                      EDIT: And probably group inaccessible.

                                                                      1. 1

                                                                        This was a bug later introduced, (from what I remember). Unfortunately, while I can access the OpenSolaris source, the history stops at the OpenSolaris launch. As you can see, by comparison to the rm.c from the unix-history-repo, the file has been refactored and reworked quite extensively.

                                                                      2. 1

                                                                        Yes it is.

                                                                        From glob(7):

                                                                           Long ago, in UNIX V6, there was a program /etc/glob that would expand
                                                                           wildcard patterns.  Soon afterward this became a shell built-in.
                                                                        
                                                                           These days there is also a library routine glob(3) that will perform
                                                                           this function for a user program.
                                                                        

                                                                        I don’t know if this played a role in the alleged bug.

                                                                    1. 4

                                                                      Wow, they used to run NFS over the Internet?! Admittedly I’m not quite old enough to have used NFS extensively, but I never would have thought that was a good idea… was this more common back in the day?

                                                                      1. 9

                                                                        My favorite strange twist on this was Sun’s brief experimentation in the mid-‘90s with public NFS mounts intended to be mounted from web browsers. They developed a simplified protocol, WebNFS, in order to make it easier for Java applets to NFS-mount remote filesystems. Seems a bit nutty now, but in a certain way it makes sense, if you think of the technology stack Sun had been building, which used SunRPC as the basis for distributed computing, in contrast to today’s set of technologies built around various vaguely REST-style APIs. Distributed object stores accessed via REST API didn’t exist at the time, but if you wanted a way of making filesystems network-transparent via SunRPC, that already did exist, as that’s what NFS is, so Sun simplified it and reused it for the web.

                                                                        I don’t think I personally ever ran across public NFS servers intended for general software distribution though; that’s what FTP was for, even in the ‘90s. Wonder what kernel.org’s reason for that was. Within organizations it was more common, with people mounting their organizations’ NFS drives over the internet. Back in the days when you might refer to this as a WAN setup, in contrast to a LAN. And there were various projects intending to make a more next-gen version of the idea with better authentication and file distribution, like AFS, which did have a much more commonly used idea of a “public” AFS site.

                                                                        1. 2

                                                                          I like the idea. I’m sure most today would say just use HTTP and be done, but having filesystem semantics could be darn handy at times.

                                                                        2. 5

                                                                          The src.doc.ic.ac.uk FTP repository was, IIRC, available over NFS, at least on the SuperJANET network, if not globally. At some point when active/passive FTP was being a pain (I forget the details) I used it a few times.

                                                                          1. 5

                                                                            Not old enough? I was born in ‘91 and I still use NFS quite extensively :)

                                                                            1. 1

                                                                              The NFS protocol is alive and well. I can tell you at least 1 new product by a major cloud service provider uses it as the bread and butter interface for the service cough cough.

                                                                          1. 10

                                                                            Title is slightly misleading. This isn’t about batteries that last a lifetime without recharging, this is about combating the decay that comes from discharging/recharging. So no charger-less laptops from this, unfortunately (but maybe your battery will no longer slowly become worthless).

                                                                            1. 8

                                                                              Yeah, as you say. I still expect this to have a nice practical effect - you know the phenomenon where your phone’s battery seems to die a lot faster a year later, and you don’t know whether to blame it on hardware or software? Once this is commercialized, it’ll definitely be software. :)

                                                                              1. 13

                                                                                You’re right, the title is misleading. Yet, this is still a great achievement. Battery’s lifetime is a real issue. Especially with Apple hardware, having to bring them every two years to replace the battery is a chore, and a waste of natural resources.

                                                                                1. 5

                                                                                  I see what you mean but at the same time I don’t.

                                                                                  Is it (even slightly) misleading when, I assume, no one will interpret “last a lifetime” as “will store a lifetime worth of energy” instead of “will keep storing the same amount of energy for a lifetime”?

                                                                                  1. 2

                                                                                    Yeah, but I kinda figured that Li-ion batteries weren’t wasting so much energy that making them more efficient would make them last a lifetime. Process of elimination.