Ugh, another one bites the dust — without giving the BSDs a chance. Try OpenBSD, the out of the box experience is the most pleasant one I’ve had yet, more so than macOS even, because it does exactly what I want it to.
I have to use a MacBook for work, and while the UI is pretty and all, I find its tiny quirks and overall rigidity very annoying. I can’t do things the way I’d like to.
I gave FreeBSD a serious chance for months. I hacked a little on i915 trying to fix some backlight bugs. I contributed to what became the implementation of the utimensat syscall implementation for the Linuxulator. I was even featured on an episode of the original BSDNow in 2015.
I suppose you could argue some value of stability and release engineering still exist in FreeBSD, but they’re even more hostile to older hardware and the desktop experience is even worse as more and more software assumes Linux.
One reason I like my OpenBSD Desktop is that it is still running great on my 2012 Toshiba Portege - I’ve upgraded the RAM to 16Gb and the HDD to an SSD - the only draw back is getting decent batteries for portability is now becoming an issue. But I have heard that OpenBSD may be coming to the M1 in the near future :~)
macppc supports all New World models, including the G3.
I think NetBSD is probably the best of the BSDs for running on PowerPC, if we’re talking about it; in addition to macppc they have ofppc, amigappc, bebox, etc.
It wasn’t mentioned by the author, but another point of make not being portable is the differences between GNU and BSD make, and whether there are any differences between the “fragments” provided by the operating system that can be included in another makefile (e.g. .include <bsd.prog.mk>).
I was giving this a shot earlier after it first entered -current snapshots and was unable to get the default IPv4 route set with the gateway from my dhcp server. This is after removing dhcp from my hostname.if and setting AUTOCONF4 on the interface. (I’m not certain if removing dhcp is correct, but /etc/netstart will still use dhclient if you specify that, and so I tried without.)
You need to remove ‘dhcp’ and add ‘inet autoconf’ to your /etc/hostname.if file. Then the system uses dhcpleased. If ‘dhcp’ is present, dhclient will be started
Perhaps lobsters is not the best place to report bugs.. but I did both of these and it was still not setting the route. Currently on:
kern.version=OpenBSD 6.9-beta (GENERIC.MP) #368: Sun Feb 28 21:10:13 MST 2021
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
When I up the interface, I see logged:
Mar 1 09:13:02 drifter dhcpleased[73796]: failed to send route message: No such process
Mar 1 09:13:06 drifter dhcpleased[73796]: failed to send route message: Network is unreachable
Even stranger is that I cannot manually add the route anymore.
# route add default 10.0.0.1
add net default: gateway 10.0.0.1: Network is unreachable
This also implies that you need to have a system that keeps track of what library versions are used in individual programs.
This seems relatively easy to implement compared to making everyone have stable ABIs for everything, including languages that can’t have one.
The nu-school package managers have machine-readable lockfiles with exact versions of all libraries used. A distro could create and archive such lockfiles to be able to later know exactly which of their statically-built binaries contain vulnerable dependencies.
In Go’s case, you can simply inspect the executable (using go version -m path/to/exe) to display the Go version used in the build, and all of the statically-linked module’s versions and checksums, so you don’t even need to keep around any extra build artifacts to determine which executable are vulnerable.
I am unfamiliar with Gentoo’s tooling, but I do know that OpenBSD’s Go ports contain the complete package list of all dependency modules that will be required for the build to succeed (this is necessary to download everything ahead of time, so the build itself can be done offline and in a chroot). This makes it rather trivial to discover all ports which need to be patched and rebuilt for any given vulnerability fix.
Wow, this blog post is so lacking in empathy for users that I’m surprised it made it on a reputable distro’s blog. Instead of spilling 1000 words on why “static linking is bad”, maybe spend a little time thinking about why people (like me) and platforms (like go/rust et al) choose it. The reason people like it is that it actually works and it won’t suddenly stop working when you change the version of openssl in three months. It doesn’t even introduce security risks! The only difference is you have to rebuild everything on that new version, which seems like a small price to have software that works, not to mention that rebuilding everything will also re-run the tests on that new version. I can build a go program on nixos, ship it to any of my coworkers and it actually just works. We are on a random mix of recent ubuntu, centos 7 and centos 8 and it all just works together. That is absolutely not possible with dynamic linking
It works well if all you care about is deploying your application. As a distro maintainer, I’m to keeping track of 500 programs, and having to care about vendored/bundled versions and statically linked in dependencies multiplies the work I have to do.
But … that’s a choice you make for yourself? No application author is asking you to do that, and many application authors actively dislike that you’re doing that, and to be honest I think most users don’t care all that much either.
I’ve done plenty of packaging of FreeBSD ports back in the day, and I appreciate it can be kind of boring thankless “invisible” gruntwork and, at times, be frustrating. I really don’t want to devalue your work or sound thankless, but to be honest I feel that a lot of packagers are making their own lives much harder than it needs to be by sticking to a model that a large swath of the software development community has, after due consideration and weighing all the involved trade-offs, rejected and moved away from.
Both Go and Rust – two communities with pretty different approaches to software development – independently decided to prefer static linking. There are reason for that.
Could there be some improvements in tooling? Absolutely! But static linking and version pinning aren’t going away. If all the time and effort spent on packagers splitting things up would be spent on improving the tooling, then we’d be in a much better situation now.
…but to be honest I feel that a lot of packagers are making their own lives much harder than it needs to be by sticking to a model that a large swath of the software development community has, after due consideration and weighing all the involved trade-offs, rejected and moved away from.
I think this is a common view but it results from sampling bias. If you’re the author of a particular piece of software, you care deeply about it, and the users you directly interact with also care deeply about it. So you will tend to see benefits that apply to people for whom your software is of particular importance in their stack. You will tend to be blind to the users for whom your software is “part of the furniture”. From the other side, that’s the majority of the software you use.
Users who benefit from the traditional distribution packaging model for most of their software also find that same model to be painful for some “key” software. The problem is that what software is key is different for different classes of user.
A big reason people ship binaries statically linked is so it’s easier to use without frills, benefiting especially users who aren’t deeply invested in the software.
For me personally as an end user, if a program is available in apt-get then I will install it from apt-get first, every time. I don’t want to be responsible for tracking updates to that program manually!
I believe static PIE can only randomize a single base address for a statically linked executable, unlike dynamically linked PIE executable where all loaded PIC objects receive a randomized base address.
Did a bit of rummaging in the exe header but I’m not 100% sure what I’m looking for to confirm there, but it had a relocation section and all symbols in it were relative to the start of the file as far as I could tell.
Edit: Okay, it appears the brute-force way works. I love C sometimes.
aslr.c:
#include <stdio.h>
int main() {
int (*p)() = main;
printf("main is %p\n", p);
return 0;
}
Testing:
$ gcc aslr.c
$ ldd a.out
linux-vdso.so.1 (0x00007ffe47d2f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2de631b000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2de6512000)
$ ./a.out; ./a.out; ./a.out
main is 0x564d9cf42135
main is 0x561c0b882135
main is 0x55f84a94f135
$ gcc -static aslr.c
$ ldd a.out
not a dynamic executable
$ a.out; a.out; a.out
main is 0x401c2d
main is 0x401c2d
main is 0x401c2d
$ gcc -static-pie aslr.c
$ ldd a.out
statically linked
$ a.out; a.out; a.out
main is 0x7f4549c07db5
main is 0x7f5c6bce5db5
main is 0x7fd0a2aaedb5
Note ldd distinguishing between “not a dynamic executable” and “statically linked”.
I don’t know much about security, and this is an honest question. My understanding is that ASLR is made to mostly protect when the executable is compromised due to stack overflow and similar attacks, right? Aren’t these problems mostly a thing of past for the two languages that abhor static linking, like go and rust?
There are a lot of code-injection vulnerabilities that can occur as a result of LD_LIBRARY_PATH shenanigans. If you’re not building everything relro, dynamic linking has a bunch of things like the PLT GOT that contain function pointers that the program will blindly jump through, making exploiting memory-safety vulnerabilities easier.
As an author of open source malware specifically targeting the PLT/GOT for FreeBSD processes, I’m familiar with PLT/GOT. :)
The good news is that the llvm toolchain (clang, lld) enables RELRO by default, but not BIND_NOW. HardenedBSD enables BIND_NOW by default. Though, on systems that don’t disable unprivileged process debugging, using BIND_NOW can open a new can of worms: making PLT/GOT redirection attacks easier over the ptrace boundary.
Easy money. Cease & Desist letter in Germany come with a fine attached if you follow them. You need to pay the the lawyers fees (~800+ EUR). This is pretty unique. So there’s “cease & desist” mills who trawl people, e.g. off bittorrent and all other networks. This means that some of these cases will end up at a court. As all things digital have no place of service or occurrence, the filing side can pick any court to go to. Which is usually Hamburg or Cologne, which tend to be the most eager to stretch the law to the rights holder side. But that’s actually not the process intended, what they want is people to be frightened and paying the lawyers fee on the first letter. They will even lower it if you even look at them like you might defend yourself.
They, like the US, have a highly information based economy. If you can put a value amount on copyright, you can put a penalty amount. You can decide if a lawyer is worth it. You can make laws about it.
This is unsurprising. Most international media companies here are basically “importing” and rarely “exporting”, which means (distribution) licensing, so all their local orgs have a high focus on rights management and lobbying for better terms. And if you have staff lawyers around all the time… you might as well use them?
The original repo is on the WayBack machine 1400 times with the most recent being 5 days ago. If you’re forking just to a copy lives around and not to continue development, I would just snag it there to be sure of the source.
Edited to make it clear I was linking to the original repo.
The minimalism is appealing, simply because there’s less you need to understand to feel confident in your systems. (same reason why I prefer to run OpenBSD over Linux.)
FreeBSD is I think the right choice if you want to have a good base with ZFS and every software out there packaged and up to date.
OpenBSD is like a study in simplicity. I really enjoy things like how they manage wireless networks, no networkmanager or wpa_supplicant, sound without pulseaudio (you can use sndio on FreeBSD as well but you need to compile your own packages).
OpenBSD feels more like things done right/simple, while FreeBSD for many purposes feels more like “a better linux distribution”, with both stable base and up to date packages, without having to deal with third party repositories
Another differencen for the decision can be whether you want to use just base. While on FreeBSD you can do some basic stuff like running an NFS server without any packages, OpenBSD will give you an httpd server, complete with the means to get a letsencrypt or other acme based certificate, you will also get a pretty nice smtp server (no imap server though) and also typical user facing things, like tmux, X with a couple of window managers are part of the base system.
But then again very big reason for choosing any general purpose OS is taste.
Edit: To expand a little, now that I’ve had some coffee, let me repeat one of soc’s points a little more vigorously: C++ will not get better until they start taking things away as well as adding things. I still have vivid memories of reading through a C++ book in undergrad, back in 2004 or something, and thinking “ok, I can’t just entirely avoid pointers in favor of references… so what good are references besides making more ways of doing the same thing?”
References serve an entirely different purpose than pointers. A pointer is a type that can be NULL, so it may represent an optional object, whereas references are (theoretically) never NULL and always valid. These are valuable safety features. However, you would find it extremely difficult to write C++ code that exclusively uses references and no pointers. So both have their place in the language, though I would replace almost all pointers with std::unique_ptr if C++11 is available.
I agree with the sentiment that the relentless growth of C++ and the reluctance of the committee to deprecate features is the source of many problems. Even though I know and write C++ for many years, I still find C++ features that weren’t previously known to me on a daily (!) basis. In other words, for thousands of days, I am daily surprised by this language. The complexity is unfathomable.
references are (theoretically) never NULL and always valid.
They are never NULL and often invalid. Like when you saved a reference to something that either (a) went out of scope; (b) was the result of dereferencing a pointer to memory that was freed afterwards. You can also make a null reference if you try very hard, but that doesn’t happen in practice, whereas invalid references sure do.
I’ve seen it in the wild; debugged a core dump that had it. The evil thing is that the program crashes at the point where the NULL-reference is used, not where NULL is dereferenced (in compiled assembly, dereferencing a pointer to store it in a reference is a no-op). These two points may be far apart.
They shouldn’t be invalid, but it’s not entirely enforced by the compiler. The idea is that by using references, a programmer signals to the reader of the code that the value is intended to be a non-NULL valid object, and if that is not the case, the error is at whatever line of code that creates this invalid reference, not the one that uses it. In the wild, even inexperienced C++ programmers rarely create invalid references simply because it is harder to do so, whereas they routinely create invalid pointers. In addition to that, invalid references are almost always a programmer error, but invalid pointers are a regular occurrence and often part of a design that allows and accounts for such states.
The idea is that by using references, a programmer signals to the reader of the code that the value is intended to be a non-NULL valid object, and if that is not the case, the error is at whatever line of code that creates this invalid reference, not the one that uses it.
Yes, nonetheless it happens. And it doesn’t matter that “somebody else” made the error. The error is still there and it’s your code that crashes. I feel much better using a language that doesn’t put me in such situations.
In the wild, even inexperienced C++ programmers rarely create invalid references simply because it is harder to do so
For some definition of “rarely”. I’ve often seen code that passed references to objects wrapped in std::shared_ptr (often to avoid ARC overhead) into function arguments where those functions captured those by reference in lambdas which were executed after the std::shared_ptr freed the memory. Similarly, this regardless of its type is in practice used like a reference. There’s a lot of code in the wild that does something like PostTask([this] { /* do smth */ });.
At the end of the day it doesn’t matter that “the error is somewhere else”. If the language makes something remotely straightforward, I’m sure to have to deal with that in a shared codebase at one point no matter how few errors I make myself.
Yes, whatever the compiler doesn’t prohibit is likely to end up in the codebase. However, I still think that references are valuable and provide better safety than pointers. Obviously it would be better if making mistakes was impossible. I’m not whitewashing C++’s serious shortcomings here.
It’s important to keep in perspective that the backwards incompatibility issue with non-module builds that was patched into older releases was done so all the way back into the Go 1.9 release branch. Go 1.9 was originally released in August of 2017. I don’t know of anyone that is still using a Go release from back then, which is mostly a product of how well the Go team prioritizes not introducing breaking changes. The Go team also only provides security patches for the latest two version branches of Go (currently, 1.14 and 1.15), so nobody should be using a release that old regardless.
It was painful from the developer perspective because your dependencies and deps-of-deps might have not had module support, and might already be a v2/v3 etc. (thus blocked you from sanely supporting non-module and module at the transition period)
There was never a need for your dependencies, or their dependencies, to be or understand modules in order to consume them in a module build. They just worked. The only module “feature” that wouldn’t have been possible with this code is the inability to contain multiple major versions of a package, since that does rely on semantic import versioning (what this author is challenging).
Being the maintainer of three of these tools and a long time command-line enthusiast, it’s just sad to read a comment like this.
I can understand that people do not want to try out new tools. Fine. But why criticize them without even taking a proper look?
As others have already pointed out, most of these tools are not breaking pipeline usage. The README for bat has a whole section on “integration with other tools”, showing multiple examples of pipeline usage. fd’s README also points out several examples on how to pipe fd’s output to other programs (e.g. xargs, parallel, as-tree).
I have a feeling that a lot of people react allergic to colorized command-line output, and I absolutely do not understand why. Just because a tool doesn’t look like 1970 doesn’t mean that it’s not a proper citizen of the command-line world. But that’s not even the point. My tools do not provide colorized output to look “fancy”. They use colorized output because I think it’s actually pretty damn useful. Also, there is always the possibility to easily switch them off if you really don’t like it (set NO_COLOR or use --color=never in an alias).
Finally, I am really sick of the implied “rewritten in Rust” stigma. No, I did not write these tools in Rust because I wanted to demonstrate something. Or to show off. I simply like to try different programming languages from time to time and I think that Rust is an awesome fit for command-line tools. fd was actually started as a C++ project. I just moved it to Rust at some point. Mainly because of boost packaging issues and because I discovered the awesome Rust libraries by @burntsushi. Criticizing a tool for the language it’s written in is fine as long as there is an actual argument why that programming language was a poor choice for the job.
I have a feeling that a lot of people react allergic to colorized command-line output, and I absolutely do not understand why.
First, let me say that I really appreciate the effort in trying to keep moving things forward. That means change, and change is not always good.
I’m pretty allergic to colors in command line tools - but that is mostly because of awful default logging/outpu formats from ruby/rails - and not for tools like these that properly detect piping etc.
I’m also slow to adopt new tools - I still tend to prefer gnu find to fd, haven’t quite been able to put fzf to work - but I’m not quite hopeless - I do use ripgrep quite a bit.
I think quality matters (more) when using colorization - and it takes a village to get there. The recent generations of colorized interactive ruby output is a boon (on Linux) - it looks good, and if I copy from gnome terminal and paste into nvim - the escape séquences vanish. But on windows servers with the standard windows console - even the line breaks do not work properly, and colorization is hit and miss (colors or escape sequences? It depends…).
Many of us need to work on various heterogeneous deployment servers not under our control - and that makes us thirsty for the simplest tools - to the point of sometimes preferring a predictable dull rock to a scalpel that might shatter at any moment. But it sometimes makes the transitionfrom skinning rabbit to preformingbbrain surgery… More challenging.
I have a feeling that a lot of people react allergic to colorized command-line output, and I absolutely do not understand why.
First, let me say that I really appreciate the effort in trying to keep moving things forward. That means change, and change is not always good.
I’m pretty allergic to colors in command line tools - but that is mostly because of awful default logging/outpu formats from ruby/rails - and not for tools like these that properly detect piping etc.
I’m also slow to adopt new tools - I still tend to prefer gnu find to fd, haven’t quite been able to put fzf to work - but I’m not quite hopeless - I do use ripgrep quite a bit.
I think quality matters (more) when using colorization - and it takes a village to get there. The recent generations of colorized interactive ruby output is a boon (on Linux) - it looks good, and if I copy from gnome terminal and paste into nvim - the escape séquences vanish. But on windows servers with the standard windows console - even the line breaks do not work properly, and colorization is hit and miss (colors or escape sequences? It depends…).
Many of us need to work on various heterogeneous deployment servers not under our control - and that makes us thirsty for the simplest tools - to the point of sometimes preferring a predictable dull rock to a scalpel that might shatter at any moment. But it sometimes makes the transitionfrom skinning rabbit to preformingbbrain surgery… More challenging.
sad as in trump-sad or you are sad? if the latter, i’m sorry.
Sorry, this seems completely uncalled for. One of the reasons I like this site is its relative lack of political drama and focus on technology. I don’t know what Donald Trump has to do with this technical disagreement, but I don’t think he’s relevant to it, and I don’t think the mention of him will resolve anything.
sad as in trump-sad or you are sad? if the latter, i’m sorry.
It makes me sad because you invest countless hours into a project and then someone disregards it within seconds with a low-effort comment like this. Other people looking at this thread will see the wrong statements (“breaking pipelines”) and might take them for granted.
i dislike it because it’s an ugly hack on top of emulated 70s hardware, and breaks.
I absolutely agree with you on this point. I just don’t see any alternative. Is someone working on a completely redesigned, modern terminal API? It could probably still support ANSI escape codes for backwards compatibility.
somehow git diff is broken on one of my machines where the automatically invoked “less” isn’t rendering escapes but printing them verbatim.
I know you are probably not looking for a solution, but the problem might be that the -R/--RAW-CONTROL-CHARS command-line option is not passed to less on that system. It tells less to interpret those escape sequences. I guess most distributions set LESS="-R" by default.
the title of the article says “rewritten in rust” as if it’s a seal of quality. i wouldn’t otherwise have made fun of that.
That’s your interpretation, but okay. I see your point. It’s definitely not a seal of quality per se. There are, however, a few things that can be inferred from a choice of programming language (expected startup latency, memory safety guarantees, portability, ease of distribution, …). As stated above, I think that Rust is actually an excellent choice for command line tools. Because it delivers on all of the mentioned aspects. That doesn’t mean that it’s the only choice though.
I maintain a terminal library in D and yeah it is interesting to do it well on Windows (especially if you want to do a gui and a cli in the same exe… I thought that was impossible until I did it with a pile of hacks) but I didn’t even try the msys then, yikes, what a hassle.
But one of the decisions I made with my library is it is invalid to use the Terminal structure on a pipe. So I force you to branch outside instead of trying to transparently handle both. I kinda regret that since so many programs can benefit from the “just-works” aspect, but it is true you can often do much better UIs if the application just branches at the top level.
Thanks! I like the SW_HIDE idea, though I went with a different approach. What I ended up doing is making a .bat file that calls the exe and just instructed the cli users to use that instead of the exe itself which worked cuz a bunch of cli users were used to a bat file anyway (it used to do chcp and when I looked at it, I told them about WriteConsoleW and that chcp hack is terrible, but the batch file was still there so that meant I could reuse it here).
Then the exe checks AttachConsole parent and reopens the stdin/out to it… as long as they were already null (otherwise they were previously redirected and don’t want to overwrite that).
Otherwise though the exe itself is the Windows subsystem so the double click and shortcut users just work without any flicker. Then the bat file causes the command shell to wait until the program finishes so the prompt and the output don’t stomp on each other.
The only thing kinda iffy to me is if the output is piped. For the console, WriteConsoleW with utf-16 just works. But for the pipe, I output utf-8 and it seems to be fine and the users happy with it, but I remain unsure what it should really do. I suspect Windows detects based on the first two bytes from some tests but I’m not fully happy with that part yet. Especially since you can pipe binary data too so idk.
Though, even if the fancy looking output was a better user experience (usually, I don’t think it is), switching output is also not particularly friendly; Now I need to think about two different formats when I use the tool.
Breaking pipelines is not an issue if you still have the coreutils implementations still around. e.g: use bat and exa for regular use but when you need pipes use cat and ls. I don’t see a problem with this and I think the concepts nushell (also mentioned in the article) brings to the table are definitely useful, albeit not implemented there for the first time.
i just think it’s a bit weird to hype the console (look ma, no gui!) and at the same time ignore the concepts which make it so powerful. some of these tools are okish, but colorization and emulating guis are great at preventing the piping of output to further processing.
tools and pipelines together are bigger than their parts.
and i think that rust is overhyped, but that’s my problem ;)
and at the same time ignore the concepts which make it so powerful
If this were true, then we wouldn’t have used tty detection to change the output format to be compatible with pipelines.
Which, by the way, is exactly what ls has been doing for years. Its default output format to a tty isn’t pipeline friendly. Maybe we’ve been learning from older tools more than you think.
It’s just funny when I see people raging about this for the Rust tools (it seems to happen quite frequently), but nobody thinks to do it for any of the older tools. And it’s even better when people use this as a launching point to accuse us of ignorance. Give me a break.
I think you will find people (suckless type people) do complain about gnu coreutils doing stuff like this. I admit it is useful and nice lots of the time too.
Plan 9 ls doesn’t do anything different, and in fact, has no way to do it short of becoming a graphical program: there’s no concept of “terminal width”. You need to open the draw device and get the window size, look at the (possibly variable-width) font, and compute line widths.
There are also no control characters that handle cursor positioning, other than ‘\b’. The plan 9 console is a simple textual transcript. If you want graphics, use graphics – it doesn’t pop open a new window, so there’s no reason to support semigraphical features when full graphics are available.
Correct – Plan 9 doesn’t. The philosophy behind it is discussed in cat -v Considered Harmful by Pike and Kernighan (page 5).
Though, to be technical, the Plan9 console isn’t really a tty, so Plan9 ls was designed for a slightly different environment. However, the design of ls (and the Plan 9 console) are rooted in this philosophy.
it’s not that i dislike the functionality but how it’s implemented. i’d be interested in having such tools build in a composable way, out of multiple commands. default invokations then could be build via shellscript (and shipped together with the commands).
And do you use Plan9’s version of ls to avoid the “nice” display output of the BSD and GNU versions of ls? If not, why do you use tools that ignore the concepts of composition?
it’s not that i dislike the functionality but how it’s implemented
That’s surprising, and even re-reading your comments above, I don’t really see this coming across. You seem to be ranting against the functionality itself and how it breaks pipelining. But we handle those cases by disabling the “nice” functionality so that pipelining works.
i’d be interested in having such tools build in a composable way
Easier said than done. Everyone worships at the alter of perfect composition (including myself), but actually achieving it is another matter entirely. It seems pretty unfair to hold that against us. It’s not unreasonable to think that in certain environments, composition, development effort and the user experience are at odds with one another. (Where the development effort to make some of these nicer features truly compositional is probably on the order of “build a new kind of shell environment and convince everyone to use it,” which really isn’t feasible.) People actually building these tools have to pick a design point in this space, and we get rewarded with people calling us ignorant. Go figure.
And do you use Plan9’s version of ls to avoid the “nice” display output of the BSD and GNU versions of ls? If not, why do you use tools that ignore the concepts of composition?
i’ve tried to use the tools of plan9port by putting them in front of $PATH, unfortunatly the assumptions about the environment are sufficiently different from linux so that i’ve had other problems then.
That’s surprising, and even re-reading your comments above, I don’t really see this coming across. You seem to be ranting against the functionality itself and how it breaks pipelining. But we handle those cases by disabling the “nice” functionality so that pipelining works.
i dislike the complexity such tools introduce by putting together unrelated things like colorization and git handling (sorry, bat is the first thing in the list..). i dislike magic as i’ve put so much time into debugging it when it breaks.
Easier said than done. Everyone worships at the alter of perfect composition (including myself), but actually achieving it is another matter entirely.
indeed.
It seems pretty unfair to hold that against us. It’s not unreasonable to think that in certain environments, composition, development effort and the user experience are at odds with one another. (Where the development effort to make some of these nicer features truly compositional is probably on the order of “build a new kind of shell environment and convince everyone to use it,” which really isn’t feasible.) People actually building these tools have to pick a design point in this space, and we get rewarded with people calling us ignorant. Go figure.
i don’t know, you got rewarded with upvotes :) i’ve never called people ignorant, i’ve said that i find it weird to use an environment which lives by simplicity and then building things which combine several complicated tasks into one.
i just think it’s a bit weird to hype the console (look ma, no gui!) and at the same time ignore the concepts which make it so powerful
Emphasis mine. Sure looks like you’re calling us ignorant to me.
It’d just be nice to see folks be consistent. If you’re going to get all cranky about new tools introducing features that aren’t compositional, then at least get cranky about all the old tools that do it too. Because then, just maybe, it’s not just a bunch of overhyped new-fangled nonsense, and maybe there’s actually something to it. The giants we stand on also made choices that sacrifice composition.
the funny thing is that no one has thrown the old “then write it yourself” in my direction, which only would be fair.
I try to avoid saying things like that in discussions such as these. I think it’s overused as a way to dismiss others’ concerns. And of course, most people are already aware that such a thing is possible and don’t need to be reminded of it. I might as well just say nothing at all at that point.
I just find your particular form of criticism to be popular yet shallow. People love to jump all over these new fangled tools as if their authors are blubbering idiots that totally ignored the entire history of computing or are blissfully unaware of the Unix philosophy. It’s a lazy critique because violating composition is nothing new. We’re just doubling down, people don’t like it and for some reason like to insult the authors of these tools or at least denigrate their work. It’s annoying. (It’s a pretty common tactic used in lazy online banter, and is used in all manner of scenarios. I think we’d all be a lot better off if we held ourselves to a higher standard of discourse.)
Let’s assume you were used to using cat somefile | less (ignoring for a moment that you could just less somefile) and swapped in an alternate tool for cat, let’s say mycat. Now mycat has a bunch of cool features like colorized output or wizbang progress bars (again, ignoring that many find colored output or wizbang features distasteful….). When just using mycat somefile everything looks great! But then you pipe mycat somefile | less and get all kinds of garbage, if less even works at all!
This is one way to breaki pipelining. The reason it happens is mycat is probably using a bunch of TTY control codes, or ANSI color codes to implement all those features. However, when that output is piped through to less, the pipe isn’t a TTY it’s just a raw text stream. So those control codes don’t get interpreted and instead just appear on the screen in raw form.
mycat can fix this by first checking if stdout is a TTY or not, and if so printing it’s control codes, but if not just printing raw text.
There are other ways you can break pipelining, such as not handling a SIGPIPE correctly (trying to write to pipe who’s reading end has been closed).
Rust has facilities to handle these cases, and most common Rust applications handle these cases just fine. It is however a common footgun since if you’re implementing a new tool and not familiar with how all this traditionally works you may not know to check for these kinds of things.
But rc.conf.local is the only configuration file an administrator touches, and it never conflicts with changes to the rc scripts under rc.d nor the default configuration found in rc.conf.
I’ve occasionally written my own rc.d scripts, but have never needed to modify an existing one provided by the base system or packages (any configuration I would want to do would be in the daemon flags in rc.conf.local).
I’m not sure where we disagree. You’ve just described the situation that led the author to suggest rc.d doesn’t belong in etc. The best case I can muster against that view is as you describe, that a local administrator might want to write a script and drop it in rc.d, but most will just use rc.local for that.
rc.conf.local exists becuase it’s generally not wise for local administrators to modify rc.d.
The crux of the issue as I understood it is that (on netbsd) rc.conf is both used to modify configuration, and is also touched by system upgrades, and therefore can create merge conflicts. With OpenBSD splitting out this local configuration into a separate file only touched manually by the user or with rcctl, there are never merge conflicts here, or to any of the rc infrastructure (unless you changed a system/package rc.d script I suppose).
I’ve taken a stance of not running any AGPL software, period, and especially not any software that might be usable via the network (triggering the clauses that make the AGPL unique vs the GPL). My reasoning is simple: I’m lazy. I may patch a piece of software to fit my needs, but by doing so, I may patch in secrets or details about my infrastructure that I don’t want public. I’m not keeping the patch proprietary to turn it a business model or anything that goes against the spirit of the GPL; I just don’t find some patches particularly valuable or appropriate for public consumption. The GPL doesn’t cross this line, because it is easy to not redistribute my modifications in binary form (why would I with these patches?), but it can be potentially disastrous to flirt with these modifications in AGPL software.
If I wasn’t so lazy, I may patch the software “correctly”, introducing configuration options which don’t hardcode pieces of my own infrastructure, but I find this unreasonably burdensome for a so-called free software license, and it slows down proof-of-concept patches that I might otherwise want to deploy to canary instances to check for faults that integration and unit tests may not cover.
I don’t have an opinion on your overall argument. Just: ‘unreasonably burdensome for a so-called free software license’. This does not make sense to me, nobody ever said free software licenses are there to reduce burdens. That most of the permissive licenses actually do that is more a side-effect than anything else, and misses the point, IMHO.
Because Go doesn’t support operator overloading or define operators in terms of methods, there’s no way to use interface constraints to specify that a type must support the < operator (as an example). In the proposal, this is done using a new feature called “type lists”, an example of which is shown below:
// Ordered is a type constraint that matches any ordered type.
// An ordered type is one that supports the <, <=, >, and >= operators.
type Ordered interface {
type int, int8, int16, int32, int64,
uint, uint8, uint16, uint32, uint64, uintptr,
float32, float64,
string
}
In practice, a constraints package would probably be added to the standard library which pre-defined common constraints like Ordered. Type lists allow developers to write generic functions that use built-in operators:
// Smallest returns the smallest element in a slice of "Ordered" values.
func Smallest(type T Ordered)(s []T) T {
r := s[0]
for _, v := range s[1:] {
if v < r { // works due to the "Ordered" constraint
r = v
}
}
return r
}
Hmm, to me this looks like the Smallest function would only accept the built-in types with a less-than operator <. Don’t you lose a whole lot of usefulness if you’re not allowed to write your own Ordered type which can be used with a Smallest function?
People will forced to decide whether their ordering-based data structures (TreeMaps, RB-Trees, …) will use Ordered and therefore only work on a small set of “blessed” types, or whether they define their own, which … well … just imagine how great it will be to have two dozen replacement types for Ordered-for-my-own-types.
This is not true. Using the type syntax with builtin types allows for operator usage on all types (both builtin and user-defined) where the either the builtin type appears in the constraint list, or a custom type with the same underlying type.
This doesn’t work for struct types, but Go never had operating overloading in the first place, so no struct types would implement these operators anyways.
Yeah, but soc’s point stands: if you’re building an ordering-based function like Smallest() or data structure like a binary tree, you’ll either:
Choose the much more limiting Ordered constraint and use <, but then your function or tree will only be able to use built-in ordered types (or types with an underlying type that’s a built-in ordered type). That’ll work great for Tree(int) or Tree(MyInt), but won’t work for Tree(MyStruct).
Or you’ll have the constraint be an interface with a Less method, and you won’t be able to use the built-in types like Tree(int) at all … you’d be required to define a MyInt type that implements Less, and convert everything to MyInt before putting it into the tree.
Option #2 is definitely more flexible as it least allows any type to go into the map, but with the conversions it’ll be more boilerplate than you want for simple types.
The latest generics draft handles this by passing in a “compare function”, for example see Map in the Containers section. That kind of skirts the constraint/interface issue … but maybe that’s okay?
This is not true. Using the type syntax with builtin types allows for operator usage on all types (both builtin and user-defined) where the either the builtin type appears in the constraint list, or a custom type with the same underlying type.
I see. I misunderstood what you meant there because that is not the correct Go terminology. Embedding is only possible when defining struct and interface types. In the type newtype oldtype definition, oldtype is referred to as the “base type” and doesn’t inherit any of its methods, but does gain its operators.
Go if it doesn’t have to execute in a browser and the runtime is not a hindrance (and it usually isn’t).
C, C++, or Rust if a more minimal runtime is needed. Which is/are picked likely depends on what libraries are available and needed.
C and C++ are non-starters if the application must talk to the internet. I consider it malpractice to write internet-facing software in languages with manual memory management, given how hard we know it is to do correctly. (edit: I’m referring to building new software from scratch, not maintaining the existing software we rely on)
I’d find something else to work on if the problem lies outside of these domains.
C++ has automatic memory management with smart pointers. In fact, most of the languages that are garbage collected behind the scenes have runtimes written in C++ to manage memory.
Is it malpractice to be using operating systems, language runtimes, and other software mostly written by C and C++?
The real malpractice is proprietary software if anything. Or the fact that we have allowed private companies to install backdoors in all our computer systems.
See the edit (I assume you replied before seeing it).
C++ smart pointers still have safety issues. They do not prevent use-after-free bugs. They do not prevent data races (admittedly, Go programs can have data races as well, with all of the same safety concerns). And they don’t do anything about the other large uncharted areas of the C and C++ standards labeled Undefined Behavior.
It’s important to minimize the existence of this unsafe code if we care about secure systems, particularly for internet-facing applications. Some higher-level languages that promise memory and type safety do have runtimes built in unsafe languages, or use the unsafe features of their host language, and failure to keep invariants has caused real issues with these runtimes and code generation in the past. However, these are much smaller and relatively easy to audit for correctness, than auditing an entire ecosystem of C code.
I guess it depends on who we are talking about when we say “we care about secure system”.
If we are talking about regular people: I would argue social engineering (phishing) and backdoors from government agencies via corporations are a bigger concern than random hackers on the internet.
If we are talking about tech employees: I would argue social engineering and backdoors from government agencies are a bigger concern than random hackers on the internet.
Doesn’t matter how safe your system is memory-wise if it is weakly engineered from a human perspective.
I watched a great presentation that showed how to replicate the FastPass UI within the browser at a pixel perfect level. FastPass can be written in the most secure language, but it’s UI is flawed.
I’m not sure I agree with you. A chain of zero-days could have you pnwed just by visiting a compromised website—certainly something we want to avoid. However, I’m not convinced that Rust is the solution to all our security woes. When Redox OS asked for people to test it, they found all sorts of security bugs. Maybe writing it in Rust minimized some of them, maybe not.
I don’t think backdoors from government agencies are nearly as big of an issue as zero-days stocked up by government agencies.
If that came across as trolling, I didn’t mean it that way. This was a post asking for opinions, and that is my honest opinion. I’m also not completely against the use of C and C++, but you must consider attack surface, and the internet is an extremely large one.
Speaking specifically about the flag anomaly, I much prefer Go’s package because it removes ambiguity. In a getopt(1) program, when I see -abc, this could mean two separate things: either there are three flags -a -b -c written in a shortened form, or there is a single flag -a which takes a string argument bc. Go’s flag removes this ambiguity entirely.
It doesn’t, because you as an end user don’t know if the program is written in Go, or if the author wrote their own argument handling because they didn’t like the flag package.
Still, clarity is something to strive towards as we develop software.
There are other reasons I prefer flag as well. For example, it is possible to provide boolean flags with true defaults, and to disable them with -name=false. This is in contrast to getopt(1)-style flag handling where individual character flags can only be toggled on.
I personally am a fan of having both — short and long options — around.
I use the short options when typing in a shell, because it’s convenient. I also usually bunch together multiple options, e.g. ls -ltr, just because of typing efficiency. (I also type python3 -mvenv env without space between the option and the argument, sue me!) For shell scripts on the other hand long options might make code more readable/self-explanatory, especially if co-workers also have to maintain it.
That’s why I like having the single-dash prefix reserved for short options and double-dash for long options, because both variants have their place.
Interesting find! So the service in question was not only running as root, but the webserver/app wasn’t even chrooted. Reminds me a bit of the Apache phf exploit from 1996. People, always wear your seatbelt!
Just as an aside, and I know this article isn’t about any particular language, but Go makes it trivially easy to get a webserver running - but it’s not chrooted and runs as root. Would be nice if it were trivially easy to get a webserver running with basic defenses.
but Go makes it trivially easy to get a webserver running - but it’s not chrooted and runs as root.
If it was dev you’d be at a port above 1024, so a regular user would run it. And if it was prod, wouldn’t you drop privileges after start via your service manager?
Or were you just saying how easy it is to just sudo ./web-server
One of the downsides of chroot is that it’s not implemented on Windows, so it’ll make your webserver not cross-platform. I’m not sure what alternatives there are for Windows?
Docker containers usually run their local system as root, and most compiler services I know of run in Docker containers or something similar. Eventually we’ll get to a capability-based reality where you have to explicitly grant permissions rather than explicitly taking them away, but we’re not quite there yet.
There seems to be a belief amongst memory safety advocates that it is not one out of many ways in which software can fail, but the most critical ones in existance today, and that, if programmers can’t be convinced to switch languages, maybe management can be made to force them.
I didn’t see this kind of zeal when (for example) PHP software fell pray to SQL injections left and right, but I’m trying to understand it. The quoted statistics about found vulnerabilities seem unconvincing, and are just as likely to indicate that static analysis tools have made these kind of programming errors easy to find in existing codebases.
Not all vulnerabilities are equal. I prioritize those that give attackers full control over my computer. They’re the worst. They can lead to every other problem. Plus, their rootkits or damage might not let you have it back. You can lose the physical property, too. Alex’s field evidence shows memory unsafety causes around 70-80% of this. So, worrying about hackers hitting native code, it’s rational to spend 70-80% of one’s effort eliminating memory unsafety.
More damning is that languages such as Go and D make it easy to write high-performance, maintainable code that’s also memory safe. Go is easier to learn with a huge ecosystem behind it, too. Ancient Java being 10-15x slower than C++ made for a good reason not to use it. Now, most apps are bloated/slow, the market uses them anyway, some safe languages are really lean/fast, using them brings those advantages, and so there’s little reason left for memory-unsafe languages. Even in intended use cases, one can often use a mix of memory-safe and -unsafe languages with unsafe used on performance-sensitive or lowest-level parts of the system. Moreover, safer languages such as Ada and Rust give you guarantees by default on much of that code allowing you to selectively turn them off only where necessary.
If using unsafe languages and having money, there’s also tools that automatically eliminate most of the memory unsafety bugs. That companies pulling in 8-9 digits still have piles of them show total negligence. Same with those in open-source development who aren’t doing much better. So, on that side of things, whatever tool you encourage should lead to memory safety even with apathetic, incompetent, or rushed developers working on code with complex interactions. Double true if it’s multi-threaded and/or distributed. Safe, orderly-by-default setup will prevent loads of inevitable problems.
The quoted statistics about found vulnerabilities seem unconvincing
If studies by security teams at Microsoft and Google, and analysis of Apple’s software is not enough for you, then I don’t know what else could convince you.
These companies have huge incentives to prevent exploitable vulnerabilities in their software. They get the best developers they can, they are pouring many millions of dollars into preventing these kinds of bugs, and still regularly ship software with vulnerabilities caused by memory unsafety.
“Why bother with one class of bugs, if another class of bugs exists too” position is not conductive to writing secure software.
“Why bother with one class of bugs, if another class of bugs exists too” position is not conductive to writing secure software.
No - but neither is pretending that you can eliminate a whole class of bugs for free. Memory safe languages are free of bugs caused by memory unsafety - but at what cost?
What other classes of bugs do they make more likely? What is the development cost? Or the runtime performance cost?
I don’t claim to have the answers but a study that did is the sort of thing that would convince me. Do you know of any published research like this?
No - but neither is pretending that you can eliminate a whole class of bugs for free. Memory safe languages are free of bugs caused by memory unsafety - but at what cost?
What other classes of bugs do they make more likely? What is the development cost? Or the runtime performance cost?
The principle cost of memory safety in Rust, IMO, is that the set of valid programs is more heavily constrained. You often here this manifest as “fighting with the borrow checker.” This is definitely an impediment. I think a large portion of folks get past this stage, in the sense that “fighting the borrow checker” is, for the most part, a temporary hurdle. But there are undoubtedly certain classes of programs that Rust will make harder to write, even for Rust experts.
Like all trade offs, the hope is that the juice is worth the squeeze. That’s why there has been a lot of effort in making Rust easier to use, and a lot of effort put into returning good error messages.
I don’t claim to have the answers but a study that did is the sort of thing that would convince me. Do you know of any published research like this?
I’ve seen people ask this before, and my response is always, “what hypothetical study would actually convince you?” If you think about it, it is startlingly difficult to do such a study. There are many variables to control for, and I don’t see how to control for all of them.
IMO, the most effective way to show this is probably to reason about vulnerabilities due to memory safety in aggregate. But to do that, you need a large corpus of software written in Rust that is also widely used. But even this methodology is not without its flaws.
If you think about it, it is startlingly difficult to do such a study. There are many variables to control for, and I don’t see how to control for all of them.
That’s true - but my comment was in response to one claiming that the bug surveys published by Microsoft et al should be convincing.
I could imagine something similar being done with large Rust code bases in a few years, perhaps.
I don’t have enough Rust experience to have a good intuition on this so the following is just an example. I have lots of C++ experience with large code bases that have been maintained over many years by large teams. I believe that C++ makes it harder to write correct software: not (just) because of memory safety issues, undefined behavior etc. but also because the language is so large, complex and surprising. It is possible to write good C++ but it is hard to maintain it over time. For that reason, I have usually promoted C rather than C++ where there has been a choice.
That was a bit long-winded but the point I was trying to make is that languages can encourage or discourage different classes of bugs. C and C++ have the same memory safety and undefined behavior issues but one is more likely than the other to engender other bugs.
It is possible that Rust is like C++, i.e. that its complexity encourages other bugs even as its borrow checker prevents memory safety bugs. (I am not now saying that is true, just raising the possibility.)
This sort of consideration does not seem to come up very often when people claim that Rust is obviously better than C for operating systems, for example. I would love to read an article that takes this sort of thing into account - written by someone with more relevant experience than me!
I’ve been writing Rust for over 4 years (after more than a decade of C), and in my experience:
For me Rust has completely eliminated memory unsafety bugs. I don’t even use debuggers or Valgrind any more, unless I’m integrating Rust with C.
I used to have, at least during development, all kinds of bugs that spray the heap, corrupt some data somewhere, use uninitialized memory, use-after-free. Now I get compile-time errors or panics (which are safe, technically like C++ exceptions).
I get fewer bugs overall. Lack of NULL and mandatory error handling are amazing for reliability.
Built-in unit test framework, richer standard library and easy access to 3rd party dependencies help too (e.g. instead of hand-rolling another own buggy hash table, I use a well-tested well-optimized one).
My Rust programs are much faster. Single-threaded Rust is 95% as fast as single-threaded C, but I can easily parallelize way more than I’d ever dare in C.
The costs:
Rust’s compile times are not nice.
It took me a while to become productive in Rust. “Getting” ownership requires unlearning C and a lot of practice. However, I’m not fighting the borrow checker any more, and I’m more productive in Rust thanks to higher-level abstractions (e.g. I can write map/reduce iterator that collects something into a btree — in 1 line).
Of course older software, mostly written in memory-unsafe languages, sometimes written in a time when not every device was connected to a network, contains more known memory vulnerabilities. Especially when it’s maintained and audited by companies with excellent security teams.
These statistics don’t say much at all about the overall state of our software landscape. It doesn’t say anything about the relative quality of memory-unsafe codebases versus memory-safe codebases. It also doesn’t say anything about the relative sizes of memory-safe and memory-unsafe codebases on the internet.
iOS and Android aren’t “older software”. They’ve been born to be networked, and supposedly secure, from the start.
Memory-safe codebases have 0% memory-unsafety vulnerabilities, so that is easily comparable. For example, check out the CVE database. Even within one project — Android — you can easily see whether the C or the Java layers are responsible for the vulnerabilities (spoiler: it’s C, by far). There’s a ton of data on all of this.
That starts to look like a No True Scotsman. When real-world C codebases have vulnerabilities, they’re somehow not proper C codebases. Even when they’re part of flagship products of top software companies.
I’m actually not arguing that good programmers are able to write memory-safe code in unsafe languages. I’m arguing vulnerabilities happen at all levels in programming, and that, while memory safety bugs are terrible, there are common classes of bugs in more widely used (and more importantly, more widely deployed languages), that make it just one class of bugs out of many.
When XSS attacks became common, we didn’t implore VPs to abandon Javascript.
We’d have reached some sort of conclusion earlier if you’d argued with the point I was making rather than with the point you wanted me to make.
When XSS attacks became common, we didn’t implore VPs to abandon Javascript.
Actually did. Sites/companies that solved XSS did so by banning generation of markup “by hand”, and instead mandated use of safe-by-default template engines (e.g. JSX). Same with SQL injection: years of saying “be careful, remember to escape” didn’t work, and “always use prepared statements” worked.
These classes of bugs are prevalent only where developers think they’re not a problem (e.g. they’ve been always writing pure PHP, and will continue to write pure PHP forever, because there’s nothing wrong with it, apart from the XSS and SQLi, which are a force of nature and can’t be avoided).
This kind of makes me think of someone hearing others talk about trying to lower the murder rate and then hysterically going into a rant about how murder is only one class of crime
That analogy is really terrible and absolutely not fitting the context here. It’s also very skewed, the murder rate is not the reason for calls for bans.
Although I mostly agree, I’ll note Android was originally built by a small business acquired by Google that continued to work on it probably with extra resources from Google. That makes me picture a move fast and break things kind of operation that was probably throwing pre-existing stuff together with their own as quickly as possible to get the job done (aka working phones, market share).
And that’s exactly what illustrates “eliminates a class of bugs”. We’re not saying that we’ll end up in utopia. We just don’t need that class of bugs anymore.
Correct, but the author is arguing that this is an exceptionally grievous class of security bugs, and (in another article) that developers’ judgement should not be trusted on this matter.
Today, the vast majority of new code is written for a platform where execution of untrusted memory-safe code is a core feature, and the safety of that platform relies on a stack of sandboxes written mostly in C++ (browser) and Objective C/C++/C (system libraries and kernel)
Replacing that stack completely is going to be a multi-decade effort, and the biggest players in the industry are just starting to dip their toes in memory-safe languages.
What purpose does it serve to talk about this problem as if it were an urgent crisis?
Replacing that stack completely is going to be a multi-decade effort, and the biggest players in the industry are just starting to dip their toes in memory-safe languages.
Hm, so. Apple has developed Swift, which is generally considered a systems programming language, to replace Objective-C, which was their main programming language and already had safety features like baked in ARC. Google has implemented Go. Mozilla Rust. Google uses tons of Rust in Fuchsia and has recently imported the Rust compiler into the Android source tree.
Microsoft has recently been blogging about Rust quite a lot and is often seen hanging around and blogs about how severe memory problems are to their safety story. Before that, Microsoft has spent tons of engineering effort into Haskell as a research base and C#/.Net as a replacement for their C/C++ APIs.
Amazon has implemented firecracker in Rust and bragged about it on their AWS keynote.
Come again about “dipping toes”? Yes, there’s huge amounts of stack around, but there’s also huge amounts to be written!
What purpose does it serve to talk about this problem as if it were an urgent crisis?
Because it’s always been a crisis and now we have the tech to fix it.
P.S.: In case this felt a bit like bragging Rust over the others: it’s just where I’m most aware of things happening. Go and Swift are doing fine, I just don’t follow as much.
The same argument was made for Java, which on top of its memory safety, was presented as a pry bar against the nearly complete market dominance of the Wintel platform at the time. Java evangelism managed to convert new programmers - and universities - to Java, but not the entire world.
Oracle’s deadly embrace of Java didn’t move it to rewrite its main cash cow in Java.
Rust evangelists should ask themselves why.
I think that of all the memory-safe languages, Microsoft’s C++/CLI effort comes closest to understanding what needs to be done to entice coders to move their software into a memory-safe environment.
At my day job, I actually try to spend my discretionary time trying to move our existing codebase to a memory-safe language. It’s mostly about moving the pieces into place so that green-field software can seamlessly communicate with our existing infrastructure. Then seeing what parts of our networking code can be replaced, slowly reinforcing the outer layers while the inner core remains memory unsafe.
Delicate stuff, not something you want the VP of Engineering to issue edicts about. In the meantime, I’m still a C++ programmer, and I really don’t appreciate this kind of article painting a big target on my back.
Java and Rust are vastly different ball parks for what you describe. And yet, Java is used successfully in the database world, so it is definitely to be considered. The whole search engine database world is full of Java stacks.
Oracle didn’t rewrite its cashcow, because - yes, they are risk-averse and that’s reasonable. That’s no statement on the tech they write it in. But they did write tons of Java stacks around Oracle DB.
It’s an argument on the level of “Why isn’t everything at Google Go now?” or “Why isn’t Apple using Swift for everything?”.
What purpose does it serve to talk about this problem as if it were an urgent crisis?
To start the multi-decade effort now, and not spend more decades just saying that buffer overflows are fine, or that—despite of 40 years of evidence to the contrary—programmers can just avoid causing them.
I didn’t see this kind of zeal when (for example) PHP software fell pray to SQL injections left and right
You didn’t? SQL injections are still #1 in the OWASP top 10. PHP had to retrain an entire generation of engineers to use mysql_real_escape_string over vulnerable alternatives. I could go on…
I think we have internalized arguments the SQL injection but have still not accepted memory safety arguments.
The articles use the language of whistleblowers. It suggests that counter-arguments are made in bad faith, that developers are trying to hide this ‘dirty secret’. Consider that C/C++ programmers skew older, have less rosy employment prospects, and that this article feeds nicely into the ageist prejudices already present in our industry.
Arguments aimed at programmers, like this one at least acknowledge the counter-arguments, and frame the discussion as one of industry maturity, which I think is correct.
I do not see it as bad faith. There are a non-zero number of people who say they can write memory safe C++ despite there being a massive amount of evidence that even the best programmers get tripped up by UB and threads.
Consider that C/C++ programmers skew older, have less rosy employment prospects, and that this article feeds nicely into the ageist prejudices already present in our industry.
There’s an argument to be made that the resurging interest in systems programming languages through Rust, Swift and Go futureproofs experience in those areas.
Memory safety advocate here. It is the most pressing issue because it invokes undefined behavior. At that point, your program is entirely meaningless and might do anything. Security issues can still be introduced without memory unsafety of course, but you can at least reason about them, determine the scope of impact, etc.
The CHANGELOG doesn’t mention this rubygems incident and it ALSO breaks BC. Maybe I’m overly pessimistic and paranoid but I’d have republished 0.0.6 unchanged as 0.0.8 and released anything new as 0.1.0.
Pre-1.0.0 releases in semantic versioning have no defined compatibility requirements with any other version.
Though, if this module is being used in production, a 1.0.0 release should be cut. Even more so since this is open source and you don’t know all of the consumers.
With a 0.0.x I wouldn’t call that a problem, really. It just irks me that I suppose a lot of people might think not to upgrade from 0.0.7 although they should. Or would gem warn you that the installed version was yanked?
I am fine if people want to make money with my software, as long as whatever
they add on to my software is also open source.
People are willing to pay for software, even open source, as getting from the
source code to a finished binary can be a pain. I want an open source license
that that doesnt allow people to use my stuff as part of a closed source
project, unless they pay me.
I dont know if such a thing exists, so for now I am using this:
You don’t need a single license for this. As long as you maintain sole copyright, you can license the software under any additional license someone is willing to pay for. That license applies to their copy, while the open source (likely copyleft) license applies to everyone else who receives the license without pay.
Ugh, another one bites the dust — without giving the BSDs a chance. Try OpenBSD, the out of the box experience is the most pleasant one I’ve had yet, more so than macOS even, because it does exactly what I want it to.
I have to use a MacBook for work, and while the UI is pretty and all, I find its tiny quirks and overall rigidity very annoying. I can’t do things the way I’d like to.
I gave FreeBSD a serious chance for months. I hacked a little on i915 trying to fix some backlight bugs. I contributed to what became the implementation of the
utimensatsyscall implementation for the Linuxulator. I was even featured on an episode of the original BSDNow in 2015.I suppose you could argue some value of stability and release engineering still exist in FreeBSD, but they’re even more hostile to older hardware and the desktop experience is even worse as more and more software assumes Linux.
The entire time reading this article I was thinking that NetBSD would be a much better fit for you than Linux
One reason I like my OpenBSD Desktop is that it is still running great on my 2012 Toshiba Portege - I’ve upgraded the RAM to 16Gb and the HDD to an SSD - the only draw back is getting decent batteries for portability is now becoming an issue. But I have heard that OpenBSD may be coming to the M1 in the near future :~)
did we read the same article
The article complains Go’s PowerPC baseline is POWER8. OpenBSD’s PowerPC baseline is POWER9…
There are two power ports: powerpc64 (POWER9) and macppc (G4/G5 and maybe G3).
Browser support in macppc was pretty abysmal last I checked though.
macppc supports all New World models, including the G3.
I think NetBSD is probably the best of the BSDs for running on PowerPC, if we’re talking about it; in addition to macppc they have ofppc, amigappc, bebox, etc.
As a Mac user who mostly likes the way it works on the surface, I wish it were as straightforward beneath the surface as a BSD.
It wasn’t mentioned by the author, but another point of
makenot being portable is the differences between GNU and BSDmake, and whether there are any differences between the “fragments” provided by the operating system that can be included in another makefile (e.g..include <bsd.prog.mk>).I was giving this a shot earlier after it first entered -current snapshots and was unable to get the default IPv4 route set with the gateway from my dhcp server. This is after removing
dhcpfrom my hostname.if and setting AUTOCONF4 on the interface. (I’m not certain if removing dhcp is correct, but /etc/netstart will still use dhclient if you specify that, and so I tried without.)You need to remove ‘dhcp’ and add ‘inet autoconf’ to your /etc/hostname.if file. Then the system uses dhcpleased. If ‘dhcp’ is present, dhclient will be started
Perhaps lobsters is not the best place to report bugs.. but I did both of these and it was still not setting the route. Currently on:
When I up the interface, I see logged:
Even stranger is that I cannot manually add the route anymore.
e: just saw this, and latest snap i’m on isn’t build with this commit yet: https://marc.info/?l=openbsd-cvs&m=161458351925096&w=2
I never understood why this doesn’t work like in C where you can define constants this way, not variables.
Build times, perhaps. It allows objects in the build cache to be reused and modified at link time, rather than needing to recompile.
This seems relatively easy to implement compared to making everyone have stable ABIs for everything, including languages that can’t have one.
The nu-school package managers have machine-readable lockfiles with exact versions of all libraries used. A distro could create and archive such lockfiles to be able to later know exactly which of their statically-built binaries contain vulnerable dependencies.
In Go’s case, you can simply inspect the executable (using
go version -m path/to/exe) to display the Go version used in the build, and all of the statically-linked module’s versions and checksums, so you don’t even need to keep around any extra build artifacts to determine which executable are vulnerable.I am unfamiliar with Gentoo’s tooling, but I do know that OpenBSD’s Go ports contain the complete package list of all dependency modules that will be required for the build to succeed (this is necessary to download everything ahead of time, so the build itself can be done offline and in a chroot). This makes it rather trivial to discover all ports which need to be patched and rebuilt for any given vulnerability fix.
Wow, this blog post is so lacking in empathy for users that I’m surprised it made it on a reputable distro’s blog. Instead of spilling 1000 words on why “static linking is bad”, maybe spend a little time thinking about why people (like me) and platforms (like go/rust et al) choose it. The reason people like it is that it actually works and it won’t suddenly stop working when you change the version of openssl in three months. It doesn’t even introduce security risks! The only difference is you have to rebuild everything on that new version, which seems like a small price to have software that works, not to mention that rebuilding everything will also re-run the tests on that new version. I can build a go program on nixos, ship it to any of my coworkers and it actually just works. We are on a random mix of recent ubuntu, centos 7 and centos 8 and it all just works together. That is absolutely not possible with dynamic linking
It works well if all you care about is deploying your application. As a distro maintainer, I’m to keeping track of 500 programs, and having to care about vendored/bundled versions and statically linked in dependencies multiplies the work I have to do.
But … that’s a choice you make for yourself? No application author is asking you to do that, and many application authors actively dislike that you’re doing that, and to be honest I think most users don’t care all that much either.
I’ve done plenty of packaging of FreeBSD ports back in the day, and I appreciate it can be kind of boring thankless “invisible” gruntwork and, at times, be frustrating. I really don’t want to devalue your work or sound thankless, but to be honest I feel that a lot of packagers are making their own lives much harder than it needs to be by sticking to a model that a large swath of the software development community has, after due consideration and weighing all the involved trade-offs, rejected and moved away from.
Both Go and Rust – two communities with pretty different approaches to software development – independently decided to prefer static linking. There are reason for that.
Could there be some improvements in tooling? Absolutely! But static linking and version pinning aren’t going away. If all the time and effort spent on packagers splitting things up would be spent on improving the tooling, then we’d be in a much better situation now.
I think this is a common view but it results from sampling bias. If you’re the author of a particular piece of software, you care deeply about it, and the users you directly interact with also care deeply about it. So you will tend to see benefits that apply to people for whom your software is of particular importance in their stack. You will tend to be blind to the users for whom your software is “part of the furniture”. From the other side, that’s the majority of the software you use.
Users who benefit from the traditional distribution packaging model for most of their software also find that same model to be painful for some “key” software. The problem is that what software is key is different for different classes of user.
A big reason people ship binaries statically linked is so it’s easier to use without frills, benefiting especially users who aren’t deeply invested in the software.
For me personally as an end user, if a program is available in apt-get then I will install it from apt-get first, every time. I don’t want to be responsible for tracking updates to that program manually!
I do that as well, but I think “apt-get or manual installs” is a bit of a false dilemma: you can have both.
Static linking does introduce a security risk: ASLR made ineffective. Static linking creates a deterministic memory layout, thus making moot ASLR.
Untrue, look up static-PIE executables. Looks like OpenBSD did it first, of course.
I believe static PIE can only randomize a single base address for a statically linked executable, unlike dynamically linked PIE executable where all loaded PIC objects receive a randomized base address.
I’m very familiar with static PIE. I’m unsure of any OS besides OpenBSD that supports it.
Rustc + musl, supports it on linux, since gcc has a flag for it I imagine it’s possible to use it for C code too but I don’t know how.
It was added to GNU libc in 2.27 from Feb 2018. I think it should work on Linux?
Looks like it works on Linux with gcc 10.
Did a bit of rummaging in the exe header but I’m not 100% sure what I’m looking for to confirm there, but it had a relocation section and all symbols in it were relative to the start of the file as far as I could tell.
Edit: Okay, it appears the brute-force way works. I love C sometimes.
aslr.c:
Testing:
Note
ldddistinguishing between “not a dynamic executable” and “statically linked”.Doesn’t it stop other kind of attacks on the other hand?
What attacks would static linking mitigate?
I don’t know much about security, and this is an honest question. My understanding is that ASLR is made to mostly protect when the executable is compromised due to stack overflow and similar attacks, right? Aren’t these problems mostly a thing of past for the two languages that abhor static linking, like go and rust?
I really have no idea, that was an honest question!
I mean, the obvious example is tricking binaries into loading the wrong dynamic library, e.g.
https://us-cert.cisa.gov/ncas/current-activity/2010/11/09/Insecure-Loading-Dynamic-Link-Libraries-Windows-Applications
and
https://www.contextis.com/en/blog/linux-privilege-escalation-via-dynamically-linked-shared-object-library
Those would be examples of local attacks. ASLR does not protect against local attacks. So we need to talk about the same threat vectors. :)
There are a lot of code-injection vulnerabilities that can occur as a result of
LD_LIBRARY_PATHshenanigans. If you’re not building everything relro, dynamic linking has a bunch of things like the PLT GOT that contain function pointers that the program will blindly jump through, making exploiting memory-safety vulnerabilities easier.As an author of open source malware specifically targeting the PLT/GOT for FreeBSD processes, I’m familiar with PLT/GOT. :)
The good news is that the llvm toolchain (clang, lld) enables RELRO by default, but not BIND_NOW. HardenedBSD enables BIND_NOW by default. Though, on systems that don’t disable unprivileged process debugging, using BIND_NOW can open a new can of worms: making PLT/GOT redirection attacks easier over the ptrace boundary.
It would be terrible if this repo was replicated across the Internet.
Remember kids, don’t copy that floppy!
Another mirror: https://gitee.com/mirrors/youtube-downloader
Codeberg is hosted in Germany, I wouldn’t count on this repo staying up.
I wouldn’t be so sure. In germany youtube-dl would probably be seen as a tool for making a personal copy, which is allowed in germany.
Historically, why is Germany so anal about copyright compared to other countries?
Easy money. Cease & Desist letter in Germany come with a fine attached if you follow them. You need to pay the the lawyers fees (~800+ EUR). This is pretty unique. So there’s “cease & desist” mills who trawl people, e.g. off bittorrent and all other networks. This means that some of these cases will end up at a court. As all things digital have no place of service or occurrence, the filing side can pick any court to go to. Which is usually Hamburg or Cologne, which tend to be the most eager to stretch the law to the rights holder side. But that’s actually not the process intended, what they want is people to be frightened and paying the lawyers fee on the first letter. They will even lower it if you even look at them like you might defend yourself.
They, like the US, have a highly information based economy. If you can put a value amount on copyright, you can put a penalty amount. You can decide if a lawyer is worth it. You can make laws about it.
Germany somehow seems even worse than the US though, despite producing less media so I don’t understand that discrepancy
This is unsurprising. Most international media companies here are basically “importing” and rarely “exporting”, which means (distribution) licensing, so all their local orgs have a high focus on rights management and lobbying for better terms. And if you have staff lawyers around all the time… you might as well use them?
I think that whole sentence can be substituted as “Historically, why is Germany so anal about everything compared to other countries?”…
(Just kidding, sorry Germans!)
Der Freud wegen.
The original repo is on the WayBack machine 1400 times with the most recent being 5 days ago. If you’re forking just to a copy lives around and not to continue development, I would just snag it there to be sure of the source.
Edited to make it clear I was linking to the original repo.
https://github.com/plredmond/yt-download
please do not click the fork button
Do not save copies outside Github as well. Wait until Github takes down all the forks in one go (it’s not hard to do technically).
When would someone prefer OpenBSD to FreeBSD?
For me, better defaults.
I just installed it, the only configuration change I needed to do was to activate
apm.The minimalism is appealing, simply because there’s less you need to understand to feel confident in your systems. (same reason why I prefer to run OpenBSD over Linux.)
Security, cohesion.
I would probably use openbsd (rather than freebsd) if not for zfs.
All the time? ;D
FreeBSD is I think the right choice if you want to have a good base with ZFS and every software out there packaged and up to date.
OpenBSD is like a study in simplicity. I really enjoy things like how they manage wireless networks, no networkmanager or wpa_supplicant, sound without pulseaudio (you can use sndio on FreeBSD as well but you need to compile your own packages).
OpenBSD feels more like things done right/simple, while FreeBSD for many purposes feels more like “a better linux distribution”, with both stable base and up to date packages, without having to deal with third party repositories
Another differencen for the decision can be whether you want to use just base. While on FreeBSD you can do some basic stuff like running an NFS server without any packages, OpenBSD will give you an httpd server, complete with the means to get a letsencrypt or other acme based certificate, you will also get a pretty nice smtp server (no imap server though) and also typical user facing things, like tmux, X with a couple of window managers are part of the base system.
But then again very big reason for choosing any general purpose OS is taste.
OpenBSD is simpler in all the good ways, but lacks some important tech. Jails and ZFS at least.
There was talk of porting HAMMER from Dragonfly years ago. I expect this to happen with HAMMER2 at some point.
That would be very cool indeed. Figuring out an openbsdy way to do jails would be amazing as well.
Yyyyyyyep.
Edit: To expand a little, now that I’ve had some coffee, let me repeat one of soc’s points a little more vigorously: C++ will not get better until they start taking things away as well as adding things. I still have vivid memories of reading through a C++ book in undergrad, back in 2004 or something, and thinking “ok, I can’t just entirely avoid pointers in favor of references… so what good are references besides making more ways of doing the same thing?”
References serve an entirely different purpose than pointers. A pointer is a type that can be
NULL, so it may represent an optional object, whereas references are (theoretically) neverNULLand always valid. These are valuable safety features. However, you would find it extremely difficult to write C++ code that exclusively uses references and no pointers. So both have their place in the language, though I would replace almost all pointers withstd::unique_ptrif C++11 is available.I agree with the sentiment that the relentless growth of C++ and the reluctance of the committee to deprecate features is the source of many problems. Even though I know and write C++ for many years, I still find C++ features that weren’t previously known to me on a daily (!) basis. In other words, for thousands of days, I am daily surprised by this language. The complexity is unfathomable.
They are never
NULLand often invalid. Like when you saved a reference to something that either (a) went out of scope; (b) was the result of dereferencing a pointer to memory that was freed afterwards. You can also make a null reference if you try very hard, but that doesn’t happen in practice, whereas invalid references sure do.That’s just not true.
is not at all hard, nor is it it even that uncommon.
It has never shown up in my 4 years of commercial C++ experience + 5 years non-commercial before that.
I’ve seen it in the wild; debugged a core dump that had it. The evil thing is that the program crashes at the point where the
NULL-reference is used, not where NULL is dereferenced (in compiled assembly, dereferencing a pointer to store it in a reference is a no-op). These two points may be far apart.This is undefined behavior.
They shouldn’t be invalid, but it’s not entirely enforced by the compiler. The idea is that by using references, a programmer signals to the reader of the code that the value is intended to be a non-
NULLvalid object, and if that is not the case, the error is at whatever line of code that creates this invalid reference, not the one that uses it. In the wild, even inexperienced C++ programmers rarely create invalid references simply because it is harder to do so, whereas they routinely create invalid pointers. In addition to that, invalid references are almost always a programmer error, but invalid pointers are a regular occurrence and often part of a design that allows and accounts for such states.Yes, nonetheless it happens. And it doesn’t matter that “somebody else” made the error. The error is still there and it’s your code that crashes. I feel much better using a language that doesn’t put me in such situations.
For some definition of “rarely”. I’ve often seen code that passed references to objects wrapped in
std::shared_ptr(often to avoid ARC overhead) into function arguments where those functions captured those by reference in lambdas which were executed after thestd::shared_ptrfreed the memory. Similarly,thisregardless of its type is in practice used like a reference. There’s a lot of code in the wild that does something likePostTask([this] { /* do smth */ });.At the end of the day it doesn’t matter that “the error is somewhere else”. If the language makes something remotely straightforward, I’m sure to have to deal with that in a shared codebase at one point no matter how few errors I make myself.
Yes, whatever the compiler doesn’t prohibit is likely to end up in the codebase. However, I still think that references are valuable and provide better safety than pointers. Obviously it would be better if making mistakes was impossible. I’m not whitewashing C++’s serious shortcomings here.
As for the discussion about invalid references, I’ll leave you with this short story: http://www.gotw.ca/conv/002.htm
The official (D&E of C++) purpose of references is to enable operator overloading.
Very interesting. Found the source: https://www.stroustrup.com/bs_faq2.html#pointers-and-references
It’s important to keep in perspective that the backwards incompatibility issue with non-module builds that was patched into older releases was done so all the way back into the Go 1.9 release branch. Go 1.9 was originally released in August of 2017. I don’t know of anyone that is still using a Go release from back then, which is mostly a product of how well the Go team prioritizes not introducing breaking changes. The Go team also only provides security patches for the latest two version branches of Go (currently, 1.14 and 1.15), so nobody should be using a release that old regardless.
It was painful from the developer perspective because your dependencies and deps-of-deps might have not had module support, and might already be a v2/v3 etc. (thus blocked you from sanely supporting non-module and module at the transition period)
There was never a need for your dependencies, or their dependencies, to be or understand modules in order to consume them in a module build. They just worked. The only module “feature” that wouldn’t have been possible with this code is the inability to contain multiple major versions of a package, since that does rely on semantic import versioning (what this author is challenging).
Being the maintainer of three of these tools and a long time command-line enthusiast, it’s just sad to read a comment like this.
I can understand that people do not want to try out new tools. Fine. But why criticize them without even taking a proper look?
As others have already pointed out, most of these tools are not breaking pipeline usage. The README for
bathas a whole section on “integration with other tools”, showing multiple examples of pipeline usage.fd’s README also points out several examples on how to pipefd’s output to other programs (e.g.xargs,parallel,as-tree).I have a feeling that a lot of people react allergic to colorized command-line output, and I absolutely do not understand why. Just because a tool doesn’t look like 1970 doesn’t mean that it’s not a proper citizen of the command-line world. But that’s not even the point. My tools do not provide colorized output to look “fancy”. They use colorized output because I think it’s actually pretty damn useful. Also, there is always the possibility to easily switch them off if you really don’t like it (set
NO_COLORor use--color=neverin an alias).Finally, I am really sick of the implied “rewritten in Rust” stigma. No, I did not write these tools in Rust because I wanted to demonstrate something. Or to show off. I simply like to try different programming languages from time to time and I think that Rust is an awesome fit for command-line tools.
fdwas actually started as a C++ project. I just moved it to Rust at some point. Mainly because of boost packaging issues and because I discovered the awesome Rust libraries by @burntsushi. Criticizing a tool for the language it’s written in is fine as long as there is an actual argument why that programming language was a poor choice for the job.First, let me say that I really appreciate the effort in trying to keep moving things forward. That means change, and change is not always good.
I’m pretty allergic to colors in command line tools - but that is mostly because of awful default logging/outpu formats from ruby/rails - and not for tools like these that properly detect piping etc.
I’m also slow to adopt new tools - I still tend to prefer gnu find to fd, haven’t quite been able to put fzf to work - but I’m not quite hopeless - I do use ripgrep quite a bit.
I think quality matters (more) when using colorization - and it takes a village to get there. The recent generations of colorized interactive ruby output is a boon (on Linux) - it looks good, and if I copy from gnome terminal and paste into nvim - the escape séquences vanish. But on windows servers with the standard windows console - even the line breaks do not work properly, and colorization is hit and miss (colors or escape sequences? It depends…).
Many of us need to work on various heterogeneous deployment servers not under our control - and that makes us thirsty for the simplest tools - to the point of sometimes preferring a predictable dull rock to a scalpel that might shatter at any moment. But it sometimes makes the transitionfrom skinning rabbit to preformingbbrain surgery… More challenging.
First, let me say that I really appreciate the effort in trying to keep moving things forward. That means change, and change is not always good.
I’m pretty allergic to colors in command line tools - but that is mostly because of awful default logging/outpu formats from ruby/rails - and not for tools like these that properly detect piping etc.
I’m also slow to adopt new tools - I still tend to prefer gnu find to fd, haven’t quite been able to put fzf to work - but I’m not quite hopeless - I do use ripgrep quite a bit.
I think quality matters (more) when using colorization - and it takes a village to get there. The recent generations of colorized interactive ruby output is a boon (on Linux) - it looks good, and if I copy from gnome terminal and paste into nvim - the escape séquences vanish. But on windows servers with the standard windows console - even the line breaks do not work properly, and colorization is hit and miss (colors or escape sequences? It depends…).
Many of us need to work on various heterogeneous deployment servers not under our control - and that makes us thirsty for the simplest tools - to the point of sometimes preferring a predictable dull rock to a scalpel that might shatter at any moment. But it sometimes makes the transitionfrom skinning rabbit to preformingbbrain surgery… More challenging.
Sorry, this seems completely uncalled for. One of the reasons I like this site is its relative lack of political drama and focus on technology. I don’t know what Donald Trump has to do with this technical disagreement, but I don’t think he’s relevant to it, and I don’t think the mention of him will resolve anything.
*sigh
i’ll just let myself out. thanks for all the fish.
Thank you for the response.
It makes me sad because you invest countless hours into a project and then someone disregards it within seconds with a low-effort comment like this. Other people looking at this thread will see the wrong statements (“breaking pipelines”) and might take them for granted.
I absolutely agree with you on this point. I just don’t see any alternative. Is someone working on a completely redesigned, modern terminal API? It could probably still support ANSI escape codes for backwards compatibility.
I know you are probably not looking for a solution, but the problem might be that the
-R/--RAW-CONTROL-CHARScommand-line option is not passed tolesson that system. It tellslessto interpret those escape sequences. I guess most distributions setLESS="-R"by default.That’s your interpretation, but okay. I see your point. It’s definitely not a seal of quality per se. There are, however, a few things that can be inferred from a choice of programming language (expected startup latency, memory safety guarantees, portability, ease of distribution, …). As stated above, I think that Rust is actually an excellent choice for command line tools. Because it delivers on all of the mentioned aspects. That doesn’t mean that it’s the only choice though.
Most of the plumbing-replacement tools actually detect pipes and switch to machine-friendly output.
I can confirm this for
fd,batandexa. I just had a quick look at those and at a first glance all of them seem to support pipelines just fine.let me flame in peace!!1
thanks, i wasn’t aware of that magick.
Its pretty common actually https://man7.org/linux/man-pages/man3/isatty.3.html not unique to rust for that matter.
Simply as easy as if (isatty(fileno(stdout))) … else …
But! Quite annoying to do correctly on Windows in a way that works in both the cmd and MSYS shells: https://github.com/softprops/atty/blob/7b5df17888997d57c2c1c8f91da1db5691f49953/src/lib.rs#L101-L154
That one was quite fun to figure out. I cribbed it from some other project, but can’t remember which one.
I maintain a terminal library in D and yeah it is interesting to do it well on Windows (especially if you want to do a gui and a cli in the same exe… I thought that was impossible until I did it with a pile of hacks) but I didn’t even try the msys then, yikes, what a hassle.
But one of the decisions I made with my library is it is invalid to use the Terminal structure on a pipe. So I force you to branch outside instead of trying to transparently handle both. I kinda regret that since so many programs can benefit from the “just-works” aspect, but it is true you can often do much better UIs if the application just branches at the top level.
I struggled with this as well, and made a post about it which may be of interest: https://www.devever.net/~hl/win32con
Thanks! I like the SW_HIDE idea, though I went with a different approach. What I ended up doing is making a .bat file that calls the exe and just instructed the cli users to use that instead of the exe itself which worked cuz a bunch of cli users were used to a bat file anyway (it used to do
chcpand when I looked at it, I told them about WriteConsoleW and that chcp hack is terrible, but the batch file was still there so that meant I could reuse it here).Then the exe checks AttachConsole parent and reopens the stdin/out to it… as long as they were already null (otherwise they were previously redirected and don’t want to overwrite that).
Otherwise though the exe itself is the Windows subsystem so the double click and shortcut users just work without any flicker. Then the bat file causes the command shell to wait until the program finishes so the prompt and the output don’t stomp on each other.
The only thing kinda iffy to me is if the output is piped. For the console, WriteConsoleW with utf-16 just works. But for the pipe, I output utf-8 and it seems to be fine and the users happy with it, but I remain unsure what it should really do. I suspect Windows detects based on the first two bytes from some tests but I’m not fully happy with that part yet. Especially since you can pipe binary data too so idk.
I’ve never used windows so I’ll take your word for it >.<
Though, even if the fancy looking output was a better user experience (usually, I don’t think it is), switching output is also not particularly friendly; Now I need to think about two different formats when I use the tool.
Breaking pipelines is not an issue if you still have the coreutils implementations
stillaround. e.g: usebatandexafor regular use but when you need pipes usecatandls. I don’t see a problem with this and I think the concepts nushell (also mentioned in the article) brings to the table are definitely useful, albeit not implemented there for the first time.Edit: typo, crossed out extra word
of course there are the standard tools.
i just think it’s a bit weird to hype the console (look ma, no gui!) and at the same time ignore the concepts which make it so powerful. some of these tools are okish, but colorization and emulating guis are great at preventing the piping of output to further processing.
tools and pipelines together are bigger than their parts.
and i think that rust is overhyped, but that’s my problem ;)
If this were true, then we wouldn’t have used tty detection to change the output format to be compatible with pipelines.
Which, by the way, is exactly what
lshas been doing for years. Its default output format to a tty isn’t pipeline friendly. Maybe we’ve been learning from older tools more than you think.It can also be quite confusing to have the output change when you aren’t looking at it.
It’s just funny when I see people raging about this for the Rust tools (it seems to happen quite frequently), but nobody thinks to do it for any of the older tools. And it’s even better when people use this as a launching point to accuse us of ignorance. Give me a break.
I think you will find people (suckless type people) do complain about gnu coreutils doing stuff like this. I admit it is useful and nice lots of the time too.
BSD
lsdoes it too. Or at least, the BSDlscommand on my mac mini box does.Does there exist a version of
lsthat doesn’t do this?The coffee is still kicking in but I don’t think Plan9 ls does this.
Plan 9 ls doesn’t do anything different, and in fact, has no way to do it short of becoming a graphical program: there’s no concept of “terminal width”. You need to open the draw device and get the window size, look at the (possibly variable-width) font, and compute line widths.
There are also no control characters that handle cursor positioning, other than ‘\b’. The plan 9 console is a simple textual transcript. If you want graphics, use graphics – it doesn’t pop open a new window, so there’s no reason to support semigraphical features when full graphics are available.
Correct – Plan 9 doesn’t. The philosophy behind it is discussed in
cat -vConsidered Harmful by Pike and Kernighan (page 5).Though, to be technical, the Plan9 console isn’t really a tty, so Plan9
lswas designed for a slightly different environment. However, the design ofls(and the Plan 9 console) are rooted in this philosophy.I don’t think sbase ls does:
https://git.suckless.org/sbase/file/ls.c.html
OpenBSD and NetBSD’s ls do not have this flag (-G).
plan9 ls? ;)
it’s not that i dislike the functionality but how it’s implemented. i’d be interested in having such tools build in a composable way, out of multiple commands. default invokations then could be build via shellscript (and shipped together with the commands).
And do you use Plan9’s version of
lsto avoid the “nice” display output of the BSD and GNU versions ofls? If not, why do you use tools that ignore the concepts of composition?That’s surprising, and even re-reading your comments above, I don’t really see this coming across. You seem to be ranting against the functionality itself and how it breaks pipelining. But we handle those cases by disabling the “nice” functionality so that pipelining works.
Easier said than done. Everyone worships at the alter of perfect composition (including myself), but actually achieving it is another matter entirely. It seems pretty unfair to hold that against us. It’s not unreasonable to think that in certain environments, composition, development effort and the user experience are at odds with one another. (Where the development effort to make some of these nicer features truly compositional is probably on the order of “build a new kind of shell environment and convince everyone to use it,” which really isn’t feasible.) People actually building these tools have to pick a design point in this space, and we get rewarded with people calling us ignorant. Go figure.
i’ve tried to use the tools of plan9port by putting them in front of $PATH, unfortunatly the assumptions about the environment are sufficiently different from linux so that i’ve had other problems then.
i dislike the complexity such tools introduce by putting together unrelated things like colorization and git handling (sorry, bat is the first thing in the list..). i dislike magic as i’ve put so much time into debugging it when it breaks.
indeed.
i don’t know, you got rewarded with upvotes :) i’ve never called people ignorant, i’ve said that i find it weird to use an environment which lives by simplicity and then building things which combine several complicated tasks into one.
You said:
Emphasis mine. Sure looks like you’re calling us ignorant to me.
It’d just be nice to see folks be consistent. If you’re going to get all cranky about new tools introducing features that aren’t compositional, then at least get cranky about all the old tools that do it too. Because then, just maybe, it’s not just a bunch of overhyped new-fangled nonsense, and maybe there’s actually something to it. The giants we stand on also made choices that sacrifice composition.
i didn’t know that ignoring things, as in “conscious decision” is the same as being ignorant, but then, i’m no native speaker.
believe me, i’m cranky about most old (like the gnu stuff?) tools too. that’s why i’d like to see new tools which aren’t like the old ones.
the funny thing is that no one has thrown the old “then write it yourself” in my direction, which only would be fair.
I try to avoid saying things like that in discussions such as these. I think it’s overused as a way to dismiss others’ concerns. And of course, most people are already aware that such a thing is possible and don’t need to be reminded of it. I might as well just say nothing at all at that point.
I just find your particular form of criticism to be popular yet shallow. People love to jump all over these new fangled tools as if their authors are blubbering idiots that totally ignored the entire history of computing or are blissfully unaware of the Unix philosophy. It’s a lazy critique because violating composition is nothing new. We’re just doubling down, people don’t like it and for some reason like to insult the authors of these tools or at least denigrate their work. It’s annoying. (It’s a pretty common tactic used in lazy online banter, and is used in all manner of scenarios. I think we’d all be a lot better off if we held ourselves to a higher standard of discourse.)
Could you elaborate a bit more on breaking pipelines?
Let’s assume you were used to using
cat somefile | less(ignoring for a moment that you could justless somefile) and swapped in an alternate tool forcat, let’s saymycat. Nowmycathas a bunch of cool features like colorized output or wizbang progress bars (again, ignoring that many find colored output or wizbang features distasteful….). When just usingmycat somefileeverything looks great! But then you pipemycat somefile | lessand get all kinds of garbage, iflesseven works at all!This is one way to breaki pipelining. The reason it happens is
mycatis probably using a bunch of TTY control codes, or ANSI color codes to implement all those features. However, when that output is piped through toless, the pipe isn’t a TTY it’s just a raw text stream. So those control codes don’t get interpreted and instead just appear on the screen in raw form.mycatcan fix this by first checking if stdout is a TTY or not, and if so printing it’s control codes, but if not just printing raw text.There are other ways you can break pipelining, such as not handling a
SIGPIPEcorrectly (trying to write to pipe who’s reading end has been closed).Rust has facilities to handle these cases, and most common Rust applications handle these cases just fine. It is however a common footgun since if you’re implementing a new tool and not familiar with how all this traditionally works you may not know to check for these kinds of things.
Isn’t this problem solved in OpenBSD by rc.conf.local?
Nope. OpenBSD has both rc.conf.local and rc.d.
But rc.conf.local is the only configuration file an administrator touches, and it never conflicts with changes to the rc scripts under rc.d nor the default configuration found in rc.conf.
I’ve occasionally written my own rc.d scripts, but have never needed to modify an existing one provided by the base system or packages (any configuration I would want to do would be in the daemon flags in rc.conf.local).
I’m not sure where we disagree. You’ve just described the situation that led the author to suggest rc.d doesn’t belong in etc. The best case I can muster against that view is as you describe, that a local administrator might want to write a script and drop it in rc.d, but most will just use rc.local for that.
rc.conf.local exists becuase it’s generally not wise for local administrators to modify rc.d.
The crux of the issue as I understood it is that (on netbsd) rc.conf is both used to modify configuration, and is also touched by system upgrades, and therefore can create merge conflicts. With OpenBSD splitting out this local configuration into a separate file only touched manually by the user or with rcctl, there are never merge conflicts here, or to any of the rc infrastructure (unless you changed a system/package rc.d script I suppose).
That may be an issue; I don’t think it’s germane to the OP at all.
I’ve taken a stance of not running any AGPL software, period, and especially not any software that might be usable via the network (triggering the clauses that make the AGPL unique vs the GPL). My reasoning is simple: I’m lazy. I may patch a piece of software to fit my needs, but by doing so, I may patch in secrets or details about my infrastructure that I don’t want public. I’m not keeping the patch proprietary to turn it a business model or anything that goes against the spirit of the GPL; I just don’t find some patches particularly valuable or appropriate for public consumption. The GPL doesn’t cross this line, because it is easy to not redistribute my modifications in binary form (why would I with these patches?), but it can be potentially disastrous to flirt with these modifications in AGPL software.
If I wasn’t so lazy, I may patch the software “correctly”, introducing configuration options which don’t hardcode pieces of my own infrastructure, but I find this unreasonably burdensome for a so-called free software license, and it slows down proof-of-concept patches that I might otherwise want to deploy to canary instances to check for faults that integration and unit tests may not cover.
I don’t have an opinion on your overall argument. Just: ‘unreasonably burdensome for a so-called free software license’. This does not make sense to me, nobody ever said free software licenses are there to reduce burdens. That most of the permissive licenses actually do that is more a side-effect than anything else, and misses the point, IMHO.
Hmm, to me this looks like the Smallest function would only accept the built-in types with a less-than operator
<. Don’t you lose a whole lot of usefulness if you’re not allowed to write your own Ordered type which can be used with a Smallest function?Yes.
People will forced to decide whether their ordering-based data structures (TreeMaps, RB-Trees, …) will use
Orderedand therefore only work on a small set of “blessed” types, or whether they define their own, which … well … just imagine how great it will be to have two dozen replacement types forOrdered-for-my-own-types.(All modulo embedding, as far as I remember.)
This is not true. Using the
typesyntax with builtin types allows for operator usage on all types (both builtin and user-defined) where the either the builtin type appears in the constraint list, or a custom type with the same underlying type.https://go2goplay.golang.org/p/59oHxvIEp0A
This doesn’t work for struct types, but Go never had operating overloading in the first place, so no struct types would implement these operators anyways.
Yeah, but soc’s point stands: if you’re building an ordering-based function like
Smallest()or data structure like a binary tree, you’ll either:Orderedconstraint and use<, but then your function or tree will only be able to use built-in ordered types (or types with an underlying type that’s a built-in ordered type). That’ll work great forTree(int)orTree(MyInt), but won’t work forTree(MyStruct).Lessmethod, and you won’t be able to use the built-in types likeTree(int)at all … you’d be required to define aMyInttype that implementsLess, and convert everything toMyIntbefore putting it into the tree.Option #2 is definitely more flexible as it least allows any type to go into the map, but with the conversions it’ll be more boilerplate than you want for simple types.
The latest generics draft handles this by passing in a “compare function”, for example see
Mapin the Containers section. That kind of skirts the constraint/interface issue … but maybe that’s okay?Here’s a Go2Go Playground example that shows what I’m talking about: https://go2goplay.golang.org/p/Rbs374BqPWw
Exactly as I said:
I see. I misunderstood what you meant there because that is not the correct Go terminology. Embedding is only possible when defining struct and interface types. In the
type newtype oldtypedefinition, oldtype is referred to as the “base type” and doesn’t inherit any of its methods, but does gain its operators.Go if it doesn’t have to execute in a browser and the runtime is not a hindrance (and it usually isn’t).
C, C++, or Rust if a more minimal runtime is needed. Which is/are picked likely depends on what libraries are available and needed.
C and C++ are non-starters if the application must talk to the internet. I consider it malpractice to write internet-facing software in languages with manual memory management, given how hard we know it is to do correctly. (edit: I’m referring to building new software from scratch, not maintaining the existing software we rely on)
I’d find something else to work on if the problem lies outside of these domains.
C++ has automatic memory management with smart pointers. In fact, most of the languages that are garbage collected behind the scenes have runtimes written in C++ to manage memory.
Is it malpractice to be using operating systems, language runtimes, and other software mostly written by C and C++?
The real malpractice is proprietary software if anything. Or the fact that we have allowed private companies to install backdoors in all our computer systems.
See the edit (I assume you replied before seeing it).
C++ smart pointers still have safety issues. They do not prevent use-after-free bugs. They do not prevent data races (admittedly, Go programs can have data races as well, with all of the same safety concerns). And they don’t do anything about the other large uncharted areas of the C and C++ standards labeled Undefined Behavior.
It’s important to minimize the existence of this unsafe code if we care about secure systems, particularly for internet-facing applications. Some higher-level languages that promise memory and type safety do have runtimes built in unsafe languages, or use the unsafe features of their host language, and failure to keep invariants has caused real issues with these runtimes and code generation in the past. However, these are much smaller and relatively easy to audit for correctness, than auditing an entire ecosystem of C code.
I guess it depends on who we are talking about when we say “we care about secure system”.
If we are talking about regular people: I would argue social engineering (phishing) and backdoors from government agencies via corporations are a bigger concern than random hackers on the internet.
If we are talking about tech employees: I would argue social engineering and backdoors from government agencies are a bigger concern than random hackers on the internet.
Doesn’t matter how safe your system is memory-wise if it is weakly engineered from a human perspective.
I watched a great presentation that showed how to replicate the FastPass UI within the browser at a pixel perfect level. FastPass can be written in the most secure language, but it’s UI is flawed.
I’m not sure I agree with you. A chain of zero-days could have you pnwed just by visiting a compromised website—certainly something we want to avoid. However, I’m not convinced that Rust is the solution to all our security woes. When Redox OS asked for people to test it, they found all sorts of security bugs. Maybe writing it in Rust minimized some of them, maybe not.
I don’t think backdoors from government agencies are nearly as big of an issue as zero-days stocked up by government agencies.
It’s much easier for said agencies and companies to simply add the backdoor instead of trying to find zero-days don’t you think?
You don’t have to like or use C or C++. But to describe using them as malpractice is just offensive trolling.
If that came across as trolling, I didn’t mean it that way. This was a post asking for opinions, and that is my honest opinion. I’m also not completely against the use of C and C++, but you must consider attack surface, and the internet is an extremely large one.
Speaking specifically about the
flaganomaly, I much prefer Go’s package because it removes ambiguity. In agetopt(1)program, when I see-abc, this could mean two separate things: either there are three flags-a -b -cwritten in a shortened form, or there is a single flag-awhich takes a string argumentbc. Go’sflagremoves this ambiguity entirely.It doesn’t, because you as an end user don’t know if the program is written in Go, or if the author wrote their own argument handling because they didn’t like the flag package.
Still, clarity is something to strive towards as we develop software.
There are other reasons I prefer
flagas well. For example, it is possible to provide boolean flags with true defaults, and to disable them with-name=false. This is in contrast togetopt(1)-style flag handling where individual character flags can only be toggled on.I personally am a fan of having both — short and long options — around.
I use the short options when typing in a shell, because it’s convenient. I also usually bunch together multiple options, e.g.
ls -ltr, just because of typing efficiency. (I also typepython3 -mvenv envwithout space between the option and the argument, sue me!) For shell scripts on the other hand long options might make code more readable/self-explanatory, especially if co-workers also have to maintain it.That’s why I like having the single-dash prefix reserved for short options and double-dash for long options, because both variants have their place.
Interesting find! So the service in question was not only running as root, but the webserver/app wasn’t even chrooted. Reminds me a bit of the Apache phf exploit from 1996. People, always wear your seatbelt!
Just as an aside, and I know this article isn’t about any particular language, but Go makes it trivially easy to get a webserver running - but it’s not chrooted and runs as root. Would be nice if it were trivially easy to get a webserver running with basic defenses.
You can listen on high ports as an unprivileged user and redirect traffic at the firewall (I do this for my Go servers in production).
If it was dev you’d be at a port above 1024, so a regular user would run it. And if it was prod, wouldn’t you drop privileges after start via your service manager?
Or were you just saying how easy it is to just
sudo ./web-server…or use
authbindand not worry about privileges at all.One of the downsides of chroot is that it’s not implemented on Windows, so it’ll make your webserver not cross-platform. I’m not sure what alternatives there are for Windows?
Docker containers usually run their local system as root, and most compiler services I know of run in Docker containers or something similar. Eventually we’ll get to a capability-based reality where you have to explicitly grant permissions rather than explicitly taking them away, but we’re not quite there yet.
There seems to be a belief amongst memory safety advocates that it is not one out of many ways in which software can fail, but the most critical ones in existance today, and that, if programmers can’t be convinced to switch languages, maybe management can be made to force them.
I didn’t see this kind of zeal when (for example) PHP software fell pray to SQL injections left and right, but I’m trying to understand it. The quoted statistics about found vulnerabilities seem unconvincing, and are just as likely to indicate that static analysis tools have made these kind of programming errors easy to find in existing codebases.
Not all vulnerabilities are equal. I prioritize those that give attackers full control over my computer. They’re the worst. They can lead to every other problem. Plus, their rootkits or damage might not let you have it back. You can lose the physical property, too. Alex’s field evidence shows memory unsafety causes around 70-80% of this. So, worrying about hackers hitting native code, it’s rational to spend 70-80% of one’s effort eliminating memory unsafety.
More damning is that languages such as Go and D make it easy to write high-performance, maintainable code that’s also memory safe. Go is easier to learn with a huge ecosystem behind it, too. Ancient Java being 10-15x slower than C++ made for a good reason not to use it. Now, most apps are bloated/slow, the market uses them anyway, some safe languages are really lean/fast, using them brings those advantages, and so there’s little reason left for memory-unsafe languages. Even in intended use cases, one can often use a mix of memory-safe and -unsafe languages with unsafe used on performance-sensitive or lowest-level parts of the system. Moreover, safer languages such as Ada and Rust give you guarantees by default on much of that code allowing you to selectively turn them off only where necessary.
If using unsafe languages and having money, there’s also tools that automatically eliminate most of the memory unsafety bugs. That companies pulling in 8-9 digits still have piles of them show total negligence. Same with those in open-source development who aren’t doing much better. So, on that side of things, whatever tool you encourage should lead to memory safety even with apathetic, incompetent, or rushed developers working on code with complex interactions. Double true if it’s multi-threaded and/or distributed. Safe, orderly-by-default setup will prevent loads of inevitable problems.
If studies by security teams at Microsoft and Google, and analysis of Apple’s software is not enough for you, then I don’t know what else could convince you.
These companies have huge incentives to prevent exploitable vulnerabilities in their software. They get the best developers they can, they are pouring many millions of dollars into preventing these kinds of bugs, and still regularly ship software with vulnerabilities caused by memory unsafety.
“Why bother with one class of bugs, if another class of bugs exists too” position is not conductive to writing secure software.
No - but neither is pretending that you can eliminate a whole class of bugs for free. Memory safe languages are free of bugs caused by memory unsafety - but at what cost?
What other classes of bugs do they make more likely? What is the development cost? Or the runtime performance cost?
I don’t claim to have the answers but a study that did is the sort of thing that would convince me. Do you know of any published research like this?
The principle cost of memory safety in Rust, IMO, is that the set of valid programs is more heavily constrained. You often here this manifest as “fighting with the borrow checker.” This is definitely an impediment. I think a large portion of folks get past this stage, in the sense that “fighting the borrow checker” is, for the most part, a temporary hurdle. But there are undoubtedly certain classes of programs that Rust will make harder to write, even for Rust experts.
Like all trade offs, the hope is that the juice is worth the squeeze. That’s why there has been a lot of effort in making Rust easier to use, and a lot of effort put into returning good error messages.
I’ve seen people ask this before, and my response is always, “what hypothetical study would actually convince you?” If you think about it, it is startlingly difficult to do such a study. There are many variables to control for, and I don’t see how to control for all of them.
IMO, the most effective way to show this is probably to reason about vulnerabilities due to memory safety in aggregate. But to do that, you need a large corpus of software written in Rust that is also widely used. But even this methodology is not without its flaws.
That’s true - but my comment was in response to one claiming that the bug surveys published by Microsoft et al should be convincing.
I could imagine something similar being done with large Rust code bases in a few years, perhaps.
I don’t have enough Rust experience to have a good intuition on this so the following is just an example. I have lots of C++ experience with large code bases that have been maintained over many years by large teams. I believe that C++ makes it harder to write correct software: not (just) because of memory safety issues, undefined behavior etc. but also because the language is so large, complex and surprising. It is possible to write good C++ but it is hard to maintain it over time. For that reason, I have usually promoted C rather than C++ where there has been a choice.
That was a bit long-winded but the point I was trying to make is that languages can encourage or discourage different classes of bugs. C and C++ have the same memory safety and undefined behavior issues but one is more likely than the other to engender other bugs.
It is possible that Rust is like C++, i.e. that its complexity encourages other bugs even as its borrow checker prevents memory safety bugs. (I am not now saying that is true, just raising the possibility.)
This sort of consideration does not seem to come up very often when people claim that Rust is obviously better than C for operating systems, for example. I would love to read an article that takes this sort of thing into account - written by someone with more relevant experience than me!
I’ve been writing Rust for over 4 years (after more than a decade of C), and in my experience:
NULLand mandatory error handling are amazing for reliability.The costs:
Of course older software, mostly written in memory-unsafe languages, sometimes written in a time when not every device was connected to a network, contains more known memory vulnerabilities. Especially when it’s maintained and audited by companies with excellent security teams.
These statistics don’t say much at all about the overall state of our software landscape. It doesn’t say anything about the relative quality of memory-unsafe codebases versus memory-safe codebases. It also doesn’t say anything about the relative sizes of memory-safe and memory-unsafe codebases on the internet.
iOS and Android aren’t “older software”. They’ve been born to be networked, and supposedly secure, from the start.
Memory-safe codebases have 0% memory-unsafety vulnerabilities, so that is easily comparable. For example, check out the CVE database. Even within one project — Android — you can easily see whether the C or the Java layers are responsible for the vulnerabilities (spoiler: it’s C, by far). There’s a ton of data on all of this.
Android is largely cobbled together from older software, as is IOS. I think Android still needs a Fortran compiler to build some dependencies.
That starts to look like a No True Scotsman. When real-world C codebases have vulnerabilities, they’re somehow not proper C codebases. Even when they’re part of flagship products of top software companies.
I’m actually not arguing that good programmers are able to write memory-safe code in unsafe languages. I’m arguing vulnerabilities happen at all levels in programming, and that, while memory safety bugs are terrible, there are common classes of bugs in more widely used (and more importantly, more widely deployed languages), that make it just one class of bugs out of many.
When XSS attacks became common, we didn’t implore VPs to abandon Javascript.
We’d have reached some sort of conclusion earlier if you’d argued with the point I was making rather than with the point you wanted me to make.
Actually did. Sites/companies that solved XSS did so by banning generation of markup “by hand”, and instead mandated use of safe-by-default template engines (e.g. JSX). Same with SQL injection: years of saying “be careful, remember to escape” didn’t work, and “always use prepared statements” worked.
These classes of bugs are prevalent only where developers think they’re not a problem (e.g. they’ve been always writing pure PHP, and will continue to write pure PHP forever, because there’s nothing wrong with it, apart from the XSS and SQLi, which are a force of nature and can’t be avoided).
This kind of makes me think of someone hearing others talk about trying to lower the murder rate and then hysterically going into a rant about how murder is only one class of crime
I think a better analogy is campaigning aggressively to ban automatic rifles when the vast majority of murders are committed using handguns.
Yes, automatic rifles are terrible. But pointing them out as the main culprit behind the high murder rate is also incorrect.
That analogy is really terrible and absolutely not fitting the context here. It’s also very skewed, the murder rate is not the reason for calls for bans.
Although I mostly agree, I’ll note Android was originally built by a small business acquired by Google that continued to work on it probably with extra resources from Google. That makes me picture a move fast and break things kind of operation that was probably throwing pre-existing stuff together with their own as quickly as possible to get the job done (aka working phones, market share).
Yes, if you zoom in on code bases written in memory-unsafe languages, you unsurprisingly get a large number of memory-unsafety vulnerabilities.
And that’s exactly what illustrates “eliminates a class of bugs”. We’re not saying that we’ll end up in utopia. We just don’t need that class of bugs anymore.
Correct, but the author is arguing that this is an exceptionally grievous class of security bugs, and (in another article) that developers’ judgement should not be trusted on this matter.
Today, the vast majority of new code is written for a platform where execution of untrusted memory-safe code is a core feature, and the safety of that platform relies on a stack of sandboxes written mostly in C++ (browser) and Objective C/C++/C (system libraries and kernel)
Replacing that stack completely is going to be a multi-decade effort, and the biggest players in the industry are just starting to dip their toes in memory-safe languages.
What purpose does it serve to talk about this problem as if it were an urgent crisis?
Hm, so. Apple has developed Swift, which is generally considered a systems programming language, to replace Objective-C, which was their main programming language and already had safety features like baked in ARC. Google has implemented Go. Mozilla Rust. Google uses tons of Rust in Fuchsia and has recently imported the Rust compiler into the Android source tree.
Microsoft has recently been blogging about Rust quite a lot and is often seen hanging around and blogs about how severe memory problems are to their safety story. Before that, Microsoft has spent tons of engineering effort into Haskell as a research base and C#/.Net as a replacement for their C/C++ APIs.
Amazon has implemented firecracker in Rust and bragged about it on their AWS keynote.
Come again about “dipping toes”? Yes, there’s huge amounts of stack around, but there’s also huge amounts to be written!
Because it’s always been a crisis and now we have the tech to fix it.
P.S.: In case this felt a bit like bragging Rust over the others: it’s just where I’m most aware of things happening. Go and Swift are doing fine, I just don’t follow as much.
The same argument was made for Java, which on top of its memory safety, was presented as a pry bar against the nearly complete market dominance of the Wintel platform at the time. Java evangelism managed to convert new programmers - and universities - to Java, but not the entire world.
Oracle’s deadly embrace of Java didn’t move it to rewrite its main cash cow in Java.
Rust evangelists should ask themselves why.
I think that of all the memory-safe languages, Microsoft’s C++/CLI effort comes closest to understanding what needs to be done to entice coders to move their software into a memory-safe environment.
At my day job, I actually try to spend my discretionary time trying to move our existing codebase to a memory-safe language. It’s mostly about moving the pieces into place so that green-field software can seamlessly communicate with our existing infrastructure. Then seeing what parts of our networking code can be replaced, slowly reinforcing the outer layers while the inner core remains memory unsafe.
Delicate stuff, not something you want the VP of Engineering to issue edicts about. In the meantime, I’m still a C++ programmer, and I really don’t appreciate this kind of article painting a big target on my back.
Java and Rust are vastly different ball parks for what you describe. And yet, Java is used successfully in the database world, so it is definitely to be considered. The whole search engine database world is full of Java stacks.
Oracle didn’t rewrite its cashcow, because - yes, they are risk-averse and that’s reasonable. That’s no statement on the tech they write it in. But they did write tons of Java stacks around Oracle DB.
It’s an argument on the level of “Why isn’t everything at Google Go now?” or “Why isn’t Apple using Swift for everything?”.
Looking at https://news.ycombinator.com/item?id=18442941 it seems that it was too late for a rewrite when Java matured.
To start the multi-decade effort now, and not spend more decades just saying that buffer overflows are fine, or that—despite of 40 years of evidence to the contrary—programmers can just avoid causing them.
You didn’t? SQL injections are still #1 in the OWASP top 10. PHP had to retrain an entire generation of engineers to use mysql_real_escape_string over vulnerable alternatives. I could go on…
I think we have internalized arguments the SQL injection but have still not accepted memory safety arguments.
I remember arguments being presented to other programmers. This article (and another one I remembered, which, as it turns out, is written by the same author: https://www.vice.com/en_us/article/a3mgxb/the-internet-has-a-huge-cc-problem-and-developers-dont-want-to-deal-with-it ) explicitly target the layperson.
The articles use the language of whistleblowers. It suggests that counter-arguments are made in bad faith, that developers are trying to hide this ‘dirty secret’. Consider that C/C++ programmers skew older, have less rosy employment prospects, and that this article feeds nicely into the ageist prejudices already present in our industry.
Arguments aimed at programmers, like this one at least acknowledge the counter-arguments, and frame the discussion as one of industry maturity, which I think is correct.
I do not see it as bad faith. There are a non-zero number of people who say they can write memory safe C++ despite there being a massive amount of evidence that even the best programmers get tripped up by UB and threads.
There’s an argument to be made that the resurging interest in systems programming languages through Rust, Swift and Go futureproofs experience in those areas.
Memory safety advocate here. It is the most pressing issue because it invokes undefined behavior. At that point, your program is entirely meaningless and might do anything. Security issues can still be introduced without memory unsafety of course, but you can at least reason about them, determine the scope of impact, etc.
ugh. https://github.com/bdmac/strong_password/blob/master/CHANGELOG#L1
The CHANGELOG doesn’t mention this rubygems incident and it ALSO breaks BC. Maybe I’m overly pessimistic and paranoid but I’d have republished 0.0.6 unchanged as 0.0.8 and released anything new as 0.1.0.
Exactly. If you’re doing breaking changes, you shouldn’t increment the patch version…
Pre-1.0.0 releases in semantic versioning have no defined compatibility requirements with any other version.
Though, if this module is being used in production, a 1.0.0 release should be cut. Even more so since this is open source and you don’t know all of the consumers.
e: sp
fair, that’s a good point.
With a 0.0.x I wouldn’t call that a problem, really. It just irks me that I suppose a lot of people might think not to upgrade from 0.0.7 although they should. Or would
gemwarn you that the installed version was yanked?I am fine if people want to make money with my software, as long as whatever they add on to my software is also open source.
People are willing to pay for software, even open source, as getting from the source code to a finished binary can be a pain. I want an open source license that that doesnt allow people to use my stuff as part of a closed source project, unless they pay me.
I dont know if such a thing exists, so for now I am using this:
https://choosealicense.com/licenses/osl-3.0
You don’t need a single license for this. As long as you maintain sole copyright, you can license the software under any additional license someone is willing to pay for. That license applies to their copy, while the open source (likely copyleft) license applies to everyone else who receives the license without pay.