1.  

    A small Ryzen PC, I had most of the parts lying around anyway so just needed the CPU+mobo. Originally it was a mini-ITX Intel Atom board. It’s steadily grown in storage capacity since being made, since I use it mainly for file hosting and backups.

    1.  

      Doesn’t work for me on x86_64 Debian linux. Just outputs:

      Failed to execute process './redbean-2021-02-25.com'. Reason:
      exec: Exec format error
      The file './redbean-2021-02-25.com' is marked as an executable but could not be run by the operating system.
      

      Looks like this is because I use fish as my default shell, which doesn’t do the (slightly insane) Bourne shell thing of “dunno what this file is, let’s just try executing it as a shell script”. Running the thing using Bash works.

      Also, despite the immense cleverness, I’m not sure that writing tools using self-modifying executable code is something we should encourage even as a joke.

      1. 5

        I’m not sure that writing tools using self-modifying executable code is something we should encourage even as a joke.

        Honestly, I’m afraid in my eyes the “even as a joke” phrase here changes a reasonable and objective question into something that sounds uncharitably and completely unnecessarily condescending, and makes me sorry for the author to have to hear that. I know by myself that it’s sometimes hard to resist smug quips, yet personally I find lobste.rs to be teaching me how to strive for sky high civility, and love the community for that.

        As to the actual matter at hand, personally I assume it by default to be the case of a practical approach I see as required in programming: if we can, we do stuff pretty way; but if we can’t, if the ugly way is the only way, buckling up and getting the hands dirty is IMO a much needed display of skill and maturity as well. Then to add to that, trailblazing a new path is often an even dirtier job. Sure, when it’s done, I welcome the cleaners to come and search for what can be improved and perfected. I totally like to do that myself at various times. But in a “hackathon mode” of PoC-ing, my explicit hard rule is to push forward like crazy and don’t look back - the only exception being when the mess distracts my own thoughts too much, stopping my forward motion.

        Based on that - if someone finds a way to achieve the APE goal with no self-modifying code, that would for sure be an interesting improvement, esp. from a W^X security perspective I guess. But if that’s the only way to do that, I find it a perfectly fine solution - another tool in the toolbelt (and one of the most beautiful ones this), with some specific set of compromises as each one of them is. Edit: yeah, and and as I actually suspected, the author doesn’t especially love this approach either, but that’s the only way she could find to do that.

        Uhh… sorry for the rant, I just find the job and the author so amazing, I really felt the need to put down a strong voice of support that I’m backing with all my self.

        1.  

          Also, despite the immense cleverness, I’m not sure that writing tools using self-modifying executable code is something we should encourage even as a joke.

          Could you elaborate?

          1.  

            On Linux, as far as I can tell, the executable runs with bash on the first run, alters its own header to be a valid ELF file on the first execution, and then executes itself.

            Any security or robustness issues aside, it means that after running it, you can’t pick up the executable and then give it to someone else and have it have the same properties of “you can run it on anywhere”. After running on Linux, it is no longer a universal executable, it is a Linux executable.

            I’m sorry for coming off as more critical than I really intended, but I was quite unpleasantly surprised to discover that you have to run it in bash or sh (doing /bin/bash redbean doesn’t work for some reason), that file reports a different file type for the thing before and after running, and that it doesn’t work properly when set read-only or on a read-only filesystem. The executable format is extremely clever, but apparently (currently) comes with a lot of hidden gotcha’s, and I hate hidden gotcha’s. Whether or not they’re gotchas one cares about is up to the user.

            1.  

              I’m sorry for coming off as more critical than I really intended…

              I think this might be more directed at jart and/or akavel than me. For my part, I sometimes see people on the cusp of saying something interesting, but they stop for whatever reason. So I wasn’t trying to criticize you; I was just trying to get you to say whatever seemed to be on your mind. :]

              Any security or robustness issues aside, it means that after running it, you can’t pick up the executable and then give it to someone else and have it have the same properties of “you can run it on anywhere”. After running on Linux, it is no longer a universal executable, it is a Linux executable.

              I hadn’t actually tried to run this. It’s still a very neat feature, but maybe the name is a bit misleading. (Though Actually Portable (Once) Executable is probably not as catchy.)

          2.  

            I upstreamed a patch with zsh. Looks like I need to upstream a patch with fish too. https://github.com/zsh-users/zsh/commit/326d9c203b3980c0f841bc62b06e37134c6e51ea

            1.  

              I’m not sure that writing tools using self-modifying executable code is something we should encourage even as a joke.

              Why deny ourselves some of the most interesting bits of programming? I suppose you aren’t a fan of currying or lambdas or partial application either then, eh?

              1.  

                I suppose you aren’t a fan of currying or lambdas or partial application either then, eh?

                I was assuming it was a security issue more than an issue of expressivity. That said, it’s not totally clear what icefox is worried about.

            1. 21

              I mostly agree with this, but it’s an interesting culture shift. Back when I started to become involved, open source was largely maintained by poor people. Niche architectures were popular because a student could get a decent SPARC or Alpha machine that a university department was throwing away. It wasn’t as fast as a decent x86 machine but it was a lot faster than an x86 machine you could actually afford. The best-supported hardware was the hardware that was cheap for developers to buy.

              Over the last couple of decades, there’s been a shift towards people being paid to maintain open source things and a big split between the projects with big corporate-backed foundations behind them and the ones that are still maintained by volunteers. The Z-series experience from the article highlights this split. There’s a big difference in perspective on this kind of thing is very different if you’re a volunteer who has been lovingly supporting Alpha for years versus someone paid by IBM to maintain Z-series support.

              Even among volunteers there’s sometimes a tension. I remember a big round of applause for Marcel at BSDCan some years ago when he agreed to drop support for Itanium. He’s been maintaining it almost single-handed and there were a bunch of things that were Itanium-only and could be cleaned up once Itanium was removed. It’s difficult to ask a volunteer to stop working on a thing because it’s requiring other volunteers to invest more effort.

              1. 9

                Speaking from experience: PowerPC is in a bit of an interesting state regarding that, with varying factions:

                • IBM selling Power systems to legacy-free Linux customers; they only support and would prefer you use 64-bit little endian
                • IBM also has to support AIX and i, which are 64-bit big endian operating systems
                • Embedded network equipment on 32-bit or crippled 64-bit cores
                • Power Mac hobbyists with mostly 32-bit and some 64-bit equipment
                • Talos users, using their fast owner controlled POWER9 systems which are mostly little endian Linux users, some big endian

                Sometimes drama happens because the “sell PPC Linux” people only care about ppc64le and don’t do any of the work (or worse, actively discontinue support) for 32-bit/BE/older cores (i.e Go dropped all non-POWER8/9 support, which is like only support Haswell or newer), which pisses off not just hobbyists, but also the people who make deep embedded networking equipment or supporting AIX.

                1.  

                  I am duty bound to link to https://github.com/rust-lang/rust/issues/59932. Rust currently sets wrong baseline for embedded network equipment PowerPC systems.

                  1.  

                    I resemble that remark, having a long history with old POWER, Power Macs, a personal POWER6 and of course several POWER9 systems. It certainly is harder to do development with all those variables, and the 4K/64K page size schism in Linux is starting to get worse as well. I think it’s good discipline to be conscious of these differences but I don’t dispute it takes up developer time. It certainly does for me even with my small modest side projects.

                  2. 6

                    The best-supported hardware was the hardware that was cheap for developers to buy.

                    That fact contributes to the shift away from non mainstream ISAs, I suppose? In this decade the hardware that is cheapest for developers to buy is the most mainstream stuff.

                    Nobody buys and discards large amounts of POWER or SPARC or anything. If you are a freegan and want to acquire computers only out of dumpster(s), what I believe you will find is going to be amd64, terrible Android phones and obsolete raspis? Maybe not even x86 - it’s on the cusp of being too old for organisations to be throwing it away in bulk. e.g. the core2duo was discontinued in 2012 so anyone throwing that away would be doing so on like an 8 year depreciation cycle.

                    Just doing a quick skim of eBay, the only cheap non-mainstream stuff I can see is old g4 mac laptops going for about the price of a new raspi. Some of the PPC macs are being priced like expensive antiques rather than cheap discards. It looks like raspis are about as fast anyway. https://forums.macrumors.com/threads/raspberry-pi-vs-power-mac-g5.2111057/

                    It’s difficult to ask a volunteer to stop working on a thing because it’s requiring other volunteers to invest more effort.

                    Ouuuch :/

                    1.  

                      Nobody buys and discards large amounts of POWER or SPARC or anything.

                      You can (because businesses do buy them), but they’re be very heavyweight 4U boxes at minimum. The kind that nerds like me would buy and run. They’re cheapish (like, $200 if you get a good deal) for old boxes, but I doubt anyone in poverty is getting one.

                      You’d also be surprised what orgs ewaste after a strict 5-10 year life cycle. I hear Haswell and adjacent business desktops/laptops are getting pretty damn cheap.

                      1.  

                        I got burned trying to get Neat Second Hand Hardware once in the late oughts. Giant UltraSPARC workstation, $40. Turns out that the appropriate power supply, Fiber Channel hard drives, plus whatever the heck it took to connect a monitor to the thing… high hundreds of dollars. I think I ended up giving it to a friend who may or may not have strictly wanted it, still in non-working condition.

                        1.  

                          They’re cheapish (like, $200 if you get a good deal) for old boxes

                          I was seeing prices about 2x higher than that skimming eBay. I also see very few of them - it’s not like infiniband NICs where there are just stacks and stacks of them. Either way, that’s much more money than trash amd64.

                          You’d also be surprised what orgs ewaste after a strict 5-10 year life cycle. I hear Haswell and adjacent business desktops/laptops are getting pretty damn cheap.

                          Sure. I was bringing up Core2Duo as a worst-case for secondhand Intel parts that someone would probably throw in a dumpster.

                          1.  

                            Sure. I was bringing up Core2Duo as a worst-case for secondhand Intel parts that someone would probably throw in a dumpster.

                            Yeah, absolutely. It freaks me out, but a Core 2 Duo is about a decade and a half old. It’s about as old as a 486 was in 2006. These are gutter ewaste, but they’re still pretty useful and probably the real poverty baseline. Anything older is us nerds.

                    1. 7

                      There’s some value in supporting odd platforms, because it excercises portability of programs, like the endianness issue mentioned in the post. I’m sad that the endian wars were won by the wrong endian.

                      1.  

                        I’m way more happy about the fact that the endian wars are over. I agree it’s a little sad that it is LE that won, just because BE is easier to read when you see it in a hex dump.

                        1.  

                          Big Endian is easy for us only because we ended up with some weird legacy of using Arabic (right-to-left) numbers in Latin (left-to-write) text. Arabic numbers in Arabic text are least-significant-digit first. There are some tasks in computing that are easier on little-endian values, none that are easier on big-endian, so I’m very happy that LE won.

                          If you want to know the low byte of a little-endian number, you read the first byte. If you want to know the top byte of a little-endian number, you need to know its width. The converse is true of a big-endian number, but if you want to know the top byte of any number and do anything useful with it then you generally do know its width because otherwise ‘top’ doesn’t mean anything meaningful.

                          1.  

                            Likewise, there are some fun bugs only big endian can expose, like accessing a field with the wrong size. On little endian it’s likely to work with small values, but BE would always break.

                        2.  

                          Apart from “network byte order” looking more intuitive to me at first sight, could you eloborate why big endian is better than little endian? I’m genuinely curious (and hope this won’t escalate ;)).

                          1. 10

                            My favorite property of big-endian is that lexicographically sorting encoded integers preserves the ordering of the numbers itself. This can be useful in binary formats. Since you have to use big-endian to get this property, a big-endian system doesn’t need to do byte swapping before using the bytes as an integer.

                            Also, given that we write numbers with the most significant digits first, it just makes more “sense” to me personally.

                            1. 5

                              Also, given that we write numbers with the most significant digits first, it just makes more “sense” to me personally.

                              A random fact I love: Arabic text is right-to-left, but writes its numbers with the same ordering of digits as Latin texts… so in Arabic, numbers are little-endian.

                              1.  

                                Speaking of Endianness: In Arabic, relationships are described from the end farthest from you to the closest, as in if you were to naively describe the husband of a second cousin, instead of saying “my mother’s cousin’s daughter’s husband” you would say “the husband of the daughter of the cousin of my mother” and it makes it insanely hard to hold it in your head without a massive working memory (because you need to reverse it to actually grok the relationship) but I always wonder if it’s because I’m not the most fluent Arabic speaker or if it’s a problem for everyone that speaks it.

                                1.  

                                  My guess is that it is harder for native speakers as well, but they don’t notice it because they are used to it. A comparable case I can think of is a friend of mine who is a native German speaker who came to the States for a post-doc. He commented that after speaking English consistently for a while, he realized that German two digit numbers are needlessly complicated. Three and twenty is harder to keep in your head than twenty three for the same reason.

                                  1.  

                                    German has nothing to Danish.

                                    95 is “fem og halvfems” - “five and half-five”, where the final five refers to five twentys (100), and the “half” refers to half of 20, i.e. 10, giving 90.

                                    It’s logical once you get the hang of it…

                                    In Swedish it’s “nittiofem”.

                            2.  

                              Little-endian vs. big-endian has a good summary of the trade-offs.

                              1.  

                                I wondered this often and figured everyone just did the wrong thing, because BE seems obviously superior. Just today I’ve been reading RISC-V: An Overview of the Instruction Set Architecture and noted this comment on endianness:

                                Notice that with a little endian architecture, the first byte in memory always goes into the same bits in the register, regardless of whether the instruction is moving a byte, halfword, or word. This can result in a simplification of the circuitry.

                                It’s the first time I’ve noticed something positive about LE!

                                1.  

                                  From what I hear, it mostly impacted smaller/older devices with small buses. The impact isn’t as big nowadays.

                                2.  

                                  That was a bit of tongue-in-cheek, so I don’t really want to restart the debate :)

                                  1.  

                                    Whichever endianness you prefer, it is the wrong one. ;-)

                                    Jokes aside, my understanding is that either endianness makes certain types of circuits/components/wire protocols easier and others harder. It’s just a matter of optimizing for the use case the speaker cares about more.

                                  2.  

                                    Having debugged on big-endian for the longest time, I miss “sane” memory dumps on little-endian. It takes a bit more thought to parse them.

                                    But I started programming on the 6502, and little-endian clearly makes sense when you’re cascading operations 8 bits at a time. I had a little trouble transitioning to the big-endian 16-bit 9900 as a result.

                                  1. 19

                                    I love plain text protocols, but … HTTP is neither simple to implement nor neither fast to parse.

                                    1. 7

                                      Yeah the problem of parsing text-based protocols in an async style has been floating around my head for a number of years. (I prefer not to parse in the async or push style, but people need to do both, depending on the situation.)

                                      This was motivated by looking at the nginx and node.js HTTP parsers, which are both very low level C. Hand-coded state machines.


                                      I just went and looked, and this is the smelly and somewhat irresponsible code I remember:

                                      https://github.com/nodejs/http-parser/blob/master/http_parser.c#L507

                                      /* Proxied requests are followed by scheme of an absolute URI (alpha).

                                      • All methods except CONNECT are followed by ‘/’ or ‘*’.

                                      I say irresponsible because it’s network-facing code with tons of state and rare code paths, done in plain C. nginx has had vulnerabilities in the analogous code, and I’d be surprised if this code didn’t.


                                      Looks like they have a new library and admit as much:

                                      https://github.com/nodejs/llhttp

                                      Let’s face it, http_parser is practically unmaintainable. Even introduction of a single new method results in a significant code churn.

                                      Looks interesting and I will be watching the talk and seeing how it works!

                                      But really I do think there should be text-based protocols that are easy to parse in an async style (without necessarily using Go, where goroutines give you your stack back)

                                      Awhile back I did an experiment with netstrings, because length-prefixed protocols are easier to parse async than delimiter-based protocols (like HTTP and newlines). I may revisit that experiment, since Oil will likely grow netstrings: https://www.oilshell.org/release/0.8.7/doc/framing.html


                                      OK wow that new library uses a parser generator I hadn’t seen:

                                      https://llparse.org/

                                      https://github.com/nodejs/llparse

                                      which does seem like the right way to do it: do the inversion automatically, not manually.

                                      1. 4

                                        Was going to say this. Especially when you have people misbehaving around things like Content-Length, Transfer-Encoding: chunked and thus request smuggling seems to imply it’s too complex. Plus, I still don’t know which response code is appropriate for every occasion.

                                        1. 2

                                          Curious what part of HTTP you think is not simple? And on which side (client, server)

                                          1. 4

                                            There’s quite a bit. You can ignore most of it, but once you get to HTTP/1.1 where chunked-encoding is a thing, it starts getting way more complicated.

                                            • Status code 100 (continue + expect)
                                            • Status code 101 - essentially allowing hijacking of the underlying connection to use it as another protocol
                                            • Chunked transfer encoding
                                            • The request “method” can technically be an arbitrary string - protocols like webdav have added many more verbs than originally intended
                                            • Properly handling caching/CORS (these are more browser/client issues, but they’re still a part of the protocol)
                                            • Digest authentication
                                            • Redirect handling by clients
                                            • The Range header
                                            • The application/x-www-form-urlencoded format
                                            • HTTP 2.0 which is now a binary protocol
                                            • Some servers allow you specify keep-alive to leave a connection open to make more requests in the future
                                            • Some servers still serve different content based on the User-Agent header
                                            • The Accept header

                                            There’s more, but that’s what I’ve come up with just looking quickly.

                                            1. 3

                                              Would add to this that it’s not just complicated because all these features exist, it’s very complicated because buggy halfway implementations of them are common-to-ubiquitous in the wild and you’ll usually need to interoperate with them.

                                              1. 1

                                                And, as far as I know, there is no conformance test suite.

                                                1. 1

                                                  Ugh, yes. WPT should’ve existed 20 years ago.

                                              2.  

                                                Heh, don’t forget HTTP/1.1 Pipelining. Then there’s caching, and ETags.

                                            2. 2

                                              You make a valid point. I find it easy to read as a human being though which is also important when dealing with protocols.

                                              I’ve found a lot of web devs I’ve interviewed have no idea that HTTP is just plain text over TCP. When the lightbulb finally goes on for them a whole new world opens up.

                                              1. 4

                                                It’s interesting to note that while “original HTTP” was plain text over TCP, we’re heading toward a situation where HTTP is a binary protocol run over an encrypted connection and transmitted via UDP—and yet the semantics are still similar enough that you can “decode” back to something resembling HTTP/1.1.

                                                1. 1

                                                  UDP? I thought HTTP/2 was binary over TCP. But yes, TLS is a lot easier thanks to ACME cert issues and LetsEncrypt for sure.

                                                  1.  

                                                    HTTP/3 is binary over QUIC, which runs over UDP.

                                              2. 1

                                                SIP is another plain text protocol that is not simple to implement. I like it and it is very robust though. And it was originally modeled after HTTP.

                                              1. 3

                                                Frankly it looks less like a stack machine and more like a Forth machine. Has a separate data and return stack, an opcode to toggle reading literals, etc. When I read “stack machine” I usually think something a bit lower level and more general.

                                                1. 1

                                                  separate data and return stack

                                                  This is a good idea generally; if you don’t do this you tend to be vulnerable to ROP.

                                                1. 1

                                                  I confess that I do it whenever I get a new hard drive/do a new OS install and have to generate a new key to replace one lost. The slight downside to this is that I recently realized that my main terminal server still would, in theory, accept SSH logins from my account on a particular computer in a particular lab at my grad school.

                                                  1. 2

                                                    oracle tests only make sense if the oracle is more likely to be correct than what you’re testing, which implies the oracle is simpler than the function under test. But if it’s simpler and more correct, why not just use the oracle as your implementation?

                                                    Several more reasons:

                                                    • the oracle covers only part of the functionality. E.g. it always correctly predicts the first element of the resulting tuple, but not the others.
                                                    • the oracle doesn’t cover orthogonal requirements, such as security, logging, being written in a language the implementation maintainers can read, …
                                                    • the purpose of the oracle is to be a second implementation, written independently using methodology to avoid making the same mistakes implementers are likely to have made

                                                    This is related to my job, which is in ‘model based testing’. We provide our customers tooling and coach them in writing oracles, that are simultaneously readable, executable, specifications, and making it easy to run them against implementations. Sometimes it happens that the model replaces the implementation or is used temporarily until the implementation is done.

                                                    1. 1

                                                      …which implies the oracle is simpler than the function under test…

                                                      This seems like a very erroneous assumption to me. In my experience the “known-good previous implementation” of a piece of software is likely to be scarred, scuffed, battle-tested, missing a few parts, have poor design assumptions, and also has exactly one configuration that is known to be good under almost any circumstance.

                                                      1. 2

                                                        This seems like a very erroneous assumption to me.

                                                        This is covered in the article:

                                                        Sometimes there’s a good reason to not use a simpler oracle:

                                                        • The oracle is too slow
                                                        • The oracle is a reference implementation
                                                        • The oracle only works on a subset of expected inputs
                                                        • The oracle only covers the happy path
                                                        • The oracle is the old code you’re trying to refactor
                                                    1. 4

                                                      It is interesting mail is the only service to get its own explicit record type. Modern protocols like XMPP will use either SRV or TXT records.

                                                      1. 13

                                                        MX was basically the first SRV record type, and then after that they went “we should really make a general case tool for this”.

                                                        1. 3

                                                          It helps that mail predated DNS. DNS was standardised in 1983, SMTP (which wasn’t the first email protocol) was standardised in 1982.

                                                          1. 2

                                                            Yes, modern services don’t get their own DNS RR, but plenty services have: https://en.m.wikipedia.org/wiki/List_of_DNS_record_types

                                                          1. 32

                                                            Wow, this blog post is so lacking in empathy for users that I’m surprised it made it on a reputable distro’s blog. Instead of spilling 1000 words on why “static linking is bad”, maybe spend a little time thinking about why people (like me) and platforms (like go/rust et al) choose it. The reason people like it is that it actually works and it won’t suddenly stop working when you change the version of openssl in three months. It doesn’t even introduce security risks! The only difference is you have to rebuild everything on that new version, which seems like a small price to have software that works, not to mention that rebuilding everything will also re-run the tests on that new version. I can build a go program on nixos, ship it to any of my coworkers and it actually just works. We are on a random mix of recent ubuntu, centos 7 and centos 8 and it all just works together. That is absolutely not possible with dynamic linking

                                                            1. 19

                                                              It works well if all you care about is deploying your application. As a distro maintainer, I’m to keeping track of 500 programs, and having to care about vendored/bundled versions and statically linked in dependencies multiplies the work I have to do.

                                                              1. 24

                                                                But … that’s a choice you make for yourself? No application author is asking you to do that, and many application authors actively dislike that you’re doing that, and to be honest I think most users don’t care all that much either.

                                                                I’ve done plenty of packaging of FreeBSD ports back in the day, and I appreciate it can be kind of boring thankless “invisible” gruntwork and, at times, be frustrating. I really don’t want to devalue your work or sound thankless, but to be honest I feel that a lot of packagers are making their own lives much harder than it needs to be by sticking to a model that a large swath of the software development community has, after due consideration and weighing all the involved trade-offs, rejected and moved away from.

                                                                Both Go and Rust – two communities with pretty different approaches to software development – independently decided to prefer static linking. There are reason for that.

                                                                Could there be some improvements in tooling? Absolutely! But static linking and version pinning aren’t going away. If all the time and effort spent on packagers splitting things up would be spent on improving the tooling, then we’d be in a much better situation now.

                                                                1. 13

                                                                  …but to be honest I feel that a lot of packagers are making their own lives much harder than it needs to be by sticking to a model that a large swath of the software development community has, after due consideration and weighing all the involved trade-offs, rejected and moved away from.

                                                                  I think this is a common view but it results from sampling bias. If you’re the author of a particular piece of software, you care deeply about it, and the users you directly interact with also care deeply about it. So you will tend to see benefits that apply to people for whom your software is of particular importance in their stack. You will tend to be blind to the users for whom your software is “part of the furniture”. From the other side, that’s the majority of the software you use.

                                                                  Users who benefit from the traditional distribution packaging model for most of their software also find that same model to be painful for some “key” software. The problem is that what software is key is different for different classes of user.

                                                                  1. 10

                                                                    A big reason people ship binaries statically linked is so it’s easier to use without frills, benefiting especially users who aren’t deeply invested in the software.

                                                                    1. 9

                                                                      For me personally as an end user, if a program is available in apt-get then I will install it from apt-get first, every time. I don’t want to be responsible for tracking updates to that program manually!

                                                                      1. 2

                                                                        I do that as well, but I think “apt-get or manual installs” is a bit of a false dilemma: you can have both.

                                                              2. 8

                                                                Static linking does introduce a security risk: ASLR made ineffective. Static linking creates a deterministic memory layout, thus making moot ASLR.

                                                                1. 5

                                                                  Untrue, look up static-PIE executables. Looks like OpenBSD did it first, of course.

                                                                  1. 3

                                                                    I believe static PIE can only randomize a single base address for a statically linked executable, unlike dynamically linked PIE executable where all loaded PIC objects receive a randomized base address.

                                                                    1. 2

                                                                      I’m very familiar with static PIE. I’m unsure of any OS besides OpenBSD that supports it.

                                                                      1. 4

                                                                        Rustc + musl, supports it on linux, since gcc has a flag for it I imagine it’s possible to use it for C code too but I don’t know how.

                                                                        1. 2

                                                                          It was added to GNU libc in 2.27 from Feb 2018. I think it should work on Linux?

                                                                          1. 5

                                                                            Looks like it works on Linux with gcc 10.

                                                                            $ uname -a
                                                                            Linux phoenix 5.10.0-2-amd64 #1 SMP Debian 5.10.9-1 (2021-01-20) x86_64 GNU/Linux
                                                                            $ gcc -static-pie hello.c
                                                                            $ ./a.out
                                                                            Hello world!
                                                                            $ ldd a.out
                                                                            	statically linked
                                                                            

                                                                            Did a bit of rummaging in the exe header but I’m not 100% sure what I’m looking for to confirm there, but it had a relocation section and all symbols in it were relative to the start of the file as far as I could tell.

                                                                            Edit: Okay, it appears the brute-force way works. I love C sometimes.

                                                                            aslr.c:

                                                                            #include <stdio.h>
                                                                            
                                                                            int main() {
                                                                                int (*p)() = main;
                                                                                printf("main is %p\n", p);
                                                                                return 0;
                                                                            }
                                                                            

                                                                            Testing:

                                                                            $ gcc aslr.c
                                                                            $ ldd a.out
                                                                            	linux-vdso.so.1 (0x00007ffe47d2f000)
                                                                            	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2de631b000)
                                                                            	/lib64/ld-linux-x86-64.so.2 (0x00007f2de6512000)
                                                                            $ ./a.out; ./a.out; ./a.out
                                                                            main is 0x564d9cf42135
                                                                            main is 0x561c0b882135
                                                                            main is 0x55f84a94f135
                                                                            
                                                                            $ gcc -static aslr.c
                                                                            $ ldd a.out
                                                                            	not a dynamic executable
                                                                            $ a.out; a.out; a.out
                                                                            main is 0x401c2d
                                                                            main is 0x401c2d
                                                                            main is 0x401c2d
                                                                            
                                                                            $ gcc -static-pie aslr.c
                                                                            $ ldd a.out
                                                                            	statically linked
                                                                            $ a.out; a.out; a.out
                                                                            main is 0x7f4549c07db5
                                                                            main is 0x7f5c6bce5db5
                                                                            main is 0x7fd0a2aaedb5
                                                                            

                                                                            Note ldd distinguishing between “not a dynamic executable” and “statically linked”.

                                                                      2. 3

                                                                        Doesn’t it stop other kind of attacks on the other hand?

                                                                        1. 3

                                                                          What attacks would static linking mitigate?

                                                                          1. 3

                                                                            I really have no idea, that was an honest question!

                                                                            1. 3

                                                                              I don’t know much about security, and this is an honest question. My understanding is that ASLR is made to mostly protect when the executable is compromised due to stack overflow and similar attacks, right? Aren’t these problems mostly a thing of past for the two languages that abhor static linking, like go and rust?

                                                                                1. 6

                                                                                  Those would be examples of local attacks. ASLR does not protect against local attacks. So we need to talk about the same threat vectors. :)

                                                                                2. 1

                                                                                  There are a lot of code-injection vulnerabilities that can occur as a result of LD_LIBRARY_PATH shenanigans. If you’re not building everything relro, dynamic linking has a bunch of things like the PLT GOT that contain function pointers that the program will blindly jump through, making exploiting memory-safety vulnerabilities easier.

                                                                                  1. 1

                                                                                    As an author of open source malware specifically targeting the PLT/GOT for FreeBSD processes, I’m familiar with PLT/GOT. :)

                                                                                    The good news is that the llvm toolchain (clang, lld) enables RELRO by default, but not BIND_NOW. HardenedBSD enables BIND_NOW by default. Though, on systems that don’t disable unprivileged process debugging, using BIND_NOW can open a new can of worms: making PLT/GOT redirection attacks easier over the ptrace boundary.

                                                                          1. 11

                                                                            There are basically 5 classes of programs that the author discusses

                                                                            1. Fully dynamically linked programs. Only python programs are of this form.
                                                                            2. Partially dynamically linked programs. This describes C and C++ using dynamic libraries. The contents of the .c files are dynamically linked, and the contents of the .h files are statically linked. We can assume that the .h files are picked up from the system that the artifact is built on and not pinned or bundled in any way.
                                                                            3. Statically linked programs without dependency pinning. This describes rust binaries that don’t check in their Cargo.lock file to the repository, for instance.
                                                                            4. Statically linked programs with dependency pinning. This describes rust binaries that do check in their Cargo.lock file to the repository. (For simplicity sake we can include bundled but easily replaceable dependencies in this category)
                                                                            5. Programs with hard to replace bundled dependencies (statically or dynamically linked, for instance they complain about rustc llvm which is dynamically linked).

                                                                            I think it’s pretty clear that what the author is interested in isn’t actually the type of linking, they are interested in the ease of upgrading dependencies. This is why they don’t like python programs despite the fact that they are the most dynamically linked. They happen to have tooling that works for the case of dynamically linked C/C++ programs (as long as the header files don’t change, and if they do, sucks to be the user) so they like them. They don’t have tooling that works for updating python/rust/go/… dependencies, so they don’t like them.

                                                                            They do have a bit of a legitimate complaint here that it takes longer to relink all the statically link dependencies than the dynamically linked ones, but this strikes me as very minor. Builds don’t take that long in the grand scheme of things (especially if you keep around intermediate artifacts from previous builds). The benefit that we don’t have the C/C++ problem where the statically linked parts and the dynamically linked parts can come from different code bases and not line up strikes me as more than worth it.

                                                                            They seem to be annoyed with case 3 because it requires they update their tooling, and maybe because it makes bugs resulting from the equivalent of header file changes more immediately their problem. As you can guess, I’m not sympathetic to this complaint.

                                                                            They seem to be annoyed with case 4 because it also makes it makes the responsibility for breaking changes in dependencies shift slightly from code authors to maintainers, and their tooling is even less likely to support it. This complaint mostly strikes me as entitled, the people who develop the code they are packaging for the most part are doing so for free (this is open source after all) and haven’t made some commitment to support you updating their dependencies, why should it be their problem? If you look at any popular C/C++ library on github, you will find issues asking for support for exactly this sort of thing.

                                                                            Category 5 does have some interesting tradeoffs in both directions depending on the situation, but I don’t think this article does justice to either side… and I think getting into them here would detract from the main point.

                                                                            1. 5

                                                                              I was especially surprised to see this article on a gentoo blog, given that as I remember gentoo (admittedly from like 10-15 years ago), it was all about recompiling everything from source code, mainly For Better Performance IIRC. And if you recompile everything from source anyway, I’d think that should solve this issue for “static linkage” too? But maybe gentoo changed their way since?

                                                                              Looking at some other modern technologies, I believe Nix (and NixOS) actually also provide this feature of basically recompiling from source, and thus should make working with “static” vs. “dynamic” linking mostly the same? I’m quite sure arbitrary patches can be (and are) applied to apps distributed via Nix. And anytime I nix-channel --upgrade, I’m getting new versions of everything AFAIK, including statically linked stuff (obviously also risking occasional breakage :/)

                                                                              edit: Hm, Wikipedia does seem to also say Gentoo is about rebuilding from source, so I’m now honestly completely confused why this article is on gentoo’s blog, of all the distros…

                                                                              Unlike a binary software distribution, the source code is compiled locally according to the user’s preferences and is often optimized for the specific type of computer. Precompiled binaries are available for some larger packages or those with no available source code.

                                                                              1. 11

                                                                                “Build from source” doesn’t really solve the case of vendored libraries or pinned dependencies. If my program ships with liblob-1.15 and it turns out that version has a security problem, then a recompile will just compile that version again.

                                                                                You need upstream to update it to liblob-1.16 which fixes the problem, or maybe even liblob-2.0. This is essentially the issue; to quote the opening sentence of this article: “One of the most important tasks of the distribution packager is to ensure that the software shipped to our users is free of security vulnerabilities”. They don’t want to be reliant on upstream for this, so they take care to patch this in their packages, but it’s all some effort. You also need to rebuild all packages that use liblob<=1.15.

                                                                                I don’t especially agree with this author, but no one can deny that recompiling only a system liblob is a lot easier.

                                                                                1. 2

                                                                                  AIUI the crux of Gentoo is that it provides compile-time configuration - if you’re not using e.g. Firefox’s Kerberos support, then instead of compiling the Kerberos code into the binary and adding “use_kerberos=false” or whatever, you can just not compile that dead code in the first place. And on top of that, you can skip a dependency on libkerberos or whatever, that might break! And as a slight side-effect, the smaller binary might have performance improvements. Also, obviously, you don’t need libkerberos or whatever loaded in RAM. Or even on disk.

                                                                                  These compile-time configuration choices have typically been the domain of distro packagers, but Gentoo gives the choice to users instead. So I think it makes a lot of sense for a Gentoo user to have strong opinions about how upstream packaging works.

                                                                                  1. 2

                                                                                    But don’t they also advertise things like --with-sse2 etc., i.e. specific flags to tailor the packages to one’s specific hardware? Though I guess maybe hardwares are uniform enough nowadays that a typical gentoo user wants exactly the same flags as most others?

                                                                                2. 4

                                                                                  This complaint mostly strikes me as entitled, the people who develop the code they are packaging for the most part are doing so for free (this is open source after all) and haven’t made some commitment to support you updating their dependencies, why should it be their problem?

                                                                                  Maybe I’m reading too much into the post, but the complaints about version pinning seem to be implying that the application maintainers should be responsible for maintaining compatibility with any arbitrary version of any dependency the application pulls in. Of course application maintainers want to specify which versions there compatible with; it’s completely unrealistic to expect an application to put in the work to maintain compatibility with any old version that one distro or another might be stuck on. The alternative is a combinatoric explosion of headaches.

                                                                                  Am I misreading this? I’m trying to come up with a more charitable reading but it’s difficult.

                                                                                  1. 3

                                                                                    I’m not sure. When I wrote a Lua wrapper for libtls, I attempted to support older versions, but the authors of libtls didn’t do a good job of versioning macros. I eventually gave up on older versions when I switched to a different libtls. I am not happy about this.

                                                                                  2. 3

                                                                                    Don’t JVM and CLR programs also do all dynamic linking all the time, or almost so?

                                                                                    1. 2

                                                                                      Er, when I said “Only python programs are of this form.” I just meant of the languages mentioned in the article. Obviously various other languages including most interpreted languages are similar in nature.

                                                                                      I think the JVM code I’ve worked on packaged all it’s (java) dependencies inside the jar file - which seems roughly equivalent to static linking. I don’t know what’s typical in the open source world though. I’ve never worked with CLR/.net.

                                                                                      1. 3

                                                                                        It depends…

                                                                                        • Desktop or standalone Java programs usually consists of a collection of JAR files and you can easily inspect them and replace/upgrade particular libraries if you wish.
                                                                                        • Many web applications that are deployed on a web container (e.g. Tomcat) or an application server (e.g. Payara) as WAR files, have libraries bundled inside. This is bit ugly and I do not like it much (you have to upload big files to servers on each deploy), however you still can do the same as in the first case – just need to unzip and zip the WAR file.
                                                                                        • Modular applications have only their own code inside + they declare dependencies in a machine readable form. So you deploy small files e.g. on a OSGi container like Karaf and dependencies are resolved during the deploy (the metadata contain needed libraries and their supported version ranges). In this case you may have installed a library in many versions and proper one is linked to your application (other versions and other libraries are invisible despite they are present in the runtime environment). The introspection is very nice and you can watch how the application is starting, whether it is waiting for some libraries or other resources, you can install or configure them and then the starting process continues.

                                                                                        So it is far from static-linking and even if everything is bundled in a single JAR/WAR, you can easily replace or upgrade the libraries or do some other hacking or studying.

                                                                                  1. 2

                                                                                    The only two things that keep me on Windows are games and OBS.

                                                                                    1. 9

                                                                                      OBS works on Linux and Mac too

                                                                                      1. 1

                                                                                        What sort of OBS problems have you been having on Linux? I use OBS on both and the experience has seemed the same to me. Maybe there’s stuff on Windows that I’m not even aware that I’m missing.

                                                                                        1. 2

                                                                                          Last time I used it, the builtin browser was missing. Also on macOS the audio stuff is clunkier, you need third party apps to create virtual audio devices and connect them manually to sources. I don’t know what’s the status of audio stuff on linux.

                                                                                          In general I have the impression that OBS is developed “windows-first”.

                                                                                          1. 1

                                                                                            I’m not a sophisticated OBS user, but it’s generally worked fine for me on Linux.

                                                                                          2. 1

                                                                                            I’m also using it on linux. But for example the virtualcam plugin is windows only.

                                                                                        1. 2

                                                                                          Going sledding!

                                                                                          1. 1

                                                                                            I think it was playing around with Vulkan and GLFW, to see how well the Rust bindings held up. I stopped at getting a triangle on the screen, but it worked pretty well.

                                                                                            1. 13

                                                                                              The whole blog seems to have disappeared. It also takes an extreme amount of time to load while some JS spinner is turning. I want the old web back, where this would take milliseconds to load.

                                                                                              1. 9

                                                                                                I was gonna make a crack or two about how slow the web was on dialup, but then I looked at this site and no, no, you’re totally right. It does a heck of a lot of work to load minimally-styled text and a few images.

                                                                                                We’re sorry but frontend doesn’t work properly without JavaScript enabled. Please enable it to continue.

                                                                                                Beautiful.

                                                                                              1. 1

                                                                                                It is nice, but the statement that human has always capabilities to write faster code in assembly than compiler from C is utter bullshit. In theory - maybe, but in reality compilers optimisations can reason about the code way beyond human capabilities.

                                                                                                1. 4

                                                                                                  Not beyond human capabilities, just usually beyond human patience.

                                                                                                  1. 1

                                                                                                    Let’s assume you are right.

                                                                                                    Let Z be the program optimized better by a compiler than a human.

                                                                                                    For Z to exist, there must be a logical or computational structure in the program that no human world recognize, but the compiler would. We assume such a structure exists.

                                                                                                    However the compiler only does optimizations based on rules. Thus for Z to exist, there must be a rule a human can not understand.

                                                                                                    Is there any rule written in to a compiler that a human can not understand? I will posit thus: a compiler rule is written or verified by a human. Thus for any given rule, there exists a human who can understand it and is able to calculate its correctness themselves.

                                                                                                    Thus there is no rule beyond human comprehension.

                                                                                                    There is no computational or logical structure in a program that a human can not parse as well as the optimizing compiler.

                                                                                                    And a human optimised program can always be equally fast as a compiler optimized program. Z does not exist, and you are not correct.

                                                                                                  1. 1

                                                                                                    Why is this tagged lisp?

                                                                                                    1. 1

                                                                                                      I generally just select the “accept cookies” popup in the Firefox inspector and delete it. Works pretty well.

                                                                                                      1. 6

                                                                                                        What is the benefit of adding const to a function? Its surprising to see what feels like a relatively minor change make the release notes.

                                                                                                        1. 17

                                                                                                          Adding ‘const’ to a rust function makes it possible to execute the function in a “const context”. Const contexts are those which are computed at compile time. A const variable’s initialization is one example of a const context. Having more things computed at compile time can have significant effects on the run time behavior of code, effectively eliminating the run time cost of producing such a value, which mattes a lot when that code may be in a tight loop.

                                                                                                          The other thing to know is that the const feature was added to rust after much of the standard library was written. In fact the language is in the process of making more and more expressions have the ability to be const. There are even some typically heap allocated values which can be const evaluated. As those features get added the standard library is adding const to functions which can be made const.

                                                                                                          So although I don’t write the release notes and I don’t know the person who does. My guess would be that they appear there because it’s a way of tracking const’s progress as an important language that has serious runtime benefits.

                                                                                                          1. 1

                                                                                                            Hm interesting, so it’s basically constexpr from C++?

                                                                                                            I started using constexpr recently, and I noticed how it has progressed over C++11, 14, 17, 20. At first you could only use functions with a single return statement. Now they are adding certain STL algorithms, etc.

                                                                                                            That is, making more and more language features valid in constexpr contexts until the STL itself can run at compile time. And probably recasting the STL itself in terms of newer features that can run in both contexts. (Memory allocation is an issue there too, but I don’t remember the exact mechanisms)

                                                                                                            Sounds like Rust is undergoing a similar evolution?

                                                                                                            1. 7

                                                                                                              Const contexts are those which are computed at compile time.

                                                                                                              I believe it more that they can be compiled at compile time, but whether they are or aren’t is up to the optimizer. But it allows the optimizer a lot more leeway in how things work; a const function is basically pure, so a lot more reduction can be done.

                                                                                                              Hm interesting, so it’s basically constexpr from C++?

                                                                                                              Pretty much exactly that, yes. Rust’s const is currently less powerful than constexpr, but it is kind of slowly creeping up on feature parity.

                                                                                                              1. 2

                                                                                                                I think const functions can also be used in types in ways that non const functions cannot. But I don’t remember all the details

                                                                                                        1. 3

                                                                                                          It was only a few months ago that I learned all Python integers are heavyweight, heap-allocated objects. This kind of boggled my mind. This post says that small ints in the 8-bit range are cached, but that’s not much of an optimization.

                                                                                                          Anyone know why Python never started using pointer-tagging for ints, the way tons of other dynamic languages (from LISP to Smalltalk to Lua) do?

                                                                                                          (Fun fact: Smalltalk-80 used pointer-tagging for ints up to +/-16384, and allocated objects for bigger ones. This was really apparent in the GUI: the text view would start to massively slow down as soon as the length of the text exceeded 16KB, because the layout and display would start allocating more and more numbers.)

                                                                                                          1. 6

                                                                                                            Virtually everything in Python is a “heavyweight” object, and neither ultra-low memory use nor blazing-fast bare-metal performance are particularly important to the average Python user. So there’s no real incentive to complicate the implementation with multiple differently-sized or differently-allocated backing types for int. In fact, Python moved away from that. In the old old days, Python had two integer types: one could only handle 32-bit values, and the other was a bignum. You could request the bignum with a suffix on an integer literal – 42 versus 42L – but despite some effort they were never quite interchangeable. The unification to a single (bignum) integer type began in Python 2.2 (released 2001) and now the old L suffix is a syntax error.

                                                                                                            1. 1

                                                                                                              neither ultra-low memory use nor blazing-fast bare-metal performance are particularly important to the average Python user

                                                                                                              They’re not THAT important, but they would sure be nice. It would probably result in me writing more Python and less, say, Rust.

                                                                                                            2. 2

                                                                                                              If you launch an interpreter and use sys.getrefcount, you can see that the first 10 numbers already sum up to over 800 references (Windows, Python 3.9.1). I’m guessing it does help, or it probably wouldn’t be there.