Many, if not most, ISPs will only give you a single /64 that they constantly change through DHCP-PD, so you can’t count on anything and have to either use IPv4 internally (which is what everyone does) or resort to IPv6 NAT (which also defeats the purpose). That’s now even about hosting servers, it’s about any communication within the home network unless can fully rely on mDNS and friends for everything.
Many ISPs in many places don’t provide even that, they don’t provide any working residential IPv6 at all.
Countries with Internet censorship are hindering IPv6 adoption because their mass surveillance and censorship technologies were designed for IPv4 and ban tunnel brokers (once a viable alternative) because they also allow people to bypass their censorship.
I’m not exactly hopeful for the future of IPv6, but at least the first part can be fixed by regulation in more-or-less normal countries.
The post has at leaset some connection to “the IPv6 internet” and why it is broken[0], but this sounds more like mosts IPSs are broken.
Many, if not most, ISPs will only give you a single /64 that they constantly change through DHCP-PD
So your ISP don’t give you proper access to the internet? In other words your ISP is broken not the IPv6 Internet.
Most ISPs I know (german home ISP, I don’t know about mobil) give you a /56 via DHCP-PD. Yes the change the prefix regularly for privacy reasons. On request you get a static prefix. For the most private (and smal non IT business) this exactly what they need.
it’s about any communication within the home network
You know about ULAs (private IPv6 address space) and DHCPv6? Setup your local service to do dhcpv6 and configure a ULA prefix on your network. This works the same way as in IPv4 and has also the same downsides (clients can’t dhcp, only have static dns servers, …).
Many ISPs in many places don’t provide even that, they don’t provide any working residential IPv6 at all.
Again the ISP don’t provider proper IPv6 internet and not the IPv6 internet is broken.
Countries with Internet censorship are hindering IPv6 adoption
Yes Countries with Internet censorship censor the Internet. Still don’t see how this means the Internet is broken.
Yes there are Issues with ISPs providing proper Internet (v4 and v6), but this doesn’t mean the Internet is broken. It bugs me when it is claimed inverted. Because then the details get lost over time and some people only remember ipv6 is bad. Also most of the hacks we have in place for working ipv4 are often forgotten.
[0] but the question is: is ipv6 is the cause of the problem?
Many, if not most, ISPs will only give you a single /64 that they constantly change through DHCP-PD, so you can’t count on anything and have to either use IPv4 internally (which is what everyone does) or resort to IPv6 NAT (which also defeats the purpose). That’s now even about hosting servers, it’s about any communication within the home network unless can fully rely on mDNS and friends for everything.
This isn’t worse than the situation with IPv4, where a lot of ISPs give you a new IP address and NAT your network. Or, worse, add carrier-grade NAT on top of local NAT.
Oh, and you aren’t restricted to using v4 internally. IPv6 was designed to support multiple IP addresses per client device, so you can have a static address and (changing) globally-reachable address pretty easily with most mainstream operating systems.
… why would you use your public IPv6 address for LAN communication? You can have multiple v6 addresses per interface, use private v6 addresses for LAN same as you do for private v4 addresses. Doesn’t require NAT since you also have the global v6 address for internet things
Yeah I’m confused as well, this is what site-local ipv6 addresses are for, to decouple your public ipv6 from your site ipv6. Or I guess now just use unique local addresses instead, same diff. Think ipv4 is polluting people’s minds into thinking you need to have a single ip address per machine. ipv6 is really not like ipv4 and the more you try to treat it that way the more you start just repeating the sins of the past in it.
Bonus is with ipv6 ULA’s you get to know exactly who you’re connecting to to boot.
If you want to route them you can even with a /64 from upstream if they’re not playing via the rfc and providing at least a /56 or /48. But even a /64 is pretty gihugic.
It may be worth noting here that keeping Ctrl-C functional is quite tractable in Haskell thanks to its async exception support and its structured parallelism/concurrency utilities. In a Haskell program, the default SIGINT handler will just throw an async exception (just like the ones you can throw to a running thread) to one of the running (lightweight) threads and as long as you follow some established best practices not to swallow async exceptions and use the standard structured concurrency utilities that propagate exceptions to your parent thread, your program will exit gracefully collecting all the resources and print any shutdown messages etc.
In my experience, all the Haskell programs I’ve interacted with seemed to follow these best practices since they handle Ctrl-C gracefully.
your program will exit gracefully collecting all the resources and print any shutdown messages etc.
The article is not advocating for Ctrl+C to shut down the program, it’s advocating for Ctrl+C to pause/reset the still-running program and allow execution to continue from that point, with the program’s running state at the moment of the Ctrl+C preserved as much as possible.
Ah yeah brainfart for some reason I was thinking TERM. Still don’t see a point to it compared to SIGSTOP/CONT though, used those for years to pause an executable and resume it later (notably firefox) to save battery.
The only problem with lots of custom aliases (or custom keybindings in other programs like editors), is that the muscle memory burns you every time you have to work on a remote machine. I used to go custom-to-the-max with my config, but I’ve gradually shifted back to fewer and fewer aliases except for the most prevalent build/version control commands I run dozens of times each day.
When I need to remote into machines where I can’t set up my shell profile for whatever reason, I just config ssh to run my preferred shell setup commands (aliases, etc) as I’m connecting.
Yeah, single session only. There are a bunch of different ways to skin this cat — LocalCommand and RemoteCommand along with ForceTTY in ssh_config can help.
Conceptually you want to do something like (syntax probably wrong, I’m on my phone)
which you could parameterize with a shell function or set up via LocalCommand and RemoteCommand above, or skip the temp file entirely with clever use of an env variable to slurp the rc file in and feed it into the remote bash (with a heredoc or SendEnv/SetEnv)
Nope. At least I’m not aware of any Linux distro installing it by default.
But being installed by default is IMHO totally overrated. The main point is that it is available in many Linux distribution’s repos without having to add 3rd party repos—at least in Debian and all derivatives like Devuan, Kali oder Ubuntu.
I understand, but it’s not the same. If I don’t have a shell regularly there, and not my own dotfiles, I likely want to avoid installing and removing system packages on other people’s systems. When stuff breaks, I want the minimum amount of blame :)
Ok, granted. Working as a system administrator it’s usually me who has to fix things anyway. And it happens only very, very seldom that something breaks just because you install a commandline tool. (Saying this with about 25 years of Linux system administration experience.)
Only zutils can theoretically have an impact as it renames commandline system tools and replaces them with wrappers. But so far in the past decade, I’ve never seen any system break due to zutils. (I only swa things not working properly because it was not installed. But that was mostly because I’m so used to it that I take it as given that zutils is installed. :-)
Yep, different role. I did some freelance work a long ago, and learned on (fortunately) my predecessor’s mistake: they hired me to do some work, because I guess someone before me updated some stuff, and that broke… probably PHP version? Anyway, their shop didn’t work any more and they were bleeding money till I fixed it. It was one of my early freelance jobs, so that confirmed the age-old BOFH mantra of if it ain’t broke, don’t fix it. So given time, I would always explicitly ask permission to do this or that or install the other, if needed.
But I went a different route anyway, so even though I am still better than average, I think, I’m neither good nor professional. But I think old habits die hard, so that’s why I’m saying “if this stuff isn’t there by default, you’ll just have to learn your tar switches” :)
In my opinion, server settings are where NixOS shines the most!
I was curious about that one recently. On a Mac if I need to install something, I just do it and if it’s big enough I’ll check back in 10min - it mostly works. (ignoring all the other problems) Maybe run gc once a week and clean up 40GB of trash. But I wouldn’t want to attempt that on an RPi class hardware. How do people deal with that? Some external compilation service/cache? I mean cases where it turns out you’re about to compile llvm on your underpowered Celeron based NAS.
I’m on a rolling release, but it’s not perfect. Specifically, some updates are not built yet at the time of installation. Also changing since configs means recompiling anyway (for example if you want to enable jit in Ruby)
Specifically, some updates are not built yet at the time of installation.
I guess for something like a raspberry pi your best option would be to stick to a release instead of nixos-unstable. For unstable or configs which are not in upstream caches you’d need some kind of external cache (which could be your desktop machine), yes.
One pitfall I ran into when using a stable release was using release-21.11 which also includes upgrades which are not yet built, switching to nixos-21.11 solved that.
Great article… In other words, Nix is a leaky abstraction and it bottoms out at shell scripts, e.g. a wrapper to munge the linker command line to set RPATH to /nix/store.
This doesn’t appear in your code / package definitions and is hard to debug when it goes wrong.
Nix was designed and developed before Linux containers, so it had to use the /nix/store mechanism to achieve its goals (paths should identify immutable content, not mutable “places”).
So I wish we had a Nix like system based on containers … (and some kind of OverlayFS and /nix/store hybrid)
I don’t, I like running nix on macos. How would your “solution” of running linux binaries and filesystems on macos work exactly? This whole blog post amounted to using the wrong lld, I don’t see how “lets use containers” is a good fix for the problem at hand.
See this reply … I have a specific problem of running lightweight containers locally and remotely, and transfering them / storing them, and am not trying to do anything with OS X:
The background is that Oil contributors already tried to put Oil’s dev env in shell.nix, and run it on CI.
However my own shell scripts that invoke Docker/podman end up working better across Github Actions, sourcehut, local dev, etc. Docker definitely has design bugs, but it does solve the problem … I like that it is being “refactored away” and orthogonalized
However my own shell scripts that invoke Docker/podman end up working better across Github Actions, sourcehut, local dev, etc. Docker definitely has design bugs, but it does solve the problem … I like that it is being “refactored away” and orthogonalized
Its not though, you’re not solving the problem for “not linux” if you put everything in docker/containers. nix the package manager is just files and symlinks at the end of the day. It runs on more than just linux, any linux only “solution” is just that, not a solution that actually fits the problem space as nix. Which includes freebsd, macos, and linux.
The background is that Oil contributors already tried to put Oil’s dev env in shell.nix, and run it on CI.
And you can create containers from nix package derivations, and your linked comment doesn’t really explain your problem. I’ve used nix to create “lightweight” containers, the way nix stores files makes it rather nice as it avoids all that layering rubbish. But without a clear understanding of what exactly you’re talking about it really seems to be unrelated to this post entirely. How do you test Oil on macos/freebsd via CI? Even with those operating systems and nix, its its own world so you’d still have to test in and out of docker. Or am I misunderstanding? I’m still unclear what problem you are trying to accomplish and how it relates to rpath in a linker on nix and how containers solve it.
Yeah it’s true, if you have a OS X or BSD requirement then what I’m thinking of isn’t going to help, or at least it has all the same problems that Docker does on OS X (I think it runs in a VM)
I would have liked to have used Nix but it didn’t solve the problem well … It apparently doesn’t solve the “it works on my machine” problem. Apparently Nix Flakes solves that better? That is, whether it’s isolated/hermetic apparently depends on the build configuration. In contrast Bazel (with all its limitations) does pretty much solve that problem, independent of your build configuration is.
I think the setup also depended on some kind of experimental Nix support for Travis CI (and cachix?), and then Travis CI went down, etc.
I would be happy if some contributor told me this is all wrong and just fixed everything. But I don’t think that will happen because there are real problems it doesn’t address. Maybe Nix flakes will do it but in the meantime I think containers solved the problem more effectively.
It’s better to get into the details on Zulip, but shells are extremely portable so about 5% of the build needs to run on OS X, and that can be done with tarballs and whatnot, because it has few dependencies. The other 95% is a huge build and test matrix that involves shells that don’t exist on OS X like busybox ash, and many many tools for quality and metaprogramming.
Shells are also very low level, so we ran into the issue where Nix can’t sandbox libc. The libc on OS X pokes through, and that’s kind of fundamental as far as I can tell.
This problem is sufficiently like most of the other problems I’ve had in the past that I’m willing to spend some time on it … e.g. I’m interested in distributed systems and 99% of those run Linux kernels everywhere. If you’re writing OS X software or simply like using OS X a lot then it won’t be interesting. (I sometimes use OS X, but use a Linux VM for development.)
paths should identify immutable content, not mutable “places”
Dynamic linking to immutable content isn’t actually dynamic linking, its more like “deferred linking”. Optimize the deferred part away and you are back at static linking. The next optimization is to dedup libraries by static linking multiple programs into multicall binaries.
Do containers allow any kind of fs merging at the moment? I.e. similar results to overlayfs or merged symlinks dir from nix itself? I thought namespaces only allow basic mapping, so I’m curious where they could help.
Yeah I think you might want a hybrid of OverlayFS and bind mounts. There was an experiment mentioned here, and it was pointed out that it’s probably not a good idea to have as many layers as packages. Because you could have more than 128 packages that a binary depends on, and the kernel doesn’t like that many layers:
Here is what I’m thinking with bind mounts. I haven’t actually done this, but I’ve done similar things, and maybe some system already works like this – I wouldn’t be surprised. Let me know if it has already been done :)
So the point is to avoid all the RPATH stuff, and have more a “stock” build. IME the RPATH hacks are not necessarily terrible for C, but gets worse when you have dynamic modules in Python and R, which are shared libraries, which depend on other shared libraries. The build systems for most languages are annoying in this respect.
So you build inside a container, bind mounting both the tarball and the output /mydistro, which is analogous to /nix/store. And then do something like:
configure --prefix /mydistro/python && make && make install
But on the host side, /mydistro/python is actually /repo/python-3.9 or /repo/python-3.10.
So then at runtime, you do the same thing – bind mount /repo/python-3.9 as /mydistro/python
So then on the host you can have python 3.9 and 3.10 simultaneously. This is where the package manager downloads data to.
But apps themselves run inside a container, with their custom namespace. So with this scheme I think you should be able to mix and match Python versions dynamically because the built artifacts don’t have version numbers or /nix/store/HASH in their paths.
I would try bubblewrap first – I just tried it and it seemed nice.
There are obviously a bunch of other issues. The distri experiment has some notes on related issues, but I think this removes the RPATH hacks while still letting you have multiple versions installed.
If anyone tries it let me know :) It should be doable with a 20 line shell script and bubblewrap.
I actually have this problem because I want to use Python 3.10 pattern matching syntax to write a type checker! That was released in October and my distro doesn’t have it.
Right now I just build it outside the container, which is fine. But I think having apps explicitly limited to a file system with their dependencies mounted has a lot of benefits. It is a middleground between the big blob of Docker and more precise dependencies of Nix.
Yeah I think you might want a hybrid of OverlayFS and bind mounts. There was an experiment mentioned here, and it was pointed out that it’s probably not a good idea to have as many layers as packages. Because you could have more than 128 packages that a binary depends on, and the kernel doesn’t like that many layers:
The problem with overlay / union filesystems is that the problem that they’re trying to solve is intrinsically very hard. If a file doesn’t exist in the top layer then you need to traverse all lower layers to try to find it. If you can guarantee that the lower layers are immutable then you can cache this traversal and build a combined view of a directory once but if they might be mutated then you need to provide some cache invalidation scheme. You have to do the traversal in order because a file can be created in one layer, deleted in a layer above (which requires you to support some notion of whiteout: the intermediate FS layer needs to track the fact that the file was deleted), and then re-added at a layer above. You also get exciting behaviours if only part of a file is modified: if I have a 1GiB file and I modify the header, my overlay needs to either copy the whole thing to the top layer, or it needs to manage the diff. In the latter case, this gets very exciting if something in the lower layer modifies the file. There are a lot of corner cases like this that mean you have to either implement things in a very inefficient way that scales poorly, or you have surprising semantics.
This is why containerd uses snapshots as the abstraction, rather than overlays. If you have an overlay / union FS, then you can implement snapshots by creating a new immutable layer and, because the layer is immutable, you won’t hit any of the painful corner cases in the union FS. If you have a CoW filesystem, then snapshots are basically free. With something like ZFS, you can write a bunch of files, snapshot the filesystem, create a mutable clone of the snapshot, write / delete more files, and snapshot the result, and so on. Each of the snapshot layers is guaranteed immutable and any file that is unmodified from the previous snapshot shares storage. This means that the top ‘layers’ just have reference-counted pointers to the data and so accesses are O(1) in terms of the number of layers.
The one thing that you lose with the snapshot model is the ability to arbitrarily change the composition order. For example, if I have one layer that installs package A, one that installs packages B on top, and one that installs package C on top, and I want a layer that installs packages A and C, then I can’t just combine the top and bottom layers, I need to start with the first one and install package C. Something like Nix can probably make the guarantees that would make this safe (that modified in the middle layer is modified by the application of the top layer), but that’s not possible in general.
Hm yeah I have seen those weird issues with OverlayFS but not really experienced them … The reason I’m interseted in it is that I believe Docker uses it by default on most Linux distros. I think it used to use block-based solutions but I’m not entirely clear why they switched.
The other reason I like it is because the layers are “first class” more amenable to shell scripting than block devices.
And yes the idea behind the “vertical slices” is that they compose and don’t have ordering, like /nix/store.
The idea behind the “horizontal layer” is that I don’t want to bootstrap the base image and the compiler myself :-/ I just want to do apt-get install build-essential.
This is mainly for “rationalizing” the 5 containers I have in the Oil build, but I think it could be used to solve many problems I’ve had in the past.
And also I think it is simple enough to do from shell; I’m not buying into a huge distro, although this could evolve into one.
Basically I want to make the containers more fine-grained and composable. Each main() program should have its own lightweight container, a /bin/sh exec wrapper, and then you can compose those with shell scripts! (The continuous build is already a bunch of portable shell scripts.)
Also I want more sharing, which gives you faster transfers over the network and smaller overall size.
I am pretty sure this can solve my immediate problem – whether it generalizes I’m not sure, but I don’t see why not. For this project, desktop apps and OS X are out of scope.
(Also I point out in another comment that I’d like to learn about the overlap between this scheme and what Flatpak already does? i.e. the build tools and runtime, and any notion of repository and network transfer. I’ve already used bubblewrap)
Hm yeah I have seen those weird issues with OverlayFS but not really experienced them … The reason I’m interseted in it is that I believe Docker uses it by default on most Linux distros. I think it used to use block-based solutions but I’m not entirely clear why they switched.
Docker now is a wrapper around containerd and so uses the snapshot abstraction. OCI containers are defined in terms of layers that define deltas on existing layers (starting with an empty one). containerd provides caching for these layers by delegating to a snapshotting service, which can apply the deltas as an overlay layer (which it then never modifies, so avoiding all of the corner cases) or to a filesystem with CoW snapshots.
The other reason I like it is because the layers are “first class” more amenable to shell scripting than block devices.
I’m not sure what this means. ZFS snapshots, for example, can be mounted in .zfs/{snapshot name} as read-only trees.
Basically I want to make the containers more fine-grained and composable. Each main() program should have its own lightweight container, a /bin/sh exec wrapper, and then you can compose those with shell scripts! (The continuous build is already a bunch of portable shell scripts.)
To do this really nicely, I want some of the functionality from capsh, so I can use Capsicum, not jails, and have the shell open file descriptors easily for the processes that it spawns, rather than relying on trying to shim all of this into a private view of the global namespace.
I think you’re only speaking about BSD. containerd has the notion of “storage drivers”, and “overlay2” is the default storage driver on Linux. I think it changed 3-4 years ago
When I look at /var/lib/docker on my Ubuntu machine, it seems to confirm this – On Linux, Docker uses file level “differential” layers, not block-level snapshots. (And all this /var/lib/docker nonsense is what I’m criticizing on the blog. Docker is “anti-Unix”. Monolithic and code-centric not data-centric.)
So basically I want to continue what Red Hat and others are doing and continue “refactoring away” Docker, and just use OverlayFS. From my point of view they did a good job of getting that into the kernel, so now it is reasonable to rely on it. (I think there were 2 iterations of OverlayFS – the second version fixes or mitigates the problems you noted – I agree it is hard, but I also think it is solved.)
I think I wrote about it on the other thread, but I’m getting at a “remote/mobile process abstraction” with explicit data dependencies, mostly for batch processes. You need the data dependencies to be mobile. And I don’t want to introduce more concepts than necessary (according to the Perlis-Thompson principle and narrow waists), so just tarballs of files as layers, rather than block devices, seem ideal.
The blocks are dependent on a specific file system, i.e. ext3 or ext4. And also I don’t think you can do anything with an upper layer without the lower layers. With the file-level abstraction you can do that.
So it seems nicer not to introduce the constraint that all nodes have to be running the same file system – they merely all have to have OverlayFS, which is increasingly true.
None of this is going to be built directly into Oil – it’s a layer on top. So presumably BSDs could use Docker or whatever, or maybe the remote process abstraction can be ported.
Right now I’m just solving my own problem, which is very concrete, but as mentioned this is very similar to lots of problems I’ve had.
Of course Kubernetes and dozens of other systems going back years have remote/mobile process abstractions, but none of them “won”, and they are all coupled to a whole lot of other stuff. I want something that is minimal and composable from the shell, and that basically leads into “distributed shell scripting”.
I think all these systems were not properly FACTORED in the Unix sense. They were not narrow waists and didn’t compose with shell. They have only the most basic integration with shell.
For example our CI is just 5 parallel jobs with 5 Dockerfiles now:
I believe most CIs are like this – dumb, racy, without data dependencies, and with hard-coded schedules. So I would like to turn into something more fine-grained, parallel, and thus faster (but also more coarse-grained than Nix.) Basically by framing it in terms of shell, you get LANGUAGE-oriented composition.
(And of course, as previous blog posts say, a container-based build system should be the same thing as a CI system; there shouldn’t be anything you can only run remotely.)
I looked at Capsicum many years ago but haven’t seen capsh… For better or worse Oil is stuck on the lowest common denominator of POSIX, but the remote processes can be built on top, and right now that part feels Linux-only. I wasn’t really aware that people used Docker on BSD and I don’t know anything about it … (I did use NearlyFreeSpeech and their “epochs” based on BSD jails – it’s OK but not as flexible as what I want. It’s more on the admin side than the user side.)
I think you’re only speaking about BSD. containerd has the notion of “storage drivers”, and “overlay2” is the default storage driver on Linux. I think it changed 3-4 years ago
No, I’m talking about the abstractions that containerd uses. It can use overlay filesystems to implement a snapshot abstraction. Docker tried to do this the other way around and use snapshots to implement an overlay abstraction but this doesn’t work well and so containerd inverted it. This is in the docs.
When I look at /var/lib/docker on my Ubuntu machine, it seems to confirm this – On Linux, Docker uses file level “differential” layers, not block-level snapshots
Snapshots don’t have to be at the block level, they can be at the file level. There are various snapshotters in containerd that implement the same abstraction in different ways. The key point is that each layer is a delta that is applied to one specific immutable thing below.
I’m not really sure what the rest of your post is talking about. You seem to be conflating abstractions and implementation.
OK I think you were misunderstanding what I was talking about in the original message. What I’m proposing uses OverlayFS with immutable layers. Any mutable state is outside the container and mounted in at runtime. It’s more like an executable than a container.
Apparently it is mostly for desktop apps? I don’t see why that would be since CLI apps and server apps should be strictly easier.
I think the main difference again would be the mix of layers and slices, so you have less build configuration. And also naming them as first class on the file system and dynamically mixing and matching. What I don’t like is all the “boiling the ocean” required for packaging, e.g. RPATH but also a lot of other stuff …
I have Snap on my Ubuntu desktop but I am trying to avoid it… Maybe Flatpak is better, not sure.
That sounds like there’s a generic “distro python” though, which… is not necessarily true. You could definitely want environments with both python3.10 and 3.9 installed and not conflicting at the same time.
The model I’m going for is that you’re not really “inside” a container … But each main() program uses a lightweight container for its dependencies. So I can’t really imagine any case where a single main() uses both Python 3.9 and 3.10.
If you have p39 and p10 and want to pipe them together, you pipe together two DIFFERENT containers. You don’t pipe them together inside the container. It’s more like the model of iOS or Android, and apps are identified by a single hash that is a hash of the dependencies, which are layers/slices.
BUT importantly they can share layers / “slices” underneath, so it’s not as wasteful as snap/flatpak and such.
I’ve only looked a little at snap / flatpak, but I think they are more heavyweight, it’s like you’re inside a “machine” and not just assembling namespaces. I imagine an exec wrapper that makes each script isolated
# this script behaves exactly like p310.py and can be piped together with other scripts
exec bwrap --mount foo foo --mount p310.py p310.py -- /bin/p310.py "$@"
Your idea kinda sounds like GoboLinux Runner, but I can’t tell if it’s exactly the same, since it’s been a long time since I played with GoboLinux. It’s a very interesting take on Linux, flipping the FHS on it’s head just like Nix or Guix, but still keeping the actual program store fully user accessible, and mostly manageable without special commands.
Ah interesting, I heard about GoboLinux >10 years ago but it looks like they made some interesting developments.
They say they are using a “custom mount table” and that is essentially what bubblewrap lets you do. You just specify a bunch of --mount flags and it makes mount() syscalls before exec-ing the program.
Its package ecosystem is in excellent condition and packages such as org-mode and eglot / lsp-mode make even the most demanding programming languages a joy to work with in Emacs.
I work on a large C/C++ codebase as part of my day job and use lsp-mode/eglot (currently eglot) to navigate the code, with very few extensions. I also use the latest mainline Emacs with native compilation. I have been using Emacs for over 25 years and my customization is best categorized as “very light”. In short, my Emacs set up is not much beyond what ships with it.
And it’s still just… slow. GCC has some pretty big files and opening them can take up to 10 seconds thanks to font-lock mode. (Yes, I know I can configure it to be less decorative, but I find that decoration useful.) It’s much worse when you open a file that is the output from preprocessor expansion (easily 20000+ lines in many cases).
Log files that are hundreds of megabytes are pretty much a guaranteed way to bring Emacs to a crawl. Incremental search in such a buffer is just painful, even if you M-x find-file-literally.
I had to turn off nearly everything in lsp-mode/eglot because it does nothing but delay my input. I can start typing and it will be 3-4 characters behind as it tries to find all the completions I’m not asking for. Company, flymake, eldoc are all intolerably slow when working with my codebase, and I have turned them all off or not installed them in the first place.
M-x term is really useful, but do not attempt to run something that will produce a lot of output to the terminal. It is near intolerable. Literally orders of magnitude slower to display than an xterm or any other terminal emulator. (M-x eterm is no better.)
The problem, of course, is that Elisp is simply not performant. At all. It’s wonderfully malleable and horribly slow. It’s been this way since I started using it. I had hopes for native compilation, but I’ve been running it for a few months now and it’s still bad. I love Emacs for text editing and will continue to use it. I tried to make it a “lifestyle choice” for a while and realized it’s not a good one if I don’t want to be frustrated all the time. Emacs never seems to feel fast, despite the fast hardware I run it on.
The performance was the reason for me to leave Emacs. I was an evil mode user anyways so the complete switch to (neo)Vim was simple for me. I just could not accept the slowness of Emacs when in Vim everything is instant.
E.g. Magit is always named as one of the prime benefits of Emacs. While its functionality is truly amazing its performance is not. Working on a large code base and repository I was sometimes waiting minutes! for a view to open.
I actually use Emacs because I found it really fast compared to other options. For example, the notmuch email client is really quick on massive mailboxes.
Some packages might be slow, though. I think the trick is to have a minimal configuration with very well chosen packages. I am particularly interested in performance because my machine is really humble (an old NUC with a slow SATA disk).
To be fair it was some time ago and I don’t remember all the details but using LSPs for code completion/inspection was pretty slow e.g.
Compared to IDEs it might not even have been slow but similar. I however have to compare to Vim where I have equal capabilities but pretty much everything is instant.
Thanks for the notice! I may try it again in the future but currently I am very happy with my Neovim setup, which took me a long time to setup/tweak :)
Out of curiosity, were you using Magit on Windows?
I use Magit every day and my main machine is very slow. (1.5GHz 4 core cortex A53) Magit never struck me as particularly slow, but I’ve heard that on Windows where launching subprocesses takes longer it’s a different story.
but I’ve heard that on Windows where launching subprocesses takes longer
Ohh you have no idea how slow in a corporate environment. Going through MSYS2, Windows defender, with windows being windows and a corporate security system on top, it takes… ages. git add a single file? 20 seconds. Create a commit? Over a minute. It’s bonkers if you hit the worst case just right. (On a private Windows install, MSYS2 + Exceptions set in Windows Defender it’s fine though, not much slower as my FreeBSD laptop)
I asked around and there is a company wide, hardcoded path on every laptop, that has exceptions in all the security systems just to make life less miserable for programmers. Doesn’t solve it completly, but helps.
Either wait an eternity or make a mokery of the security concept. Suffice to say I stopped using Windows and Cross-Compile from now on.
With Windows I think it’s it’s particularly git that is slow, and magit spawns git repeatedly. It used also to be very slow on Mac OS as well because of problems with fork performance. On linux, it used to be slow with tramp. There are some tuning suggestions for all of these in the magit manual I think.
Nope on Linux. As mentioned our code base is big and has many branches etc. Not sure where exactly Magit’s bottleneck was. It was quite some time ago. I just remember that I found similar reports online and no real solution to them.
I now use Lazygit when I need something more than git cli and it’s a fantastic tool for my purpose. I also can use it from within Vim.
Working on a large code base and repository I was sometimes waiting minutes! for a view to open.
This happens for me as well with large changes. I really like Magit but when there are a lot of files it’s nearly unusable. You literally wait for minutes for it to show you an update.
I actually switched to M-x shell because I found the line/char mode entry in term-mode to be annoying (and it seems vterm is the same in this respect). shell-mode has all the same slowness of term-mode, of course. I’ve found doing terminal emulation in Emacs to be a lost cause and have given up on it after all these years. I think shell-mode is probably the most usable since it’s more like M-x shell-command than a terminal (and that’s really its best use case).
If you need ansi/curses there’s no good answer and while I like term it was too slow in the end and I left. I do think that for “just” using a shell that eshell is fine though.
Do you use the jit branch of emacs? I found once I switched to that and it had jit compiled things my emacs isn’t “fast” but its pretty boring now in that what used to be slow is now at least performant enough for me not to care.
I use the emacs-plus1 package. it compiles the version you specify. currently using emacs-plus@29 with --with-native-comp for native compilation, and probably some other flags.
Awesome! also, check out pixel-scroll-precision-mode for the sexiest pixel-by-pixel scrolling. seems to be a little buggy in info-mode, can’t replicate with emacs -Q though, so YMMV.
I gotta convert more of my config over but that was enough to build it and get my existing ~/.emacs.d working with it and speedy to the point I don’t care about emacs slowness even on macos anymore.
Here you go. It changes a little bit here and there with some experiments.The packages I currently have installed and use are: which-key, fic-mode, counsel, smartparens, magit, and solarized-theme. There may be a few others that I was trying out or are only installed for some language support (markdown, yaml, and so forth).
Quick addendum on the config: that’s my personal config, which morphs into my work setup. My work one actually turns off flymake and eldoc when using eglot.
Is there anything that has prevented a Neovim-style rewrite of Emacs? A Neomacs?
I keep hearing about the byzantine C-layer of Emacs and the slowness of Elisp. And Emacs definitely has the community size to develop such an alternative.
Why do you think no one has attempted such an effort? Or maybe I should add “recently” to the question. As I know there are other Emacs implementations.
As crusty as Emacs source can be, it’s nowhere near as bad Vim source was, which was a rat’s nest of #ifdef. That’s why Neovim had to basically rewrite their way to a fork. The Emacs implementation is surprisingly clean, as long as you can tolerate some of the aged decisions (and GNU bracing).
There is Climacs, which isn’t exactly the same, but is close.
The problem for any new Emacs clone will that it has to run all the Elisp out there. Unless there is a substantial speed improvement to Elisp or a very good automatic translation tool, any such project will be doomed from the start.
The interesting part about Copilot is precisely that it usually doesn’t just copypaste. I wouldn’t go so far as to say it has true understanding of code flow, but its level of understanding is somewhere between Ctrl-V and a human.
So roughly the same level of understanding as a human that only takes a cursory glance at the problem and that uses Ctrl-V as their main tool. Which is the point of the article.
Now do you see why I tell this story in relation to a question about Copilot? It’s like, yes, there’s now a coder tool which does this. But… there have been “coder tool” people who have been ripping off code and jamming it deep into their company’s code bases for a very long time.
So there are worries that a dumb tool like Copilot bypasses the safeguards that companies put up to prevent using copyrighted code, but the author points out that these safeguards have been bypassed by lazy coders and their Ctrl-V for ages.
That’s true. But then, at a certain level of lazy ripoff it stops being copyright violation and starts edging into creativity anyways. Wine may ask their developers to avoid reading the leaked Windows source code, but I don’t know of any copyright suit that was ever won on such a basis, rather than demonstrating substantial actual replication. And especially at the scale level and usecases of Copilot, the creativity of the code it (rips off / learns from) is usually already marginal to start.
Cases like Carmack’s fast inverse square root aside. But again, everyone rips that one off. :)
It doesn’t magically remove copyright. I imagine bit of a spectrum here:
Ideas reduced to concrete form are protected by copyright
Derivatives of those concrete forms are sometimes protected by copyright, or have varying degrees of protection
…lots of space here…
And all the way at the other end, you have pure ideas, of the sort that people learn by going to school and working as developers and all the other ways we learn. This is outside of the realm of copyright, but is in the realm of patents.
Copilot inhabits some uncharacterized range in between, and anything it provides to you might have been taken from any point along that spectrum. It could have copied the code. It could be a derivative of a copy. It could be something so derivative that the only connection remaining is “concepts and understanding of code”, or whatever that looks like for a machine.
So if you’re lucky and its output truly is from the “ideas” end of the spectrum, then yes it has “removed copyright” in the same way that a human mind that has learned about code is not violating the copyright of what the human read. But it might be from the other end, and you don’t know and can’t prove it either way. Not for yourself, not for your company, not in a court. At least not without some serious forensic analysis that costs a bunch of money.
The result is that everything is contaminated with Shroedinger’s copyrighted-ness and you have no way of telling. (Not to mention what patents you might be violating anywhere along that spectrum.)
Knowing judges/lawyers I’d be more inclined for them to go with: cute idea, the entire thing is copyright laundering by another name. That “a machine” does it for you with statistics doesn’t seem to obscure the end result.
I find the parallel between IP as a narrow waist of the internet and bytes/text as the narrow waist of shell commands a bit lacking. An email client or web browser never has to concern itself with IP, as pointed out in the article, whilst Unix command line tools all drop down to the rawest of formats with no way to negotiate a higher level protocol like SMTP or HTTP.
I don’t think any of them support a Content-Type. But the point is that you could build that if you wanted to! Pipelines are unstructured but you can give them whatever structure you want.
Every such project must include its own serialising/deserialising routines to pass that narrow waist, which is the equivalent of every email client and web browser implementing its own TCP/IP stack.
Not really all that parsing/etc… can just be a library call away a la https://www.unix.com/man-page/freebsd/3/libxo/ as an example for cli programs on freebsd to make a way to have structured output as well as normal text output. That programmers choose to serialize/deserialize all the time in bespoke ways is on them.
Hm I don’t agree, because sockets also work on bytes. So if you have a word document, to send it across the network, you have to serialize it. Ditto for a spreadsheet, a 3D model, etc.
I think the more accurate analogy is if the kernel didn’t have pipes or files at all! And applications had to “fend for themselves” to interoperate.
Actually you have a very good example of this with iOS and Android! Have you ever seen the “Send With” or “Share” buttons in mobile apps? They ask you to share to APPLICATIONS directly – share with text message, with Google hangouts, etc. It’s an M x N explosion although I’m not sure if they implement with M x N code or some kind of plugin system (Intents or whatever?)
So these OSes are not data-centric. Unix is data centric.
I think you are suggesting that the kernel have support for structured data. There is no other way to avoid parsing and serializing.
However my basic point is that nobody can agree on what the narrow waist is. Every language chooses a different one.
This might be on the blog, but here are very long discussions here with the authors of Elvish and Rash, two alternative shells.
My basic viewpoint is that the shell needs support for structured data, and Oil is getting that. It will have routines to make parsing easier, so every shell script doesn’t have to parse.
But the structure is optional and “projected” or “backward compatible” with byte streams, via JSON, QTT (TSV upgrade), and HTML.
So Oil doesn’t dream of having structured data in the kernel, unlike other shells … But that seems mostly academic because I don’t think they will get it :-/ Also even if the kernel had support, it’s impossible for sockets to grow support. You must parse and serialize when using sockets. So you would end up with 2 tiers again!
Slogans that will appear on the blog:
The lowest common denominator between a Elvish, Rash, and nushell program is a shell script (operating on byte streams). Because they don’t agree on their data model!
This is not supposed to be a criticism of those shells – they could be more convenient than shell or Oil in dealing with certain types of data in certain domains. But I predict the need to bridge them is real and not theoretical and would be done with shell scripts.
The lowest common denominator between a Racket, Clojure, and Common Lisp is a shell script as well (operating on byte streams). Not s-expressions! Because again they don’t agree on the model.
Byte streams are the model that everyone agrees on, in part because the kernel and network support it, but I think it’s more fundamental than that even.
It requires O(M + N) code to parse and serialize, not O(M x N).
Again, practically speaking, Oil will help you with this problem. It’s the only Bourne-style shell with JSON support, etc.
This is great, the first time in a while that I read something about Nix and come out less confused than I was going in.
Flakes seems like a great tool. Normally, when I hear someone say they use Nix with $TOOL, my reaction is “I have a hard enough time figuring out Nix on its own, tyvm”. Why is Flakes its own thing? And what is experimental about it?
Flakes is a thing because the model that Nix uses kinda accidentally assumed that the whole world could be a part of nixpkgs. Turns out that’s sadly not the case, but this explains why external dependencies (such as random git repos such as my website’s source code) were never really considered as a core part of the model. Not to mention just giving people a .nix file to build the software is basically like giving someone a Dockerfile. A Dockerfile will get you to build the software yeah, but overall it doesn’t give you details on how to use it.
I believe that the main innovation with flakes is that it doesn’t just give you build instructions, but also it gives you a way to give out usage instructions with the software. This is the same way that a docker-compose.yml file gives you instructions on how to use stuff that is built with docker.
I’m pretty sure that flakes are still experimental because they want to get feedback about it and potentially make breaking changes to the implementation at some point. However, from my view the net improvements on giving more semantic information and direct ways to use things (eg: nix run github:Xe/gohello) are so great that this kind of thing really needs to be in the hands of more people. In my decision calculus it’s a case of the benefits outweighing the risks.
I’m going to use this style of teaching people how to Nix in the future. I think the next post is purely going to be on how to write Nix packages (and split up a bigger package into a few smaller ones) using way too many contrived Go programs as examples. I may also work in Xeact and Xess. IDK, still working out the formula. This is the kind of style that I’d like to use for a book on Nix/NixOS if I ever get my ADHD to let off enough to let me commit to a longer term project like that.
To pivot off of the note there on nix run, I’ll give an example, nixpkgs doesn’t have htwatch yet (I might submit a pr for it who knows), but if you want to run it anywhere with nix, aka darwin/linux with nix you can run this:
https://github.com/mitchty/nixos/blob/master/flake.nix#L44-L71
nix run github:mitchty/nixos#hwatch -- -n 300 -N -d somecommandorwhatever
And It will build the same hwatch I am running at home now. I’m working on porting pikvm as well in that repo (i got side tracked so… don’t expect anything soon) but I love having nix flakes for a bit of a mini “personal nixpkgs” repo where I can test out experiments easily.
I do have gripes about some of this mostly with the command MUST be the same as the package name, which isn’t quite good when you’re testing out something that has a name like blahfs and has a subcommand blah.mount that you want to just test out but I just haven’t looked at how hard that is to do or not.
For my personal tooling I use flakes for individual repositories b/c the built-in template and integrations (like poetry2nix for python applications) is a direct upgrade to what we had before.
For my nixos configurations though I dislike using flakes. My main gripe with flakes is that it effectively becomes the entrypoint or ‘main’ of the configuration, which it makes it annoying to use with other tools like morph and the outputs section forces a certain structure to the code where I would prefer it work like niv where it sets up dependencies but the end result is just imports to your regular code.
@cadey has your position on flakes changed? My earlier interpretation from earlier writing is that you favored tools like niv and morph which eschew the flake model completely. I also see that nix.dev still recommends niv (https://nix.dev/tutorials/towards-reproducibility-pinning-nixpkgs) so wondering what the community thinks about flakes in general
My position on flakes is that they are slightly less terrible than using NixOS normally. They ain’t perfect but they could be so much worse than they are. I’m happy with the usability as-is. Also see here for more words about the kind of mindset change that I had once I hit the satori moment with flakes.
wondering what the community thinks about flakes in general
No idea about “the community” (I really hate those terms), but I found flakes easier to get into than anything that came before that always felt “tacked on like spaghetti stapled together”.
I’m doing more with nix now that I ever have now that flakes are “officially” here. Yes experimental but i now have a repo for my entire home config, I have that setup so I can build auto installing iso images, have secrets setup too, and build a system with my home config all in one deploy step via deploy-rs.
In my view flakes are already useful enough to make me not care about what came before which I avoided to be honest. Perfect no but good enough for government work as they say.
Apparently you can buy IPv6 addresses, use them for the servers on your home network, and then if you change your ISP, continue to use the same IP addresses?
You need to be a RIR (RIPE/ARIN/LACNIC/APNIC/AfriNIC) member for that. The membership fee alone is within thousands/year. Then you need to arrange routing with the hosting providers, and those that are ready to do that will also charge at least hundreds per month. No public cloud I’m aware of supports that at all, so you also need your own hardware in a datacenter where your transit provider is present.
In other words, owning your IPv6 network is completely out of reach for individuals and small projects. I believe it shouldn’t be that way and that RIRs are basically rent-seeking organizations now that resources they still can distribute (32-bit ASNs and IPv6 addresses) are anything but scarce, but I can’t see why it may change any soon.
In the RIPE area at least, you can obtain a provider-independent IPv6 assignment via an LIR - you don’t have to go directly to RIPE. A cheap option is Snapserv, who offer an IPv6 PI assignment for 99 EUR/year and an ASN for a one-off fee of 99 EUR. These can both be transferred to another LIR if, for example, Snapserv went out of business, or you wanted to switch LIR for some other reason. They also offer IPv6 PA assignments for less money, but the trade-off is that a PA assignment is tied to the LIR.
You do need to be multi-homed to justify the PI/ASN assignments, so you’d need to find another upstream provider in addition to Vultr. Someone I know uses Vultr and a HE tunnel to justify it.
In other words, owning your IPv6 network is completely out of reach for individuals and small projects. I believe it shouldn’t be that way and that RIRs are basically rent-seeking organizations now that resources they still can distribute (32-bit ASNs and IPv6 addresses) are anything but scarce, but I can’t see why it may change any soon.
I suspect the problem is routing tables. It would be trivial to assign every person a /64 without making a dent in the address space but then you’d end up with every router on any Internet backbone needing a few billion entries in its routing table. That would completely kill current (and near-future) hardware. Not to mention the fact that if everyone moving between ISPs required a BGP update, the total amount of BGP traffic would overwhelm networks’ abilities to handle the update rate.
You need some mechanism to ration the number of routable networks and money tends to be how we ration things.
I doubt this will ever be a problem in practice. Even among those who host their own servers, the number of people who want to own their address space is always going to be small.
I’m also not advocating for making addresses free or charge, only for making them available for less than the current exorbitant prices that RIRs charge for membership.
TIL, that’s really interesting. I just remember many, many years ago that people were entertaining this, but also with Sixxs and HE tunnels that kinda worked for a while.
Yes of course, but at least the way it works you could in theory use it longer despite switching ISPs. And I think my Sixxs account was nearly a decade old at the end. Some people might have moved cities three times in that time.
welp for some reason my ISP provides every customer a /64, I don’t know what the reason for that is. There is no single person the internet that needs a /64 and I’m certain no german household needs. But yeah waste tons of network space for no reason. IPv8 we’re coming..
Its the minimum routing size and if you stray from it a lot of the protocol breaks, making it smaller would be insane. And its not wasteful, you could give every atom on the planet a /64, the address space is REALLY BIG. Ipv4 this is not. For it to show up in BGP it needs to be a /48, /32 is the minimum allocation. And there is as many of those as there are ip’s. It should be a /48 you’re given actually, not a /64 (or /60 /56 in comcast home/business cases)
Why do you believe ipv8 is needed because of /64 allocations? Can you back that up with some numbers?
I haven’t done the math but I’ll let the last apnic report speak for itself in that regard (you’ll have to serach, its long and there’s no way to mark some chapter).
However, before we go too far down this path it is also useful to bear in mind that the 128 bits of address space in IPv6 has become largely a myth. We sliced off 64 bits in the address span for no particularly good reason, as it turns out. We then sliced off a further 48 bits for, again, no particularly good reason. So, the vastness of the address space represented by 128 bits in IPv6 is in fact, not so vast.
And
Today’s IPv6 environment has some providers using a /60 end site allocation unit, many using a /56, and many others using a /48
So It’s not really a standard that breaks things, because then things would already break.
I just don’t see a reason why we’re throwing away massive address ranges, even my private server gets a /64, and that’s one server, not a household or such thing.
The main reason your LAN must be a /64 is that the second half of each address can contain a MAC address (SLAAC) or a big random number (privacy extension).
So It’s not really a standard that breaks things, because then things would already break.
For routing, not in general, but going below /64 does break things like SLAAC. The original guidance was a /48, its been relaxed somewhat since the original rfc but can go down to a /64. Doing work or i’d pull up the original rfc. Going below /64 does break things, but not at that level being referenced.
I just don’t see a reason why we’re throwing away massive address ranges, even my private server gets a /64, and that’s one server, not a household or such thing.
Have to get out of the ipv4 conservation mindset, a /64 is massive yes, but 64 bits of address is ipv4 to the power of ipv4, that is… a large amount of space. It also enables things like having ephemeral ip addresses that change every 8 hours. Its better to think of a /64 as the minimum addressable/routable subnet, not a single /32 like you would have in ipv4. And there is A LOT of them, we aren’t at risk of running out even if we get allocation crazy. And thats not hyperbole, we could give every single device, human, animal, place, thing a /64 and still not approach running out.
Today’s IPv6 environment has some providers using a /60 end site allocation unit, many using a /56, and many others using a /48
Also just realized that you might be confusing /60 or /56 as being smaller than a /64, its unintuitive but this is the mask of the subnets not the size. So smaller than /64 would be a CIDR above 64, not below. aka a /96 would break in my example. Its also why assigning “just* a /64 is a bit evil on the part of isp’s and the allocation should be larger.
For a while (see: SO question) I’ve been curious about “dehydrating” a fully-pushed repo into either a script or some metadata for restoring it (as in, with the same remotes, maybe branches). Has anyone seen existing tooling for this?
(Not urgent, so it needn’t be done/polished. I took a swing at some Shell scripting for it, though I put it on hold when I realized the logic for when to actually use it or not wasn’t quite as automatable as I’d hoped. I’ll need to do some more thinking/designing there before it’s actionable.)
The best backups are the ones that happen automatically in the future.
Add one for GitHub, one for sr.ht, and one for a local NAS or ssh server. Now, whenever you push, you have two remote services and a local service storing it. The local NAS can be a single raspberry pi with the default 8GB SD card for most people.
This is a good solution, but on the one repo I used it with I switched to a different setup. I now use a local gitea to pull from sr.ht and push that same repo to GitHub, with my origin only pushing to sr.ht.
Why do you need xargs here at all? Just add more remote urls to the same remote name and then git push remote –all. Programmers seem to love to overcomplicate this stuff with layers that aren’t needed.
git remote set-url –add remote some/url
Also setup backups, this remote nonsense won’t do jack for not yet committed stashes or changes.
asdf works fine for pinning runtimes until you have system libraries, etc that extensions link against which aren’t versioned with asdf. Then you’re back in the same boat as you are with brew, etc where upgrading might break existing workdirs.
It sounds like Nix can also build your containers for you based on your project definition?
Yep basically just something like this, lots of assumptions given with this and that you “want” to containerize the hello world program gnu ships but eh its an example:
There are caveats to using the docker builds (can’t build on macos) and you’ll need to learn the nix programming language at some point but its a rather droll affair IME once you get that its all just data and functions. And before you ask why is it so big, the short answer is everything that hello defined it depends on is included, which includes jq/pigz/jshon/perl/moreutils etc… for some reason. But its basically lifted straight out of the nix store verbatim.
everything that hello defined it depends on is included, which includes jq/pigz/jshon/perl/moreutils etc… for some reason
I recognise this list. These are the dependencies used in the shell scripts which build the Docker image. They shouldn’t be included in the image itself.
Of course if I used musl libc than glibc and its dependencies would go away automatically.
What’s better is that if you use buildLayeredImage each of these is a separate layer so that rebuilding for example the word list, or the binary doesn’t require rebuilding other layers. (This is actually better than docker itself because docker only supports linear layering, so you would have to decide if the word list or the binary is the top layer and rebuilding the lower would force a rebuild of the higher one.)
Without it, technically the government can just seize corporate assets as an example.
Also legal personhood != human person even under the law. They are distinct. Note this all is from my laywer drinking buddy but the one time I asked him about it he had a bit of a sigh that the nuances are a lot more complex than “corporations are people under law” like it was some sort of soylent green situation. The Tillman act as an example is a good example of why you want personhood for businesses. Unless we want to not be able to collectively barter as a group I would argue that is a good thing.
Out of curiosity, why is corporate personhood necessary to prevent arbitrary seizure of corporate assets given that a natural person (human) owns the corporation and therefore owns the assets? It seems like that would have been a much simpler way of making that work. So it feels like there must be a lot more to it.
The author of the post works at Cray, so he would have access to the very best Fortran developers in the world (along with the best compilers and OSes for running it) and knows exactly why climate models are still done in Fortran. He also knows how to write a headline that gets a lot of clicks :).
I used to work there, and got to bs with Bill Long at lunch about Fortran (not the seattle office). Talking to Fortran compiler guys is fun.
Fortran is cool in that the Cray Fortran compiler had ENV switches for, well basically everything, and i’m not kidding. So its “easy” to tune every single compilation to what you need to do. That and you can mix old and new fortran into the same binary. Try that with c++/c. Rust is only approaching what fortran longevity has had in the past.
The chief complaint with Fortran is that it’s unreadable but also that its buggy and slow, which it is. Being less readable than ASM is just the cherry on top of the mound of ****.
The very benchmarks the author cites show that Fortran is slower than C++ and Rust in all tasks and slower than C in all but one. Conceptually Rust, C++, Fortran and C can aim to have about the same speed, C and C++ have the advantage in terms of native APIs and ABIs designed with them in mind, Rust has the advantage of safety when doing weird stuff, C++ has the advantage of library ecosystem, compile-time abstraction and abstraction in general, C has simplicity. Fortran is inferior to all 3 in any way imaginable.
It should be noted that the speed difference here is very significant, with fortran being 2-10x times slower than the leader on virtually all problems and when the results are taken in aggregate, it barely beats C# (you know, the whole .NET language made to develop web apps?): https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html
And again, these are the benchmarks the author reference, so I assume they are better than average.
Fortran only gets written in fields that are status-driven, where skill and merit are irrelevant, it’s used to keep the “old guard” into academic positions.
If you want to know what the guys that need efficiency are doing (AWS, GCP, Cloudflare, Dropbox… etc), they are doing C++ and some C and some are considering using some Rust, if you want to know what the guys that need fast complex math are doing (Hedge funds, exchanges, brokers, applied ML services… etc) they are doing C++ and some C.
Nobody is arguing for turning Fortran code into python, they are arguing for turning it into code that works as intended, can be read by normal humans and compiles to something that is 2-10x times as fast.
The chief complaint with Fortran is that it’s unreadable but also that its buggy and slow, which it is. Being less readable than ASM is just the cherry on top of the mound of ****.
The very benchmarks the author cites show that Fortran is slower than C++ and Rust in all tasks and slower than C in all but one. Conceptually Rust, C++, Fortran and C can aim to have about the same speed, C and C++ have the advantage in terms of native APIs and ABIs designed with them in mind, Rust has the advantage of safety when doing weird stuff, C++ has the advantage of library ecosystem, compile-time abstraction and abstraction in general, C has simplicity. Fortran is inferior to all 3 in any way imaginable.
It should be noted that the speed difference here is very significant, with fortran being 2-10x times slower than the leader on virtually all problems and when the results are taken in aggregate, it barely beats C# (you know, the whole .NET language made to develop web apps?): https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html
You know that LAPACK (itself based on BLAS which is also written in Fortran) is written in Fortran 90 right? Almost all scientific code running on every platform used by every academy, government, or corporation is using LAPACK through their programming language/runtime of choice. If you want to push scientific computing out of the Fortran orbit, there’s a lot of work to be done.
LAPACK is not very widely used for compute intensive problems, CUDA doesn’t support it, it support CUBLAS (which si written in C/ASM), so that’s 90% of high speed computing ruled out.
Then you have CPU based stuff, for which LAPACK will be used, but not the original implementation, but rather e.g. intel’s MK, or whatever the guys with supercomputers are using. LAPACK is more of a standard than a library, and people have their own implementations.
The fortran one is indeed used by… ahm? Who exactly? A few procedures on Android phones that lack GPUs?
I’m not talking about SOTA. I’m talking about, well, every other application of linear algebra. If you think everything but Android has moved to using CUDA and MKL, I think you need to take a broader look at academia and industry. ARM cores don’t even have MKL available.
I’m not sure if you’re intentionally dense, but let me try to take a stab at this again:
For applications where speed matters and where complex linear algebra is needed (e.g. what the author is talking about), NOBODY is using the default LAPACK implementation.
For applications where you need 2-3 LAPACK function calls every hour, LAPACK is a reasonable choice, since it’s lightweight and compatible on a lot of platforms and hard enough to write that nobody bothered to try and rewrite a free and open-source cross-hardware version.
However, 99% of usage is not 99% of devices, so the fact that LAPACK is running on most devices doesn’t matter, since this device run <1% of all LAPACK-interface based computations.
This goes back to my point of Fortran is slow and buggy, so people that have skin in the game don’t use it for mathematical applications. It might be reasonable for a simple decision tree used in bejewelled but not, e.g., for meteorological models (at least not if the meteorologists were concerned about speed or accuracy, but see the problem of responsibility/skin-in-the-game).
You seem to be attacking a strawman of my position here. I’m not arguing that libraries written in fortran don’t work anymore, just that they are very bad, very bad in that they are slower than alternatives written in Rust, C++ and C, and more on part with fast GCed language (e.g. C#, Java, Switf)
I’m not sure if you’re intentionally dense, but let me try to take a stab at this again:
Not everyone on the internet is trying to fight you, please relax. Assume good faith.
You seem to be attacking a strawman of my position here. I’m not arguing that libraries written in fortran don’t work anymore, just that they are very bad, very bad in that they are slower than alternatives written in Rust, C++ and C, and more on part with fast GCed language (e.g. C#, Java, Switf)
I was responding in particular to your claim that we should just rewrite these routines out of Fortran. When we need to use specific CPU or GPU features, we reach for MKL or CUBLAS. For applications that need the speed or CPU/GPU optimizations, this is fine. For all other applications, Fortran is sufficient. I’m only saying that rewriting Fortran into other languages is a lot of work for dubious gain. Folks that need the extra speed can get it. It sounds like we largely agree on this.
I am a bit dubious about your claim that Fortran is chased by academic status-seekers, because it makes the assumption that industry is smart and efficient while academia is bureaucratic and not-efficient and uses results from the Benchmarks game to highlight it. I realize TFA linked to these benchmarks, but I took a cursory glance at them, and some of them don’t even parallelize across CPUs and some don’t even compile! Given that Fortran is a niche language, an online benchmark shootout is probably not the best place. That said, I don’t have a justified idea of Fortran’s numbers either so I can’t do anything more than say I’m dubious.
Based in part on the comment orib made, and looking at some of the code myself, I actually agree with you that those benchmarks might not be a fair comparison.
I guess the thing I disagree on still is that there’s little purpose for most scientific applications if people can’t easily read and understand their code. Especially in domains where practical applications are 30-years away, I feel like it would make sense for 3rd parties to be able to validate the logic as easily as possible.
Much like I would protest to someone using mathematical notation from the late 19th century, even though in principal it would be sufficient for a vast swath of all equations being represented in physics.
The chief complaint with Fortran is that it’s unreadable but also that its buggy and slow, which it is.
Prove it, how is it buggy and slow for its domain? Give 3 examples for each of buggy and slow, should be easy for you with how determined your claim is. Showing benchmarkgames isn’t showing real world code/simulations in any language. Matrix multiplication is what you want to benchmark, why are you using benchmark “games”? Those “games” are hardly readable after people golf them to insanity. If your argument is that I should trust anything from those “games”, well I’ve got ice cubes in antarctica to sell you.
And demonstrate how the restrict keyword in c, which is relatively recent mind you can overcome what fortran can guarantee inherently. And how that affects aliasing and ultimately generated code and how your statement of fortran being inferior to all of c/c++/rust. Have you ever talked to a scientist writing fortran for simualtion? Ever ask them why? Methinks the answer to those last two questions is no. Because I have, and the people writing fortran for simulations are insanely smart, and bear in mind the goal isn’t 100% “it benchmarks faster than language xyz”, its “I run this simulation for $N months/weeks on end, it needs to be fast enough, and I’ll profile it to see what could be improved”. And for each supercomputer they rebuild and retune every library and object for the new processor as needed. I gotta be honest the amount of ignorance of the general computer programming population about how traditional HPC works is disheartening.
And how are simulations that are being run by scientists using the literal fortran COMPLEX type by definition not “guys that need fast complex math”? Fortran specifically handles complex math better than c. My mind boggles at some of the statements here.
Less supposition and hearsay and more demonstrable proof please, this comment is atrocious for basically being without any actual depth or substance to its claims.
I provided examples of fortran being bad within the terms the author set for himself. Have you heard of arbitrary demands for rigor? If every time an opinion you hold is challenge you start clamouring for more evidence instead of considering whether or not you have any counter-evidence, well, you see the problem.
Are academics with significant clout even writing code these days? I assume they should have a small team of grad students and postdocs to do that in their lab for them…. is is the fact that everything is written in Fortran the cause of them being forced to maintain old code?
Fortran only gets written in fields that are status-driven
Our industry isn’t immune to status-driven endeavors. Think of all the “bad” technology that gets adopted because FAANG uses it or the proliferation of flavor-of-the-month frameworks in the web space.
Even if his assertion were true, assuming that all ‘complex math’ is the same and a language that is good for one is good for another is a pretty good indication that you should disregard the post. For example, a lot of financial regulation describes permitted errors in terms of decimal digits and so to comply you must perform the calculations in BCD. If you’re doing scientific computing in BCD, you have serious problems. Conversely, a lot of scientific computing needs the full range of IEEE floating point rounding modes and so on, yet this is rarely if ever needed for financial analysis.
With C99, C grew the full support for floating point weirdness that you need to be able to port Fortan code to C, but so few people care about this in C that most C compilers are only now starting to grow acceptable support (GCC had it for a while, LLVM is gaining it now, in both cases the majority of the work is driven by the needs of the Fortran front end and then just exposed in C). Fortran has much stricter restrictions on aliasing, which make things like autovectorisation far easier than for C/C++. C/C++ developers now use modern OpenMP vectorisation pragmas to tell the compiler what assumptions it is allowed to make (and is unable to check, so miscompiles may happen if you get it wrong), whereas a half-decent Fortan compiler can get the same information directly from the source.
Along with the effort of rewriting, there’s also distrust of new models of rewrites of existing ones. There are decades of work in the literature publishing and critiquing results from existing models, and there are…not that for anything new. It’s much less risky and thus easier to accept to port an existing model to a new HPC platform or architecture than it is to adopt a new one.
Additionally, HPC is a weird little niche of design and practice at both the software and hardware levels. There are a few hundred customers who occasionally have a lot of money and usually have no money. Spinning that whole ecosystem around is difficult, and breaking a niche off the edge of it (if you decided to take climate modelling into Julia without also bringing petrochemicals, bioinformatics, automotive, etc) is a serious risk.
When NREL finally open sourced SAM (GitHub), one of the standard tools for calculating PV output based on location and other factors, a friend of mine decided to take a look at the code. On his first compilation, it was clear no one had ever built it using -Wall, it had thousands of warnings. When he looked more closely he could tell it has been translated, badly, from (probably) MATLAB, and had many errors in the translation, like (this is C++)
arr[x,y]
to access a 2D coordinate in arrays - for anyone playing at home, a,b in C++ means evaluate a, then evaluate b and return its result, so this code was accessing only the y coordinate.
This would be find if this was undergrad code, but this code had been around for a very long time (decades?), had dozens of papers based on it, and plenty of solar projects relied on it for estimating their ROI. I bring up this anecdote as a counterexample that the age of these libraries and programs does not mean they are high quality, and in fact their age lulls people into a false sense of trust that they actually implement the algorithms they claim to.
He’s since submitted many patches to resolve all the warnings and made sure it compiles with a number of compilers, but I wonder how valid the results over the years actually are- maybe they got lucky and it turns out the simplified model they accidentally wrote was sufficient.
All governments take actions because of these models. They do affect lives of every person on the planet ant future generations to come. “If it’s not broken, don’t fix it” approach doesn’t fit here. Rewriting these models is could be made for a cost of a few international conferences, flights and accommodations of participants.
critiquing results from existing models
The models should be scrutinized, no the results they give?
As someone that has interacted with people that write these models, optimistic is putting it lightly. I think whomever thinks that rewriting a bunch of fortran will be productive is entirely underselling both fortran, and the effort to make fortran super fast for simulations.
Rewriting this stuff in javascript isn’t realistic nor will it be fast. And any rewrite is going to have the same problem in 50 years. What are you going to rewrite it again then? How do you know its the same simulation and results?
Sometimes I think us computer programmers don’t really think through the delusions we tell ourselves.
But by rewriting we may check if the implementation follows the specification - see this as a reproductivity issue, do you recall when a bug in excel compromised thousands of researches? And by not changing anything we may find ourselves in a situation where no one knows how these models work, nor be able to improve them. Something similar to banking and cobol situation, but much worse.
In general, we don’t know, and in the abstract, that’s a problem (though specifically in the case of weather forecasting, “did you fail to predict a storm over this shipping lane” is a bigger deal than “did you predict this storm that actually happened for reasons that don’t stand up to scrutiny”, and many meteorological models are serviceably good). There are recent trends to push for more software engineering involvement in computational research (my own PhD topic, come back to me in three years!), and for greater discoverability and accessibility of source code used in computational research.
But turning “recent trends to push” into “international shift in opinion across a whole field of endeavour” is a slow process, much slower than the few international flights and a fancy dinner some software engineers think it should take. And bear in mind none of that requires replatforming anyone off of Fortran, which is not necessary, sufficient, desirable, feasible, or valuable to anyone outside the Rust evangelism strike force.
Most climate models are published in papers and are widely distributed. If you would like to remake these models in other languages and runtimes, you absolutely could (and perhaps find gaps in the papers along the way, but that’s a separate matter.) The problem is, getting the details right is very tough here. Is your library rounding in the same places the previous library was rounding at, in the same ways? How accurate is its exponential arithmetic? What’s the epsilon you need to verify against to be confident that the results are correct?
The article links CliMA as a project for Julia based climate models, but remember, most scientific computing libraries use Fortran one way or another. We’re just pushing the Fortran complexity down to LAPACK rather than up into our model code. Though that’s probably enough to greatly increase explainability, maintainability, and iterative velocity on these models.
“If it’s not broken, don’t fix it” approach doesn’t fit here.
What are you even trying to say here? If it’s not broken we should rewrite it because… reasons? It’s not broken, so why would we waste public money rewriting it when we could use that money to further improve its ability to help us?
Rewriting these models is could be made for a cost of a few international conferences, flights and accommodations of participants.
The article is low on these specific details and I’m not too familiar with climate modelling but I bet that the models aren’t just fancy Excel sheets on steroids – there’s at least one large iterative solver involved, for example, and these things account for most of the Fortran code. In that case, this estimate is off by at least one order of magnitude.
Even if it weren’t, and a few international conferences and flights were all it takes, what actual problem would this solve? This isn’t a mobile app. If a 30 year-old piece of.code is still in use right now, that’s because quite a stack of taxpayer money has been spent on solving that particular problem (by which I mean several people have spent 3-5 of their most productive years in solving it), and the solution is still satisfactory. Why spend money on something that has not been a problem for 30 years instead of spending it on all the problems we still have?
Depends on how they’re configured. If they’re configured to step (aka jump) they can. If you configure them to slew then they do the +/- ppm adjustment of roughly at most 500ppm.
The old ntpd client in most unix distros defaulted to step as a note. Well if i remember right. The issue with slewing though is that if your computer’s time say is wicked out of date, like January 1 1970, its impossible to eventually get your time in sync.
As if it was the only way it’s broken!
I’m not exactly hopeful for the future of IPv6, but at least the first part can be fixed by regulation in more-or-less normal countries.
The post has at leaset some connection to “the IPv6 internet” and why it is broken[0], but this sounds more like mosts IPSs are broken.
So your ISP don’t give you proper access to the internet? In other words your ISP is broken not the IPv6 Internet.
Most ISPs I know (german home ISP, I don’t know about mobil) give you a /56 via DHCP-PD. Yes the change the prefix regularly for privacy reasons. On request you get a static prefix. For the most private (and smal non IT business) this exactly what they need.
You know about ULAs (private IPv6 address space) and DHCPv6? Setup your local service to do dhcpv6 and configure a ULA prefix on your network. This works the same way as in IPv4 and has also the same downsides (clients can’t dhcp, only have static dns servers, …).
Again the ISP don’t provider proper IPv6 internet and not the IPv6 internet is broken.
Yes Countries with Internet censorship censor the Internet. Still don’t see how this means the Internet is broken.
Yes there are Issues with ISPs providing proper Internet (v4 and v6), but this doesn’t mean the Internet is broken. It bugs me when it is claimed inverted. Because then the details get lost over time and some people only remember ipv6 is bad. Also most of the hacks we have in place for working ipv4 are often forgotten.
[0] but the question is: is ipv6 is the cause of the problem?
This isn’t worse than the situation with IPv4, where a lot of ISPs give you a new IP address and NAT your network. Or, worse, add carrier-grade NAT on top of local NAT.
Oh, and you aren’t restricted to using v4 internally. IPv6 was designed to support multiple IP addresses per client device, so you can have a static address and (changing) globally-reachable address pretty easily with most mainstream operating systems.
… why would you use your public IPv6 address for LAN communication? You can have multiple v6 addresses per interface, use private v6 addresses for LAN same as you do for private v4 addresses. Doesn’t require NAT since you also have the global v6 address for internet things
Yeah I’m confused as well, this is what site-local ipv6 addresses are for, to decouple your public ipv6 from your site ipv6. Or I guess now just use unique local addresses instead, same diff. Think ipv4 is polluting people’s minds into thinking you need to have a single ip address per machine. ipv6 is really not like ipv4 and the more you try to treat it that way the more you start just repeating the sins of the past in it.
Bonus is with ipv6 ULA’s you get to know exactly who you’re connecting to to boot.
https://datatracker.ietf.org/doc/html/rfc4193
If you want to route them you can even with a /64 from upstream if they’re not playing via the rfc and providing at least a /56 or /48. But even a /64 is pretty gihugic.
It may be worth noting here that keeping Ctrl-C functional is quite tractable in Haskell thanks to its async exception support and its structured parallelism/concurrency utilities. In a Haskell program, the default SIGINT handler will just throw an async exception (just like the ones you can throw to a running thread) to one of the running (lightweight) threads and as long as you follow some established best practices not to swallow async exceptions and use the standard structured concurrency utilities that propagate exceptions to your parent thread, your program will exit gracefully collecting all the resources and print any shutdown messages etc.
In my experience, all the Haskell programs I’ve interacted with seemed to follow these best practices since they handle Ctrl-C gracefully.
The article is not advocating for Ctrl+C to shut down the program, it’s advocating for Ctrl+C to pause/reset the still-running program and allow execution to continue from that point, with the program’s running state at the moment of the Ctrl+C preserved as much as possible.
Thats not what SIGTERM is for though, for that there is SIGSTOP/SIGCONT. If the goal is to use SIGTERM wrong I’m not sure this article is worth much.
Ctrl+C sends
SIGINT.Ah yeah brainfart for some reason I was thinking TERM. Still don’t see a point to it compared to SIGSTOP/CONT though, used those for years to pause an executable and resume it later (notably firefox) to save battery.
The only problem with lots of custom aliases (or custom keybindings in other programs like editors), is that the muscle memory burns you every time you have to work on a remote machine. I used to go custom-to-the-max with my config, but I’ve gradually shifted back to fewer and fewer aliases except for the most prevalent build/version control commands I run dozens of times each day.
When I need to remote into machines where I can’t set up my shell profile for whatever reason, I just config ssh to run my preferred shell setup commands (aliases, etc) as I’m connecting.
My tools work for me, I don’t work for my tools.
You mean, could single session only? Care to share that lifehack? I’m assuming something in ssh_config?
Yeah, single session only. There are a bunch of different ways to skin this cat — LocalCommand and RemoteCommand along with ForceTTY in ssh_config can help.
Conceptually you want to do something like (syntax probably wrong, I’m on my phone)
scp .mypreferedremoterc me@remote:.tmprc; ssh -t me@remote “bash —rcfile ~/.tmprc -l; rm .tmprc”
which you could parameterize with a shell function or set up via LocalCommand and RemoteCommand above, or skip the temp file entirely with clever use of an env variable to slurp the rc file in and feed it into the remote bash (with a heredoc or SendEnv/SetEnv)
every time i have to work on a remote machine i do the commands through ssh or write a script to do it for me.
naming a meta-archive-extracter, “atool” doesn’t help either. OP used
unzipfor this but it is overloaded.uncompressalso is taken.What word would you guys use for aliasing it?
I use extract as a function that just calls the right whatever based on the filename.
I think prezto comes with
xalias, and I like it a lot. It’s burns easily into the muscle memory.To defeat muscle memory when changing tools, I make sure the muscle memory command fails:
alias unzip = “echo ‘use atool’”
It doesn’t take many times to break the muscle memory. Then I remove the alias.
Is
atoolthere by default on Linux boxes?Nope. At least I’m not aware of any Linux distro installing it by default.
But being installed by default is IMHO totally overrated. The main point is that it is available in many Linux distribution’s repos without having to add 3rd party repos—at least in Debian and all derivatives like Devuan, Kali oder Ubuntu.
I understand, but it’s not the same. If I don’t have a shell regularly there, and not my own dotfiles, I likely want to avoid installing and removing system packages on other people’s systems. When stuff breaks, I want the minimum amount of blame :)
Not that this is not a useful tool.
Ok, granted. Working as a system administrator it’s usually me who has to fix things anyway. And it happens only very, very seldom that something breaks just because you install a commandline tool. (Saying this with about 25 years of Linux system administration experience.)
Only
zutilscan theoretically have an impact as it renames commandline system tools and replaces them with wrappers. But so far in the past decade, I’ve never seen any system break due tozutils. (I only swa things not working properly because it was not installed. But that was mostly because I’m so used to it that I take it as given that zutils is installed. :-)Yep, different role. I did some freelance work a long ago, and learned on (fortunately) my predecessor’s mistake: they hired me to do some work, because I guess someone before me updated some stuff, and that broke… probably PHP version? Anyway, their shop didn’t work any more and they were bleeding money till I fixed it. It was one of my early freelance jobs, so that confirmed the age-old BOFH mantra of if it ain’t broke, don’t fix it. So given time, I would always explicitly ask permission to do this or that or install the other, if needed.
But I went a different route anyway, so even though I am still better than average, I think, I’m neither good nor professional. But I think old habits die hard, so that’s why I’m saying “if this stuff isn’t there by default, you’ll just have to learn your tar switches” :)
Note that this doesn’t apply for eshell as the OP is using: If you cd to a remote machine in eshell, your aliases are still available.
Command history and completion suggestions have really helped me avoid new aliases.
I was curious about that one recently. On a Mac if I need to install something, I just do it and if it’s big enough I’ll check back in 10min - it mostly works. (ignoring all the other problems) Maybe run gc once a week and clean up 40GB of trash. But I wouldn’t want to attempt that on an RPi class hardware. How do people deal with that? Some external compilation service/cache? I mean cases where it turns out you’re about to compile llvm on your underpowered Celeron based NAS.
Basically yeah, its pretty easy to setup and defer to something else to build: https://sgt.hootr.club/molten-matter/nix-distributed-builds/
Unless I’m misunderstanding, you want to target revisions of nixpkgs channels which are already built as per: https://status.nixos.org
I’m on a rolling release, but it’s not perfect. Specifically, some updates are not built yet at the time of installation. Also changing since configs means recompiling anyway (for example if you want to enable jit in Ruby)
One pitfall I ran into when using a stable release was using
release-21.11which also includes upgrades which are not yet built, switching tonixos-21.11solved that.Great article… In other words, Nix is a leaky abstraction and it bottoms out at shell scripts, e.g. a wrapper to munge the linker command line to set RPATH to /nix/store.
This doesn’t appear in your code / package definitions and is hard to debug when it goes wrong.
Nix was designed and developed before Linux containers, so it had to use the
/nix/storemechanism to achieve its goals (paths should identify immutable content, not mutable “places”).So I wish we had a Nix like system based on containers … (and some kind of OverlayFS and /nix/store hybrid) Related: https://lobste.rs/s/psfsfo/curse_nixos#c_ezjimo
https://lobste.rs/s/ui7wc4/nix_idea_whose_time_has_come#c_5j3zmc
I don’t, I like running nix on macos. How would your “solution” of running linux binaries and filesystems on macos work exactly? This whole blog post amounted to using the wrong lld, I don’t see how “lets use containers” is a good fix for the problem at hand.
this is the proper sentiment
See this reply … I have a specific problem of running lightweight containers locally and remotely, and transfering them / storing them, and am not trying to do anything with OS X:
https://lobste.rs/s/vadunt/rpath_why_lld_doesn_t_work_on_nixos#c_pb8cpo
The background is that Oil contributors already tried to put Oil’s dev env in
shell.nix, and run it on CI.However my own shell scripts that invoke Docker/podman end up working better across Github Actions, sourcehut, local dev, etc. Docker definitely has design bugs, but it does solve the problem … I like that it is being “refactored away” and orthogonalized
Its not though, you’re not solving the problem for “not linux” if you put everything in docker/containers. nix the package manager is just files and symlinks at the end of the day. It runs on more than just linux, any linux only “solution” is just that, not a solution that actually fits the problem space as nix. Which includes freebsd, macos, and linux.
And you can create containers from nix package derivations, and your linked comment doesn’t really explain your problem. I’ve used nix to create “lightweight” containers, the way nix stores files makes it rather nice as it avoids all that layering rubbish. But without a clear understanding of what exactly you’re talking about it really seems to be unrelated to this post entirely. How do you test Oil on macos/freebsd via CI? Even with those operating systems and nix, its its own world so you’d still have to test in and out of docker. Or am I misunderstanding? I’m still unclear what problem you are trying to accomplish and how it relates to rpath in a linker on nix and how containers solve it.
Yeah it’s true, if you have a OS X or BSD requirement then what I’m thinking of isn’t going to help, or at least it has all the same problems that Docker does on OS X (I think it runs in a VM)
The history of the problem is long, it’s all on https://oilshell.zulipchat.com/ and there is a shell.nix here
https://github.com/oilshell/oil/blob/master/shell.nix
which is not widely used. Instead the CI now uses 5 parallel jobs with 5 Dockerfiles, which I want to refactor into something more fine-grained.
https://github.com/oilshell/oil/tree/master/soil
I would have liked to have used Nix but it didn’t solve the problem well … It apparently doesn’t solve the “it works on my machine” problem. Apparently Nix Flakes solves that better? That is, whether it’s isolated/hermetic apparently depends on the build configuration. In contrast Bazel (with all its limitations) does pretty much solve that problem, independent of your build configuration is.
I think the setup also depended on some kind of experimental Nix support for Travis CI (and cachix?), and then Travis CI went down, etc.
I would be happy if some contributor told me this is all wrong and just fixed everything. But I don’t think that will happen because there are real problems it doesn’t address. Maybe Nix flakes will do it but in the meantime I think containers solved the problem more effectively.
It’s better to get into the details on Zulip, but shells are extremely portable so about 5% of the build needs to run on OS X, and that can be done with tarballs and whatnot, because it has few dependencies. The other 95% is a huge build and test matrix that involves shells that don’t exist on OS X like busybox ash, and many many tools for quality and metaprogramming.
Shells are also very low level, so we ran into the issue where Nix can’t sandbox libc. The libc on OS X pokes through, and that’s kind of fundamental as far as I can tell.
This problem is sufficiently like most of the other problems I’ve had in the past that I’m willing to spend some time on it … e.g. I’m interested in distributed systems and 99% of those run Linux kernels everywhere. If you’re writing OS X software or simply like using OS X a lot then it won’t be interesting. (I sometimes use OS X, but use a Linux VM for development.)
Dynamic linking to immutable content isn’t actually dynamic linking, its more like “deferred linking”. Optimize the deferred part away and you are back at static linking. The next optimization is to dedup libraries by static linking multiple programs into multicall binaries.
Do containers allow any kind of fs merging at the moment? I.e. similar results to overlayfs or merged symlinks dir from nix itself? I thought namespaces only allow basic mapping, so I’m curious where they could help.
Yeah I think you might want a hybrid of OverlayFS and bind mounts. There was an experiment mentioned here, and it was pointed out that it’s probably not a good idea to have as many layers as packages. Because you could have more than 128 packages that a binary depends on, and the kernel doesn’t like that many layers:
https://lobste.rs/s/psfsfo/curse_nixos#c_muaunf
Here is what I’m thinking with bind mounts. I haven’t actually done this, but I’ve done similar things, and maybe some system already works like this – I wouldn’t be surprised. Let me know if it has already been done :)
Say you want to install Python 3.9 and Python 3.10 together, which Nix lets you do. (As an aside, the experiment distri also lets you do that, so it also has these RPATH issues to solve, mentioned here: https://michael.stapelberg.ch/posts/2020-05-09-distri-hermetic-packages/ )
So the point is to avoid all the RPATH stuff, and have more a “stock” build. IME the RPATH hacks are not necessarily terrible for C, but gets worse when you have dynamic modules in Python and R, which are shared libraries, which depend on other shared libraries. The build systems for most languages are annoying in this respect.
So you build inside a container, bind mounting both the tarball and the output /mydistro, which is analogous to /nix/store. And then do something like:
But on the host side, /mydistro/python is actually /repo/python-3.9 or /repo/python-3.10.
So then at runtime, you do the same thing – bind mount /repo/python-3.9 as /mydistro/python
So then on the host you can have python 3.9 and 3.10 simultaneously. This is where the package manager downloads data to.
But apps themselves run inside a container, with their custom namespace. So with this scheme I think you should be able to mix and match Python versions dynamically because the built artifacts don’t have version numbers or /nix/store/HASH in their paths.
I would try bubblewrap first – I just tried it and it seemed nice.
So the runtime would end up as something like
and
There are obviously a bunch of other issues. The distri experiment has some notes on related issues, but I think this removes the RPATH hacks while still letting you have multiple versions installed.
If anyone tries it let me know :) It should be doable with a 20 line shell script and bubblewrap.
I actually have this problem because I want to use Python 3.10 pattern matching syntax to write a type checker! That was released in October and my distro doesn’t have it.
Right now I just build it outside the container, which is fine. But I think having apps explicitly limited to a file system with their dependencies mounted has a lot of benefits. It is a middleground between the big blob of Docker and more precise dependencies of Nix.
The problem with overlay / union filesystems is that the problem that they’re trying to solve is intrinsically very hard. If a file doesn’t exist in the top layer then you need to traverse all lower layers to try to find it. If you can guarantee that the lower layers are immutable then you can cache this traversal and build a combined view of a directory once but if they might be mutated then you need to provide some cache invalidation scheme. You have to do the traversal in order because a file can be created in one layer, deleted in a layer above (which requires you to support some notion of whiteout: the intermediate FS layer needs to track the fact that the file was deleted), and then re-added at a layer above. You also get exciting behaviours if only part of a file is modified: if I have a 1GiB file and I modify the header, my overlay needs to either copy the whole thing to the top layer, or it needs to manage the diff. In the latter case, this gets very exciting if something in the lower layer modifies the file. There are a lot of corner cases like this that mean you have to either implement things in a very inefficient way that scales poorly, or you have surprising semantics.
This is why
containerduses snapshots as the abstraction, rather than overlays. If you have an overlay / union FS, then you can implement snapshots by creating a new immutable layer and, because the layer is immutable, you won’t hit any of the painful corner cases in the union FS. If you have a CoW filesystem, then snapshots are basically free. With something like ZFS, you can write a bunch of files, snapshot the filesystem, create a mutable clone of the snapshot, write / delete more files, and snapshot the result, and so on. Each of the snapshot layers is guaranteed immutable and any file that is unmodified from the previous snapshot shares storage. This means that the top ‘layers’ just have reference-counted pointers to the data and so accesses are O(1) in terms of the number of layers.The one thing that you lose with the snapshot model is the ability to arbitrarily change the composition order. For example, if I have one layer that installs package A, one that installs packages B on top, and one that installs package C on top, and I want a layer that installs packages A and C, then I can’t just combine the top and bottom layers, I need to start with the first one and install package C. Something like Nix can probably make the guarantees that would make this safe (that modified in the middle layer is modified by the application of the top layer), but that’s not possible in general.
Hm yeah I have seen those weird issues with OverlayFS but not really experienced them … The reason I’m interseted in it is that I believe Docker uses it by default on most Linux distros. I think it used to use block-based solutions but I’m not entirely clear why they switched.
The other reason I like it is because the layers are “first class” more amenable to shell scripting than block devices.
And yes the idea behind the “vertical slices” is that they compose and don’t have ordering, like /nix/store.
The idea behind the “horizontal layer” is that I don’t want to bootstrap the base image and the compiler myself :-/ I just want to do
apt-get install build-essential.This is mainly for “rationalizing” the 5 containers I have in the Oil build, but I think it could be used to solve many problems I’ve had in the past.
And also I think it is simple enough to do from shell; I’m not buying into a huge distro, although this could evolve into one.
Basically I want to make the containers more fine-grained and composable. Each main() program should have its own lightweight container, a /bin/sh exec wrapper, and then you can compose those with shell scripts! (The continuous build is already a bunch of portable shell scripts.)
Also I want more sharing, which gives you faster transfers over the network and smaller overall size.
I am pretty sure this can solve my immediate problem – whether it generalizes I’m not sure, but I don’t see why not. For this project, desktop apps and OS X are out of scope.
(Also I point out in another comment that I’d like to learn about the overlap between this scheme and what Flatpak already does? i.e. the build tools and runtime, and any notion of repository and network transfer. I’ve already used bubblewrap)
Docker now is a wrapper around
containerdand so uses the snapshot abstraction. OCI containers are defined in terms of layers that define deltas on existing layers (starting with an empty one).containerdprovides caching for these layers by delegating to a snapshotting service, which can apply the deltas as an overlay layer (which it then never modifies, so avoiding all of the corner cases) or to a filesystem with CoW snapshots.I’m not sure what this means. ZFS snapshots, for example, can be mounted in
.zfs/{snapshot name}as read-only trees.To do this really nicely, I want some of the functionality from capsh, so I can use Capsicum, not jails, and have the shell open file descriptors easily for the processes that it spawns, rather than relying on trying to shim all of this into a private view of the global namespace.
I think you’re only speaking about BSD. containerd has the notion of “storage drivers”, and “overlay2” is the default storage driver on Linux. I think it changed 3-4 years ago
https://docs.docker.com/storage/storagedriver/select-storage-driver/
When I look at /var/lib/docker on my Ubuntu machine, it seems to confirm this – On Linux, Docker uses file level “differential” layers, not block-level snapshots. (And all this /var/lib/docker nonsense is what I’m criticizing on the blog. Docker is “anti-Unix”. Monolithic and code-centric not data-centric.)
So basically I want to continue what Red Hat and others are doing and continue “refactoring away” Docker, and just use OverlayFS. From my point of view they did a good job of getting that into the kernel, so now it is reasonable to rely on it. (I think there were 2 iterations of OverlayFS – the second version fixes or mitigates the problems you noted – I agree it is hard, but I also think it is solved.)
I think I wrote about it on the other thread, but I’m getting at a “remote/mobile process abstraction” with explicit data dependencies, mostly for batch processes. You need the data dependencies to be mobile. And I don’t want to introduce more concepts than necessary (according to the Perlis-Thompson principle and narrow waists), so just tarballs of files as layers, rather than block devices, seem ideal.
The blocks are dependent on a specific file system, i.e. ext3 or ext4. And also I don’t think you can do anything with an upper layer without the lower layers. With the file-level abstraction you can do that.
So it seems nicer not to introduce the constraint that all nodes have to be running the same file system – they merely all have to have OverlayFS, which is increasingly true.
None of this is going to be built directly into Oil – it’s a layer on top. So presumably BSDs could use Docker or whatever, or maybe the remote process abstraction can be ported.
Right now I’m just solving my own problem, which is very concrete, but as mentioned this is very similar to lots of problems I’ve had.
Of course Kubernetes and dozens of other systems going back years have remote/mobile process abstractions, but none of them “won”, and they are all coupled to a whole lot of other stuff. I want something that is minimal and composable from the shell, and that basically leads into “distributed shell scripting”.
I think all these systems were not properly FACTORED in the Unix sense. They were not narrow waists and didn’t compose with shell. They have only the most basic integration with shell.
For example our CI is just 5 parallel jobs with 5 Dockerfiles now:
https://github.com/oilshell/oil/tree/master/soil
So logically it looks like this:
I believe most CIs are like this – dumb, racy, without data dependencies, and with hard-coded schedules. So I would like to turn into something more fine-grained, parallel, and thus faster (but also more coarse-grained than Nix.) Basically by framing it in terms of shell, you get LANGUAGE-oriented composition.
(And of course, as previous blog posts say, a container-based build system should be the same thing as a CI system; there shouldn’t be anything you can only run remotely.)
I looked at Capsicum many years ago but haven’t seen capsh… For better or worse Oil is stuck on the lowest common denominator of POSIX, but the remote processes can be built on top, and right now that part feels Linux-only. I wasn’t really aware that people used Docker on BSD and I don’t know anything about it … (I did use NearlyFreeSpeech and their “epochs” based on BSD jails – it’s OK but not as flexible as what I want. It’s more on the admin side than the user side.)
No, I’m talking about the abstractions that
containerduses. It can use overlay filesystems to implement a snapshot abstraction. Docker tried to do this the other way around and use snapshots to implement an overlay abstraction but this doesn’t work well and socontainerdinverted it. This is in the docs.Snapshots don’t have to be at the block level, they can be at the file level. There are various snapshotters in
containerdthat implement the same abstraction in different ways. The key point is that each layer is a delta that is applied to one specific immutable thing below.I’m not really sure what the rest of your post is talking about. You seem to be conflating abstractions and implementation.
OK I think you were misunderstanding what I was talking about in the original message. What I’m proposing uses OverlayFS with immutable layers. Any mutable state is outside the container and mounted in at runtime. It’s more like an executable than a container.
Adding to my own comment, if anyone has experience with Flatpak I’d be interested (since it uses bubblewrap):
https://dev.to/bearlike/flatpak-vs-snaps-vs-appimage-vs-packages-linux-packaging-formats-compared-3nhl
Apparently it is mostly for desktop apps? I don’t see why that would be since CLI apps and server apps should be strictly easier.
I think the main difference again would be the mix of layers and slices, so you have less build configuration. And also naming them as first class on the file system and dynamically mixing and matching. What I don’t like is all the “boiling the ocean” required for packaging, e.g. RPATH but also a lot of other stuff …
I have Snap on my Ubuntu desktop but I am trying to avoid it… Maybe Flatpak is better, not sure.
That sounds like there’s a generic “distro python” though, which… is not necessarily true. You could definitely want environments with both python3.10 and 3.9 installed and not conflicting at the same time.
The model I’m going for is that you’re not really “inside” a container … But each main() program uses a lightweight container for its dependencies. So I can’t really imagine any case where a single main() uses both Python 3.9 and 3.10.
If you have p39 and p10 and want to pipe them together, you pipe together two DIFFERENT containers. You don’t pipe them together inside the container. It’s more like the model of iOS or Android, and apps are identified by a single hash that is a hash of the dependencies, which are layers/slices.
BUT importantly they can share layers / “slices” underneath, so it’s not as wasteful as snap/flatpak and such.
I’ve only looked a little at snap / flatpak, but I think they are more heavyweight, it’s like you’re inside a “machine” and not just assembling namespaces. I imagine an exec wrapper that makes each script isolated
Your idea kinda sounds like GoboLinux Runner, but I can’t tell if it’s exactly the same, since it’s been a long time since I played with GoboLinux. It’s a very interesting take on Linux, flipping the FHS on it’s head just like Nix or Guix, but still keeping the actual program store fully user accessible, and mostly manageable without special commands.
Ah interesting, I heard about GoboLinux >10 years ago but it looks like they made some interesting developments.
They say they are using a “custom mount table” and that is essentially what bubblewrap lets you do. You just specify a bunch of
--mountflags and it makes mount() syscalls before exec-ing the program.https://github.com/containers/bubblewrap/blob/main/demos/bubblewrap-shell.sh
I will look into it, thanks!
I work on a large C/C++ codebase as part of my day job and use lsp-mode/eglot (currently eglot) to navigate the code, with very few extensions. I also use the latest mainline Emacs with native compilation. I have been using Emacs for over 25 years and my customization is best categorized as “very light”. In short, my Emacs set up is not much beyond what ships with it.
And it’s still just… slow. GCC has some pretty big files and opening them can take up to 10 seconds thanks to font-lock mode. (Yes, I know I can configure it to be less decorative, but I find that decoration useful.) It’s much worse when you open a file that is the output from preprocessor expansion (easily 20000+ lines in many cases).
Log files that are hundreds of megabytes are pretty much a guaranteed way to bring Emacs to a crawl. Incremental search in such a buffer is just painful, even if you
M-x find-file-literally.I had to turn off nearly everything in lsp-mode/eglot because it does nothing but delay my input. I can start typing and it will be 3-4 characters behind as it tries to find all the completions I’m not asking for. Company, flymake, eldoc are all intolerably slow when working with my codebase, and I have turned them all off or not installed them in the first place.
M-x termis really useful, but do not attempt to run something that will produce a lot of output to the terminal. It is near intolerable. Literally orders of magnitude slower to display than an xterm or any other terminal emulator. (M-x etermis no better.)The problem, of course, is that Elisp is simply not performant. At all. It’s wonderfully malleable and horribly slow. It’s been this way since I started using it. I had hopes for native compilation, but I’ve been running it for a few months now and it’s still bad. I love Emacs for text editing and will continue to use it. I tried to make it a “lifestyle choice” for a while and realized it’s not a good one if I don’t want to be frustrated all the time. Emacs never seems to feel fast, despite the fast hardware I run it on.
The performance was the reason for me to leave Emacs. I was an evil mode user anyways so the complete switch to (neo)Vim was simple for me. I just could not accept the slowness of Emacs when in Vim everything is instant.
E.g. Magit is always named as one of the prime benefits of Emacs. While its functionality is truly amazing its performance is not. Working on a large code base and repository I was sometimes waiting minutes! for a view to open.
What did you find slow on Emacs aside from Magit?
I actually use Emacs because I found it really fast compared to other options. For example, the notmuch email client is really quick on massive mailboxes.
Some packages might be slow, though. I think the trick is to have a minimal configuration with very well chosen packages. I am particularly interested in performance because my machine is really humble (an old NUC with a slow SATA disk).
To be fair it was some time ago and I don’t remember all the details but using LSPs for code completion/inspection was pretty slow e.g.
Compared to IDEs it might not even have been slow but similar. I however have to compare to Vim where I have equal capabilities but pretty much everything is instant.
My machine was BTW pretty good hardware.
lsp-mode became much more efficient during the last year or so. Eglot is even more lightweight, I think. Perhaps it is worth giving it another go.
I think there was some initial resistance to LSP in the Emacs community and therefore they were not given the attention they deserve.
Thanks for the notice! I may try it again in the future but currently I am very happy with my Neovim setup, which took me a long time to setup/tweak :)
Out of curiosity, were you using Magit on Windows?
I use Magit every day and my main machine is very slow. (1.5GHz 4 core cortex A53) Magit never struck me as particularly slow, but I’ve heard that on Windows where launching subprocesses takes longer it’s a different story.
Ohh you have no idea how slow in a corporate environment. Going through MSYS2, Windows defender, with windows being windows and a corporate security system on top, it takes… ages. git add a single file? 20 seconds. Create a commit? Over a minute. It’s bonkers if you hit the worst case just right. (On a private Windows install, MSYS2 + Exceptions set in Windows Defender it’s fine though, not much slower as my FreeBSD laptop) I asked around and there is a company wide, hardcoded path on every laptop, that has exceptions in all the security systems just to make life less miserable for programmers. Doesn’t solve it completly, but helps.
Either wait an eternity or make a mokery of the security concept. Suffice to say I stopped using Windows and Cross-Compile from now on.
Can confirm. I use Magit on both Linux and Windows, and it takes quite a bit of patience on Windows.
With Windows I think it’s it’s particularly git that is slow, and magit spawns git repeatedly. It used also to be very slow on Mac OS as well because of problems with fork performance. On linux, it used to be slow with tramp. There are some tuning suggestions for all of these in the magit manual I think.
Nope on Linux. As mentioned our code base is big and has many branches etc. Not sure where exactly Magit’s bottleneck was. It was quite some time ago. I just remember that I found similar reports online and no real solution to them.
I now use Lazygit when I need something more than git cli and it’s a fantastic tool for my purpose. I also can use it from within Vim.
This happens for me as well with large changes. I really like Magit but when there are a lot of files it’s nearly unusable. You literally wait for minutes for it to show you an update.
I know you’re not looking to customise much but wrt. terminals, vterm is a lot better in that regard.
I actually switched to
M-x shellbecause I found the line/char mode entry in term-mode to be annoying (and it seems vterm is the same in this respect). shell-mode has all the same slowness of term-mode, of course. I’ve found doing terminal emulation in Emacs to be a lost cause and have given up on it after all these years. I think shell-mode is probably the most usable since it’s more likeM-x shell-commandthan a terminal (and that’s really its best use case).If you need ansi/curses there’s no good answer and while I like term it was too slow in the end and I left. I do think that for “just” using a shell that eshell is fine though.
Do you use the jit branch of emacs? I found once I switched to that and it had jit compiled things my emacs isn’t “fast” but its pretty boring now in that what used to be slow is now at least performant enough for me not to care.
Is there a brew recipe or instructions on compiling on Mac? Or does checking out the source and running make do the business?
I use the emacs-plus1 package. it compiles the version you specify. currently using emacs-plus@29 with
--with-native-compfor native compilation, and probably some other flags.Thanks again, this is appreciably faster and I’m very pleased 😃
Awesome! also, check out
pixel-scroll-precision-modefor the sexiest pixel-by-pixel scrolling. seems to be a little buggy ininfo-mode, can’t replicate withemacs -Qthough, so YMMV.Thank you that sounds perfect
I’m a Mac user and I found it very hard to compile Emacs.
This might be a good starting point however:
https://github.com/railwaycat/homebrew-emacsmacport
I honestly don’t know I use nix+home-manager to manage my setup on macos, this is all I did to make it work across nixos/darwin:
Added it as a flake input: https://github.com/mitchty/nix/blob/7e75d7373e79163f665d7951829d59485e1efbe2/flake.nix#L42-L45
Then added the overlay nixpkgs setup: https://github.com/mitchty/nix/blob/7e75d7373e79163f665d7951829d59485e1efbe2/flake.nix#L84-L87
Then just used it like so: https://github.com/mitchty/nix/blob/6fd1eaa12bbee80b6e80f78320e930d859234cd4/home/default.nix#L87-L90
I gotta convert more of my config over but that was enough to build it and get my existing ~/.emacs.d working with it and speedy to the point I don’t care about emacs slowness even on macos anymore.
Yes. I’ve been using the libgccjit/native compilation version for some time now.
That’s half of it. Another half is that, IIRC, Emacs has rather poor support for asynchrony: most of elisp that runs actually blocks UI.
Can share your config? I’m curious to know how minimal you made it.
Here you go. It changes a little bit here and there with some experiments.The packages I currently have installed and use are: which-key, fic-mode, counsel, smartparens, magit, and solarized-theme. There may be a few others that I was trying out or are only installed for some language support (markdown, yaml, and so forth).
Thank you very much.
Quick addendum on the config: that’s my personal config, which morphs into my work setup. My work one actually turns off flymake and eldoc when using eglot.
Is there anything that has prevented a Neovim-style rewrite of Emacs? A Neomacs?
I keep hearing about the byzantine C-layer of Emacs and the slowness of Elisp. And Emacs definitely has the community size to develop such an alternative. Why do you think no one has attempted such an effort? Or maybe I should add “recently” to the question. As I know there are other Emacs implementations.
As crusty as Emacs source can be, it’s nowhere near as bad Vim source was, which was a rat’s nest of
#ifdef. That’s why Neovim had to basically rewrite their way to a fork. The Emacs implementation is surprisingly clean, as long as you can tolerate some of the aged decisions (and GNU bracing).There is Climacs, which isn’t exactly the same, but is close.
The problem for any new Emacs clone will that it has to run all the Elisp out there. Unless there is a substantial speed improvement to Elisp or a very good automatic translation tool, any such project will be doomed from the start.
The interesting part about Copilot is precisely that it usually doesn’t just copypaste. I wouldn’t go so far as to say it has true understanding of code flow, but its level of understanding is somewhere between Ctrl-V and a human.
(The same goes for GPT, which it is based on.)
So roughly the same level of understanding as a human that only takes a cursory glance at the problem and that uses Ctrl-V as their main tool. Which is the point of the article.
I’d say the same level as a human that takes a cursory glance, then uses Ctrl-V and hacks the copied code to work in their codebase.
Which is a massive step up from “mere” Ctrl-V, as it requires at least some level of conceptual modelling of the program.
It might be. But the point of the post was:
So there are worries that a dumb tool like Copilot bypasses the safeguards that companies put up to prevent using copyrighted code, but the author points out that these safeguards have been bypassed by lazy coders and their Ctrl-V for ages.
That’s true. But then, at a certain level of lazy ripoff it stops being copyright violation and starts edging into creativity anyways. Wine may ask their developers to avoid reading the leaked Windows source code, but I don’t know of any copyright suit that was ever won on such a basis, rather than demonstrating substantial actual replication. And especially at the scale level and usecases of Copilot, the creativity of the code it (rips off / learns from) is usually already marginal to start.
Cases like Carmack’s fast inverse square root aside. But again, everyone rips that one off. :)
To me, the interesting part about Copilot is precisely that it is a form of copypaste that might magically remove copyright.
It doesn’t magically remove copyright. I imagine bit of a spectrum here:
Copilot inhabits some uncharacterized range in between, and anything it provides to you might have been taken from any point along that spectrum. It could have copied the code. It could be a derivative of a copy. It could be something so derivative that the only connection remaining is “concepts and understanding of code”, or whatever that looks like for a machine.
So if you’re lucky and its output truly is from the “ideas” end of the spectrum, then yes it has “removed copyright” in the same way that a human mind that has learned about code is not violating the copyright of what the human read. But it might be from the other end, and you don’t know and can’t prove it either way. Not for yourself, not for your company, not in a court. At least not without some serious forensic analysis that costs a bunch of money.
The result is that everything is contaminated with Shroedinger’s copyrighted-ness and you have no way of telling. (Not to mention what patents you might be violating anywhere along that spectrum.)
No company wants to open that box.
Knowing judges/lawyers I’d be more inclined for them to go with: cute idea, the entire thing is copyright laundering by another name. That “a machine” does it for you with statistics doesn’t seem to obscure the end result.
I find the parallel between IP as a narrow waist of the internet and bytes/text as the narrow waist of shell commands a bit lacking. An email client or web browser never has to concern itself with IP, as pointed out in the article, whilst Unix command line tools all drop down to the rawest of formats with no way to negotiate a higher level protocol like SMTP or HTTP.
The commonality is the O(M x N) property.
But I would say you are not limited to fiddling with raw text in shell. There are many projects that use pipelines of CSV, JSON, and HTML here:
https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil#projects
I don’t think any of them support a
Content-Type. But the point is that you could build that if you wanted to! Pipelines are unstructured but you can give them whatever structure you want.Every such project must include its own serialising/deserialising routines to pass that narrow waist, which is the equivalent of every email client and web browser implementing its own TCP/IP stack.
Not really all that parsing/etc… can just be a library call away a la https://www.unix.com/man-page/freebsd/3/libxo/ as an example for cli programs on freebsd to make a way to have structured output as well as normal text output. That programmers choose to serialize/deserialize all the time in bespoke ways is on them.
Hm I don’t agree, because sockets also work on bytes. So if you have a word document, to send it across the network, you have to serialize it. Ditto for a spreadsheet, a 3D model, etc.
I think the more accurate analogy is if the kernel didn’t have pipes or files at all! And applications had to “fend for themselves” to interoperate.
Actually you have a very good example of this with iOS and Android! Have you ever seen the “Send With” or “Share” buttons in mobile apps? They ask you to share to APPLICATIONS directly – share with text message, with Google hangouts, etc. It’s an M x N explosion although I’m not sure if they implement with M x N code or some kind of plugin system (Intents or whatever?)
So these OSes are not data-centric. Unix is data centric.
I think you are suggesting that the kernel have support for structured data. There is no other way to avoid parsing and serializing.
However my basic point is that nobody can agree on what the narrow waist is. Every language chooses a different one.
This might be on the blog, but here are very long discussions here with the authors of Elvish and Rash, two alternative shells.
https://lobste.rs/s/ww7fw4/unix_shell_history_trivia#c_cpfoyj
https://lobste.rs/s/pdpjvo/google_zx_3_0_release#c_wbt31y
My basic viewpoint is that the shell needs support for structured data, and Oil is getting that. It will have routines to make parsing easier, so every shell script doesn’t have to parse.
But the structure is optional and “projected” or “backward compatible” with byte streams, via JSON, QTT (TSV upgrade), and HTML.
So Oil doesn’t dream of having structured data in the kernel, unlike other shells … But that seems mostly academic because I don’t think they will get it :-/ Also even if the kernel had support, it’s impossible for sockets to grow support. You must parse and serialize when using sockets. So you would end up with 2 tiers again!
Slogans that will appear on the blog:
Byte streams are the model that everyone agrees on, in part because the kernel and network support it, but I think it’s more fundamental than that even.
It requires O(M + N) code to parse and serialize, not O(M x N).
Again, practically speaking, Oil will help you with this problem. It’s the only Bourne-style shell with JSON support, etc.
This is great, the first time in a while that I read something about Nix and come out less confused than I was going in.
Flakes seems like a great tool. Normally, when I hear someone say they use Nix with
$TOOL,my reaction is “I have a hard enough time figuring out Nix on its own, tyvm”. Why is Flakes its own thing? And what is experimental about it?I’m glad it had the effect I was going for :)
Flakes is a thing because the model that Nix uses kinda accidentally assumed that the whole world could be a part of nixpkgs. Turns out that’s sadly not the case, but this explains why external dependencies (such as random git repos such as my website’s source code) were never really considered as a core part of the model. Not to mention just giving people a .nix file to build the software is basically like giving someone a Dockerfile. A Dockerfile will get you to build the software yeah, but overall it doesn’t give you details on how to use it.
I believe that the main innovation with flakes is that it doesn’t just give you build instructions, but also it gives you a way to give out usage instructions with the software. This is the same way that a docker-compose.yml file gives you instructions on how to use stuff that is built with docker.
I’m pretty sure that flakes are still experimental because they want to get feedback about it and potentially make breaking changes to the implementation at some point. However, from my view the net improvements on giving more semantic information and direct ways to use things (eg:
nix run github:Xe/gohello) are so great that this kind of thing really needs to be in the hands of more people. In my decision calculus it’s a case of the benefits outweighing the risks.I’m going to use this style of teaching people how to Nix in the future. I think the next post is purely going to be on how to write Nix packages (and split up a bigger package into a few smaller ones) using way too many contrived Go programs as examples. I may also work in Xeact and Xess. IDK, still working out the formula. This is the kind of style that I’d like to use for a book on Nix/NixOS if I ever get my ADHD to let off enough to let me commit to a longer term project like that.
To pivot off of the note there on nix run, I’ll give an example, nixpkgs doesn’t have htwatch yet (I might submit a pr for it who knows), but if you want to run it anywhere with nix, aka darwin/linux with nix you can run this: https://github.com/mitchty/nixos/blob/master/flake.nix#L44-L71
And It will build the same hwatch I am running at home now. I’m working on porting pikvm as well in that repo (i got side tracked so… don’t expect anything soon) but I love having nix flakes for a bit of a mini “personal nixpkgs” repo where I can test out experiments easily.
I do have gripes about some of this mostly with the command MUST be the same as the package name, which isn’t quite good when you’re testing out something that has a name like blahfs and has a subcommand blah.mount that you want to just test out but I just haven’t looked at how hard that is to do or not.
For my personal tooling I use flakes for individual repositories b/c the built-in template and integrations (like
poetry2nixfor python applications) is a direct upgrade to what we had before.For my
nixosconfigurations though I dislike using flakes. My main gripe with flakes is that it effectively becomes the entrypoint or ‘main’ of the configuration, which it makes it annoying to use with other tools likemorphand the outputs section forces a certain structure to the code where I would prefer it work likenivwhere it sets up dependencies but the end result is just imports to your regular code.@cadey has your position on flakes changed? My earlier interpretation from earlier writing is that you favored tools like
nivandmorphwhich eschew the flake model completely. I also see that nix.dev still recommendsniv(https://nix.dev/tutorials/towards-reproducibility-pinning-nixpkgs) so wondering what the community thinks about flakes in generalMy position on flakes is that they are slightly less terrible than using NixOS normally. They ain’t perfect but they could be so much worse than they are. I’m happy with the usability as-is. Also see here for more words about the kind of mindset change that I had once I hit the satori moment with flakes.
No idea about “the community” (I really hate those terms), but I found flakes easier to get into than anything that came before that always felt “tacked on like spaghetti stapled together”.
I’m doing more with nix now that I ever have now that flakes are “officially” here. Yes experimental but i now have a repo for my entire home config, I have that setup so I can build auto installing iso images, have secrets setup too, and build a system with my home config all in one deploy step via deploy-rs.
In my view flakes are already useful enough to make me not care about what came before which I avoided to be honest. Perfect no but good enough for government work as they say.
You need to be a RIR (RIPE/ARIN/LACNIC/APNIC/AfriNIC) member for that. The membership fee alone is within thousands/year. Then you need to arrange routing with the hosting providers, and those that are ready to do that will also charge at least hundreds per month. No public cloud I’m aware of supports that at all, so you also need your own hardware in a datacenter where your transit provider is present.
In other words, owning your IPv6 network is completely out of reach for individuals and small projects. I believe it shouldn’t be that way and that RIRs are basically rent-seeking organizations now that resources they still can distribute (32-bit ASNs and IPv6 addresses) are anything but scarce, but I can’t see why it may change any soon.
Vultr will let you do BGP with them for (as far as I know) no additional cost above the price of your VPS: https://www.vultr.com/docs/configuring-bgp-on-vultr/
In the RIPE area at least, you can obtain a provider-independent IPv6 assignment via an LIR - you don’t have to go directly to RIPE. A cheap option is Snapserv, who offer an IPv6 PI assignment for 99 EUR/year and an ASN for a one-off fee of 99 EUR. These can both be transferred to another LIR if, for example, Snapserv went out of business, or you wanted to switch LIR for some other reason. They also offer IPv6 PA assignments for less money, but the trade-off is that a PA assignment is tied to the LIR.
You do need to be multi-homed to justify the PI/ASN assignments, so you’d need to find another upstream provider in addition to Vultr. Someone I know uses Vultr and a HE tunnel to justify it.
Interesting, that’s sure an improvement. My company is a RIPE member so I haven’t been watching the PI situation closely, I’m glad to see it improve.
I suspect the problem is routing tables. It would be trivial to assign every person a /64 without making a dent in the address space but then you’d end up with every router on any Internet backbone needing a few billion entries in its routing table. That would completely kill current (and near-future) hardware. Not to mention the fact that if everyone moving between ISPs required a BGP update, the total amount of BGP traffic would overwhelm networks’ abilities to handle the update rate.
You need some mechanism to ration the number of routable networks and money tends to be how we ration things.
I doubt this will ever be a problem in practice. Even among those who host their own servers, the number of people who want to own their address space is always going to be small.
I’m also not advocating for making addresses free or charge, only for making them available for less than the current exorbitant prices that RIRs charge for membership.
TIL, that’s really interesting. I just remember many, many years ago that people were entertaining this, but also with Sixxs and HE tunnels that kinda worked for a while.
Oh, but with tunnelbroker.net and similar, the provider owns the network, you just get a temporary permission to use it and can’t take it with you.
Yes of course, but at least the way it works you could in theory use it longer despite switching ISPs. And I think my Sixxs account was nearly a decade old at the end. Some people might have moved cities three times in that time.
I always wish that addresses were more equitably distributed. With IPv6 there’s no reason not to. And yet ☹
welp for some reason my ISP provides every customer a /64, I don’t know what the reason for that is. There is no single person the internet that needs a /64 and I’m certain no german household needs. But yeah waste tons of network space for no reason. IPv8 we’re coming..
Its the minimum routing size and if you stray from it a lot of the protocol breaks, making it smaller would be insane. And its not wasteful, you could give every atom on the planet a /64, the address space is REALLY BIG. Ipv4 this is not. For it to show up in BGP it needs to be a /48, /32 is the minimum allocation. And there is as many of those as there are ip’s. It should be a /48 you’re given actually, not a /64 (or /60 /56 in comcast home/business cases)
Why do you believe ipv8 is needed because of /64 allocations? Can you back that up with some numbers?
I think we’re good to be honest: https://www.samsclass.info/ipv6/exhaustion.htm
I haven’t done the math but I’ll let the last apnic report speak for itself in that regard (you’ll have to serach, its long and there’s no way to mark some chapter).
And
So It’s not really a standard that breaks things, because then things would already break.
I just don’t see a reason why we’re throwing away massive address ranges, even my private server gets a /64, and that’s one server, not a household or such thing.
The main reason your LAN must be a /64 is that the second half of each address can contain a MAC address (SLAAC) or a big random number (privacy extension).
For routing, not in general, but going below /64 does break things like SLAAC. The original guidance was a /48, its been relaxed somewhat since the original rfc but can go down to a /64. Doing work or i’d pull up the original rfc. Going below /64 does break things, but not at that level being referenced.
Have to get out of the ipv4 conservation mindset, a /64 is massive yes, but 64 bits of address is ipv4 to the power of ipv4, that is… a large amount of space. It also enables things like having ephemeral ip addresses that change every 8 hours. Its better to think of a /64 as the minimum addressable/routable subnet, not a single /32 like you would have in ipv4. And there is A LOT of them, we aren’t at risk of running out even if we get allocation crazy. And thats not hyperbole, we could give every single device, human, animal, place, thing a /64 and still not approach running out.
Also just realized that you might be confusing /60 or /56 as being smaller than a /64, its unintuitive but this is the mask of the subnets not the size. So smaller than /64 would be a CIDR above 64, not below. aka a /96 would break in my example. Its also why assigning “just* a /64 is a bit evil on the part of isp’s and the allocation should be larger.
fun fact: it’s called 6 because it’s the 6th version (kinda). Not because of the number of bytes (which is 8 anyway). You’re rooting for IPv7!
Seems like a not-terrible place to ask:
For a while (see: SO question) I’ve been curious about “dehydrating” a fully-pushed repo into either a script or some metadata for restoring it (as in, with the same remotes, maybe branches). Has anyone seen existing tooling for this?
(Not urgent, so it needn’t be done/polished. I took a swing at some Shell scripting for it, though I put it on hold when I realized the logic for when to actually use it or not wasn’t quite as automatable as I’d hoped. I’ll need to do some more thinking/designing there before it’s actionable.)
Just use this perl script: https://myrepos.branchable.com/
Works on not “only” git too. Then what you want is “just” a config file.
You should really include adding push URLs to a repo’s default remote: https://stackoverflow.com/a/14290145/317670
The best backups are the ones that happen automatically in the future.
Add one for GitHub, one for sr.ht, and one for a local NAS or ssh server. Now, whenever you push, you have two remote services and a local service storing it. The local NAS can be a single raspberry pi with the default 8GB SD card for most people.
This is a good solution, but on the one repo I used it with I switched to a different setup. I now use a local gitea to pull from sr.ht and push that same repo to GitHub, with my origin only pushing to sr.ht.
I’ve been meaning to give this a try, just haven’t found the time yet: https://github.com/cooperspencer/gickup
That is actually very cool. I had no idea! I will update the article ASAP to include this :)
I was not aware of this, thanks!
I’ve been using multiple remotes for my repos and I setup a git alias in my
.gitconfigfor pusing to all remotes at once;Maybe it’s time to start using
pushurl.Why do you need xargs here at all? Just add more remote urls to the same remote name and then git push remote –all. Programmers seem to love to overcomplicate this stuff with layers that aren’t needed.
git remote set-url –add remote some/url
Also setup backups, this remote nonsense won’t do jack for not yet committed stashes or changes.
That’s what’s in the link. I should have done a tldr in my comment…
Hehe, that’s what I meant by
I’m in the process of starting to use this style instead :)
Been thinking about standardizing on asdf+direnv. Could anyone offer a quick comparison?
It sounds like Nix can also build your containers for you based on your project definition?
asdf works fine for pinning runtimes until you have system libraries, etc that extensions link against which aren’t versioned with
asdf. Then you’re back in the same boat as you are with brew, etc where upgrading might break existing workdirs.Yep basically just something like this, lots of assumptions given with this and that you “want” to containerize the hello world program gnu ships but eh its an example:
There are caveats to using the docker builds (can’t build on macos) and you’ll need to learn the nix programming language at some point but its a rather droll affair IME once you get that its all just data and functions. And before you ask why is it so big, the short answer is everything that hello defined it depends on is included, which includes jq/pigz/jshon/perl/moreutils etc… for some reason. But its basically lifted straight out of the nix store verbatim.
I recognise this list. These are the dependencies used in the shell scripts which build the Docker image. They shouldn’t be included in the image itself.
They won’t be included in the image if unused.
Have I just been building docker images wrong then this whole time?
Yup. Nix is a fantastic way to build docker images. For example https://gitlab.com/kevincox/dontsayit-api/-/blob/46cbc50038dfd3d76fee2e458a4503c646b8ff2c/default.nix#L23-35 (nd older project but good example because it has more than just a single binary) creates an image with:
Of course if I used musl libc than glibc and its dependencies would go away automatically.
What’s better is that if you use
buildLayeredImageeach of these is a separate layer so that rebuilding for example the word list, or the binary doesn’t require rebuilding other layers. (This is actually better than docker itself because docker only supports linear layering, so you would have to decide if the word list or the binary is the top layer and rebuilding the lower would force a rebuild of the higher one.)There is also https://nixery.dev/ which allows you to build a container with the necessary tools as easy as just properly naming them. For example:
Will bring you in a container that has a shell, git, and htop.
So much cool stuff in this release, as always!
Some things I liked in the release notes:
For me this is the biggest part of the release: boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
Finally off 5.10 kernel on my zfs system.
This seems like a reasonable barrier for now. Once machines demonstrate that they should have civil rights, then the issue can be revisited.
How would machines demonstrate that?
In the best timelines, they would politely ask. (I have retrieved the full fictional speech from the Internet Archive, for those who want to read it.)
I’m also curious, but to be honest, I’m not sure if I would want to learn that empirically ;)
Obviously this is talking about UK law and not US law, but in the US the definition of “a person” can be stretched beyond what you’d normally expect.
Legal personhood is slightly needed, that article is missing a lot of the “why” things changed: https://www.upcounsel.com/corporate-personhood
Without it, technically the government can just seize corporate assets as an example.
Also legal personhood != human person even under the law. They are distinct. Note this all is from my laywer drinking buddy but the one time I asked him about it he had a bit of a sigh that the nuances are a lot more complex than “corporations are people under law” like it was some sort of soylent green situation. The Tillman act as an example is a good example of why you want personhood for businesses. Unless we want to not be able to collectively barter as a group I would argue that is a good thing.
Out of curiosity, why is corporate personhood necessary to prevent arbitrary seizure of corporate assets given that a natural person (human) owns the corporation and therefore owns the assets? It seems like that would have been a much simpler way of making that work. So it feels like there must be a lot more to it.
The author of the post works at Cray, so he would have access to the very best Fortran developers in the world (along with the best compilers and OSes for running it) and knows exactly why climate models are still done in Fortran. He also knows how to write a headline that gets a lot of clicks :).
I used to work there, and got to bs with Bill Long at lunch about Fortran (not the seattle office). Talking to Fortran compiler guys is fun.
Fortran is cool in that the Cray Fortran compiler had ENV switches for, well basically everything, and i’m not kidding. So its “easy” to tune every single compilation to what you need to do. That and you can mix old and new fortran into the same binary. Try that with c++/c. Rust is only approaching what fortran longevity has had in the past.
This article is attacking a strawman using lies.
The chief complaint with Fortran is that it’s unreadable but also that its buggy and slow, which it is. Being less readable than ASM is just the cherry on top of the mound of ****.
The very benchmarks the author cites show that Fortran is slower than C++ and Rust in all tasks and slower than C in all but one. Conceptually Rust, C++, Fortran and C can aim to have about the same speed, C and C++ have the advantage in terms of native APIs and ABIs designed with them in mind, Rust has the advantage of safety when doing weird stuff, C++ has the advantage of library ecosystem, compile-time abstraction and abstraction in general, C has simplicity. Fortran is inferior to all 3 in any way imaginable.
It should be noted that the speed difference here is very significant, with fortran being 2-10x times slower than the leader on virtually all problems and when the results are taken in aggregate, it barely beats C# (you know, the whole .NET language made to develop web apps?): https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fastest.html
And again, these are the benchmarks the author reference, so I assume they are better than average.
Fortran only gets written in fields that are status-driven, where skill and merit are irrelevant, it’s used to keep the “old guard” into academic positions.
If you want to know what the guys that need efficiency are doing (AWS, GCP, Cloudflare, Dropbox… etc), they are doing C++ and some C and some are considering using some Rust, if you want to know what the guys that need fast complex math are doing (Hedge funds, exchanges, brokers, applied ML services… etc) they are doing C++ and some C.
Nobody is arguing for turning Fortran code into python, they are arguing for turning it into code that works as intended, can be read by normal humans and compiles to something that is 2-10x times as fast.
You know that LAPACK (itself based on BLAS which is also written in Fortran) is written in Fortran 90 right? Almost all scientific code running on every platform used by every academy, government, or corporation is using LAPACK through their programming language/runtime of choice. If you want to push scientific computing out of the Fortran orbit, there’s a lot of work to be done.
LAPACK is not very widely used for compute intensive problems, CUDA doesn’t support it, it support CUBLAS (which si written in C/ASM), so that’s 90% of high speed computing ruled out.
Then you have CPU based stuff, for which LAPACK will be used, but not the original implementation, but rather e.g. intel’s MK, or whatever the guys with supercomputers are using. LAPACK is more of a standard than a library, and people have their own implementations.
The fortran one is indeed used by… ahm? Who exactly? A few procedures on Android phones that lack GPUs?
I’m not talking about SOTA. I’m talking about, well, every other application of linear algebra. If you think everything but Android has moved to using CUDA and MKL, I think you need to take a broader look at academia and industry. ARM cores don’t even have MKL available.
I’m not sure if you’re intentionally dense, but let me try to take a stab at this again:
This goes back to my point of Fortran is slow and buggy, so people that have skin in the game don’t use it for mathematical applications. It might be reasonable for a simple decision tree used in bejewelled but not, e.g., for meteorological models (at least not if the meteorologists were concerned about speed or accuracy, but see the problem of responsibility/skin-in-the-game).
You seem to be attacking a strawman of my position here. I’m not arguing that libraries written in fortran don’t work anymore, just that they are very bad, very bad in that they are slower than alternatives written in Rust, C++ and C, and more on part with fast GCed language (e.g. C#, Java, Switf)
Not everyone on the internet is trying to fight you, please relax. Assume good faith.
I was responding in particular to your claim that we should just rewrite these routines out of Fortran. When we need to use specific CPU or GPU features, we reach for MKL or CUBLAS. For applications that need the speed or CPU/GPU optimizations, this is fine. For all other applications, Fortran is sufficient. I’m only saying that rewriting Fortran into other languages is a lot of work for dubious gain. Folks that need the extra speed can get it. It sounds like we largely agree on this.
I am a bit dubious about your claim that Fortran is chased by academic status-seekers, because it makes the assumption that industry is smart and efficient while academia is bureaucratic and not-efficient and uses results from the Benchmarks game to highlight it. I realize TFA linked to these benchmarks, but I took a cursory glance at them, and some of them don’t even parallelize across CPUs and some don’t even compile! Given that Fortran is a niche language, an online benchmark shootout is probably not the best place. That said, I don’t have a justified idea of Fortran’s numbers either so I can’t do anything more than say I’m dubious.
Based in part on the comment orib made, and looking at some of the code myself, I actually agree with you that those benchmarks might not be a fair comparison.
I guess the thing I disagree on still is that there’s little purpose for most scientific applications if people can’t easily read and understand their code. Especially in domains where practical applications are 30-years away, I feel like it would make sense for 3rd parties to be able to validate the logic as easily as possible.
Much like I would protest to someone using mathematical notation from the late 19th century, even though in principal it would be sufficient for a vast swath of all equations being represented in physics.
Prove it, how is it buggy and slow for its domain? Give 3 examples for each of buggy and slow, should be easy for you with how determined your claim is. Showing benchmarkgames isn’t showing real world code/simulations in any language. Matrix multiplication is what you want to benchmark, why are you using benchmark “games”? Those “games” are hardly readable after people golf them to insanity. If your argument is that I should trust anything from those “games”, well I’ve got ice cubes in antarctica to sell you.
And demonstrate how the restrict keyword in c, which is relatively recent mind you can overcome what fortran can guarantee inherently. And how that affects aliasing and ultimately generated code and how your statement of fortran being inferior to all of c/c++/rust. Have you ever talked to a scientist writing fortran for simualtion? Ever ask them why? Methinks the answer to those last two questions is no. Because I have, and the people writing fortran for simulations are insanely smart, and bear in mind the goal isn’t 100% “it benchmarks faster than language xyz”, its “I run this simulation for $N months/weeks on end, it needs to be fast enough, and I’ll profile it to see what could be improved”. And for each supercomputer they rebuild and retune every library and object for the new processor as needed. I gotta be honest the amount of ignorance of the general computer programming population about how traditional HPC works is disheartening.
And how are simulations that are being run by scientists using the literal fortran COMPLEX type by definition not “guys that need fast complex math”? Fortran specifically handles complex math better than c. My mind boggles at some of the statements here.
Less supposition and hearsay and more demonstrable proof please, this comment is atrocious for basically being without any actual depth or substance to its claims.
Think its time for me to call lobsters quits.
FWIW, every other commenter, except for the comment you replied to, agreed with your position on Fortran.
I provided examples of fortran being bad within the terms the author set for himself. Have you heard of arbitrary demands for rigor? If every time an opinion you hold is challenge you start clamouring for more evidence instead of considering whether or not you have any counter-evidence, well, you see the problem.
Are academics with significant clout even writing code these days? I assume they should have a small team of grad students and postdocs to do that in their lab for them…. is is the fact that everything is written in Fortran the cause of them being forced to maintain old code?
Our industry isn’t immune to status-driven endeavors. Think of all the “bad” technology that gets adopted because FAANG uses it or the proliferation of flavor-of-the-month frameworks in the web space.
https://blog.janestreet.com/why-ocaml/
Together with the two parallel comments I’m going to conclude you don’t know what you’re talking about.
Even if his assertion were true, assuming that all ‘complex math’ is the same and a language that is good for one is good for another is a pretty good indication that you should disregard the post. For example, a lot of financial regulation describes permitted errors in terms of decimal digits and so to comply you must perform the calculations in BCD. If you’re doing scientific computing in BCD, you have serious problems. Conversely, a lot of scientific computing needs the full range of IEEE floating point rounding modes and so on, yet this is rarely if ever needed for financial analysis.
With C99, C grew the full support for floating point weirdness that you need to be able to port Fortan code to C, but so few people care about this in C that most C compilers are only now starting to grow acceptable support (GCC had it for a while, LLVM is gaining it now, in both cases the majority of the work is driven by the needs of the Fortran front end and then just exposed in C). Fortran has much stricter restrictions on aliasing, which make things like autovectorisation far easier than for C/C++. C/C++ developers now use modern OpenMP vectorisation pragmas to tell the compiler what assumptions it is allowed to make (and is unable to check, so miscompiles may happen if you get it wrong), whereas a half-decent Fortan compiler can get the same information directly from the source.
Along with the effort of rewriting, there’s also distrust of new models of rewrites of existing ones. There are decades of work in the literature publishing and critiquing results from existing models, and there are…not that for anything new. It’s much less risky and thus easier to accept to port an existing model to a new HPC platform or architecture than it is to adopt a new one.
Additionally, HPC is a weird little niche of design and practice at both the software and hardware levels. There are a few hundred customers who occasionally have a lot of money and usually have no money. Spinning that whole ecosystem around is difficult, and breaking a niche off the edge of it (if you decided to take climate modelling into Julia without also bringing petrochemicals, bioinformatics, automotive, etc) is a serious risk.
When NREL finally open sourced SAM (GitHub), one of the standard tools for calculating PV output based on location and other factors, a friend of mine decided to take a look at the code. On his first compilation, it was clear no one had ever built it using
-Wall, it had thousands of warnings. When he looked more closely he could tell it has been translated, badly, from (probably) MATLAB, and had many errors in the translation, like (this is C++)to access a 2D coordinate in arrays - for anyone playing at home,
a,bin C++ means evaluate a, then evaluate b and return its result, so this code was accessing only theycoordinate.This would be find if this was undergrad code, but this code had been around for a very long time (decades?), had dozens of papers based on it, and plenty of solar projects relied on it for estimating their ROI. I bring up this anecdote as a counterexample that the age of these libraries and programs does not mean they are high quality, and in fact their age lulls people into a false sense of trust that they actually implement the algorithms they claim to.
He’s since submitted many patches to resolve all the warnings and made sure it compiles with a number of compilers, but I wonder how valid the results over the years actually are- maybe they got lucky and it turns out the simplified model they accidentally wrote was sufficient.
All governments take actions because of these models. They do affect lives of every person on the planet ant future generations to come. “If it’s not broken, don’t fix it” approach doesn’t fit here. Rewriting these models is could be made for a cost of a few international conferences, flights and accommodations of participants.
The models should be scrutinized, no the results they give?
Rarely have I read something so optimistic.
As someone that has interacted with people that write these models, optimistic is putting it lightly. I think whomever thinks that rewriting a bunch of fortran will be productive is entirely underselling both fortran, and the effort to make fortran super fast for simulations.
Rewriting this stuff in javascript isn’t realistic nor will it be fast. And any rewrite is going to have the same problem in 50 years. What are you going to rewrite it again then? How do you know its the same simulation and results?
Sometimes I think us computer programmers don’t really think through the delusions we tell ourselves.
But by rewriting we may check if the implementation follows the specification - see this as a reproductivity issue, do you recall when a bug in excel compromised thousands of researches? And by not changing anything we may find ourselves in a situation where no one knows how these models work, nor be able to improve them. Something similar to banking and cobol situation, but much worse.
The “specification” is “does this code give the same results as it always has”. HPC isn’t big on unit testing or on other forms of detailed design.
Isn’t that a problem? How do we know then they follow peer reviewed papers they were supposed to follow?
In general, we don’t know, and in the abstract, that’s a problem (though specifically in the case of weather forecasting, “did you fail to predict a storm over this shipping lane” is a bigger deal than “did you predict this storm that actually happened for reasons that don’t stand up to scrutiny”, and many meteorological models are serviceably good). There are recent trends to push for more software engineering involvement in computational research (my own PhD topic, come back to me in three years!), and for greater discoverability and accessibility of source code used in computational research.
But turning “recent trends to push” into “international shift in opinion across a whole field of endeavour” is a slow process, much slower than the few international flights and a fancy dinner some software engineers think it should take. And bear in mind none of that requires replatforming anyone off of Fortran, which is not necessary, sufficient, desirable, feasible, or valuable to anyone outside the Rust evangelism strike force.
Most climate models are published in papers and are widely distributed. If you would like to remake these models in other languages and runtimes, you absolutely could (and perhaps find gaps in the papers along the way, but that’s a separate matter.) The problem is, getting the details right is very tough here. Is your library rounding in the same places the previous library was rounding at, in the same ways? How accurate is its exponential arithmetic? What’s the epsilon you need to verify against to be confident that the results are correct?
The article links CliMA as a project for Julia based climate models, but remember, most scientific computing libraries use Fortran one way or another. We’re just pushing the Fortran complexity down to LAPACK rather than up into our model code. Though that’s probably enough to greatly increase explainability, maintainability, and iterative velocity on these models.
What are you even trying to say here? If it’s not broken we should rewrite it because… reasons? It’s not broken, so why would we waste public money rewriting it when we could use that money to further improve its ability to help us?
The article is low on these specific details and I’m not too familiar with climate modelling but I bet that the models aren’t just fancy Excel sheets on steroids – there’s at least one large iterative solver involved, for example, and these things account for most of the Fortran code. In that case, this estimate is off by at least one order of magnitude.
Even if it weren’t, and a few international conferences and flights were all it takes, what actual problem would this solve? This isn’t a mobile app. If a 30 year-old piece of.code is still in use right now, that’s because quite a stack of taxpayer money has been spent on solving that particular problem (by which I mean several people have spent 3-5 of their most productive years in solving it), and the solution is still satisfactory. Why spend money on something that has not been a problem for 30 years instead of spending it on all the problems we still have?
I thought NTP clients won’t blindly shift the date (especially with a wild shift like that), but rather slowly increment the date towards that.
Depends on how they’re configured. If they’re configured to step (aka jump) they can. If you configure them to slew then they do the +/- ppm adjustment of roughly at most 500ppm.
The old ntpd client in most unix distros defaulted to step as a note. Well if i remember right. The issue with slewing though is that if your computer’s time say is wicked out of date, like January 1 1970, its impossible to eventually get your time in sync.