You can have a wildcard domain, such as `.mydomain.com that routes to some dispatch service.
Then each process you run registers (in a database, writes to disk, etc.) itself, and the dispatch service takes myprocessname.mydmain.com and routes it to that process.
Theoretically, the dispatch service could also do a lookup of active processes running and dynamically route based on that, but I assume there might be some overhead with that approach (without caching).
Hmm, this is interesting. So if I understand you correctly, you’re saying I can create a master process on the instance (or on a separate server?), that can read a remote database, figures out the process, and routes the request to that process?
Are there any examples of this approach on GitHub, or in books you know and recommend? How might this approach change if individual processes and instances are shielded behind a load balancer?
Can you name individual UNIX processes, or do you have to search by PID or command? I’m guessing that even if PIDs don’t change over the process lifetime, if the process goes down and it restarts, then it’ll have a different PID, and relying on a failing process to properly issue an HTTP call to update a remote registry isn’t wise because you’ll be coupling failure models within your system design.
I don’t know enough about process name / PID internals to know how easy or hard it is to look up - that’s also why I suggest that each process self-registers in some shared store (DB, disk, in-memory service, etc.). Someone further up suggested Consul which fits this role well - in general, “service discovery” is probably what you should be googling.
The router (that takes incoming requests and sends them to the right process) can both live on the same instance or somewhere else, assuming you use HTTP for communication. If you want to use sockets or similar IPC, you’ll need to have it on the same instance.
To handle failing processes, you could either have a heartbeat mechanism from the router checking that the process is up (with X failures before unregistering it) or you could just have a timeout on all incoming requests and make it so that a new process registering will overwrite the old process’ registration.
It’s hard to be more specific without pulling in specific code samples or talking about actual implementation details.
Familiarize yourself with composition: how Linux looks like in a disk image, a chroot, a container, etc. Things like Packer, mkosi, casync, welder, debootstrap, etc. This will lead into package management and servicing (updates, etc.)
Then systemd. From Mattias [1], Enrico [2] or Lennart’s blog. You might want to follow the systemd releases page. You can use Debian or Fedora, but even if you use Debian I suggest you track the Fedora changelog.
A good organized collection of readings is [3] and focused in internals.
People are already giving you great eBPF resources. New titles are coming out. I would suggest you experiment with Sysdig’s Falco.
I’ve also learned a thing or three from Julia Evans’ zines and blog posts [4] and I bought her prints. And in terms of actual books, consider Kerrisk’s “The Linux Programming Interface” and the latest Nemeth et al. “UNIX and Linux System Administration Handbook”
I hope this helps. I’ve been using Linux for over 15 years, creating distros, packaging, operating large fleets of it, using it as a desktop and more. I’m surprised how much of it evolves and reinvents itself. It keeps me an eternal learner and a perennial novice!
First off, thanks for asking! One of the most demotivating things is when upstreams don’t care!
I am speaking form the perspective of an OpenBSD port maintainer, so everything will have a fairly OpenBSD centric slant.
Responses to your questions:
What can be done to minimize friction with building and installation?
Use one build tool. Know it well. Often times the build tool has already solved problems you are attempting to solve. For example:
Here the author chose to use cmake typically this is great, cmake does a fantastic job of being compatible with everything - hell, it even produces native *BSD Makefiles! However, the original author could have used this and saved a lot of hassle further down the line.
The other end of “not using one tool” is reaching for other build tools from inside an existing one. This just makes things suck as packagers have untangle a heap of things to make the program work.
Is a curl | bash pattern okay enough for packagers or should it be expected that they will always try to build things the hard way from source?
Always build from source. I call these scripts “icebergs” as they often look small until you glance under the surface. 90% of the time things that “install” via this method use #!/bin/bash which is absolutely not portable. Then, typically, the script will pull down all the dependencies needed to build $tool (this usually involves LLVM!!).. then the script will attempt to build said deps (99.9% of the time without success) with what ever build tool/wrapper the author has decided to use. Meanwhile, all the dependencies are available as packages (most of the time with system specific patches to make them build), and they could have simply been installed using the OSs package manager.
Are there entire languages or ecosystems to avoid for software that desires to be packaged?
NPM. We tried to make packages out of npm early on and ultimately hit version gridlock. There are still things we package that build npm things (firefox, chromium, kibana) - but they are shipping a pre-populated node_modules directory.
Are there projects that serve as positive or negative examples that could be referenced?
I have linked a few examples above, but if you want many, dig down to the patches directory for any port in the OpenBSD ports tree, you can see exactly what we deal with when porting various things.
My goal with the links is not to say “look how stupid these guys are!”, it’s simply to point out that we could all stand to know our tooling / target users a bit better.
In general
The best thing you can do is understand the tools you are using. Know their limits. Know when a tool is the right one for a given job.
Also don’t think of your build environment as something that end users will have to deal with. Doing so creates a false sense of “I must make this easy to build!” and results in non-portable things like curl | bash icebergs. The vast majority of the time it will be package maintainers who are working with it.
I agree with nearly everything you said here, but there is one thing I would like to push back on:
NPM. We tried to make packages out of npm early on and ultimately hit version gridlock. There are still things we package that build npm things (firefox, chromium, kibana) - but they are shipping a pre-populated node_modules directory.
I went through the very painful process of packaging my own upstream software for Debian and ran into this rule about having to package all the dependencies, recursively. I don’t think a policy that mandates NPM packages must all be separate system packages is reasonable. I ended up rewriting most of my own dependencies, even creating browserify-lite, because reimplementing Browserify from scratch with no dependencies was easier than packaging them. Mind you this is a mere build dependency.
On top this, some JS dependencies are small, and that’s fine. But Debian ftp masters didn’t accept dependencies that were smaller than 100 lines, while also not accepting pre-bundling node_modules.
The policy is outdated. I’ve heard all the arguments, I’m familiar with the DFSG, but I think it’s too strict with regards to static dependencies. Static dependencies are sometimes quite appropriate. In, particular, just bundle the node_modules folder. It’s fine. If there’s a package in particular that should be system-wide, or you need patches for the system, go for it. Make that package a system package. Allow multiple versions to solve version lock.
Static dependencies are sometimes quite appropriate. In, particular, just bundle the node_modules folder. It’s fine.
It’s not. If it’s just a “blob” basically, how do you then know when one of those modules is in need of a security upgrade? The entire concept of packaging is based on knowing which software is on your system, and to avoid having multiple copies of various dependencies scattered around.
The real problem is the proliferation of dependencies and the break-neck speed at which the NPM ecosystem is developed.
how do you then know when one of those modules is in need of a security upgrade?
This is the most common argument I hear. The answer is simple: the security upgrade is upstream’s problem.
The distribution / package manager must accept this fact. Upstream could, for example, have a security issue that is in its application code, not any of its dependencies. Due to this fact, distros already must track upstream releases in order to stay on top of security upgrades. It is then, in turn, upstream’s job to stay on top of security upgrades in its own dependencies. If upstream depends on FooDep 1.0.0, and FooDep releases 1.0.1 security update, it is then upstream’s job to upgrade and then make a new upstream security fix release. Once this is done, the distro will already be tracking upstream for security releases, and pull this fix in.
I don’t buy this security upgrade argument. Static dependencies are fine.
The real problem is the proliferation of dependencies and the break-neck speed at which the NPM ecosystem is developed.
The real problem is how productive the community is and how successful they are at code reuse? No. The real problem is outdated package management policies.
This is the most common argument I hear. The answer is simple: the security upgrade is upstream’s problem.
That’s insane. First of all, CVEs are filed against specific pieces of software. For truly critical bugs, distribution maintainers often coordinate disclosure in such a way that the bugs are fixed before details are released into the wild. This allows users to patch before being at too much risk.
Expecting that all upstreams which include said package statically will update it in a timely manner means that those upstream packages need to be conscientiously maintained. This is a pipe dream; there are lots and lots of packages which are unmaintained or maintained in a haphazard way.
As a user, this is exactly what you don’t want: if there’s a critical security update, I want to be able to do “list packages”, check if I’m running a patched one, and if not, a simple “update packages” should pull in the fix. I don’t necessarily know exactly which programs are using exactly which dependencies. Using a package manager means I don’t have to (that’s the entire point of package managers).
So, if some packages on my system included their dependencies statically, I would be at risk without even knowing it.
If upstream depends on FooDep 1.0.0, and FooDep releases 1.0.1 security update, it is then upstream’s job to upgrade and then make a new upstream security fix release.
So instead of updating 1 package, this requires the (already overworked) package maintainers to update 50 packages. I don’t think this is progress.
I agree, individual packaging of recursive deps is a nightmare.
The one thing that makes it a “feature” on OpenBSD is the forced use of ftp(1), which has extra security measures like pledge(2). I trust ftp(1) much more than npm/node/other tools when it comes to downloading things.
One super down side to the “use ftp to fetch” method is that unless the tools in question (npm, go get, gem) expose the dependency resolution in a way that can be used by the ports framework/ftp.. you are basically stuck re-implementing dependency resolution, which is stupid.
I didn’t know about the100 line limit. That’s interesting! I can see it being a pain in the butt, but I also appreciate the “minimize dependencies” approach!
This is all excellent advice. I would also add that commercial and other financially backed publishers can’t outsource this, and can’t shortchange this. It’s important to have a distro liaison if you care about the end user experience. If you don’t have this you end up in the GNOME or Skype or NPM or Docker situations. Invest in testing the packages. Monitor the bug reports. Stay ahead of changes on the dependency tree. And why not, sponsor the bug bash and the BoF and all of that.
Even though it seems as though operating system package managers are almost falling out of favor in exchange for a proliferation of language-specific package managers
if this is the case it is a mistake and a trend we should work to reverse.
To help OS maintainers, keep the software as simple as possible, avoid CMake and autotools if possible, write very simple Makefiles.
I totally agree, especially in regard to Makefiles, and am glad to see that you linked one of ours as an example. We only write Makefiles for all suckless tools and stay away from CMake, autohell or other solutions. This simplicity makes it easy to package software (see Gentoo, Alpine and NixOS as examples).
Granted, we keep the scope of our software small, but maybe we should generally question the tendency nowadays to build massive behemoths with tons of dependencies and complex build logic. If you think about it, the reasons why things like autohell exist are not present anymore. 10-20 years ago, the ecosystem was much more diverse, but nowadays, every time I see a configure-script check if my compiler supports trigonometric functions or something, I just shake my head. 99% of configure scripts are copy-pasted from GNU code and they are a huge waste of time and energy. To make matters worse, these configure-scripts effectively prevent me from easily compiling such software on a RaspberryPI (or similar) as it takes so long to run these configure-scripts every time.
In contrast to this, a Makefile takes mere seconds, and if you run a more “nonstandard” system, you just change config.mk (example taken from farbfeld) to your needs, but this is not necessary in 99% of the cases.
All suckless programs are simple enough that any experienced Unix user should be able to compile them without any Makefile at all. Most can be compiled by just cc -lX11 -lXft -o dwm *.c, or something along those lines.
It’s probably not a good solution for, say, Firefox or KDE, but not all software is Firefox or KDE.
Even though it seems as though operating system package managers are almost falling out of favor in exchange for a proliferation of language-specific package managers
if this is the case it is a mistake and a trend we should work to reverse.
Most language package managers are poorly designed, poorly implemented and modify the system in ways that are not deterministic, not reproducible and cannot be reversed. I’m thinking of pip in particular, but I believe npm suffers from similar issues as well. If we want something inbetween OS package managers and language package managers then probably something like Nix is required.
I don’t think it’s a mistake. Only 1% of open source components are packaged on a distro like Debian. Best chance to have a packaged ecosystem is CPAN and that hovers between 10-15%. Distros aren’t in that business anymore. What OS package managers should do is deliver native packaged and be good stewards of the rest of the software in the system. For example, making apt “npm aware” so that apt can singly emit a package install operation even if it was triggered by npm locally.
It doesn’t take much for simple Makefiles to not scale. Try adding a dependency on something as basic as iconv or curses while remaining portable. Autoconf is not as bad as all that and is common enough that any OS packager will know how to deal with it. I’m rather less fond of libtool, though.
I love this book. It has helped me understand a lot of what has changed on Linux itself over the last 10 years. I’m speaking next week at All Things Open and 100% recommending DDIA.
What are your needs? Why do you need all of these requirements?
A product, even, open-source, like this, would have a massive testing matrix and footprint.
Another “closest you’ll get” is the JVM ecosystem. It’s cross-platform and can be environment-oriented (e.g. tell maven, gradle, or sbt and friends where to put libraries), but language agnosticism is limited to JVM languages (Java, Scala, Kotlin, Clojure, Jython, JRuby, Freige (Haskell), Javascript, Groovy, etc.) and languages that can expose internals via FFI and JNI (C, C++, Rust easily; Go, Nim probably).
Well, ZeroInstall seems to meet the requirements. We’ll see if it works in a practice test.
I’m willing to relax on convenience, for example. Pip/Conda are probably easier for Python packages, but the advantage to also handle C++, Perl, Binaries with a single tool might outweigh that.
Package managers are more than package formats. Windows is the blocker here, and spack documents it well although I think the same applies to brew, nix and distri, and the inverse would apply to chocolatey.
ZeroInstall looks pretty cool. I can say that I’ve never had a reason to install ZeroInstall, flatpak, or anything like it. I’ve used snappy only to try it out on a Raspberry Pi installation of Ubuntu Server.
Keep in mind though that a user will hesitate to install another package management system, especially if it’s just to install your software. It’ll be a source of friction. It’s not clear what your intended audience is, so that might not really affect you or you may desire that friction to scare off users for whom you’re unprepared to support. ZeroInstall may be adequate for now but if you intend to grow a userbase, it may become a liability or a blocker to adoption unless your use provides adequate momentum to encourage users to install ZeroInstall first.
The more you meet a user in the middle, the more likely they are to use your software. Consider using existing automation that packages your software for distribution in existing formats. nebula os-package or setupbuilder in the Gradle world handle this without much effort for all of your target OSes. Maybe there’s something in your ecosystem that handles this for you.
Ah, I do not care so much about the user but about our development environment here. ;)
The use case is development of embedded products in a large company. Development requires a range of tools where some are Open Source, some internal, some bought. Essentially we write C++, but something like Conan is not fit to deliver simulation or modelling tools for example. Reproducibility is important, so the precise Python, Perl, bash, etc versions are necessary.
Currently, we use a Windows-only home-grown installation tool which cannot even do dependencies or uninstall. We increasingly use Linux for our testing infrastructure and this brings a second configuration tooling. Using the same would be an improvement.
(Using a monorepo is also considered but that is much more disruptive a people would have to change their version control system.)
You can have a wildcard domain, such as `.mydomain.com that routes to some dispatch service.
Then each process you run registers (in a database, writes to disk, etc.) itself, and the dispatch service takes
myprocessname.mydmain.comand routes it to that process.Theoretically, the dispatch service could also do a lookup of active processes running and dynamically route based on that, but I assume there might be some overhead with that approach (without caching).
Hmm, this is interesting. So if I understand you correctly, you’re saying I can create a master process on the instance (or on a separate server?), that can read a remote database, figures out the process, and routes the request to that process?
Are there any examples of this approach on GitHub, or in books you know and recommend? How might this approach change if individual processes and instances are shielded behind a load balancer?
Can you name individual UNIX processes, or do you have to search by PID or command? I’m guessing that even if PIDs don’t change over the process lifetime, if the process goes down and it restarts, then it’ll have a different PID, and relying on a failing process to properly issue an HTTP call to update a remote registry isn’t wise because you’ll be coupling failure models within your system design.
Also perhaps look into dbus: https://news.ycombinator.com/item?id=9451023
I don’t know enough about process name / PID internals to know how easy or hard it is to look up - that’s also why I suggest that each process self-registers in some shared store (DB, disk, in-memory service, etc.). Someone further up suggested Consul which fits this role well - in general, “service discovery” is probably what you should be googling.
The router (that takes incoming requests and sends them to the right process) can both live on the same instance or somewhere else, assuming you use HTTP for communication. If you want to use sockets or similar IPC, you’ll need to have it on the same instance.
To handle failing processes, you could either have a heartbeat mechanism from the router checking that the process is up (with X failures before unregistering it) or you could just have a timeout on all incoming requests and make it so that a new process registering will overwrite the old process’ registration.
It’s hard to be more specific without pulling in specific code samples or talking about actual implementation details.
Familiarize yourself with composition: how Linux looks like in a disk image, a chroot, a container, etc. Things like Packer, mkosi, casync, welder, debootstrap, etc. This will lead into package management and servicing (updates, etc.)
Then systemd. From Mattias [1], Enrico [2] or Lennart’s blog. You might want to follow the systemd releases page. You can use Debian or Fedora, but even if you use Debian I suggest you track the Fedora changelog.
A good organized collection of readings is [3] and focused in internals.
People are already giving you great eBPF resources. New titles are coming out. I would suggest you experiment with Sysdig’s Falco.
I’ve also learned a thing or three from Julia Evans’ zines and blog posts [4] and I bought her prints. And in terms of actual books, consider Kerrisk’s “The Linux Programming Interface” and the latest Nemeth et al. “UNIX and Linux System Administration Handbook”
I hope this helps. I’ve been using Linux for over 15 years, creating distros, packaging, operating large fleets of it, using it as a desktop and more. I’m surprised how much of it evolves and reinvents itself. It keeps me an eternal learner and a perennial novice!
[1] https://ma.ttias.be/learning-systemd/ [2] https://www.enricozini.org/blog/2017/debian/systemd-01-intro/ [3] https://0xax.gitbooks.io/linux-insides/ [4] https://jvns.ca/
wow, thank you so much for sharing this!
First off, thanks for asking! One of the most demotivating things is when upstreams don’t care!
I am speaking form the perspective of an OpenBSD port maintainer, so everything will have a fairly OpenBSD centric slant.
Responses to your questions:Use one build tool. Know it well. Often times the build tool has already solved problems you are attempting to solve. For example:
The other end of “not using one tool” is reaching for other build tools from inside an existing one. This just makes things suck as packagers have untangle a heap of things to make the program work.
Always build from source. I call these scripts “icebergs” as they often look small until you glance under the surface. 90% of the time things that “install” via this method use
#!/bin/bashwhich is absolutely not portable. Then, typically, the script will pull down all the dependencies needed to build $tool (this usually involves LLVM!!).. then the script will attempt to build said deps (99.9% of the time without success) with what ever build tool/wrapper the author has decided to use. Meanwhile, all the dependencies are available as packages (most of the time with system specific patches to make them build), and they could have simply been installed using the OSs package manager.NPM. We tried to make packages out of npm early on and ultimately hit version gridlock. There are still things we package that build npm things (firefox, chromium, kibana) - but they are shipping a pre-populated
node_modulesdirectory.I have linked a few examples above, but if you want many, dig down to the
patchesdirectory for any port in the OpenBSD ports tree, you can see exactly what we deal with when porting various things.My goal with the links is not to say “look how stupid these guys are!”, it’s simply to point out that we could all stand to know our tooling / target users a bit better.
In generalThe best thing you can do is understand the tools you are using. Know their limits. Know when a tool is the right one for a given job.
Also don’t think of your build environment as something that end users will have to deal with. Doing so creates a false sense of “I must make this easy to build!” and results in non-portable things like
curl | bashicebergs. The vast majority of the time it will be package maintainers who are working with it.For fun I made a quick and dirty list of ports and their respective patch count, here are the top 4:
I agree with nearly everything you said here, but there is one thing I would like to push back on:
I went through the very painful process of packaging my own upstream software for Debian and ran into this rule about having to package all the dependencies, recursively. I don’t think a policy that mandates NPM packages must all be separate system packages is reasonable. I ended up rewriting most of my own dependencies, even creating browserify-lite, because reimplementing Browserify from scratch with no dependencies was easier than packaging them. Mind you this is a mere build dependency.
On top this, some JS dependencies are small, and that’s fine. But Debian ftp masters didn’t accept dependencies that were smaller than 100 lines, while also not accepting pre-bundling node_modules.
The policy is outdated. I’ve heard all the arguments, I’m familiar with the DFSG, but I think it’s too strict with regards to static dependencies. Static dependencies are sometimes quite appropriate. In, particular, just bundle the node_modules folder. It’s fine. If there’s a package in particular that should be system-wide, or you need patches for the system, go for it. Make that package a system package. Allow multiple versions to solve version lock.
It’s not. If it’s just a “blob” basically, how do you then know when one of those modules is in need of a security upgrade? The entire concept of packaging is based on knowing which software is on your system, and to avoid having multiple copies of various dependencies scattered around.
The real problem is the proliferation of dependencies and the break-neck speed at which the NPM ecosystem is developed.
This is the most common argument I hear. The answer is simple: the security upgrade is upstream’s problem.
The distribution / package manager must accept this fact. Upstream could, for example, have a security issue that is in its application code, not any of its dependencies. Due to this fact, distros already must track upstream releases in order to stay on top of security upgrades. It is then, in turn, upstream’s job to stay on top of security upgrades in its own dependencies. If upstream depends on FooDep 1.0.0, and FooDep releases 1.0.1 security update, it is then upstream’s job to upgrade and then make a new upstream security fix release. Once this is done, the distro will already be tracking upstream for security releases, and pull this fix in.
I don’t buy this security upgrade argument. Static dependencies are fine.
The real problem is how productive the community is and how successful they are at code reuse? No. The real problem is outdated package management policies.
That’s insane. First of all, CVEs are filed against specific pieces of software. For truly critical bugs, distribution maintainers often coordinate disclosure in such a way that the bugs are fixed before details are released into the wild. This allows users to patch before being at too much risk.
Expecting that all upstreams which include said package statically will update it in a timely manner means that those upstream packages need to be conscientiously maintained. This is a pipe dream; there are lots and lots of packages which are unmaintained or maintained in a haphazard way.
As a user, this is exactly what you don’t want: if there’s a critical security update, I want to be able to do “list packages”, check if I’m running a patched one, and if not, a simple “update packages” should pull in the fix. I don’t necessarily know exactly which programs are using exactly which dependencies. Using a package manager means I don’t have to (that’s the entire point of package managers).
So, if some packages on my system included their dependencies statically, I would be at risk without even knowing it.
So instead of updating 1 package, this requires the (already overworked) package maintainers to update 50 packages. I don’t think this is progress.
I agree, individual packaging of recursive deps is a nightmare.
The one thing that makes it a “feature” on OpenBSD is the forced use of ftp(1), which has extra security measures like pledge(2). I trust ftp(1) much more than npm/node/other tools when it comes to downloading things.
One super down side to the “use ftp to fetch” method is that unless the tools in question (npm, go get, gem) expose the dependency resolution in a way that can be used by the ports framework/ftp.. you are basically stuck re-implementing dependency resolution, which is stupid.
I didn’t know about the100 line limit. That’s interesting! I can see it being a pain in the butt, but I also appreciate the “minimize dependencies” approach!
This is all excellent advice. I would also add that commercial and other financially backed publishers can’t outsource this, and can’t shortchange this. It’s important to have a distro liaison if you care about the end user experience. If you don’t have this you end up in the GNOME or Skype or NPM or Docker situations. Invest in testing the packages. Monitor the bug reports. Stay ahead of changes on the dependency tree. And why not, sponsor the bug bash and the BoF and all of that.
if this is the case it is a mistake and a trend we should work to reverse.
To help OS maintainers, keep the software as simple as possible, avoid CMake and autotools if possible, write very simple Makefiles.
I totally agree, especially in regard to Makefiles, and am glad to see that you linked one of ours as an example. We only write Makefiles for all suckless tools and stay away from CMake, autohell or other solutions. This simplicity makes it easy to package software (see Gentoo, Alpine and NixOS as examples).
Granted, we keep the scope of our software small, but maybe we should generally question the tendency nowadays to build massive behemoths with tons of dependencies and complex build logic. If you think about it, the reasons why things like autohell exist are not present anymore. 10-20 years ago, the ecosystem was much more diverse, but nowadays, every time I see a configure-script check if my compiler supports trigonometric functions or something, I just shake my head. 99% of configure scripts are copy-pasted from GNU code and they are a huge waste of time and energy. To make matters worse, these configure-scripts effectively prevent me from easily compiling such software on a RaspberryPI (or similar) as it takes so long to run these configure-scripts every time.
In contrast to this, a Makefile takes mere seconds, and if you run a more “nonstandard” system, you just change config.mk (example taken from farbfeld) to your needs, but this is not necessary in 99% of the cases.
To make it short, @xorhash: Keep it simple! :)
This is madness if you intend to support anything other than Linux.
All suckless programs are simple enough that any experienced Unix user should be able to compile them without any Makefile at all. Most can be compiled by just
cc -lX11 -lXft -o dwm *.c, or something along those lines.It’s probably not a good solution for, say, Firefox or KDE, but not all software is Firefox or KDE.
The makefile I linked is cross-platform.
I disagree, the only platform that can’t deal with a simple makefile in my experience is Windows.
Why is this a mistake?
Most language package managers are poorly designed, poorly implemented and modify the system in ways that are not deterministic, not reproducible and cannot be reversed. I’m thinking of pip in particular, but I believe npm suffers from similar issues as well. If we want something inbetween OS package managers and language package managers then probably something like Nix is required.
I don’t think it’s a mistake. Only 1% of open source components are packaged on a distro like Debian. Best chance to have a packaged ecosystem is CPAN and that hovers between 10-15%. Distros aren’t in that business anymore. What OS package managers should do is deliver native packaged and be good stewards of the rest of the software in the system. For example, making apt “npm aware” so that apt can singly emit a package install operation even if it was triggered by npm locally.
It doesn’t take much for simple Makefiles to not scale. Try adding a dependency on something as basic as iconv or curses while remaining portable. Autoconf is not as bad as all that and is common enough that any OS packager will know how to deal with it. I’m rather less fond of libtool, though.
I maintain a project that has iconv as a dependency, the Makefile supports Linux and the BSDs, trivially.
edit: disambiguate
As a user of packaging mostly, I’d like authors to include manpages to help packagers integrate their software to their OS.
What would this manpage say? What should the publisher say in that file that packagers should do?
I love this book. It has helped me understand a lot of what has changed on Linux itself over the last 10 years. I’m speaking next week at All Things Open and 100% recommending DDIA.
What are your needs? Why do you need all of these requirements?
A product, even, open-source, like this, would have a massive testing matrix and footprint.
Another “closest you’ll get” is the JVM ecosystem. It’s cross-platform and can be environment-oriented (e.g. tell maven, gradle, or sbt and friends where to put libraries), but language agnosticism is limited to JVM languages (Java, Scala, Kotlin, Clojure, Jython, JRuby, Freige (Haskell), Javascript, Groovy, etc.) and languages that can expose internals via FFI and JNI (C, C++, Rust easily; Go, Nim probably).
Well, ZeroInstall seems to meet the requirements. We’ll see if it works in a practice test.
I’m willing to relax on convenience, for example. Pip/Conda are probably easier for Python packages, but the advantage to also handle C++, Perl, Binaries with a single tool might outweigh that.
Package managers are more than package formats. Windows is the blocker here, and spack documents it well although I think the same applies to brew, nix and distri, and the inverse would apply to chocolatey.
https://github.com/spack/spack/issues/9323
ZeroInstall looks pretty cool. I can say that I’ve never had a reason to install ZeroInstall, flatpak, or anything like it. I’ve used snappy only to try it out on a Raspberry Pi installation of Ubuntu Server.
Keep in mind though that a user will hesitate to install another package management system, especially if it’s just to install your software. It’ll be a source of friction. It’s not clear what your intended audience is, so that might not really affect you or you may desire that friction to scare off users for whom you’re unprepared to support. ZeroInstall may be adequate for now but if you intend to grow a userbase, it may become a liability or a blocker to adoption unless your use provides adequate momentum to encourage users to install ZeroInstall first.
The more you meet a user in the middle, the more likely they are to use your software. Consider using existing automation that packages your software for distribution in existing formats. nebula os-package or setupbuilder in the Gradle world handle this without much effort for all of your target OSes. Maybe there’s something in your ecosystem that handles this for you.
Ah, I do not care so much about the user but about our development environment here. ;)
The use case is development of embedded products in a large company. Development requires a range of tools where some are Open Source, some internal, some bought. Essentially we write C++, but something like Conan is not fit to deliver simulation or modelling tools for example. Reproducibility is important, so the precise Python, Perl, bash, etc versions are necessary.
Currently, we use a Windows-only home-grown installation tool which cannot even do dependencies or uninstall. We increasingly use Linux for our testing infrastructure and this brings a second configuration tooling. Using the same would be an improvement.
(Using a monorepo is also considered but that is much more disruptive a people would have to change their version control system.)