i can somewhat relate to the article, but well… Sure, the concept of providing a per-process environment is nothing new, and chroot has been there forever. Heck, in the early 2000s i once maintained an apache plugin in Debian that did setup virtual chroots (libapache2-mod-chroot) within the Apache process for easier separation of virtual hosts, but thats not nearly what docker or other container solutions do for you nowadays.
Yes, the concept MAY be the same, but now its kinda chroot on steriods, if you like. The whole ecosystem around (cgroups, especially networking namespaces) make it much more usable.
Edit: if i explain the differences between containers and virtual machines, i usually also start explaining about chroot and separating process environments, so yes, the article has some point here.
I really enjoyed the article in hindsight to the debugging mechanisms available in go, but well, why not place the files in the regular filesystem as final solution?
why not place the files in the regular filesystem as final solution?
I did consider it because it was a bit of a mess to chase these issues down, but I value the ease of replication too highly. By putting everything into SQLite, I can just run Litestream, and Litestream handles replicatiing and restoring all application state. If I used the regular filesystem, then I have to roll my own solution for backup and restore.
yes,i understand litestream might be a nice solution to sync sqlite databases, but for my understanding, its just simpler to sync filesystem contents, isnt it? Plus for filesystems, its not like keeping snapshots is something like a new technoloy.
I’ve tried using cloud-backed FUSE filesystems in the past (e.g., s3fuse, gcsfuse), and I’ve found them to be much harder to work with than Litestream. They seem like they’re just a regular filesystem, but then I ran into lots of gotchas in places where they can’t fully behave like a local disk. And SQLite doesn’t support networked filesystems, so I’d still have a separate solution for replicating my database
I know there are tools like SyncThing that can sync a folder to cloud storage, but I think integrating SyncThing is more complex than Litestream.
Are there tools you’ve found that would work well here?
Created an uitlity to export github issues to mbox (https://github.com/abbbi/issue2mbox). Some people used it to import their projects issues from github to sourcehut..
would love to see the same with network communication via loopback devices (for example: MTU settings on a loopback device can have effects on software components transferring data locally only in terms of throghput).
Also note: the “pv” utility attempts to use splice(2) by default! so if you compare pv throughputs against an application that doesnt, pv will outperform.
Splice has to have at least one pipe argument. As the main use case for dd is file-to-file, there’s little point in using it. And splice is mostly fast because it can move kernel pages from one pipe (or socket) to another. File system pages are managed by the page cache, and—as far as I know—splice has to copy those pages rather than moving them.
To elaborate on what carlmjohnson said, this program is not useful as a program. To be useful it needs to chain to another process, be exposed as a library, or similar. As it is this program will run, remove the variables from its environment and then exit. This will not effect the parent process.
That is not worse. That is a better separation of concerns IMO. Probably the author thought unsetting them in its process would propagate to the parent environment. The readme is still wrong. But the repo subtitle is correct. It exits on the presence of any of them.
But as other people stated, in essence this is just a list of variable names and would probably be more useful as such than wrapping it in a go program.
its more thought as a list of known environment variables which are used for storing api tokens, for example like recent happenings in the python ctx package which attempt to steal authentication tokens from the users environment. Unfortunately the description is a little bit off.
I remember my first contact with SELinux about 2007 when I wanted to use XEN virtualization on Fedora.
I put ISO images and systems ‘disks’ at my /home/vermaden dir - then specified path to these file at XEN configs and wanted to start the machine.
I could not. I only got Permission Denied errors. Nothing more. I checked all the chmod(8)/chown(8) permissions but still no luck.
After loosing half a day and searching for the solution on the net I find out that the default SELinux policy requires that all of these files need to be under /var/lib/xen/images path … and SELinux error Permission Denied tells NOTHING about this. Its just shit. Omit like all other shitty ‘technologies’.
The system call that opens the files are unaware of why access was denied. It doesn’t say it’s because of permissions or MAC. journald has made this a bit easier by displaying the program error log next to the audit messages. However, even if you realize that it’s SELinux there’s still no easy path or documentation on how to properly resolve the problem.
There’s a related problem, which is the inverse of your specific case and which has been the root cause of some Chrome vulnerabilities on Linux.
If you get the policy wrong by making it too restrictive (your case), debugging the root cause is hard.
If you get the policy wrong by making it too permissive, you don’t get error messages of any kind, you just get security vulnerabilities.
The root cause of both of these is that the policy is completely decoupled from the application. SELinux is designed as a tool for system administrators to write policies that apply globally to software that they’re managing but it’s used as a tool for software to enforce defence-in-depth sandboxing policies. Capsicum is a much better fit for this (the Capsicum code for the Chrome sandbox was about a 10th of the SELinux code, failed closed, and was easier to debug) but the Linux version was never upstreamed.
But as the article expresses, system administrators typically don’t feel in control of SELinux policies. I think this is an agency problem. The developers are most familiar with the needs and potential vulnerabilities of a program. The administrators are most aware of the requirements the users of the software have. But the policies are written by the distributor (Fedora/Red Hat), who is aware of neither.
The usability of SELinux isn’t great either (and as a developer I much prefer a capabilities-based system), but I think that’s almost secondary to the way it is used in practice.
And app documentation is generally completely missing about the environment the app is expected to run in, i.e. they say “linux” but never goes into more depth. Stuff like the default is to read from these directories and write to these, I require these ENV variables, etc.
None of that is ever documented in any program I’ve ever found(that I remember). Shoot, just getting a list of ports a network app runs on for FW rules can be like pulling teeth sometimes.
It’s super hard to write a reasonable SELinux policy without this information, so you run stuff like audit2allow and just hope for the best, and then randomly flip on extra permissions here and there until it seems to run, call it good and move along. To do it right you need to have developer experience and sysadmin experience and be willing to do deep dives for every installed instance.
I’m a fan of Capsicum, and the pledge stuff that OpenBSD is doing, as at least the developers have a much better chance of getting it right @ runtime.
Another thing developer-managed capabilities facilitate is dynamically dropping them at runtime. The administrator has disabled the admin interface? Drop network privileges. Did you finish reading your config file? The developer knows the process never needs to read another file again.
On the other hand, these policies are not easy to audit. They sit embedded in the code, opaque to administrators and auditors. Knowing what they are requires you either trust the developer, or have access to the source code.
SELinux is a good policy system for an ecosystem where there’s an adversarial relation between the people implementing the software, and the people who run it. I don’t think it’s a natural fit for most FLOSS operating systems.
I only got Permission Denied errors. Nothing more.
This is the typical “first contact” with SELinux. You might be super well versed in Linux/Unix security with years of experience in several distros, but if you’ve never used a system with SELinux (ie RedHat), this is what you’ll see and it’s absolutely maddening. None of your regular Linux skills and knowledge transfer to an SELinux-enabled Linux, and the errors make no sense. And to ask someone to spend weeks or months studying this crap that’s typically only used in the context of one distro? I don’t think so.
try to put some libvirt virtual machine images outside of /var/lib/libvirt/images if you have apparmor enabled (for example on debian). Great fun ahead. I can understand that pain, not only related to SELinux :/
Very few, because selinux is more of a system-wide approach, so app-on-Fedora may have a different settings than app-on-Ubuntu, etc. Even docker doesn’t ship its own and that would be closest qualifying one I think.
A warning is recoverable, right? Otherwise it would be an error. So why use a non-zero exit code when the command didn’t fail? I’ve seen this pattern elsewhere, and I don’t understand it. Exit codes are limited to a single byte of information, and if you want to communicate some non-fatal issue to the user that can be done in great detail using a warning message, but please don’t treat a non-failure as a failure. That just leads to ugly hacks like puppet || [[ $? -eq 2 ]] or worse to check for non-errors.
I makes sense if you document the exit codes accordingly and scripts around the utility you are providing can take care of them, rather than parsing output messages. take rsync, for example, exit codes:
0 Success
1 Syntax or usage error
2 Protocol incompatibility
3 Errors selecting input/output files, dirs
4 Requested action not supported: an attempt was made to manipulate 64-bit files on a platform
that cannot support them; or an option was specified that is supported by the client and not by the server.
[ and many more ]
depends on the situation of course. A warning during file copy might not be “recoverable” in the sense of: the file on the target is still ok. Or maybe just ACL information missing? Or Metadata?
Vagrant is a valid way to do this, however I admit my needs are a bit self-inflicted. I don’t like Ruby (the package manager is a pain) and wanted to have it be a single command to choose distribution/version, CPU core count, disk size, memory that the VM has and what machine to run it on. I also wanted to be able to run waifuctl on a machine that is NOT the one that runs the VM. Basically I want waifud to be instrumental to making my private cloud run. Ideally I’d like it to be like having my own private EC2 that I can also let friends use.
My homelab setup is a bit weird (4 kinda powerful machines that run libvirtd instead of 1 really powerful machine to rule them all), so this software reflects that weirdness. I’d love to use Vagrant, however most Vagrant boxes do not support libvirtd, so my options are limited at best. My current biggest limiting factor in waifud’s implementation is the fact that I have to sling around base images, this will probably be done better in the future (NFS? 9P? Samba?), but for now it works enough.
I had the same issues with Vagrant and similar design goals as what you
mention:
Multiple physical hosts (in different providers or at home)
Single standard way to provision VMs from an underpowered laptop
Version controlled config files
Standard images format
I ended up using terraform + libvirtd
to achieve the above goals. Three years later and it still works fine
for provisioning VMs and networks in my laptop, desktop and various
physical hosts in datacenters.
However as you noted, I have the same issue with moving images around,
which is somewhat alleviated by creating images in each host locally
using packer.
no worries, i like the approach. Have you had a look at the generic/* images from the roboxes.org project? Theyre all available for libvirt, using the vagrant-libvirt plugin, too.
I dont like ruby, too, but with the generic/* boxes and an centralized dhcp service its also possible to easily spin up machines on multiple hosts, if required, which comes more closely to your approach (vms must run qemu agent within them for vagrant to be able to find out about the assigned ip address, and vagrant must be configured to run a remote connection to the libvirtd host)
At the risk of being unduly opinionated, I’d advise that you don’t use NFS. Sling the images around in userspace. In the medium to long run the bugs you get that way will be less nasty.
I love SQLite. But is there any off-the-shelf solutions for having multiple web servers accessing a single shared SQLite database? “running that same app all on a single machine using SQLite” is a pretty big constraint.
If you need a server, what’s wrong with using a DBMS server? SQLite is nice for local file database, but beyond that, any off-the-shelf solutions will most likely add extra complexity and lose most SQLite benefits, especially given alternative like Postgresql.
Because now you have two servers, and the DBMS needs accounts and privileges set up, and you have to configure the app server to tell it where the DBMS is, etc.
Obviously a DBMS is more scaleable, but for something that doesn’t need to scale to multiple nodes, why add all the complexity? Why drive an SUV to the corner store when you can just hop on your bike?
In the environment of the person asking, if you’ve got multiple web servers you already need to deal with accounts, privileges, address discovery, and all the relevant bits.
You’re right, I wasn’t reading back up to @nelson’s question so I missed the “multiple web servers” part.
Running SQLite as a shared db server does seem an odd choice … I agree with @isra17.
Nothing at all is wrong with using a DBMS server! But this fine article is all about how you can use SQLite instead. I’ve been wondering if someone’s built a really simple DBMS wrapper around SQLite. The other answers suggesting rqlite or reading from replicas are the kind of thing I had in mind.
From my understanding, the article makes the point that you can run a single web server and therefore, keep a sqlite database on this server to keep everything simple and fast. If for some reason you need to scale to multiple nodes, then the article points does not apply anymore. When using a project like rqlite, you are using sqlite as much as how you are using files when using postgres. rqlite is a whole different systems with a whole new performance and guarantees profiles. Calling it simply a sqlite server wrapper is an understatement. It seems like rqlite is more in the etcd and consul category than general DBMS.
Agreed, rqlite could be considered similar to etcd or Consul, but with relational modeling available. But I think it’s perfectly fair to call it a DBMS too, just a particularly lightweight and easy-to-run one.
“running that same app all on a single machine using SQLite” is a pretty big constraint.
I’m not sure it is – single machines are BIG these days. You can fit a lot on a single node before you even need to think about scaling. However, if you feel that way, don’t use sqlite. There are database servers out there.
well, not really. The WAL mode is really just based of file locking. So if you use the sqlite library for your programming language, it will make sure that during write operation changes are written to the WAL catalog instead of the original file, which is locked during that time. This works for certain amount of write concurrency, but as soon as you use a complex application which issues a lots of writes, you will soon start to notice your sqlite layer to respond with messages like “database is busy, locked”.
Thats the time you have to deal on your application backend to either handle retries, or even adjust the timeout setting a sqlite write operation is “waiting” for the lock to become free (default is a few microseconds). You can imagine this wont work well with multiple servers accessing the same sqlite database, doing concurrent writes. Really, if you want that, just use some DBMS.
Also, long running SQL write operations can really fuck up the response time for any other application thread on the same database, the SQL queries will receive the busy error and have to deal with that.
And i have seen pretty bad application designs NOT dealing with that, and just “forgetting” about the query, not even issuing an error to the user.
SQLITE has WAL, yes, but it gives you very limited write concurrency.
That FAQ entry is about having multiple readers opening the same database file. Which works great! But it’s read only. More importantly, it only works if you have access to the database file, which in practice means only a single machine. (The doc even explicitly warns against trying this with NFS.)
Normally you would expect someone to backport patches and fixes; but web browser codebases are massive and ugly, so I suspect that’s a really hard job for volunteers. They would possibly have to invent their own fixes too, as upstream might have replaced whole systems within the codebase when fixing the bugs.
Options I can see:
Suddenly summon a vast amount of manpower to backport these patches for the Debian build
Convince the bulldozer browser vendors that they should do this work (hah!)
Remove the browser packages. Then you’re left with a distro that people will complain about (trading security for social issues). This probably also breaks the “stable” idea.
Add a giant warning popup before the browser launches saying that it’s completely insecure and giving the users an option to abort launching it. It’s probably very wise to add a paragraph about why you are doing this (cultures of stable versus rolling browser releases, cost of man hours backporting packages) and another paragraph describing actual practical options to work around this problem (eg moving to Deb testing?).
Shoe-horn in an isolated updated system in a box (eg appimage,etc). It “can” work, but it can also cause a thousand other technical issues (new bugs, esp in regards to video drivers & mesa, let alone potential security ones) and it’s probably not as easy as people think. Remember that browsers are essentially a complete operating system of their own, with things like hardware accelerated video decoding that need to cross the divide to your drivers.
When I used Debian I just used the Google Chrome deb repo. I used Debian testing, which is what Google tracks internally, so Chrome is guaranteed to work. That is, if Chrome were broken on Debian testing, it would be broken for Google developers. And the Google developer workflow heavily relies on web-based tooling. That’s as close to a “blessed” configuration you can get for web browsers on Linux as far as I know.
but then you’re introducing an untrusted binary package into the system (untrusted in that it was built by a 3rd party, not from source on debian-owned servers, etc)
Yeah, but most people don’t care about that and just want their computers to work. Even as a relatively security-conscious SRE, that includes me.
On the list of “people likely to distribute malware-infected binaries,” Google is pretty far down. Unless Chrome falls under your personal definition of malware I suppose.
Very much so. It’s amazing how much the goalposts of “malware” have shifted.
Chrome is spyware. Having a EULA or opt-in was never a reason for spyware not to be listed by AV tools in the past (at best this might make them get flagged as “PUPS” instead of “spyware”). If Chrome came out from a small company in the 2000’s then it would get flagged.
No-one dares mark Chrome as malware. You cannot upset such a large company nor such a large computer base without expecting users to think you are the one at fault. We are not malware, we are an industry leader, you must be mistaken sir :)
It seems that you can, indirectly, buy your way out of being considered malware simply by being a big player.
…from a small company in the 2000’s then it would get flagged.
I get your point, but c’mon… Stuff got flagged back then because it interrupted what the user was trying to do. If you don’t launch Chrome, you don’t see it, and it doesn’t attempt to interact with you. That’s what most users care about, that’s what most users consider to be malware, and, as far as I recall, that’s (largely) what got apps flagged as malware in the 2000s.
That’s a silly distinction. If you use Chrome, then you’re already executing tons of arbitrary code from Google. In practice, whether you get Chrome from Debian or Google, you still have no choice but trust Google.
I’d go with that. Well, leave netsurf in so there’s at least a modicum of web browsing functionality OOTB. Motivated users can download Firefox themselves and the world won’t end. That’s what they have to do on windows and macOS. But trying to keep up with the merry go round is like trying to boil the ocean. Then volunteer effort can be spent on an area that the investment will recoup.
In previous Debian releases they had a section in the release notes about how the version of webkit they shipped was known to be behind on security patches and that it was only included so that you could use it to view trusted sources like your own HTML files or whatever. They were very specific about the fact that only Firefox and Chromium were safe to use with untrusted content.
But I only found out about it by a friend telling me about it in chat. I have my doubts that this could be communicated effectively.
Normally you would expect someone to backport patches and fixes; but web browser codebases are massive and ugly, so I suspect that’s a really hard job for volunteers. They would possibly have to invent their own fixes too, as upstream might have replaced whole systems within the codebase when fixing the bugs.
The article allows us an interesting glimpse into just how hard this is, and it’s not just because of the web browsers:
Debian’s official web browser is Mozilla Firefox (the ESR version). The last update of Firefox ESR in Debian stable has been version 78.15.0. This version also has quite a few unpatched security issues and the 78.x ESR branch is not maintained by Mozilla anymore. They need to update to the 91.x ESR branch, which apparently causes big problems in the current stable Debian platform. In an issue, people complain about freezing browser sessions with the 91.x release, which blocks the new Firefox ESR release from being pushed to “stable-security”. Somebody in the issue claims the reason: “Firefox-ESR 91.3 doesn’t use OpenGL GLX anymore. Instead it uses EGL by default. EGL requires at least mesa version 21.x. Debian stable (bullseye) ships with mesa version 20.3.5.”
“So just update mesa” doesn’t sound like the kind of thing you could do over just a couple of days, seeing how many packages depend on it. Assuming that even fixes the Firefox end of things, I’m not sure I want to think about how many things could break with that update, not before I’ve had my second coffee of the day in any case. Just testing the usual “I updated mesa and now it crashes/looks funny” suspects – Gnome, Plasma, a bunch of games – takes weeks. It’s something you can do in testing but it takes a while.
Large commercial vendors are hitting release management problems like these, too, this is actually part of the reason why you see so many Linux gadgets unironically using tech stacks from three years ago. It’s worse for Debian because they’re trying to build a general-purpose system out of parts that are increasingly made for special-purpose systems that you can either freeze forever (embedded devices) or overwork DevOps teams into PTSD and oblivion in order to keep them running (cloud apps).
Realize that their current model of Debian slow and “stable” will no longer work in 2021 (and beyond) and change it
Not saying Debian should drop stable releases and becoming a rolling release, but perhaps there’s some slightly more rapid cadence they could adopt with releases? Like, is the issue highlighted in the article also a problem with OpenSuSE and Red Hat?
“Stable” means different things to different distros.
To Debian, “Stable” means that bugs will be patched, but features and usage will not. This does not fit with Mozilla and Google’s monthly release cadence; all changes need to be checked over by skilled devs.
SuSE just builds whatever Mozilla hands them, as far as I can tell.
For Firefox (and some other packages iirc) Debian have already given up on that. They would package the latest Firefox ESR even if it introduced new features (and it would, of course). The issue is even that is an insurmountable amount of work. The latest ESR needs much newer LLVM and Rust toolchain versions than the last one. And Debian also wants to build all packages for a given release with other packages in that release; so that means updating all that stack too.
This is why I don’t really see the point in LTS Linux distros. By a couple of years into their lifetime, the only thing that you’re getting from the stability is needing to install most things that you actually want from a separate repo. If ‘stable’ means ‘does not get security fixes’ then it’s worse than useless. A company like Red Hat might have the resources to do security backports for a large set of packages but even they don’t have that ability for everything in their package repos.
It works a bit better in the BSD world, where there’s a strict distinction between the base system and third-party packages, so the base system can provide ABI stability over a multi-year period within a single release but other things can be upgraded. The down side of this is that the stability applies only to the base system. This is great if you’re building an appliance but for anything else you’re likely to have dependencies from the packages that are outside of the base system.
The Debian stable approach works really well for servers. It works moderately well for desktops, with the very notable exception of web browsers – which are, without a doubt, the most used most exposed most insanely complicated subsystem on any desktop, so much so that Google’s ChromeOS is a tiny set of Linux vital organs supporting Chrome.
Even so, Debian is working on this and within a few weeks, I think, there will be new packages for stable and oldstable and even LTS.
I used to think that the “stability” was fine for servers, but it practice it meant that every couple of years I was totally screwed when I had to urgently fix a small thing, but it couldn’t be done without a major upgrade of the whole OS that upset everything. It also encourages having “snowflake” servers, which is problematic on its own.
I feel like the total amount of problems and hassle is the same whether you use a rolling release or snapshots, but snapshot approach forces you to deal with all of them at once. You can’t never upgrade, and software is going to evolve whether you like it or not, so only choice you have is whether you deal with upgrade problems one by one, or all at once.
The Debian release cadence is about 2 years, and has been for 16 years. How much faster would work? What’s Firefox ESR’s cadence? The best I could find from Mozilla was “on average 42 weeks” but I’m not sure that’s quite the right thing. ESR 78 only came out in September this year and is already unsupported. The latest ESR has very different toolchain requirements to build. It’s a confusing picture.
not that i know of, and yes, its really really painful to create a good working sysprepped images. Lately i was looking into building Windows 11 Vagrant images for deployment on libvirt, which, was another kind of fun:
windows 11 refuses to install without UEFI/secureboot
windows 11 refuses to install without working TPM module
After working around all of that stuff by making packer pass a tpm emulation device (swtpm) to qemu and make it use tianocore uefi bios, after hours, i had an automated install going which failed during sysprep phase, because a OneDrive Appx package was unable to be uninstalled and some error messages followed where no exact reason was to be found. I went on and removed the mentioned package manually and then sysprep finally worked.
All in all it took me about a day to get a working image, and i wont touch that image.. ever.. again (until it breaks, for some reason)
Neat idea. I could imagine this as a service that tracks changed regions, and based on high watermarks, pushes the changes to some remote storage: block level incremental backups on live disks…
Wow, that repo checked out is 2.5GB already. This is definitely going to trip GitHub’s soft size limit pretty soon - https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-large-files-on-github#repository-size-limits says it’s 5GB. Expect to get an anxious email from GitHub support when that happens!
yes, fills up quicker than expected, probably gonna switch to a “one repo per day” approach.
lovely talk!
github mirror or alike?
From the included APPLE ACADEMIC LICENSE AGREEMENT:
So publishing it on Github is not covered by the agreement.
i can somewhat relate to the article, but well… Sure, the concept of providing a per-process environment is nothing new, and chroot has been there forever. Heck, in the early 2000s i once maintained an apache plugin in Debian that did setup virtual chroots (libapache2-mod-chroot) within the Apache process for easier separation of virtual hosts, but thats not nearly what docker or other container solutions do for you nowadays.
Yes, the concept MAY be the same, but now its kinda chroot on steriods, if you like. The whole ecosystem around (cgroups, especially networking namespaces) make it much more usable.
Edit: if i explain the differences between containers and virtual machines, i usually also start explaining about chroot and separating process environments, so yes, the article has some point here.
I really enjoyed the article in hindsight to the debugging mechanisms available in go, but well, why not place the files in the regular filesystem as final solution?
Thanks for reading!
I did consider it because it was a bit of a mess to chase these issues down, but I value the ease of replication too highly. By putting everything into SQLite, I can just run Litestream, and Litestream handles replicatiing and restoring all application state. If I used the regular filesystem, then I have to roll my own solution for backup and restore.
yes,i understand litestream might be a nice solution to sync sqlite databases, but for my understanding, its just simpler to sync filesystem contents, isnt it? Plus for filesystems, its not like keeping snapshots is something like a new technoloy.
I’ve tried using cloud-backed FUSE filesystems in the past (e.g., s3fuse, gcsfuse), and I’ve found them to be much harder to work with than Litestream. They seem like they’re just a regular filesystem, but then I ran into lots of gotchas in places where they can’t fully behave like a local disk. And SQLite doesn’t support networked filesystems, so I’d still have a separate solution for replicating my database
I know there are tools like SyncThing that can sync a folder to cloud storage, but I think integrating SyncThing is more complex than Litestream.
Are there tools you’ve found that would work well here?
Created an uitlity to export github issues to mbox (https://github.com/abbbi/issue2mbox). Some people used it to import their projects issues from github to sourcehut..
would love to see the same with network communication via loopback devices (for example: MTU settings on a loopback device can have effects on software components transferring data locally only in terms of throghput).
Also note: the “pv” utility attempts to use splice(2) by default! so if you compare pv throughputs against an application that doesnt, pv will outperform.
Curious, I was expecting GNU coreutils dd(1) to use splice as well, but it appears that it doesn’t.
Splice has to have at least one pipe argument. As the main use case for dd is file-to-file, there’s little point in using it. And splice is mostly fast because it can move kernel pages from one pipe (or socket) to another. File system pages are managed by the page cache, and—as far as I know—splice has to copy those pages rather than moving them.
I get that, but this is a different use case. Here the reason to use dd is that status=progress shows transfer speed.
To elaborate on what carlmjohnson said, this program is not useful as a program. To be useful it needs to chain to another process, be exposed as a library, or similar. As it is this program will run, remove the variables from its environment and then exit. This will not effect the parent process.
Even worse: It doesn’t even unset them. It just exits.
Yeah that’s new since my comment.
That is not worse. That is a better separation of concerns IMO. Probably the author thought unsetting them in its process would propagate to the parent environment. The readme is still wrong. But the repo subtitle is correct. It exits on the presence of any of them.
But as other people stated, in essence this is just a list of variable names and would probably be more useful as such than wrapping it in a go program.
its more thought as a list of known environment variables which are used for storing api tokens, for example like recent happenings in the python ctx package which attempt to steal authentication tokens from the users environment. Unfortunately the description is a little bit off.
I remember my first contact with SELinux about 2007 when I wanted to use XEN virtualization on Fedora.
I put ISO images and systems ‘disks’ at my /home/vermaden dir - then specified path to these file at XEN configs and wanted to start the machine.
I could not. I only got Permission Denied errors. Nothing more. I checked all the chmod(8)/chown(8) permissions but still no luck.
After loosing half a day and searching for the solution on the net I find out that the default SELinux policy requires that all of these files need to be under /var/lib/xen/images path … and SELinux error Permission Denied tells NOTHING about this. Its just shit. Omit like all other shitty ‘technologies’.
The system call that opens the files are unaware of why access was denied. It doesn’t say it’s because of permissions or MAC. journald has made this a bit easier by displaying the program error log next to the audit messages. However, even if you realize that it’s SELinux there’s still no easy path or documentation on how to properly resolve the problem.
There’s a related problem, which is the inverse of your specific case and which has been the root cause of some Chrome vulnerabilities on Linux.
The root cause of both of these is that the policy is completely decoupled from the application. SELinux is designed as a tool for system administrators to write policies that apply globally to software that they’re managing but it’s used as a tool for software to enforce defence-in-depth sandboxing policies. Capsicum is a much better fit for this (the Capsicum code for the Chrome sandbox was about a 10th of the SELinux code, failed closed, and was easier to debug) but the Linux version was never upstreamed.
But as the article expresses, system administrators typically don’t feel in control of SELinux policies. I think this is an agency problem. The developers are most familiar with the needs and potential vulnerabilities of a program. The administrators are most aware of the requirements the users of the software have. But the policies are written by the distributor (Fedora/Red Hat), who is aware of neither.
The usability of SELinux isn’t great either (and as a developer I much prefer a capabilities-based system), but I think that’s almost secondary to the way it is used in practice.
And app documentation is generally completely missing about the environment the app is expected to run in, i.e. they say “linux” but never goes into more depth. Stuff like the default is to read from these directories and write to these, I require these ENV variables, etc.
None of that is ever documented in any program I’ve ever found(that I remember). Shoot, just getting a list of ports a network app runs on for FW rules can be like pulling teeth sometimes.
It’s super hard to write a reasonable SELinux policy without this information, so you run stuff like audit2allow and just hope for the best, and then randomly flip on extra permissions here and there until it seems to run, call it good and move along. To do it right you need to have developer experience and sysadmin experience and be willing to do deep dives for every installed instance.
I’m a fan of Capsicum, and the pledge stuff that OpenBSD is doing, as at least the developers have a much better chance of getting it right @ runtime.
Another thing developer-managed capabilities facilitate is dynamically dropping them at runtime. The administrator has disabled the admin interface? Drop network privileges. Did you finish reading your config file? The developer knows the process never needs to read another file again.
On the other hand, these policies are not easy to audit. They sit embedded in the code, opaque to administrators and auditors. Knowing what they are requires you either trust the developer, or have access to the source code.
SELinux is a good policy system for an ecosystem where there’s an adversarial relation between the people implementing the software, and the people who run it. I don’t think it’s a natural fit for most FLOSS operating systems.
So to summarize, no-one likes selinux because it’s hard for everyone
This is the typical “first contact” with SELinux. You might be super well versed in Linux/Unix security with years of experience in several distros, but if you’ve never used a system with SELinux (ie RedHat), this is what you’ll see and it’s absolutely maddening. None of your regular Linux skills and knowledge transfer to an SELinux-enabled Linux, and the errors make no sense. And to ask someone to spend weeks or months studying this crap that’s typically only used in the context of one distro? I don’t think so.
try to put some libvirt virtual machine images outside of /var/lib/libvirt/images if you have apparmor enabled (for example on debian). Great fun ahead. I can understand that pain, not only related to SELinux :/
Article has some points. Has someone good examples on software projects that ship their own selinux policies?
Very few, because selinux is more of a system-wide approach, so app-on-Fedora may have a different settings than app-on-Ubuntu, etc. Even docker doesn’t ship its own and that would be closest qualifying one I think.
Apparmor policies are more common to be included.
I think MySQL does
A warning is recoverable, right? Otherwise it would be an error. So why use a non-zero exit code when the command didn’t fail? I’ve seen this pattern elsewhere, and I don’t understand it. Exit codes are limited to a single byte of information, and if you want to communicate some non-fatal issue to the user that can be done in great detail using a warning message, but please don’t treat a non-failure as a failure. That just leads to ugly hacks like
puppet || [[ $? -eq 2 ]]or worse to check for non-errors.I makes sense if you document the exit codes accordingly and scripts around the utility you are providing can take care of them, rather than parsing output messages. take rsync, for example, exit codes:
depends on the situation of course. A warning during file copy might not be “recoverable” in the sense of: the file on the target is still ok. Or maybe just ACL information missing? Or Metadata?
it’s worth use an existed fault injection filesystem like unreliablefs.
its easily possible to simulate faulty disks using dm-setup on linux, see: https://abbbi.github.io/dd/
Right. Except dd(1) and unreliablefs there are many other tools for injecting faults. See a diagram.
Spinning up a arch, centos or nix VM with a single command is very appealing. Automatic hostname even!
well, works nicely with vagrant, though.
Vagrant is a valid way to do this, however I admit my needs are a bit self-inflicted. I don’t like Ruby (the package manager is a pain) and wanted to have it be a single command to choose distribution/version, CPU core count, disk size, memory that the VM has and what machine to run it on. I also wanted to be able to run waifuctl on a machine that is NOT the one that runs the VM. Basically I want waifud to be instrumental to making my private cloud run. Ideally I’d like it to be like having my own private EC2 that I can also let friends use.
My homelab setup is a bit weird (4 kinda powerful machines that run libvirtd instead of 1 really powerful machine to rule them all), so this software reflects that weirdness. I’d love to use Vagrant, however most Vagrant boxes do not support libvirtd, so my options are limited at best. My current biggest limiting factor in waifud’s implementation is the fact that I have to sling around base images, this will probably be done better in the future (NFS? 9P? Samba?), but for now it works enough.
I had the same issues with Vagrant and similar design goals as what you mention:
I ended up using terraform + libvirtd to achieve the above goals. Three years later and it still works fine for provisioning VMs and networks in my laptop, desktop and various physical hosts in datacenters.
However as you noted, I have the same issue with moving images around, which is somewhat alleviated by creating images in each host locally using packer.
no worries, i like the approach. Have you had a look at the generic/* images from the roboxes.org project? Theyre all available for libvirt, using the vagrant-libvirt plugin, too.
I dont like ruby, too, but with the generic/* boxes and an centralized dhcp service its also possible to easily spin up machines on multiple hosts, if required, which comes more closely to your approach (vms must run qemu agent within them for vagrant to be able to find out about the assigned ip address, and vagrant must be configured to run a remote connection to the libvirtd host)
I have had a look at them just now, but I am deep enough into making waifud a thing that it may be too late for me now :)
At the risk of being unduly opinionated, I’d advise that you don’t use NFS. Sling the images around in userspace. In the medium to long run the bugs you get that way will be less nasty.
Why not Ubuntu MaaS (metal as a service) / lxd? (I can think of a few reasons, just wondering)
My homelab runs NixOS. I didn’t want to try and get MaaS working on there. I also remembered that exists today.
I love SQLite. But is there any off-the-shelf solutions for having multiple web servers accessing a single shared SQLite database? “running that same app all on a single machine using SQLite” is a pretty big constraint.
If you need a server, what’s wrong with using a DBMS server? SQLite is nice for local file database, but beyond that, any off-the-shelf solutions will most likely add extra complexity and lose most SQLite benefits, especially given alternative like Postgresql.
Because now you have two servers, and the DBMS needs accounts and privileges set up, and you have to configure the app server to tell it where the DBMS is, etc.
Obviously a DBMS is more scaleable, but for something that doesn’t need to scale to multiple nodes, why add all the complexity? Why drive an SUV to the corner store when you can just hop on your bike?
In the environment of the person asking, if you’ve got multiple web servers you already need to deal with accounts, privileges, address discovery, and all the relevant bits.
You’re right, I wasn’t reading back up to @nelson’s question so I missed the “multiple web servers” part. Running SQLite as a shared db server does seem an odd choice … I agree with @isra17.
Nothing at all is wrong with using a DBMS server! But this fine article is all about how you can use SQLite instead. I’ve been wondering if someone’s built a really simple DBMS wrapper around SQLite. The other answers suggesting rqlite or reading from replicas are the kind of thing I had in mind.
From my understanding, the article makes the point that you can run a single web server and therefore, keep a sqlite database on this server to keep everything simple and fast. If for some reason you need to scale to multiple nodes, then the article points does not apply anymore. When using a project like rqlite, you are using sqlite as much as how you are using files when using postgres. rqlite is a whole different systems with a whole new performance and guarantees profiles. Calling it simply a sqlite server wrapper is an understatement. It seems like rqlite is more in the etcd and consul category than general DBMS.
rqlite author here.
Agreed, rqlite could be considered similar to etcd or Consul, but with relational modeling available. But I think it’s perfectly fair to call it a DBMS too, just a particularly lightweight and easy-to-run one.
https://github.com/rqlite/rqlite
Along those lines there’s also https://dqlite.io/
I’m not sure it is – single machines are BIG these days. You can fit a lot on a single node before you even need to think about scaling. However, if you feel that way, don’t use sqlite. There are database servers out there.
Isn’t this just built into SQLite?
https://www.sqlite.org/faq.html#q5
well, not really. The WAL mode is really just based of file locking. So if you use the sqlite library for your programming language, it will make sure that during write operation changes are written to the WAL catalog instead of the original file, which is locked during that time. This works for certain amount of write concurrency, but as soon as you use a complex application which issues a lots of writes, you will soon start to notice your sqlite layer to respond with messages like “database is busy, locked”.
Thats the time you have to deal on your application backend to either handle retries, or even adjust the timeout setting a sqlite write operation is “waiting” for the lock to become free (default is a few microseconds). You can imagine this wont work well with multiple servers accessing the same sqlite database, doing concurrent writes. Really, if you want that, just use some DBMS.
Also, long running SQL write operations can really fuck up the response time for any other application thread on the same database, the SQL queries will receive the busy error and have to deal with that.
And i have seen pretty bad application designs NOT dealing with that, and just “forgetting” about the query, not even issuing an error to the user.
SQLITE has WAL, yes, but it gives you very limited write concurrency.
That FAQ entry is about having multiple readers opening the same database file. Which works great! But it’s read only. More importantly, it only works if you have access to the database file, which in practice means only a single machine. (The doc even explicitly warns against trying this with NFS.)
I did a single duckduckgo search and found https://www.symmetricds.org/, i cant vouch for it though. fwiw my search terms were “sqlite mirroring”
I think LiteStream supports having multiple readers following one leader. If you’re read heavy, it can work to just consolidate writing to one big box
Litestream doesn’t support live read replication but it’s being worked on right now and should be available in the new year.
Eek.
Normally you would expect someone to backport patches and fixes; but web browser codebases are massive and ugly, so I suspect that’s a really hard job for volunteers. They would possibly have to invent their own fixes too, as upstream might have replaced whole systems within the codebase when fixing the bugs.
Options I can see:
Any other options?
When I used Debian I just used the Google Chrome deb repo. I used Debian testing, which is what Google tracks internally, so Chrome is guaranteed to work. That is, if Chrome were broken on Debian testing, it would be broken for Google developers. And the Google developer workflow heavily relies on web-based tooling. That’s as close to a “blessed” configuration you can get for web browsers on Linux as far as I know.
but then you’re introducing an untrusted binary package into the system (untrusted in that it was built by a 3rd party, not from source on debian-owned servers, etc)
Yeah, but most people don’t care about that and just want their computers to work. Even as a relatively security-conscious SRE, that includes me.
On the list of “people likely to distribute malware-infected binaries,” Google is pretty far down. Unless Chrome falls under your personal definition of malware I suppose.
Yeah I consider Chrome to be malware, but that’s beside the point.
Very much so. It’s amazing how much the goalposts of “malware” have shifted.
Chrome is spyware. Having a EULA or opt-in was never a reason for spyware not to be listed by AV tools in the past (at best this might make them get flagged as “PUPS” instead of “spyware”). If Chrome came out from a small company in the 2000’s then it would get flagged.
No-one dares mark Chrome as malware. You cannot upset such a large company nor such a large computer base without expecting users to think you are the one at fault. We are not malware, we are an industry leader, you must be mistaken sir :)
It seems that you can, indirectly, buy your way out of being considered malware simply by being a big player.
I get your point, but c’mon… Stuff got flagged back then because it interrupted what the user was trying to do. If you don’t launch Chrome, you don’t see it, and it doesn’t attempt to interact with you. That’s what most users care about, that’s what most users consider to be malware, and, as far as I recall, that’s (largely) what got apps flagged as malware in the 2000s.
Chrome is like Internet Explorer with all those nasty toolbars installed, except the toolbars are hidden by default ¯\(ツ)/¯.
That’s a silly distinction. If you use Chrome, then you’re already executing tons of arbitrary code from Google. In practice, whether you get Chrome from Debian or Google, you still have no choice but trust Google.
same here, even as a long term Debian user (20+ years), this is just the only way for me, for both the private and regular workstation.
I’d go with that. Well, leave netsurf in so there’s at least a modicum of web browsing functionality OOTB. Motivated users can download Firefox themselves and the world won’t end. That’s what they have to do on windows and macOS. But trying to keep up with the merry go round is like trying to boil the ocean. Then volunteer effort can be spent on an area that the investment will recoup.
In previous Debian releases they had a section in the release notes about how the version of webkit they shipped was known to be behind on security patches and that it was only included so that you could use it to view trusted sources like your own HTML files or whatever. They were very specific about the fact that only Firefox and Chromium were safe to use with untrusted content.
But I only found out about it by a friend telling me about it in chat. I have my doubts that this could be communicated effectively.
The article allows us an interesting glimpse into just how hard this is, and it’s not just because of the web browsers:
“So just update mesa” doesn’t sound like the kind of thing you could do over just a couple of days, seeing how many packages depend on it. Assuming that even fixes the Firefox end of things, I’m not sure I want to think about how many things could break with that update, not before I’ve had my second coffee of the day in any case. Just testing the usual “I updated mesa and now it crashes/looks funny” suspects – Gnome, Plasma, a bunch of games – takes weeks. It’s something you can do in
testingbut it takes a while.Large commercial vendors are hitting release management problems like these, too, this is actually part of the reason why you see so many Linux gadgets unironically using tech stacks from three years ago. It’s worse for Debian because they’re trying to build a general-purpose system out of parts that are increasingly made for special-purpose systems that you can either freeze forever (embedded devices) or overwork DevOps teams into PTSD and oblivion in order to keep them running (cloud apps).
Not saying Debian should drop stable releases and becoming a rolling release, but perhaps there’s some slightly more rapid cadence they could adopt with releases? Like, is the issue highlighted in the article also a problem with OpenSuSE and Red Hat?
“Stable” means different things to different distros.
To Debian, “Stable” means that bugs will be patched, but features and usage will not. This does not fit with Mozilla and Google’s monthly release cadence; all changes need to be checked over by skilled devs.
SuSE just builds whatever Mozilla hands them, as far as I can tell.
For Firefox (and some other packages iirc) Debian have already given up on that. They would package the latest Firefox ESR even if it introduced new features (and it would, of course). The issue is even that is an insurmountable amount of work. The latest ESR needs much newer LLVM and Rust toolchain versions than the last one. And Debian also wants to build all packages for a given release with other packages in that release; so that means updating all that stack too.
This is why I don’t really see the point in LTS Linux distros. By a couple of years into their lifetime, the only thing that you’re getting from the stability is needing to install most things that you actually want from a separate repo. If ‘stable’ means ‘does not get security fixes’ then it’s worse than useless. A company like Red Hat might have the resources to do security backports for a large set of packages but even they don’t have that ability for everything in their package repos.
It works a bit better in the BSD world, where there’s a strict distinction between the base system and third-party packages, so the base system can provide ABI stability over a multi-year period within a single release but other things can be upgraded. The down side of this is that the stability applies only to the base system. This is great if you’re building an appliance but for anything else you’re likely to have dependencies from the packages that are outside of the base system.
The Debian stable approach works really well for servers. It works moderately well for desktops, with the very notable exception of web browsers – which are, without a doubt, the most used most exposed most insanely complicated subsystem on any desktop, so much so that Google’s ChromeOS is a tiny set of Linux vital organs supporting Chrome.
Even so, Debian is working on this and within a few weeks, I think, there will be new packages for stable and oldstable and even LTS.
I used to think that the “stability” was fine for servers, but it practice it meant that every couple of years I was totally screwed when I had to urgently fix a small thing, but it couldn’t be done without a major upgrade of the whole OS that upset everything. It also encourages having “snowflake” servers, which is problematic on its own.
I feel like the total amount of problems and hassle is the same whether you use a rolling release or snapshots, but snapshot approach forces you to deal with all of them at once. You can’t never upgrade, and software is going to evolve whether you like it or not, so only choice you have is whether you deal with upgrade problems one by one, or all at once.
The Debian release cadence is about 2 years, and has been for 16 years. How much faster would work? What’s Firefox ESR’s cadence? The best I could find from Mozilla was “on average 42 weeks” but I’m not sure that’s quite the right thing. ESR 78 only came out in September this year and is already unsupported. The latest ESR has very different toolchain requirements to build. It’s a confusing picture.
Update mesa, then update Firefox? Fighting upstream like that is a losing battle.
Agreed, but updating Mesa is easier said than done.
Added support for single file restore / instant recovery in virtnbdbackup
I’ve heard sysprep is super janky w/ modern Windows. Has it been deprecated and replaced with something else yet?
not that i know of, and yes, its really really painful to create a good working sysprepped images. Lately i was looking into building Windows 11 Vagrant images for deployment on libvirt, which, was another kind of fun:
After working around all of that stuff by making packer pass a tpm emulation device (swtpm) to qemu and make it use tianocore uefi bios, after hours, i had an automated install going which failed during sysprep phase, because a OneDrive Appx package was unable to be uninstalled and some error messages followed where no exact reason was to be found. I went on and removed the mentioned package manually and then sysprep finally worked.
All in all it took me about a day to get a working image, and i wont touch that image.. ever.. again (until it breaks, for some reason)
Why GO if you have bpftrace and bash?
just a POC: https://github.com/abbbi/bpf-hotclone
Neat idea. I could imagine this as a service that tracks changed regions, and based on high watermarks, pushes the changes to some remote storage: block level incremental backups on live disks…
seeing the SHOGO screenshot hits me straight in the feels. Its been a long time ago i had the demo running as a kid.
“According to intelligence, eh? Then I’ve got nothing to worry about!” - Sanjuro