Apart from the meme, I’m a little disapointed when people give up and just disable SELinux.
It doesn’t help that no one wants to be told: “stop what you’re doing, spend a week to learn this other thing, make a tiny change, and then return to your real job”. They’re looking for solutions — preferably quick ones — and not a week’s worth of study.
I understand that this might be demotivating, but I personally feel that SELinux should be part of the basic skill-set of anybody deploying/running a CentOS/RHEL server. I mean most people learned about permissions, systemd, etc. before deploying their server, what’s wrong with investing a little more time into learning another necessary technology to do your job.
I get the point of the article, I also run some Debian servers, and it would never come to my mind to maintain SELinux policies and enforcement there. But Red Hat made a great effort to make SELinux seamless on their platform. Yes I agree with the author, the lack of configuration files and the necessity to use seinfo/sesearch/semanage can be frustrating, but a lot of other aspects of Linux “are just there” (to quote the author). Kernel modules being one of them, some script reads and loads what’s in /etc/modules, but I can just modprobe xxxxxx and it’s “just there”, you’ll have to use lsmod to see it.
In general, when I’ve seen people fight SELinux they’re doing something weird that could be done differently:
They’re not storing files in the directory which is specified by in the documentation.
They wrote custom policies instead of enabling an selinux boolean which would have allowed them to do what they wanted to achive.
Take my comment with a grain of salt, I like SELinux (on RHEL/CentOS! Not on other distros which don’t make the effort for it, I just use systemd hardening like the author.)
The fact that the meme page links to a 40-minute video instead of any usable documentation just proves the point in my opinion. Almost all the documentation points to kludging around with “just make it work*” commands instead of teaching how to write a policy.
I had an issue with smbd not being able to read from a ZFS mount. The only documentation I could find on the policy said there was an sebool I could set. It was wrong.
audit2allow basically set it to permissive for that syscall, which filled my logs with permitted violations. Since it was an older machine, that actually impacted performance greatly.
To be honest, watching that video has just inspired me to go set selinux to permissive, relabel stuff and try and see if I’ll be able to live with it from now on.
Although, not running servers and not seeing this as part of my job in the future, I think it’s a wasted effort. I am just doing it to stay up-to-date with my linux skills (also not needed), not because I’ll be running systems with policies and permissions and stuff.
This feeling is fed not just by a history of bad documentation, but also by a history of absolutely atrocious tooling, especially as shipped by Fedora. I don’t remember the details because it was very long ago, but at one point the handbook asked users to use a Python script that wasn’t even packaged (or maybe it was packaged in a non-obvious package – either way, what I distinctly remember my colleagues doing was literally grabbing a CentOS package to get at the damn thing). It also didn’t help that, for the longest time, SELinux packages weren’t very well synchronized with other packages, so it was pretty common to do an update that involved neither SELinux nor any of its configuration, and still break something.
Another moderately fun thing is that the tools for working with SELinux are pretty CPU and memory hungry. I remember being unable to change SELinux config on CentOS 6 VMs with 1GB of RAM because the policy update tool would crash with an OOM error.
Exactly this. I tried Fedora once and needed to recursively reset some SELinux context for my home directory. This required to use some single threaded Python tool that took forever to finish the job. I just installed Arch afterwards and never looked back.
As a long-time Fedora user, I’ve “learned to stop worrying and love SELinux”. I consider the tooling and documentation atrocious, but you’re right about the two problems. Whenever I have a problem, the solution is either to go searching for the right selinux boolean (/usr/sbin/getsebool -a | grep httpd) or find the right label for whatever I just installed and relabel the place I installed it.
Very few, because selinux is more of a system-wide approach, so app-on-Fedora may have a different settings than app-on-Ubuntu, etc. Even docker doesn’t ship its own and that would be closest qualifying one I think.
Yeah, SELinux documentation sucks a lot and the error messages leave a ton to be desired. Given the rise of containers and the power available with systemd’s options, I appreciate the post’s title. You can achieve most of the practical benefits alone with either of those solutions.
Personally, I still disagree with turning it off, but having a broken and nonfunctional system is worse than an insecure one. So turning it off, temporarily I hope, is usually the right call. (Defense in depth is a nice ideal.)
One of the things I’m happy with is that I’ve been able to automate (through ansible) using my own custom policies to apply to systems. Though that did take me weeks of reading documentation and even reading source code to piece together. Ugh. Not everyone is willing to put up with that, I know.
I remember my first contact with SELinux about 2007 when I wanted to use XEN virtualization on Fedora.
I put ISO images and systems ‘disks’ at my /home/vermaden dir - then specified path to these file at XEN configs and wanted to start the machine.
I could not. I only got Permission Denied errors. Nothing more. I checked all the chmod(8)/chown(8) permissions but still no luck.
After loosing half a day and searching for the solution on the net I find out that the default SELinux policy requires that all of these files need to be under /var/lib/xen/images path … and SELinux error Permission Denied tells NOTHING about this. Its just shit. Omit like all other shitty ‘technologies’.
The system call that opens the files are unaware of why access was denied. It doesn’t say it’s because of permissions or MAC. journald has made this a bit easier by displaying the program error log next to the audit messages. However, even if you realize that it’s SELinux there’s still no easy path or documentation on how to properly resolve the problem.
There’s a related problem, which is the inverse of your specific case and which has been the root cause of some Chrome vulnerabilities on Linux.
If you get the policy wrong by making it too restrictive (your case), debugging the root cause is hard.
If you get the policy wrong by making it too permissive, you don’t get error messages of any kind, you just get security vulnerabilities.
The root cause of both of these is that the policy is completely decoupled from the application. SELinux is designed as a tool for system administrators to write policies that apply globally to software that they’re managing but it’s used as a tool for software to enforce defence-in-depth sandboxing policies. Capsicum is a much better fit for this (the Capsicum code for the Chrome sandbox was about a 10th of the SELinux code, failed closed, and was easier to debug) but the Linux version was never upstreamed.
But as the article expresses, system administrators typically don’t feel in control of SELinux policies. I think this is an agency problem. The developers are most familiar with the needs and potential vulnerabilities of a program. The administrators are most aware of the requirements the users of the software have. But the policies are written by the distributor (Fedora/Red Hat), who is aware of neither.
The usability of SELinux isn’t great either (and as a developer I much prefer a capabilities-based system), but I think that’s almost secondary to the way it is used in practice.
And app documentation is generally completely missing about the environment the app is expected to run in, i.e. they say “linux” but never goes into more depth. Stuff like the default is to read from these directories and write to these, I require these ENV variables, etc.
None of that is ever documented in any program I’ve ever found(that I remember). Shoot, just getting a list of ports a network app runs on for FW rules can be like pulling teeth sometimes.
It’s super hard to write a reasonable SELinux policy without this information, so you run stuff like audit2allow and just hope for the best, and then randomly flip on extra permissions here and there until it seems to run, call it good and move along. To do it right you need to have developer experience and sysadmin experience and be willing to do deep dives for every installed instance.
I’m a fan of Capsicum, and the pledge stuff that OpenBSD is doing, as at least the developers have a much better chance of getting it right @ runtime.
Another thing developer-managed capabilities facilitate is dynamically dropping them at runtime. The administrator has disabled the admin interface? Drop network privileges. Did you finish reading your config file? The developer knows the process never needs to read another file again.
On the other hand, these policies are not easy to audit. They sit embedded in the code, opaque to administrators and auditors. Knowing what they are requires you either trust the developer, or have access to the source code.
SELinux is a good policy system for an ecosystem where there’s an adversarial relation between the people implementing the software, and the people who run it. I don’t think it’s a natural fit for most FLOSS operating systems.
I only got Permission Denied errors. Nothing more.
This is the typical “first contact” with SELinux. You might be super well versed in Linux/Unix security with years of experience in several distros, but if you’ve never used a system with SELinux (ie RedHat), this is what you’ll see and it’s absolutely maddening. None of your regular Linux skills and knowledge transfer to an SELinux-enabled Linux, and the errors make no sense. And to ask someone to spend weeks or months studying this crap that’s typically only used in the context of one distro? I don’t think so.
try to put some libvirt virtual machine images outside of /var/lib/libvirt/images if you have apparmor enabled (for example on debian). Great fun ahead. I can understand that pain, not only related to SELinux :/
I wasn’t aware of those two utilities. Both system-config-selinux and selinux-polgengui is managed by the SELinux project but barely mentioned in any documentation.
system-config-selinux can show you labels and path mappings. It can display and manipualte SELinux policy booleans. That’s useful! You can even add your own labels and path maps. However, it can’t display or manipulate the policy modules themselves. It can list them, but it won’t tell you what they do.
selinux-polgengui is a step-by-step wizard (with no explanation of what any of the fields mean) for creating new policy modules.
In my experience, any security that comes in the way will eventually be turned off or bypassed. If one needs a PhD to work with SELinux, then it is as good as useless.
Balancing security with usability and convenience is fairly tricky (and fortunes are made trying to solve this problem), so I am not surprised many find SELinux to be a terrible and untamable beast, and either set it in “permissive” mode or simply turn it off.
I’ve always felt SELinux was a gamble, because outside of 100% streamlined vendor distributions, with only vendor-supplied packages installed, there would just be too much friction to maintain a functional setup.
I wish OpenBSD-style pledge and unveil capabilities APIs were more popular. That way it’s the software itself saying: “OK, given my startup configuration, I only need X, Y and Z, and I only need access to these files and these directories, please ban me from everything else!”
I wasn’t aware of Linux Landlock, so thanks for sharing!
Under long-term goals: “add the ability to create (file descriptor) capabilities compatible with Capsicum.”
YAY! :) I’m glad the author didn’t get stuck in NIH(where lots of Linux stuff seems to end up), and thought about the other stuff out in the world that already do this, like capsicum.
Also adopting things like Rust in kernel and essential modules, where things like capabilities are built-in, should pull in that direction and eventually help with it.
I don’t get the recommendation to turn it off. audit2allow may be tricky to understand in full, but running it and saying yes to everything still gets you in a better situation than disabling the service. If you get conflicts after major updates - you can blow away all custom modules and run audit2allow again. If you’re bothered by denied access, switch to permissive mode instead, process the failures after a few days and switch back to enforcing.
The only place where I would maybe not bother is on the desktop if someone runs into lots of issues. Then again, I’m running enforcing mode with extra policy additions for years now on Fedora and can’t remember it causing any issues for the last 2 years.
SELinux has been on my “whoa, that’s a lot of complexity; I’ll sit down and learn that one day if I find myself obligated to” list for years, and since I mostly live on the Debian/Ubuntu side of the fence, I have not yet found myself obligated to. I’ve had similar experiences with AppArmor, though, where the foo that’s been working for months broke because bar and now I have to learn exactly enough about this thing to fix it, and then I immediately forget everything I just learned as I move on to the next thing.
I could probably benefit from a more focused and systemic approach to learning these things, but I’m also hopeful that the day is approaching where I could just hire someone else to understand them for me…
You should almost never disable SELinux fully: when disabled, proper labels will not be applied to any changes to the filesystem, resulting in massive pain.
If you’re running into problems, set it to permissive mode instead, which keeps it active but not enforcing the policies.
You can try to run the magical commands to create a local policy patch, but then you might run into problems again further down the line. The best option in this situation is to, at least temporarily, put SELinux into permissive mode. Permissive mode logs denials but doesn’t enforce them the way the regular enforcing mode does. […] The thing I’ve started to realize over time is that it’s probably best to just leave SELinux in permissive mode. The enforcing mode might prevent malicious actors from taking advantage of a weak point in my system’s security. However, it’s definitely preventing me from getting my work done.
Apart from the meme, I’m a little disapointed when people give up and just disable SELinux.
I understand that this might be demotivating, but I personally feel that SELinux should be part of the basic skill-set of anybody deploying/running a CentOS/RHEL server. I mean most people learned about permissions, systemd, etc. before deploying their server, what’s wrong with investing a little more time into learning another necessary technology to do your job.
I get the point of the article, I also run some Debian servers, and it would never come to my mind to maintain SELinux policies and enforcement there. But Red Hat made a great effort to make SELinux seamless on their platform. Yes I agree with the author, the lack of configuration files and the necessity to use
seinfo/sesearch/semanagecan be frustrating, but a lot of other aspects of Linux “are just there” (to quote the author). Kernel modules being one of them, some script reads and loads what’s in/etc/modules, but I can justmodprobe xxxxxxand it’s “just there”, you’ll have to uselsmodto see it.In general, when I’ve seen people fight SELinux they’re doing something weird that could be done differently:
Take my comment with a grain of salt, I like SELinux (on RHEL/CentOS! Not on other distros which don’t make the effort for it, I just use systemd hardening like the author.)
The fact that the meme page links to a 40-minute video instead of any usable documentation just proves the point in my opinion. Almost all the documentation points to kludging around with “just make it work*” commands instead of teaching how to write a policy.
I had an issue with smbd not being able to read from a ZFS mount. The only documentation I could find on the policy said there was an sebool I could set. It was wrong.
audit2allow basically set it to permissive for that syscall, which filled my logs with permitted violations. Since it was an older machine, that actually impacted performance greatly.
That was when I decided to disable it.
To be honest, watching that video has just inspired me to go set selinux to permissive, relabel stuff and try and see if I’ll be able to live with it from now on.
Although, not running servers and not seeing this as part of my job in the future, I think it’s a wasted effort. I am just doing it to stay up-to-date with my linux skills (also not needed), not because I’ll be running systems with policies and permissions and stuff.
This feeling is fed not just by a history of bad documentation, but also by a history of absolutely atrocious tooling, especially as shipped by Fedora. I don’t remember the details because it was very long ago, but at one point the handbook asked users to use a Python script that wasn’t even packaged (or maybe it was packaged in a non-obvious package – either way, what I distinctly remember my colleagues doing was literally grabbing a CentOS package to get at the damn thing). It also didn’t help that, for the longest time, SELinux packages weren’t very well synchronized with other packages, so it was pretty common to do an update that involved neither SELinux nor any of its configuration, and still break something.
Another moderately fun thing is that the tools for working with SELinux are pretty CPU and memory hungry. I remember being unable to change SELinux config on CentOS 6 VMs with 1GB of RAM because the policy update tool would crash with an OOM error.
Exactly this. I tried Fedora once and needed to recursively reset some SELinux context for my home directory. This required to use some single threaded Python tool that took forever to finish the job. I just installed Arch afterwards and never looked back.
This is the way.
As a long-time Fedora user, I’ve “learned to stop worrying and love SELinux”. I consider the tooling and documentation atrocious, but you’re right about the two problems. Whenever I have a problem, the solution is either to go searching for the right selinux boolean (
/usr/sbin/getsebool -a | grep httpd) or find the right label for whatever I just installed and relabel the place I installed it.Article has some points. Has someone good examples on software projects that ship their own selinux policies?
Very few, because selinux is more of a system-wide approach, so app-on-Fedora may have a different settings than app-on-Ubuntu, etc. Even docker doesn’t ship its own and that would be closest qualifying one I think.
Apparmor policies are more common to be included.
I think MySQL does
Yeah, SELinux documentation sucks a lot and the error messages leave a ton to be desired. Given the rise of containers and the power available with systemd’s options, I appreciate the post’s title. You can achieve most of the practical benefits alone with either of those solutions.
Personally, I still disagree with turning it off, but having a broken and nonfunctional system is worse than an insecure one. So turning it off, temporarily I hope, is usually the right call. (Defense in depth is a nice ideal.)
One of the things I’m happy with is that I’ve been able to automate (through ansible) using my own custom policies to apply to systems. Though that did take me weeks of reading documentation and even reading source code to piece together. Ugh. Not everyone is willing to put up with that, I know.
I remember my first contact with SELinux about 2007 when I wanted to use XEN virtualization on Fedora.
I put ISO images and systems ‘disks’ at my /home/vermaden dir - then specified path to these file at XEN configs and wanted to start the machine.
I could not. I only got Permission Denied errors. Nothing more. I checked all the chmod(8)/chown(8) permissions but still no luck.
After loosing half a day and searching for the solution on the net I find out that the default SELinux policy requires that all of these files need to be under /var/lib/xen/images path … and SELinux error Permission Denied tells NOTHING about this. Its just shit. Omit like all other shitty ‘technologies’.
The system call that opens the files are unaware of why access was denied. It doesn’t say it’s because of permissions or MAC. journald has made this a bit easier by displaying the program error log next to the audit messages. However, even if you realize that it’s SELinux there’s still no easy path or documentation on how to properly resolve the problem.
There’s a related problem, which is the inverse of your specific case and which has been the root cause of some Chrome vulnerabilities on Linux.
The root cause of both of these is that the policy is completely decoupled from the application. SELinux is designed as a tool for system administrators to write policies that apply globally to software that they’re managing but it’s used as a tool for software to enforce defence-in-depth sandboxing policies. Capsicum is a much better fit for this (the Capsicum code for the Chrome sandbox was about a 10th of the SELinux code, failed closed, and was easier to debug) but the Linux version was never upstreamed.
But as the article expresses, system administrators typically don’t feel in control of SELinux policies. I think this is an agency problem. The developers are most familiar with the needs and potential vulnerabilities of a program. The administrators are most aware of the requirements the users of the software have. But the policies are written by the distributor (Fedora/Red Hat), who is aware of neither.
The usability of SELinux isn’t great either (and as a developer I much prefer a capabilities-based system), but I think that’s almost secondary to the way it is used in practice.
And app documentation is generally completely missing about the environment the app is expected to run in, i.e. they say “linux” but never goes into more depth. Stuff like the default is to read from these directories and write to these, I require these ENV variables, etc.
None of that is ever documented in any program I’ve ever found(that I remember). Shoot, just getting a list of ports a network app runs on for FW rules can be like pulling teeth sometimes.
It’s super hard to write a reasonable SELinux policy without this information, so you run stuff like audit2allow and just hope for the best, and then randomly flip on extra permissions here and there until it seems to run, call it good and move along. To do it right you need to have developer experience and sysadmin experience and be willing to do deep dives for every installed instance.
I’m a fan of Capsicum, and the pledge stuff that OpenBSD is doing, as at least the developers have a much better chance of getting it right @ runtime.
Another thing developer-managed capabilities facilitate is dynamically dropping them at runtime. The administrator has disabled the admin interface? Drop network privileges. Did you finish reading your config file? The developer knows the process never needs to read another file again.
On the other hand, these policies are not easy to audit. They sit embedded in the code, opaque to administrators and auditors. Knowing what they are requires you either trust the developer, or have access to the source code.
SELinux is a good policy system for an ecosystem where there’s an adversarial relation between the people implementing the software, and the people who run it. I don’t think it’s a natural fit for most FLOSS operating systems.
So to summarize, no-one likes selinux because it’s hard for everyone
This is the typical “first contact” with SELinux. You might be super well versed in Linux/Unix security with years of experience in several distros, but if you’ve never used a system with SELinux (ie RedHat), this is what you’ll see and it’s absolutely maddening. None of your regular Linux skills and knowledge transfer to an SELinux-enabled Linux, and the errors make no sense. And to ask someone to spend weeks or months studying this crap that’s typically only used in the context of one distro? I don’t think so.
try to put some libvirt virtual machine images outside of /var/lib/libvirt/images if you have apparmor enabled (for example on debian). Great fun ahead. I can understand that pain, not only related to SELinux :/
There’s a GUI for SELinux, have you seen it?
https://jfearn.fedorapeople.org/fdocs/en-US/Fedora/20/html/Security_Guide/sect-Security-Enhanced_Linux-Working_with_SELinux-SELinux_Management.html
I wasn’t aware of those two utilities. Both
system-config-selinuxandselinux-polgenguiis managed by the SELinux project but barely mentioned in any documentation.system-config-selinuxcan show you labels and path mappings. It can display and manipualte SELinux policy booleans. That’s useful! You can even add your own labels and path maps. However, it can’t display or manipulate the policy modules themselves. It can list them, but it won’t tell you what they do.selinux-polgenguiis a step-by-step wizard (with no explanation of what any of the fields mean) for creating new policy modules.In my experience, any security that comes in the way will eventually be turned off or bypassed. If one needs a PhD to work with SELinux, then it is as good as useless.
Balancing security with usability and convenience is fairly tricky (and fortunes are made trying to solve this problem), so I am not surprised many find SELinux to be a terrible and untamable beast, and either set it in “permissive” mode or simply turn it off.
I’ve always felt SELinux was a gamble, because outside of 100% streamlined vendor distributions, with only vendor-supplied packages installed, there would just be too much friction to maintain a functional setup.
I wish OpenBSD-style pledge and unveil capabilities APIs were more popular. That way it’s the software itself saying: “OK, given my startup configuration, I only need X, Y and Z, and I only need access to these files and these directories, please ban me from everything else!”
I hope Linux Landlock may provide such sandboxing capabilities like pledge or unveil in the future.
I wasn’t aware of Linux Landlock, so thanks for sharing!
Under long-term goals: “add the ability to create (file descriptor) capabilities compatible with Capsicum.”
YAY! :) I’m glad the author didn’t get stuck in NIH(where lots of Linux stuff seems to end up), and thought about the other stuff out in the world that already do this, like capsicum.
Also adopting things like Rust in kernel and essential modules, where things like capabilities are built-in, should pull in that direction and eventually help with it.
rust has capabilities built in?
I don’t get the recommendation to turn it off. audit2allow may be tricky to understand in full, but running it and saying yes to everything still gets you in a better situation than disabling the service. If you get conflicts after major updates - you can blow away all custom modules and run audit2allow again. If you’re bothered by denied access, switch to permissive mode instead, process the failures after a few days and switch back to enforcing.
The only place where I would maybe not bother is on the desktop if someone runs into lots of issues. Then again, I’m running enforcing mode with extra policy additions for years now on Fedora and can’t remember it causing any issues for the last 2 years.
SELinux has been on my “whoa, that’s a lot of complexity; I’ll sit down and learn that one day if I find myself obligated to” list for years, and since I mostly live on the Debian/Ubuntu side of the fence, I have not yet found myself obligated to. I’ve had similar experiences with AppArmor, though, where the foo that’s been working for months broke because bar and now I have to learn exactly enough about this thing to fix it, and then I immediately forget everything I just learned as I move on to the next thing.
I could probably benefit from a more focused and systemic approach to learning these things, but I’m also hopeful that the day is approaching where I could just hire someone else to understand them for me…
You should almost never disable SELinux fully: when disabled, proper labels will not be applied to any changes to the filesystem, resulting in massive pain.
If you’re running into problems, set it to permissive mode instead, which keeps it active but not enforcing the policies.
That’s actually what the article recommends: