1. 1

    This is a great solution if you are already running prometheus, or if you are interested in doing so. I do like the simplicity of hchk.io for cases where I don’t want to run prometheus (and related services/tooling like grafana, and push-gateway).

    Great idea and writeup though! Next time I have to run prometheus at a job, I’ll definitely keep this in mind for tracking the errant cron jobs that always seems to sneak in there somewhere.

    1. 1

      As I mentioned in https://blog.bejarano.io/alertmanager-alerts-with-amazon-ses/#sup1, I do not run Grafana or any dashboarding because I consider it worthless and time-consuming to set up.

      Thanks for the feedback!

      1. 1

        At a small scale the expression browser is sufficient (I use it for most of my work), but once you get beyond that something like Grafana is essential.

    1. 4

      Nice quick write-up, thanks! One small thing: you can self-host healthchecks ( https://github.com/healthchecks/healthchecks ), so the cost argument is a bit unfair. Promethues does have more features, and I reckon one should install it anyway, which is why the solution you outline is still probably better in a lot of circumstances.

      1. 1

        Oh wow! I didn’t know that. Thanks for the feedback!

      1. 10

        With the built-in container support in SystemD you don’t even need new tools:

        https://blog.selectel.com/systemd-containers-introduction-systemd-nspawn/

        …and with good security if you build your own containers with debootstrap instead of pulling stuff made by random strangers on docker hub.

        1. 8

          The conflict between the Docker and systemd developers is very interesting to me. Since all the Linux machines I administer already have systemd I tend to side with the Red Hat folks. If I had never really used systemd in earnest before maybe it wouldn’t be such a big deal.

          1. 5

            …and with good security if you build your own containers with debootstrap instead of pulling stuff made by random strangers on docker hub.

            I was glad to see this comment.

            I have fun playing with Docker at home but I honestly don’t understand how anyone could use Docker Hub images in production and simultaneously claim to take security even quasi-seriously. It’s like using random npm modules on your crypto currency website but with even more opaqueness. Then I see people arguing over the relative security of whether or not the container runs as root but then no discussion of far more important security issues like using Watchtower to automatically pull new images.

            I’m no security expert but the entire conversation around Docker and security seems absolutely insane.

            1. 4

              That’s the road we picked as well, after evaluating Docker for a while. We still use Docker to build and test our containers, but run them using systemd-nspawn.

              To download and extract the containers into folders from the registry, we wrote a little go tool: https://github.com/seantis/roots

              1. 2

                From your link:

                Inside these spaces, we can launch Linux-based operating systems.

                This keeps confusing me. When I first saw containers, I saw them described as light weight VM’s. Then I saw people clarifying that they are really just sandboxed Linux processes. If they are just processes, then why do containers ship with different distros like Alpine or Debian? (I assume it’s to communicate with the process in the sandbox.) Can you just run a container with a standalone executable? Is that desirable?

                EDIT

                Does anyone know of any deep dives into different container systems? Not just Docker, but a survey of various types of containers and how they differ?

                1. 4

                  Containers are usually Linux processes with their own filesystem. Sandboxing can be good or very poor.

                  Can you just run a container with a standalone executable? Is that desirable?

                  Not desirable. An advantage of containers over VMs is in how easily the host can inspect and modify the guest filesystem.

                  1. 5

                    Not desirable.

                    Minimally built containers reduce attack surface, bring down image size, serve as proof that your application builds in a sterile environment and act as a list with all runtime dependencies, which is always nice to have.

                    May I ask why isn’t it desirable?

                    1. 1

                      You can attach to a containerized process just fine from the host, if the container init code doesn’t go out of it’s way to prevent it.

                      gdb away.

                    2. 3

                      I’m not sure if it’s as deep as you’d like, but https://www.ianlewis.org/en/tag/container-runtime-series might be part of what you’re looking for.

                      1. 1

                        This looks great! Thank you for posting it.

                      2. 3

                        I saw them described as light weight VM’s.

                        This statement is false, indeed.

                        Then I saw people clarifying that they are really just sandboxed Linux processes.

                        This statement is kinda true (my experience is limited to Docker containers). Keep in mind more than one process can run on a container, as containers have their own PID namespace.

                        If they are just processes, then why do containers ship with different distros like Alpine or Debian?

                        Because containers are spun up based on a container image, which is essentially a tarball that gets extracted to the container process’ root filesystem.

                        Said filesystem contains stuff (tools, libraries, defaults) that represents a distribution, with one exception: the kernel itself, which is provided by the host machine (or a VM running on the host machine, à la Docker for Mac).

                        Can you just run a container with a standalone executable? Is that desirable?

                        Yes, see my prometheus image’s filesystem, it strictly contains the prometheus binary and a configuration file.

                        In my experience, minimising a container image’s contents is a good thing, but for some cases you may not want to. Applications written in interpreted languages (e.g. Python) are very hard to reduce down to a few files in the image, too.

                        I’ve had most success writing minimal container images (check out my GitHub profile) with packages that are either written in Go, or that have been around for a very long time and there’s some user group keeping the static building experience sane enough.

                        1. 3

                          I find the easier something is to put into a docker container, the less point there is. Go packages are the ideal example of this: building a binary requires 1 call to a toolchain which is easy to install, and the result has no library dependencies.

                        2. 2

                          They’re not just processes: they are isolated process trees.

                          Why Alpine: because the images are much smaller than others.

                          Why Debian: perhaps because reliable containers for a certain application happen to be available based on it?

                          1. 1

                            Afaik: Yes, you can and yes, it would be desirable. I think dynamically linked libraries were the reason why people started to use full distributions in containers. For a Python environment you would probably have to collect quite a few different libraries from your OS to copy into the container so that Python can run.

                            If my words are true then in the Go environment you should see containers with only the compiled binary? (I personally installed all my go projects without containers, because it’s so simple to just copy the binary around)

                            1. 3

                              If you build a pure Go project, this is true. If you use cgo, you’ll have to include the extra libraries you link to.

                              In practice, for a Go project you might want a container with a few other bits: ca-certificates for TLS, /etc/passwd and /etc/group with the root user (for “os/user”), tzdata for timezone support, and /tmp. gcr.io/distroless/static packages this up pretty well.

                              1. 1

                                You can have very minimal containers. Eg. Nix’s buildLayeredImage builds layered Docker images from a package closure. I use it to distribute some NLP software, the container only contains glibc, libstdc++, libtensorflow, and the program binaries.

                          1. 2

                            The “only one RUN block” thing really irks me, every time I see it recommended. Trying to minimize layers this way is an uphill battle, and if you are concerned about it you will be tempted to make bad decisions like omitting LABELs or ENVs etc that make your image much more useful at the expense of layers. Use multiple RUN commands; benefit from individual RUN command caching and more legible Dockerfiles; use a tool to squash your layers post-build. Consider using something like cekit as your image source instead of Dockerfiles.

                            “Build your image FROM scratch” is bad advice (and a misleading heading, because the section talks about not doing this, but using multi-stage builds. Which is a good idea.)

                            I particularly like the “Keep your images in two registries simultaneously” advice.

                            1. 2

                              “Build your image FROM scratch” is bad advice (and a misleading heading, because the section talks about not doing this, but using multi-stage builds. Which is a good idea.)

                              May you elaborate on why using FROM scratch as the last stage is bad advice? I sincerely wouldn’t want something to be wrong in the post. Thanks!

                              1. 2

                                What you are actually suggesting in the article is fine advice, IMHO: but the heading “Build your image FROM scratch” is a little misleading because you aren’t building FROM scratch; your build phase is doing so from (in this case) Debian; which is fine.

                                1. 2

                                  Oh, alright!

                                  I’ll try to change it up for something more precise. Thanks!

                            1. 1

                              Hey @ricardbejarano – just wanted to flag that your code portions are unreadable. It’s turning the background grey with a white text on top.

                              1. 1

                                Hey, thanks for pointing it out!

                                I’m working on it.

                                Edit: should be ok by now.

                              1. 1

                                I’m loving the “log stdout” part, everything else can basically be ignored.

                                1. 2

                                  That’s definitely an improvement over the syslog situation, at least for our deployments. The native Prometheus export is neat as well; saves having to build an adapter to run alongside for metrics.

                                  1. 2

                                    I was partly joking. Not being able to log to stderr or stdout has caused so many problems because it’s basically impossible to debug haproxy without syslog being present (and HAProxy has the annoying tendence to stop logging if syslog hangs up such as happens when the network has a hiccup in an rsyslog sitaution)

                                    1. 1

                                      That exporter is one of the oldest: https://github.com/prometheus/haproxy_exporter

                                      1. 2

                                        Nope, this is a new, exporter-less endpoint, built into HAProxy itself: https://www.haproxy.com/blog/haproxy-exposes-a-prometheus-metrics-endpoint/

                                  1. 7

                                    but most Docker images that use it are badly configured.

                                    Man, this has just been the trend lately hasn’t it? Official Java images using “mystery meat” builds, root users improperly configured in Alpine for years.

                                    This isn’t meant to be an off-topic rant, but I think simply backs up the central point of the article. Containers are subtly different than a VM, but on the other hand, they’re also not “just a process” as some would like to believe. The old rules of system administration still apply and pretending that containers will fix all your problems puts you in situations like this.

                                    I’m a huge advocate of containers, but they’re definitely easy to learn and difficult to master. If you want to run containers in production reliably, you need to have a solid understanding of all the components which support them.

                                    1. 6

                                      I’m a huge advocate of containers, but they’re definitely easy to learn and difficult to master. If you want to run containers in production reliably, you need to have a solid understanding of all the components which support them.

                                      Yes, it’s a massive amount of details. I’m working on a prepackaged template for Dockerizing Python applications (https://pythonspeed.com/products/pythoncontainer/), and to build good images you need to understand:

                                      1. Dockerfile format, including wacky stuff about “use syntax A not syntax B if you want signals handling to work”. (see https://hynek.me/articles/docker-signals/ for that and all the other tiny details involved).
                                      2. Docker layer-image format and its impact on image size
                                      3. Docker’s caching model, as it interacts with 1 and 2.
                                      4. The way CI will break caching (wrote about this bit in earlier post: https://pythonspeed.com/articles/faster-multi-stage-builds/)
                                      5. The details of the particular base image you’re choosing, and why. E.g. people choose Alpine when it’s full of jagged broken edges for Python (and probably other langauges)—https://pythonspeed.com/articles/base-image-python-docker-images/
                                      6. Operational processes you need to keep system packages and Python packages up-to-date.
                                      7. Enough Python packaging to ensure you get reproducible builds instead of random updates every time.
                                      8. Random stuff like the gunicorn config in this post (or if you’re using uWSGI, the 7 knobs you have to turn off to make uWSGI not broken-out-of-the-box).
                                      9. Be wary enough of bash to either avoid it, or know about bash strict mode and shellcheck and when to switch to Python.
                                      10. Not to run container as root.

                                      And then there’s nice to haves like enabling faulthandler so when you segfault you can actually debug it.

                                      And then there’s continuous attention to details even if you do know all the above, like “oh wait why is caching not happening suddenly… Oh! I’m adding timestamp as metadata at top of the Dockerfile and that invalidates the whole cache cause it changes every time.”

                                      Some of this is Dockerfiles being no good, but the majority is just the nature of ops work.

                                      1. 2

                                        At the risk of going even further off-topic: do you have any recommendations for properly applying the old rules of system administration to containers? For example, frequent updating of docker containers could be a cron job that stops the container, then rebuilds with the dockerfile (unless there’s a better way to do live-updating?), but how do you handle things like setting a strong root password or enabling ssh key auth only (with pre-configured accepted keys) when the container configuration is under public source control?

                                        1. 2

                                          Typically you don’t run ssh in a container, so that’s less of an issue. For rebuilds: Dockerfile is more of an input, so update process is usually:

                                          1. Build new image from Dockerfile.
                                          2. When that’s done, kill old container.
                                          3. Start new container.

                                          And you really want to rebuild the image from scratch (completely, no caching) once a week at least to get system package updates.

                                          1. 1

                                            There are legitimate cases where running ssh in a container is desired (e.g. securely transferring data).

                                            Anyways, what about the root password bit of my question?

                                            1. 2

                                              First thing that comes to mind: you can copy in a sshd config that allows public key auth only, and pass in the actual key with a build arg (https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg)

                                        2. 2

                                          Containers are subtly different than a VM, but on the other hand, they’re also not “just a process” as some would like to believe.

                                          I might be one of the “some” that “would like to believe”, as I said containers are isolated processes in a recent blog post.

                                          I still think containers are processes. Let me explain and please correct me if I’m wrong:

                                          A container might not end up as a single process, but it definitely begins as one, execing into the container’s entrypoint and maybe forking to create children. Therefore they might not be a single process but a tree, with the one that execs into the image’s entrypoint as the root of the tree.

                                          And while under my logic of container = process you could call everything a process (i.e., the operating system, because it all begins with PID 1), it’s not all that wrong, and I said so in my post so that people realised containers are more like processes than VMs.

                                          Therefore, is it right then, to define “container” as “a heavily isolated process tree”?

                                          1. 1

                                            That definition makes sense if you use the terms “docker container” and “containers” interchangeably, but that is not the case (as you point out in your article!). Containers are a collection of namespaces and cgroups; are emphasized as a container literally consists of these underlying linux-provided components.

                                            Seeing that you can create all of these items that make up a “container” without ever having a process running within it I think that’s a good reason that the “a heavily isolated process tree” definition is not accurate.

                                            1. 1

                                              Ooh… Alright, thanks!

                                        1. 1

                                          The best base image you can use is FROM scratch + Python + your source.

                                          1. 7

                                            I’m surprised that musl is bigger than glibc. Isn’t size and simplicity like the whole point of musl?

                                            1. 4

                                              Yup, I was surprised at it too!

                                              I’ll try to get the musl image to build statically, as that’s what musl was thought for, and in containers there’s no point in dynamically linking. So that might reduce size.

                                              But yeah, weird.

                                              Taking a glance with dive I see the musl-based binary takes 14MB while the glibc image has a 3.5MB HAProxy binary. The /lib folder takes 7.9MB in glibc and 5.3MB in musl.

                                              Weird. I’ll look into in over the weekend. Thanks!

                                              1. 6
                                                $ ls -lh /lib/libc-2.29.so /lib/musl/lib/libc.so /lib/libc.a /lib/musl/lib/libc.a
                                                -rwxr-xr-x 1 root root 2.1M Apr 17 21:11 /lib/libc-2.29.so*
                                                -rw-r--r-- 1 root root 5.2M Apr 17 21:11 /lib/libc.a
                                                -rw-r--r-- 1 root root 2.5M Apr 16 13:49 /lib/musl/lib/libc.a
                                                -rwxr-xr-x 1 root root 595K Apr 16 13:49 /lib/musl/lib/libc.so*
                                                

                                                Sounds like an issue with compile flags or whatnot.

                                                1. 1

                                                  The author doesn’t need the .a, do they? I thought the .a were the static libraries. I don’t think I’ve seen them since I did a Linux From Scratch build over 10 years ago.

                                                  1. 2

                                                    I’m not sure how cc -static works to be honest, I just included it to demonstrate that musl is smaller on my system both as a dynamic library and a static one.

                                                    1. 1

                                                      cc -static should look into the .a file (which is an archive of .o files) and pick out only the parts actually needed to build the static binary and then you don’t need the .a file anymore.

                                              2. 1

                                                I’ve been looking into if during my free time. I’ve managed to get rid of all other shared objects but libc, whose removal would cause a segfault. Adding CFLAGS="-static" and LDFLAGS="-static" to the make step doesn’t help.

                                                It does not reduce binary size, though, right now it’s down to 18.6MB the image and 17.2M the binary (with the other objects statically linked, of course).

                                                See the changes in this branch.

                                              1. 7

                                                What guarantees do we have that it will be kept up to date?

                                                It might be the most secure image today but without security patches it might not matter that much.

                                                1. 7

                                                  The same guarantees that any other free project will be kept up to date.

                                                  It’s a simple dockerfile, if a new library version comes out, update the file and rebuild the container.

                                                  1. 4

                                                    Yea, but we’re depending on the author here to do so. If the author has a cool CI that checks for dependency updates and attempts to auto-rebuild and get new hashes, that’d be a cool automation step that could be open sourced and applied to a bunch of other stuff.

                                                    This is a big problem with Docker containers in general. Unless you rebuild them regularly or have tools to scan your containers for out of date libraries/dependencies, security problems can creep up. Sure if they break your app, they’re stuck in the container, but what if you’re running a Kernel with cgroup vulnerabilities? It’s unlikely, and the layers of security do help make breaching more difficult, but it also makes updating all the micro-components more challenging as well.

                                                    1. 2

                                                      It’s almost fully automated:

                                                      1. I have a system set up that checks the software I rely upon (HAProxy, PCRE, NGINX, etc.) for available upgrades, and sends me a report every day at 5AM.

                                                      2. I read my email first thing in the morning, at around 5:15AM.

                                                      3. It takes me less than a minute to tick up a version number, change a tarball checksum, commit, tag and push to GitHub.

                                                      4. I have automatic build rules set up on the Docker Hub to build both variants of the image as soon as a new tag is emitted on the GitHub repo.

                                                      5. The Docker Hub takes between 0 and 2h to start building my image, and takes about 20m to build it.

                                                      I tried to put it in code, I think it makes it easier to understand:

                                                      t_patch  = t_notice + t_fix
                                                      
                                                      t_notice = t_alert + t_delay
                                                      0 <= t_alert <= 1d
                                                      t_delay = 15m
                                                      >>> 15m <= t_notice <= 1d15m
                                                      
                                                      t_fix = t_push + t_queue + t_build
                                                      t_push <= 1m
                                                      0 <= t_queue <= 2h
                                                      t_build = 20m
                                                      >>> 21m <= t_fix <= 2h21m
                                                      
                                                      Therefore: 36m <= t_patch <= 1d2h36m
                                                      

                                                      It (theoretically) takes a minimum of 36m and a maximum of 1d2h36m, assuming there’s no other cause preventing me form pushing the changes and increasing t_delay.

                                                      If for whatever reason a build fails, I get notified via email, and the loop reiterates.

                                                      One of the main reasons of having only the files required during runtime inside the image is that I don’t have to worry about vulnerabilities found on other software packages I happen to carry from the base image, I only care about HAProxy, it’s dependencies (PCRE, zlib, OpenSSL, Lua and libreadline) and the toolchain (GCC, G++, Perl and Make).

                                                      The whole process is verified by GitHub (see the commits page, click on the green ticks next to the commit timestamp. All my commits are signed via GPG, too.

                                                      I don’t want to fully automate it because I like to read the changelogs and manually do some checks before applying the changes. I also can’t as I have to manually introduce the private’s key passphrase before commiting.

                                                    2. 1

                                                      No, Debian has been keeping my systems patched and up to date since 1995.

                                                  1. 3
                                                    1. Trying to statically build the musl-based variant of my haproxy Docker image.

                                                    2. Writing a post about popular misconceptions about containers for my blog.

                                                    3. Waiting for a response on my application for a paid summer internship position at the Barcelona Supercomputing Center.

                                                    1. 5

                                                      So I’ll have to do my own tuning via Nix. But I reckon I can get this down to a cook 1MB at a least. I’ll post back with a nix config in the next day or so.

                                                      1. 3

                                                        You can probably get it much smaller by stripping the binary. The glibc image has 8.8k symbols IIRC. Yet I’m still to find a great reason to strip the binaries besides size.

                                                        You could also replace ZLIB with LIBSLZ, which I believe is smaller and 100% compatible.

                                                        You can also remove liblua and save some extra kBs.

                                                        By smallest I meant size not in bytes but in file tree. The musl-based image is dynamically linked and has only 10 files. I’ll try to get it to build statically the next weekend and remove the whole /lib directory, leaving a total of just 5 files to run full-blown HAProxy.

                                                      1. 12

                                                        Excuse my ignorance, but how do we know this is the world’s most secure image?

                                                        1. 6

                                                          Allright, allright… It’s a little baity.

                                                          I said this (and said it with my NGINX image in the past) because every other Docker image I run into either is based in one of the OS images (Ubuntu, Debian, Alpine, etc.) therefore have one or many shells, the whole UNIX toolset, a package manager, etc. Or have a binary that doesn’t have the most basic exploit mitigations like RELRO, NX, PIE and SSP, making it somewhat vulnerable to stack overflows should {HAProxy,NGINX} have an unknown vulnerability.

                                                          1. 2

                                                            Thanks for the clarification! Appreciate the transparency.

                                                            1. 0

                                                              Removing basic tools does not improve security significantly. It’s an obsoleted concept since more than a decade, as tools like metasploit inject feature-rich shells and toolsets as attack payload. There are many reasons for an attacker to use a custom environment over the OS shells and tools.

                                                              Also, legitimate intrusion detection, vulnerability management, access control and logging systems rely on having a “full” OS to run their own daemons, probes and so on.

                                                          1. 5

                                                            This is pretty well done. Great job!

                                                            1. 2

                                                              Thank you!

                                                            1. 26

                                                              This law targets link aggregators like lobster.rs explicitly. Nobody has any idea how it should work in practice but presumably linking to the bbc now incurs an obligation to pay. A sad day for the internet as yet another promising future is crushed beneath those who use the law as a means to oppress the populace.

                                                              1. 4

                                                                How can a site like this one be affected by this law?

                                                                Correct me if I’m wrong, but, lobste.rs:

                                                                1. doesn’t make money off the content they host,
                                                                2. it hosts links (not quotes, not summaries, …), they are giving the original author more visibility;
                                                                3. it also hosts discussion, which I believe is still a human right.

                                                                If someone where to acuse Lobsters of damaging content creators (which is what this law is all about, isn’t it?) how would that differ from taking me to court for sharing a link in a closed chat group, or even IRL?

                                                                Lobsters is by the community, for the community, it’s not one large corp promoting their stuff (I could see the argument made against HN as it’s YCombinator’s site and it hosts some publicity from time to time), that does not differ IMO to sharing things IRL, and we surely won’t stop from sharing links to our friends, will we?

                                                                If this law goes agains’t Lobsters for those reasons, then I will understand all the noise made around this directive.

                                                                1. 2

                                                                  it hosts links (not quotes, not summaries, …)

                                                                  well, depends on the links you have. technically, torrent sites which host only magnet links should be fine, too, but aren’t.

                                                                  1. 3

                                                                    Torrent sites and alike are link aggregators for copyrighted material. It’s a link to something that’s already illegal to distribute, therefore torrent sites are redistributing copyrighted material, which goes against copyright.

                                                                    But lobste.rs’ submissions link to direct sources of information, where you can see the content how it’s creator wanted it to be. Sometimes paywalled articles are downvoted here because not everyone can read them. If lobste.rs were redistributing copyrighted material it wouldn’t be paywalled.

                                                                    A clear example of the opposite is https://outline.com, which displays the content of a third party without it’s consent, and without keeping the shape and form of how the author wanted it to be.

                                                                    Example:

                                                                    • This link is not illegal. It’s the primary source of the content, you are seeing the content how it was meant to be and from the creator’s own site.
                                                                    • This link is illegal, it’s a secondary source of copyrighted material, if the creator decides to paywall it, modify it, put ads on it, etc. They can’t.

                                                                    Lobsters links are to the direct (sometimes indirect, but it’s an unwritten rule to post the direct) source of the topic.

                                                                    I ignore if the EU-approved directive would put Lobsters in the “copyright infringement” bucket, if it does then I repeat my previous point: if sharing links online with other people is illegal, where do you draw the line so that sharing links with IRL friends isn’t, because that would be an obvious violation of free speech?

                                                                    1. 3

                                                                      Agreed. Threadstarter/OP is being way hyperbolic.

                                                                      This is probably not the best law that could have been enacted, but it’s also a fact that American companies like Google and Facebook have been lobbying heavily against it. A lot of online rhetoric is reflective of this.

                                                                    2. 1

                                                                      Such links are off-topic for this site.

                                                                      Only legitimate pointer to a torrent link for this site’s rules is for an open-source distribution. But in that case, it’s more appropriate to link to that project’s release page.

                                                                  2. 7

                                                                    actually lobste.rs is exempt because it earns less than 10million euros, and if its for educational or research use its exempt as well.

                                                                    just as sites like Wikipedia are exempt

                                                                    1. 26

                                                                      No.

                                                                      According to Julia Reda you have to fulfill all (not any) of those three criteria:

                                                                      • Available to the public for less than 3 years
                                                                      • Annual turnover below €10 million
                                                                      • Fewer than 5 million unique monthly visitors

                                                                      Since lobste.rs is nearly seven years old an upload filter is required.

                                                                      1. 4

                                                                        Reads like it was designed specifically to prevent newcomers to the field.

                                                                        Clever, and not subtle at all. I’m surprised to see this coming from the EU.

                                                                      2. 10

                                                                        You have to distinguish between former article 11 and former article 13 (now article 15 and article 17).

                                                                        Article 17 (requirement to try to get licenses and if you cannot get a license make sure that no licensed content gets uploaded) has the limitation regarding company size and age (as correctly stated by qznc) and Wikipedia is exempt from this.

                                                                        Article 15 however (requirement to license newspaper excerpts) does not have exemptions (according to my current knowledge). I guess however that all newspapers will again give Google a royalty-free license, because they fear that they will get less visitors without Google. Thus, in effect the only affected services are small services. Article 15 has these limits (imo not codified directly in the article, but in an annotation to the article): “The rights provided for in the first subparagraph shall not apply to private or non-commercial uses of press publications by individual users.”, but I am not sure how to interpret this “non-commercial uses by individual users” (it’s a similar grammatical construction in German).

                                                                        German Wikipedia stated that they are exempt from “article 13” (now 17). They mention their implications by article 11 (now 15), but do not mention that they are exempt from it. They state “[Article 11] could complicate our research for sources on current topics” (“Dies könnte auch unsere Quellenrecherche bei aktuellen Themen deutlich erschweren.”)

                                                                        1. 1

                                                                          I guess however that all newspapers will again give Google a royalty-free license, because they fear that they will get less visitors without Google.

                                                                          I can’t be certain but a discussion between some newspaper owners on a BBC Radio 4 weekly program on the state of media pointed out that similar laws existed already in both Germany and Spain (I think I remember the countries right) rendering Google news illegal in those countries and therefore unavailable. I don’t know if these new EU directives differ from those countries initial versions but their laws stated clearly that a license fee must be charged, therefore free licensing became illegal. The discussion revolved around how damaging it was to a number of publications whom obtained the majority if not all their revenue generating traffic from Google.

                                                                        2. 7

                                                                          Found something: I think lobste.rs is exempt from article 17 (article 13), because of the definitions in article 2. A “online content-sharing service provider” according to that definition “promotes [the content uploaded by its users] for profit-making purposes”. I think lobste.rs does not want to make any money? And then there comes the list of “educational or research use” that yakamo refers to. However, that’s only for the article 17.

                                                                          For article 15 (article 11) the relevant term is “information society service providers” and that is defined as: “information society service’ means a service within the meaning of point (b) of Article 1(1) of Directive (EU) 2015/1535”:

                                                                          Directive (EU) 2015/1535 in turn defines: “‘service’ means any Information Society service, that is to say, any service normally provided for remuneration, at a distance, by electronic means and at the individual request of a recipient of services.” (but I have troubles with “normally provided for renumeration”, because in Germany we have some terms that sound like they would mean “commercial”, but in fact they don’t).

                                                                      1. 1

                                                                        I can’t speak for my co-workers who compete and win pwn2own, but this seems like some seriously odd posturing… If the vuln is something like an overflow or a use after free it literally might be as easy as changing one character or line. Without context it’s relatively meaningless and honestly makes them seem more petty in my eyes.

                                                                        1. 9

                                                                          Pretty sure they mean fixed, built, validated and shipped across all channels and platforms.

                                                                          1. 6

                                                                            Nothing on the actual page linked has the “fixed in less than 24h” language; I think that may have been editorializing by the OP. The linked page is just a regular old security advisory.

                                                                            1. 1

                                                                              I agree, I guess this thread may need a title change.

                                                                          1. 3

                                                                            Respectfully, is that something an org can brag about?

                                                                            The time-to-patch metric heavily depends on the nature of the bug to patch.

                                                                            I don’t know the complexity of fixing these two vulns, surely fixing things fast is something to be proud of, but if they don’t want people pointing fingers at Mozilla when a bug stays more than one week in the backlog, don’t brag about it when it doesn’t in the first place.

                                                                            1. 18

                                                                              Assuming that the title refers to fixing and successfully releasing a bugfix, a turnaround of less than 24 hours is a huge accomplishment for something like a browser. Don’t forget that a single CI run can take several hours, careful release management/canarying is required, and it takes time to measure crash rates to make sure you haven’t broken anything. The 24 hours is more a measure of the Firefox release pipeline than the developer fix time; it’s also a measure of its availability and reliability.

                                                                              1. 10

                                                                                This. I remember a time when getting a release like this out took longer than a week. I think we’ve been able to do it this fast for a few years now, so still not that impressive.

                                                                              2. 6

                                                                                As far as I can tell, the org isn’t bragging; the “less than 24h” boast is not present on the security advisory.

                                                                                1. 1

                                                                                  To be fair, you’re right.

                                                                                2. 2

                                                                                  also the bugs are not viewable - even if logging in

                                                                                  so its hard to get any context on this

                                                                                  1. 2

                                                                                    Is it possible to check the revisions between both versions, and they do not seem so trivial.

                                                                                    These are the revisions (without the one that blocks some extensions):
                                                                                    https://hg.mozilla.org/mozilla-unified/rev/e8e770918af7
                                                                                    https://hg.mozilla.org/mozilla-unified/rev/eebf74de1376
                                                                                    https://hg.mozilla.org/mozilla-unified/rev/662e97c69103

                                                                                    1. 1

                                                                                      Well, sorta-the-same but with the context is them fixing pwn2own security vulnerabilties with less than 24 hours 12 months ago

                                                                                      https://hacks.mozilla.org/2018/03/shipping-a-security-update-of-firefox-in-less-than-a-day/

                                                                                    2. 2

                                                                                      Respectfully, is that something an org can brag about?

                                                                                      I always assume it’s P.R. stunt. Double true if the product is in a memory-unsafe language without lots of automated tooling to catch vulnerabilities before they ship. Stepping back from that default, Mozilla is also branding themselves on privacy. This fits into that, too.

                                                                                      EDIT: Other comments indicate the 24 hrs part might be editorializing. If so, I stand by the claim as a general case for “we patched fast after unsafe practices = good for PR.” The efforts that led to it might have been sincere.

                                                                                    1. 3

                                                                                      For CS-related books, as a student I highly recommend https://teachyourselfcs.com. I discovered thanks to Lobsters and I’ve bought maybe 1/4 of the books featured there already. There’s also recommended video lectures, if you want to.

                                                                                      1. 1

                                                                                        I found it very useful too! I’d also recommend My Favourite Book Recommendations by catonmat

                                                                                      1. 5

                                                                                        Yet nginx runs as root inside the docker container….

                                                                                        1. 5

                                                                                          Check out Docker’s user namespaces (https://docs.docker.com/engine/security/userns-remap/)

                                                                                          They map the container’s UID 0 to a host’s UID != 0.

                                                                                          TL;DR: a container’s root ain’t the host’s root.

                                                                                          1. 6

                                                                                            Security is best in layers. Yes host root != container root, but to change this image to not run as container root is 100% less work than everything you already have done here.

                                                                                            Inside of the container, it is root. Also, breaking out of Docker containers is not ridiculously easy, but it’s not known for being very difficult either. I’m sure over time it’s gotten harder, but docker is still pretty new, I’m sure there are still tons of low-hanging fruit here. The default permissions Docker gives containers is non-trivial. Adding another layer here, especially one that is completely trivial is a no-brainer.

                                                                                            This is the security person’s perspective, always assume worst case scenarios, and try to minimize those worst cases, so that in the best case or normal case, the bad people have to work super, super hard to do bad things.

                                                                                            Never, ever make it easier for them, especially when making it harder, in this case, is 100% trivial.

                                                                                            Another way to put this, from the perspective of a bad person:

                                                                                            1. Break out of Nginx
                                                                                            2. Break out of Docker
                                                                                            3. Free host root / profit!

                                                                                            With the trivial change:

                                                                                            1. Break out of Nginx
                                                                                            2. Break into root
                                                                                            3. Break out of Docker
                                                                                            4. Free host root / profit!

                                                                                            An extra step, especially one that becomes non-trivial since all they have in the container is the nginx binary for basically zero work on your part, is again, a no-brainer to implement.

                                                                                            1. 2

                                                                                              This would make sense in the traditional scenario, where the attacker would require root privileges to do other stuff, but there’s nothing to do. The container itself has no special capabilities, it’s filesystem is as tight as it can be, the network namespace should be very limited, the attacker has no tools other than the NGINX binary and the runtime deps…

                                                                                              I know it is a very easy change, I might consider changing it to something other than root, just because there’s nothing to lose, but it would not add any “security layer” whatsoever.

                                                                                              1. 5

                                                                                                keyword: should. Trusting on should is a terrible idea. You have no control over how people run your docker container, I promise it will be as bad as you can imagine at least once, where the roots are mapped identical for some stupid reason. I’ve seen docker hosts map host / into containers. It’s obviously a terrible idea, but that doesn’t stop people from doing stupid things.

                                                                                                As for nothing to do.. One has ALL of the musl library there for the picking(s). so it’s not remotely out of possibility to write out new binaries once you get past nginx, to do pretty much anything you want. Is it more difficult than having a full shell or python laying around? Of course, but that’s the ENTIRE POINT of security.. make it HARD for bad people to do things.

                                                                                                Another thing you could do: enforce and limit file permissions (read-only by root, only nginx is executable, etc)

                                                                                                Anyways, at this point I think we have very different viewpoints. I don’t really see you seeing things from the perspective I am trying to show: that of a bad person.

                                                                                                1. 5

                                                                                                  No! Trust me, I see your point.

                                                                                                  I’ll consider changing to something other than root in the next few days.

                                                                                                  Thanks for the feedback!

                                                                                                  1. 1

                                                                                                    I agree, making it run as a low priv user inside docker would improve this already great package!

                                                                                                    1. 4

                                                                                                      After considering community feedback, including yours, I’ve decided to replace root (UID 0) as the process owner of nginx in the image with a non-root user.

                                                                                                      Check out the commit here: https://github.com/ricardbejarano/nginx/commit/77047757b3e01608c93ef17e8aaaf2a526654fa4

                                                                                                      It should be live as soon as the Docker Hub finished building the image.

                                                                                                      Thanks for your help!

                                                                                                    2. 1

                                                                                                      Thank you for seeing this! Your comment gives the paranoid among us a hope that more people can come to bring secure software to the world.

                                                                                                    3. 1

                                                                                                      After considering community feedback, including yours, I’ve decided to replace root (UID 0) as the process owner of nginx in the image with a non-root user.

                                                                                                      Check out the commit here: https://github.com/ricardbejarano/nginx/commit/77047757b3e01608c93ef17e8aaaf2a526654fa4

                                                                                                      It should be live as soon as the Docker Hub finished building the image.

                                                                                                      Thanks for your help!

                                                                                                    4. 1

                                                                                                      The filesystem can be tight, but by exploiting a vulnerability in nginx, an attacker may be able to upload anything to the filesystem (as nginx is running as root). It’s not like nginx doesn’t already have code to download and store stuff.

                                                                                                  2. 2

                                                                                                    But is it treated like root in the container? If yes, that’s still a huge problem.

                                                                                                    Either way, this is a terrible solution.

                                                                                                    1. 3

                                                                                                      It’s treated like root in that it has rwx on the whole filesystem, that is the image’s contents and any volume you mount on it.

                                                                                                      But any special Linux capabilities like NET_ADMIN are not granted unless the Docker host says so.

                                                                                                      1. 1

                                                                                                        After considering community feedback, including yours, I’ve decided to replace root (UID 0) as the process owner of nginx in the image with a non-root user.

                                                                                                        Check out the commit here: https://github.com/ricardbejarano/nginx/commit/77047757b3e01608c93ef17e8aaaf2a526654fa4

                                                                                                        It should be live as soon as the Docker Hub finished building the image.

                                                                                                        Thanks for your help!

                                                                                                    1. 4

                                                                                                      Is avoiding bloat by default premature optimization? I think not. It’s just a good, engineering tradeoff in many situations. A quick example is how people might have a bloated stack running on a $10-20 VM. A lean, efficient stack might run on a $2.50 VM with 512MB of RAM. They save money. If money doesn’t matter itself, they have extra that can go to other capabilities like their next project that runs in parallel, high-availability for their older one, backups for their data, something other than just Netflix, and so on. All because one app was more efficient than another. This concept can scale even bigger if it’s a big business making these decisions. :)

                                                                                                      1. 3

                                                                                                        And that’s just avoiding bloat.

                                                                                                        Container startup time has been proved to be linked to image size, too.