1. 3

    Update: as folks here have surely seen, last month Qualys reported exploits against qmail 1.03. We fixed them somewhat thoroughly in notqmail 1.08, and for 1.09 we’re aiming to build with zero compiler warnings. This will let us ratchet up the warning level and add more static analysis tools. 1.08’s source tree also includes our first few unit tests, and when coverage improves a bit more we can start getting benefit from valgrind.

    1. 8

      Will notqmail change qmail’s accept-then-bounce behavior, or at least give the user the option to do so? I’m talking about the fact that qmail-smtpd doesn’t check that the recipient of a message actually exists, and then another component (qmail-send, I think) sends a bounce back to the sender if the message can’t be delivered. This means innocent bystanders get bounces for spam that they didn’t actually send. And this is one reason I haven’t used qmail in the past 9 years or so, though I used to be a big fan.

      1. 9

        Thanks for the reminder to explicitly include SMTP recipient validation in our list of planned features; I’ve added it. On the release roadmap, you’ll see “extension API for qmail-smtpd” (qmail-spp) slated for a few releases from now. The SMTP recipient-validation programs in my rejectutils run under qmail-spp (or the RCPTCHECK patch) in any combination and order configured by the sysadmin. Needless to say, I plan to submit rejectutils for inclusion in notqmail. :-)

        1. 5

          Just filed Configurable validation of SMTP recipients. @mwcampbell, does the proposed solution address your concerns?

        2. 9

          That behavior, when abused, is called backscatter. It is a frustrating problem in qmail, both because the software is fatally susceptible to it out of the box and because correcting the problem grates against qmail’s design. Specifically, you need to know in your SMTP session whether the mailbox you are accepting delivery for exists, and qmail-smtpd is not only not designed to know that, it may be running on a different host that the user account with the mailbox, or your system may be using any number of mechanisms (Unix users, the qmail-users mechanism, the plethora of routing and delivery methods including virtualdomains) that need to essentially be pulled forward in to qmail-smtpd–in the worst case bringing your entire system with it.

          One example of how this is solved today is Erwin Hoffmann’s recipient extension. With stock qmail you can use a default rule to deliver mail to /dev/null and fail to generate any bounces–itself certainly unsatisfying but no worse than the disease.

          I cannot say in what manner notqmail will resolve this issue–only that we’ll do so consistent with qmail’s other features or with extensions that integrate well with each other.

          1. 2

            Runtime-configurable SMTP recipient validation will likely be in the next release, reusing existing code as mentioned in my previous comment.

          1. 5

            We’ve shipped notqmail 1.08, addressing these vulnerabilities (among other things). Lobste.rs discussion of notqmail 1.08

              1. 1

                One of the questions I’ve had looking at software development and the culture around software development is this: how much do we use our tools versus how much the tools use us?

                I know that sounds a big vague and fluffy, but the point of the essay, if I understood it, was that the process of creating components changes the people who are creating them as much as it does the codebase.

                It’s a fascinating idea.

                1. 3

                  I’m fond of the related Iron Law of Tools. In Gene Hughson’s formulation, “That which does for you, also does to you.” I suppose we’re looking at a corollary: “That which you do, also does to you.” (Part of why I like switching between static and dynamic langs, IDEs and text editors, etc. Helps me see what I’m doing for, to, and with myself.)

                1. 10

                  I appreciate this article’s point of view (and not only for the notqmail shoutout ;-). Indeed, for an individual human person who knows their way around Unix, running your own mail server requires little marginal effort or attention beyond running your own server. The initial effort or attention that appears to be warranted, however, may loom large: there are many things that appear to be necessary to know inside and out, and one must at least convince oneself one has sufficiently understood enough of them to feel safe to get started.

                  It’s not only big mail vendors that got us here. We’re still paying the price for Sendmail having defined the genre. Projects like OpenSMTPD (and hopefully soon notqmail) that offer excellent defaults and just enough configurability in a human-centric way, with polished packages easily available for popular OSes, may not be sufficient to turn the tide. But there’s no question they are necessary.

                  1. 4

                    Congratulations on the release.

                    This begs a question I haven’t seen answered elsewhere though. What would be the reasons for a sysadmin to invest time and effort into (not)qmail? Other implementations (such as Postfix) seem to meet an equal standard of stability, while already providing the features you want to implement in notqmail.

                    1. 4

                      For the moment, we’re targeting sysadmins who already run (net)qmail. Much later, when we begin to approach feature parity with Postfix, notqmail might start being an interesting alternative. We hope that time will come. If you can’t figure out why you’d run it right now, you’re probably right :-)

                      1. 2

                        That makes sense, thanks :-)

                    1. 2

                      This was the script I was packaging netqmail with.

                      #!/bin/sh -ex
                      
                      mkdir -p /var/qmail
                      mkdir -p /etc/qmail
                      mkdir -p "$PREFIX/bin"
                      rm -rf /var/qmail/control
                      ln -sf /etc/qmail /var/qmail/control
                      ln -sf "$PREFIX/bin" /var/qmail
                      
                      user() {
                              if [ "$(readlink -f "$(which adduser)" | sed 's;.*/;;')" = "busybox" ]; then
                                      adduser -D -s "$1" -G "$2" -h "3" "$4"
                              else
                                      useradd -s "$1" -g "$2" -d "$3" "$4"
                              fi
                      }
                      
                      group() {
                              if [ "$(readlink -f "$(which addgroup)" | sed 's;.*/;;')" = "busybox" ]; then
                                      addgroup "$1"
                              else
                                      groupadd "$1"
                              fi
                      }
                      
                      ! group nofiles
                      ! user "$(which true)" nofiles /var/qmail/alias alias
                      ! user "$(which true)" nofiles /var/qmail qmaild
                      ! user "$(which true)" nofiles /var/qmail qmaill
                      ! user "$(which true)" nofiles /var/qmail qmailp
                      
                      ! group qmail
                      ! user "$(which true)" qmail /var/qmail qmailq
                      ! user "$(which true)" qmail /var/qmail qmailr
                      ! user "$(which true)" qmail /var/qmail qmails
                      ! user "$(which true)" qmail /var/qmail alias
                      
                      touch nroff
                      chmod +x nroff
                      export PATH="$PATH:$PWD"
                      
                      make setup
                      
                      rm -rf "$PREFIX/share/man"
                      mkdir -p "$PREFIX/share"
                      mv /var/qmail/man "$PREFIX/share/man"
                      

                      You might notice the extra difficulty caused by configuring users at packaging time, although more due to busybox choices and current systems heterogeneity.

                      I’d gladly live fine with /etc/qmail and /bin instead of /var/qmail/control and /var/qmail/bin, but two symlinks does the trick just fine.

                      Is formatting the man pages at compile time really necessary? The man command (and an nroff/groff/mandoc backend) is quite ubiquitous.

                      1. 3

                        The way I’ve packaged mail/qmail in pkgsrc (which is now notqmail-based), you get symlinks too:

                        alias -> /etc/qmail/alias
                        bin -> /opt/pkg/bin
                        boot -> /opt/pkg/share/examples/qmail/boot
                        control -> /etc/qmail/control
                        doc -> /opt/pkg/share/doc/qmail
                        man -> /opt/pkg/man
                        queue -> /var/spool/qmail
                        users -> /etc/qmail/users
                        

                        We intend to address FHS/hier(7) more directly in 1.08.

                        In pkgsrc, I’ve patched out preformatted man pages. But for notqmail, we haven’t convinced ourselves that’s the best move. I think we’re not done noodling on how to handle docs.

                        1. 2

                          Neat, thank you for sharing your packaging script.

                          Is formatting the man pages at compile time really necessary? The man command (and an nroff/groff/mandoc backend) is quite ubiquitous.

                          It has always been true that not all platforms use or make available preformatted manpages (“catpages” with a .0 extension). qmail does absolutely install them, as you observe. One significant change that has occurred in the intervening years is platforms (e.g., OpenBSD) only supporting the mandoc macro package, and not the man macro package that the qmail documentation is written in. We can no longer assume or rely on the man macro package being available even when a platform otherwise has a working man command.

                          I would like to make building the documentation optional, which would make our build-time dependency on groff optional–without rewriting the man pages we’d still have a runtime dependency on an nroff that has the man macros (groff in most cases), so the question of whether to use the mandoc/mdoc macros (now more widely supported) immediately comes up.

                          Some of those questions won’t be decidable until we’re further along with FHS support, but this is an area where both practice and tooling have changed out from under qmail, and we’ll attend to that as it makes sense to do so.

                          1. 2

                            I was about to suggest to provide the preformatted man pages in the release, but it might be better to be able to reproduce the entire thing everywhere.

                            Thank you for the description.

                        1. 8

                          Some examples of feedback we’d welcome:

                          • That you can’t try updating to notqmail yet, and what would make it possible for you
                          • That you tried updating, and it didn’t work
                          • That you tried updating, and it did work

                          Hope to hear from some folks playing with this.

                          1. 3

                            We’ve shipped our first release! Lobste.rs post here.

                            1. 10

                              The defining point of qmail was that it’s a secure mailer written by a security genius. What security-enhancing practices or tools will this project use to maintain that?

                              1. 12

                                You’re right, that’s why we’re here: thanks to DJB, the code has properties that are worth preserving.

                                Like @alynpost says, none of us can do what DJB did. One mitigating factor here is, we’re not trying to do what DJB did. He had to write qmail from scratch; we have the advantage of existing designs, interfaces, implementations, and decades of running in production.

                                Other mitigations, as @alynpost also says, include modern development techniques and tooling.

                                The biggest mitigations we have, in my opinion, are:

                                1. Working in public
                                2. Shipping frequent small releases

                                “Working in public” means we’re thinking out loud, showing our work, reacting to reviewer feedback, and making our decision-making process as explicit as possible.

                                “Shipping frequent small releases” means

                                • Our previous release will always have been fairly recent, so we’ll remember how to do it carefully
                                • Our next release is always fairly soon, so we’ll design and review a little extra carefully
                                • Any given release contains relatively few changes, so our users can probably vet them a little extra carefully

                                None of us can hold in mind all simultaneous details the way it seems DJB must have. As a community, though, we might manage to make very few terrible mistakes that survive for long.

                                1. 2

                                  Thank you for this question, @nickpsecurity. It’s a very important one, and now we’re giving a more thorough answer: https://github.com/notqmail/notqmail/wiki#why-should-i-trust-notqmail

                                2. 8

                                  In the early part of the project here, we’ve been making correctness fixes to the existing codebase: adding missing dependencies to make targets, header files to compilation units, and parameters to functions. Each of these was found by us variously and independently doing code modernization (from K&R to C89/C90) or by using tools, warnings, or errors that haven’t previously been applied to this codebase.

                                  djb is, as you say, a security genius. Unparalleled–I do not think this or any team will meet or exceed that standard working in the same manner he did. The codebase is coming on 20 years old, however, and includes features that never saw wide deployment (QMQP, the quick mail queuing protocol), behaviors that are archaic (redeclaring system functions, including build-time detection of type sizes like gid_t), and expensive design decisions (build host/build time requirement for qmail system account uid/gid to match deployment environment) that can all be improved in light of the intervening years. We’ve even got djb’s own reflection on the same, Some thoughts on security after ten years of qmail 1.0.

                                  SQLite is an outstanding example of what a modern C codebase can accomplish. Regression testing, code coverage, undefined behavior checking, and other off-the-shelf tooling can catch mistakes that a genius wouldn’t make. The security design of qmail is as good as it was when first published. We’ll continue to benefit from that while making qmail applicable to serving email on the modern Internet.

                                  1. 3

                                    SQLite is an outstanding example of what a modern C codebase can accomplish.

                                    I’m actually curious about this one. In another thread, I said we shouldn’t use as examples codebases from unbelievably-good programmers since they’re probably not representative of what most people will do in the language. SQLite is incredible in its QA. What I want to know is whether people who have seen the codebases of Hipp and his contributors think they’re several cuts above the rest to the point that their brains are driving the quality, that it’s mostly their process (esp testing), or some combo.

                                    The question is whether we can use SQLite as an example of what a random group of experienced programmers with some dedication could accomplish in C code or if it mostly derives from its core contributors. If the former, then what a “modern C codebase can accomplish” in the general case. If the latter, then it’s not unless you have world-renown talent with QA focus.

                                  2. 6

                                    Another defining point of qmail is that the author won’t accept patches. Obviously this is a reaction to that.

                                    1. 5

                                      We’ve got an import of the netqmail tree, and I’ve collected variously some of the patches that have been published against it in to branches:

                                      Notably missing here is the John M. Simpson’s DKIM patch, as it’s staged on top of his combined patchset and I haven’t pulled out the bits necessary to get it to apply to standalone netqmail. Certainly also there are patches available that implement the same features as this set, or solve different problems. I wouldn’t call this work here complete.

                                      Not everything here will go in to notqmail–it’s that much of qmail’s patches have been prepared by individual contributors and there has been relatively little work organizing, curating, or integrating it.

                                    2. 4

                                      @alynpost @schmonz @josuah @gerikson

                                      A few more Lobsters working on it increases the potential a bit. Excellent answers, too, both to my questions and the others. This looks promising. I hope you get the extra contributors once they see it’s something you’re committed to. Although yall might have read it, I’ll throw in the security strategy (pdf) of Qmail in case anyone interested in contributing is wondering what some of the comments are talking about.

                                      1. 3

                                        Dan Bernstein does carefully designed and simple interfaces that can be implemented on today’s and tomorrow’s POSIX systems.

                                        It is possible to follow the same design patterns, avoiding to be cleverer than needed, which would probably end in obfuscated, bug-prone code.

                                        For instance, Qmail lacks built-in TLS support.

                                        Some patches added TLS support through linking to Open/Libressl. Instead, notqmail uses ucspi client to implement TLS in a separate address space by an independent program. – from wiki

                                        Likewise, early DJB code does not looks like the most recent, such as for the substdio_*() functions that he later replaced by the buffer_*() library.

                                        1. 4

                                          To be clear, notqmail doesn’t have TLS support yet either. As of today, we’re a handful of commits ahead of netqmail 1.06, and have not yet published our initial notqmail 1.07 release. TLS will happen sometime at least a few releases after that, when we’ve provided sufficient extension points for TLS to be implemented outside qmail core code.

                                      1. 4

                                        Oh I do hope this finally lands DKIM support in a way that is palatable. That would be super, and one of the only things I’ve ever longed for after running qmail in prod for 12 years.

                                        1. 4

                                          I’ve always envisioned providing enough interfaces such that DKIM could be implemented as Unix filters. I believe this is exactly how mbhangui implemented DKIM in his IndiMail fork. When the time comes, that’s probably how we’ll go about it in notqmail.

                                        1. 7

                                          notqmail is a few weeks old, and we’ve been putting the finishing touches on our 1.07 release. Our early focus is on platform fixes and packaging, as users overwhelming expect software to be available as a precompiled binary or otherwise available in the distribution mechanism provided by their operating system or platform. This release supports the DESTDIR environment variable and removes the requirement for the qmail system accounts to exist at build time, permitting a binary package to be prepared without requiring root access. The build works on case-insensitive filesystems (Mac OS X) and platforms that have fully deprecated utmp in favor of utmpx (FreeBSD).

                                          After this release we’ll focus on fine-grained support for FHS and platform-specific install layout–systematically removing barriers to building and packaging. At the same time, we’ll continue making correctness fixes and adding features–In the fullness of patches that are available for qmail, many of them touch the same code. qmail-smtpd most particularly. We’ll be taking the work started in netqmail with QMAILQUEUE and making an extension interface so these features can be used without requiring them to be merged in to the codebase. This lets us use the security design qmail already has, adapted for the demonstrated problems existing patches solve for.

                                          If any of you are interested in testing, please claim a distribution and try the build+package instructions. I’m happy to make testing equipment available to you, if you need it.

                                          I’ve really enjoyed working with my co-contributors schmonz (@schmonz on Lobsters), DerDakon, and mbhangui. I look forward to our first release.

                                          1. 3

                                            Are you in touch with the netqmail people? The correctness fixes seem to be in line with the goals of that project.

                                            1. 5

                                              I’ve been in touch with – and gotten encouragement from – a few. If we’re really lucky, maybe after a few notqmail releases they’ll decide it’s worth the time and energy to join us. We’d be very happy to have them, of course, and also very unsurprised if their lives have taken them in other directions.

                                              1. 4

                                                The last netqmail release was November 30th 2007, upon djb placing qmail-1.03 in the public domain. That is over a decade ago. It’s a reasonably small community, the set of contributors to qmail, and I believe some of the almost dozen folk involved in netqmail-1.06 are aware of notqmail. I am happy to work with any of them who are still interested in the project, but that interest has not been demonstrated by any changes, patches, releases, or even news in over a decade.

                                            1. 2

                                              I wonder if the ISO9660 trick I found my way to in 2006 (described here) would have worked just as well. Miss my old NetBSD G4 mini.

                                              1. 1

                                                No hardcoded or pre-allocated numeric IDs

                                                One of my own pain points in my qmail deployment is that I build on one set of hosts and deploy to another. The fact that qmail has build-time dependencies on the role accounts it uses means these uid/gid must be the same between all of my hosts. @schmonz Will you talk about how you removed the need for pre-allocated numeric IDs? Is that patch reasonably self-contained?

                                                1. 3

                                                  That was one of my pain points, too. You want the destdir patch. If you don’t care about building to a staging area as non-root, skip that part. You’ll still get numeric IDs being looked up at runtime instead of compiled into the binaries.