1. 59
  1. 8

    What’s your plan for sustainability? This is a gigantic project; how do I know it will be maintained 5-10 years from now?

    1. 14

      I can’t know for sure, and I’ll take any tips/hints. (: I am using it for my own email, so at least there’s that incentive to keep it going. It would certainly help to be with more! I also want to keep the maintainer burden low. No separate website. Releasing is mostly just adding a tag. The tests should help keep the code base in working order. And I wondered early on how to keep all the standards/RFCs in my head, and decided to heavily cross-reference the code with the RFCs, which helped a lot. Also, email is not evolving at a high pace… Once functionality is working it may not require all that much ongoing development.

      1. 4

        This looks really impressive. The only thing on the not-yet-implemented list that I would miss is Sieve support. Do you have some documentation on your privilege-separation model?

        1. 3

          this is all one process. go is supposed to do a good part of the protection. i imagine resource (ab)use could be a issue: memory and file descriptors. i’m aware of openbsd privsep principles. would you have ideas on where separations would be good to have?

          and about sieve: i’ve never used it. how does one use it with current mail stacks? from memory, i think it is a way to match messages and take action on them, like moving them to a mailbox, or possibly set flags? how does one configure the rules? just editing a text file on a server, in a web interface, or in a mail client?

          1. 8

            go is supposed to do a good part of the protection

            That protects you against most memory safety bugs (though Go is not memory safe in the presence of concurrency - data races on slices in objects shared between goroutines can break memory safety), but that doesn’t protect you against logic bugs. A lot of these can be prevented by threading a capability model through your system and respecting the principle of intentionality everywhere, but that doesn’t mean that the principle of least privilege is something to ignore. Mail servers are among the most aggressively attacked systems on the Internet so it’s a good idea to aggressively pursue both.

            would you have ideas on where separations would be good to have?

            At a minimum, I’d consider separating the pre- and post-authentication steps. If an attacker compromises the pre-auth process but doesn’t have valid credentials then they should find that they’ve compromised a completely unprivileged process.

            The authentication should also then be a separate process. This may need some restricted filesystem access (or limited database connectivity), depending on how you store credentials (or possibly they’re loaded before the process starts before it drops privileges and the auth process is restarted whenever they change - with a target deployment of <10 users, that should be fairly simple), but it shouldn’t be allowed to create any network or IPC connections, or access most of the local filesystem.

            The post-auth process should be confined to being able to inherit the network connection created when it is started and having access only to the mail store for the specific user. This ensures that no bug in the code that communicates with a user can perform filesystem accesses for other users. If the backing store is a database, the same applies, just use the database’s ACLs instead.

            Some of the other services would also benefit from being compartmentalised. For example, you have spam filtering. A significant proportion of spam emails are trying to ship malware to the user, and compromising the mail server is a great way of doing this (and may avoid the need to compromise the client). Even the simple case of exhausting resources so that the next spam email gets through the filters or all email processing stops need to be in scope for this kind of threat model, so you probably want to process each inbound email in a separate process that returns a single value (spam probability) to the parent and runs with tight resource limits, so the worst that an attacker can do is push an email past the filter.

            The component that does Let’s Encrypt / ACME things almost certainly needs to be isolated - anyone who compromises that can sign arbitrary private keys. I don’t know how much of ACME you’re implementing, mail servers often use the DNS-based variant since a mail server may be pointed to by MX records that it does not have an A record for. A thing that can create and update DNS records is a very high-value target.

            Similarly, DMARC keys are high value (if they can be compromised then an attacker can send email that is indistinguishable from email that you sent). Signing should be done in a separate process that just does the signing and so even an attacker who compromises a client connection can do an online attack but can’t exfiltrate the keys.

            As I recall, Ben Laurie added support for Capsicum to the Go standard library some years back, so these kinds of thing are fairly easy to add in Go programs.

            This is just off the top of my head without thinking things through in too much detail. You can probably do a lot better understanding the shape of your code. I’d encourage you to think about three things:

            • What is your threat model? What is an attacker trying to do and any given point and what should you assume that they can do? This helps frame the next steps. For example, email is often used for password-reset links, so one attack to consider is someone intercepting a password reset link. What components do they need to compromise to get there? If they compromise one user, have they compromised all of them?
            • How do you enforce the principle of least privilege? For every component, what is the absolute minimum that it needs to have access to? Can you make that set smaller without losing functionality?
            • How do you enforce the principle of intentionality? If a component needs to have access to two bits of important state (for example, two users’ credentials or mail boxes, two different mail boxes, a mail box and a different folder on the system), then how do you ensure that it is exercising the one that it intends to?

            The last high-profile Exchange bug was a violation of the Principle of Intentionality: Exchange had access to a system location, but intended to write a configuration file into the configuration-file directory and did not use anything vaguely like a capability system to prevent this. Capsicum and similar systems make this easy: you would have a directory descriptor for the configuration directory and use it with openat and so be unable to create a config file anywhere else.

            and about sieve: i’ve never used it. how does one use it with current mail stacks? from memory, i think it is a way to match messages and take action on them, like moving them to a mailbox, or possibly set flags? how does one configure the rules? just editing a text file on a server, in a web interface, or in a mail client?

            ManageSieve is a protocol for sending Sieve scripts from the client to the server. Clients expose it in different ways. I’ve seen a couple of things that look like the rule editor in Outlook or Mail.app but I tend to use a Thunderbird plugin that exposes the scripts directly. It is a bit nicer than just editing the script on the server because ManageSieve doesn’t let you install a script with syntax errors and lets the server report the error to the client, so the client doesn’t need to know every extension that the server supports (and there are a lot)

            Dovecot also has support for IMAPSieve, which runs sieve scripts in response to events. This is most commonly used for detecting things being added to or removed from a spam folder to trigger learning. I think you have built-in support for that? It can also be used for things like providing a virtual mailbox that auto-files email according to your latest rules if you drop mails there, or running external scripts so you can copy an email with a calendar attachment into a mail box and have the attachment passed to your calendar, and other automation workflows.

            1. 2

              thanks, that’s a lot of good info!

              valid point about the logic bugs. i’m wondering how difficult it is to take over a go process. whatever the answer, having separated privileges as a layer of defence will certainly make it safer. pre-auth and post-auth, and per-logged-in-user-processes, and key-managing-processes all sound right. i’m going to put it on the todo list.

              I don’t know how much of ACME you’re implementing, mail servers often use the DNS-based variant since a mail server may be pointed to by MX records that it does not have an A record for. A thing that can create and update DNS records is a very high-value target.

              mox uses tls-alpn-01, which is why it needs port 443 (along with for mta-sts and autoconfig).

              but managing dns records is an interesting topic. i would like to be able to do that, mostly to make it easier to set up/manage mox (i believe many potential mox admins would be pasting dns records in some web interface zone import field. if they are lucky. and creating records one by one in a web interface otherwise. also, with dns management, mox could automatically rollover dkim keys in phases, update mtasts policy ids, etc). but i don’t know of a commonly implemented dns server api i would use. i don’t want to make it harder to set up mox. if anyone knows there is a way, please let me know!

              ManageSieve is a protocol for sending Sieve scripts from the client to the server

              make sense, although yet another protocol to implement… i personally am probably fine going to a web page and editing the script there. mox already has a web page you can manage (some of) your account settings in. it currently only has basic rules for moving messages to a mailbox when they are delivered, see Rulesets in https://pkg.go.dev/github.com/mjl-/mox/config. these can be edited in the accounts page.

              Dovecot also has support for IMAPSieve, which runs sieve scripts in response to events. This is most commonly used for detecting things being added to or removed from a spam folder to trigger learning. I think you have built-in support for that?

              yeah, i recently added a simple approach for setting (non)junk flags based on the mailbox a message is delivered/moved/copied into. see https://github.com/mjl-/mox/blob/ad51ffc3652ff19a1265fe2831c83ebf669ecdc3/config/config.go#L210. i looked at mail clients, but did not see behaviour to set those flags conveniently, e.g. “archive” in thunderbird does not mark a message as nonjunk, etc.

              It can also be used for things like providing a virtual mailbox that auto-files email according to your latest rules if you drop mails there, or running external scripts so you can copy an email with a calendar attachment into a mail box and have the attachment passed to your calendar, and other automation workflows.

              interesting. this is certainly not possible in mox. there isn’t even the notion of a user (uid) to run such scripts as. sounds like adding useful sieve support may need that. this is going a bit lower on the todo list. (:

              1. 2

                valid point about the logic bugs. i’m wondering how difficult it is to take over a go process. whatever the answer, having separated privileges as a layer of defence will certainly make it safer.

                I recommend not thinking about “Go processes” as a process is a process is a process.

                Another good example of real world recent vulnerability in mail servers is CVE-2020-7247 and its resulting security errata: a remote code exploit in OpenSMTPD from 2020.

                Priv-sep specifically didn’t mitigate against that vulnerability and it had nothing to do with memory safety, but being able to have a mental model of which parts of your program are operating with specific capabilities will make it easier to audit and easier to respond to inevitable vulnerabilities. It’s not a matter of “if”, but “when”.

              2. 1

                It should really be underscored that without doing the extra legwork with capabilities or bubblewrap or putting different apps under different users or really anything (there are a ton of methods here) that merely deciding to have extra functionality in a separate process owned by the same user offers absolutely no extra security protection and in the case of “process each inbound email in a separate process” has considerable downsides. You do mention restricting fs/ipc/net in the auth paragraph though. I guess what I’m saying is that end-users need to be aware there is extra work to be done there if they take that route. Sure different stack/heap so you aren’t hitting footgun issues, but once that process is owned the rest don’t matter. In this example mox suggests creating a mox user with seven additional setup commands. That’s great and I bet a lot of people ignore that and just sudo su their way to freedom cause it’s not enforced. If mjl decides to break this into separate processes there is going to be a lot more setup involved then. This is pointed out because there is a lot of language online about “just put in another process”, and then the end-user is not told or is unaware that they need to do all the extra work that is required to reap the benefits.

                1. 2

                  noted. ideally i would prefer do all the separation in mox itself, not requiring extra tools like bubblewrap.

                  about the additional commands, i probably should validate permissions at startup. mox currently only checks that it doesn’t run as root.

                  1. 1

                    Please ignore the above message. None of this needs to impact the end-user experience. Dropping privilege for a child process without creating different users is supported on all major operating systems.

                  2. 1

                    Almost none of what you say is true with any vaguely modern *NIX system operating system. They all provide mechanisms for a process to drop privileges and run with less than the ambient authority of the user that started as. FreeBSD has Capsicum, XNU has the sandbox framework, OpenBSD has pledge, Linux has seccomp-bpf / Drawbridge / whatever they are doing this week.

                    1. 1

                      I didn’t claim that. If you look at my very first sentence I’m pretty clear that there are plenty of methods to deal with this, however merely spawning a new process does not give you inherent added security which gets implied in many articles and discussions and which was my whole point.

          2. 2

            how do I know it will be maintained 5-10 years from now?

            You put your skin in the game and contribute.

          3. 6

            I saw a related post interactively reviewing your code the other day and was really intrigued. I intend to give it a shot in the near future; it might end up replacing my Postfix/Dovecot/rspamd/MySQL setup :)

            I also love the idea behind gobuild and might host an instance on my own infrastructure in a few days.

            1. 2

              good to hear! i’m interested in hearing from people that have set up mox. both if it succeeds and fails! i enjoyed that interactive review with jonathan. but who doesn’t love talking about code? (:

              gobuild is one of those ideas that may make it easier to maintain mox. i don’t have to build and release binaries! hopefully it doesn’t bring too much maintenance burden itself…

            2. 5

              Any plans for JMAP support? The protocol looks a lot nicer for modern email than the mess of IMAP + SMTP + CalDAV + CardDAV + ManageSieve, but unfortunately Dovecot has had it on the to-do-sometime list for years and seems to not be making any progress. I’d love to see more servers support it because currently it’s stuck in a limbo of clients not supporting it because servers don’t and servers not supporting it because clients don’t.

              1. 1

                clients not supporting it is part of the reason why i haven’t added support. my priorities so far have been to get it working with existing mail clients. but i’m surely not against it, with time it will probably happen. and i agree imap and the davs aren’t great. does jmap also replace the davs?

                1. 2

                  does jmap also replace the davs?

                  It’s been over a year since I looked at the specs, but that was my recollection, at least CardDAV. The main advantage from my perspective is that it separates identity from location for emails. I have a few smart mailboxes uses the virtual support in Dovecot that show me things like all emails from the last 5 days, all emails that are either unread or flagged, or all that are unread. Unfortunately, this means that IMAP clients end up downloading all of my emails four times (everything starts out in all three smart mailboxes, plus its canonical home). With JMAP, mailboxes are just views on a message store are closer to labels than folders. This has implications for the data model that you use for the backing store, which is why it’s somewhat hard to retrofit to things like Dovecot.

                  1. 2

                    Unfortunately, this means that IMAP clients end up downloading all of my emails four times (everything starts out in all three smart mailboxes, plus its canonical home). With JMAP, mailboxes are just views on a message store are closer to labels than folders.

                    i wonder if rfc 8474, IMAP Extension for Object Identifiers, could help with that. if clients support it that is… (mox doesn’t support it, though its storage does have unique id’s for mailboxes and messages, so it’s doable). this rfc is written by fastmail, and i think they also wrote the jmap rfc’s.

              2. 5

                How does it compare to maddy?

                1. 2

                  I haven’t looked at the code, so apologies if it’s obvious from it, but I would be interested in knowing which storage options the project supports.

                  1. 2

                    From the config it looks like the filesystem is the only option: https://pkg.go.dev/github.com/mjl-/mox/config

                    1. 3

                      correct. messages are stored into the file system in the form they are received. there is also database file that holds additional information, such as the smtp mailfrom, smtp/spf/dkim/dmarc validation status, message flags, prepended received/auth-results headers, etc. see https://github.com/mjl-/mox/blob/c65731ac56d8f92a134e2a8d5469e1927cefb3a6/store/account.go#L217

                      the database file is a boltdb file. that’s a transactional, high-performance, truly single-file key/value store. i’ve added a layer on top called bstore (https://pkg.go.dev/github.com/mjl-/bstore) for common database functionality like referential integrity, indexes, unique/nonzero constraints, etc. when i started prototyping, i used sqlite. sqlite is great, but i wanted mox to be pure go.

                      i could have added an option to store into an existing database. but then setting up mox would also require you to set up a database, and maintain it, etc. i wanted to keep it simple and self-contained.

                      i would still be interested to hear about storage option that folks would want, and why.

                      1. 2

                        But sqlite is pure go :P Either with mattn/go-sqlite3, or with CGO-less modernc.org/sqlite :)

                        I have a project where I have abstracted the storage into a couple of interfaces and it can be compiled alternatively with boltdb, sqlite, badger, or plain file-system storage support. I am curious if you’d be willing to accept changes into this direction.

                        1. 4

                          in my mind a “pure go application” does not include c code, which mattn/go-sqlite3 does. i’ve seen modernc.org/sqlite, but i’m not entirely at ease with the idea. a lot of code (sqlite), translated.

                          i’m afraid adding multiple storage backend options would require a relatively large storage interface. i think there are bigger fish to fry at this moment. (:

                          1. 1

                            OK, fair enough.

                  2. 2

                    Nice! A while ago I setup my own mail server with docker, and it was not fun. Hopefully this gives a better experience

                    1. 1

                      Looks great! Quite an undertaking, seems like an amazing amount of work must have gone into it. Congrats on getting so far already.

                      Does it/will it do blacklists (for remote IP, subdomain(s), domain, address), RBLs, or user+variant@domain.com addressing?

                      1. 2

                        thanks!

                        mox currently only allows (global) IP-based DNSBL’s to be configured. no support for url-based blocklists yet, but it’s now on the todo-list! is a (sub)domain BL just using an url blocklist to reject mail from listed domains, or is it something else?

                        classification as junk/nonjunk of incoming messages influences if new messages from those senders are accepted. both on ip (with subnet variants, the wider the stronger the signal must be), full sender address (also based on whether you’ve sent to them), sender domain, sender organizational domain, verified spf/dkim identities. this is all per-account.

                        the user+variant is indeed supported.

                        see “DNSBLs” and “LocalpartCatchallSeparator” on https://pkg.go.dev/github.com/mjl-/mox/config. and see https://github.com/mjl-/mox/blob/main/smtpserver/reputation.go for the message-based reputation, which is called as part of https://github.com/mjl-/mox/blob/main/smtpserver/analyze.go.

                        1. 1

                          Cool, thanks! I was thinking of blacklists where you can say e.g. reject anything from a connection from 1.2.3.4 (maybe even at helo/ehlo?), or messages from sub.domain.com, *.domain.com, domain.com, user@<any-of-the-above>.

                          It sounds like I’m just wanting to be able to short-cut the classification targets you’re describing, by explicitly saying “OK, I’ve just spam come in from and I know from previous patterns that this is now going to start spamming from multiple IPs on that subnet”. Or “I know this address is bad, it hasn’t been caught, I could move it to spam folder but I want to block not just that user or that whole domain but anything in subdomains of that domain”. I find myself doing that a lot with /etc/postfix/client_checks as 1.2.3.0/24 REJECT You are spam or whatever, or python /opt/iredapd/tools/wblist_admin.py --blacklist --add @.sub.domain.com.

                          Another Q, sorry, what’s the status with running multiple virtual domains?

                          1. 2

                            ah yes, make sense to have that at some point. i’m thinking about making these kinds settings (including dnsbl) also (only?) configurable per account, instead of globally. dnsbl already runs only at the end of the analysis.

                            the config generated by the quickstart has a single domain, but more domains can be added/removed by editing the config file, or through the cli or admin web page (which update the config file). you can get the required dns records printed just as with the quickstart. an account (which holds mailboxes/messages/imap subscriptions/etc) can have multiple addresses, in multiple domains.

                            1. 1

                              Sounds great. Thanks for the info, I’m sure I’ll be giving this a try some time. Cheers :-)

                      2. 1

                        Is there any ansible playbook or a way to configure it in an automated way?

                        1. 1

                          not that i know of. i suspect getting DNS set up automatically may be the hardest part, especially if you want to make it work for others.

                          mox is quite standalone, with just a binary, 2 config files (plus dkim keys), and a data directory. so that shouldn’t be hard to script.

                          1. 1

                            This was mostly for me, I have my personal DNS zone in a terraform file, so it would be great to create the server with TF, install mox with Ansible, send the output to a TF file and then launch TF again. It’s just a bit tricky, but not complex. I can probably work on it (at lease in the Ansible part) and then share with you if you want.

                        2. 1

                          Interesting but I feel as I’ve been FUD’d out of running my own email server as I used to do way back when because of this post and meta discussion like this HN thread. The problem is you’ll end up in a spam folder? I don’t know. I think I understand the technologies and work that would go into it but I guess I also understand how arbitrary a blocklist is and a request to get removed. If I’m doing transaction user email for a service I run, I don’t want this risk. That’s my FUD chunking anyway.