Today, many email providers have some kind of method of contacting them, but I have never once received a response or even evidence of action due to one of these messages… both for complaints of abuse on their end and deliverability problems
Strangely, one thing I was surprised about was how it was possible to get in touch with Postmasters when we had issues with blacklisting. It generally took a while to track down the proper way to contact them, but I think there’s only been one mailhost (Apple) who we never heard back from – although we were removed from their blacklist the day after. When blacklisted by Microsoft (Outlook, Hotmail, Live, etc.) I even got to chat with a helpful real person; an actual named person no less!
This was my experience also. A couple of years ago, I was trying to change my Warby Parker password after a breach on their end. The process failed at the send a nonce to Chris via email step. I could see that the cause of the failure was that I hadn’t added warbyparker.com to my greylister so I wasn’t going to get immediate delivery. Even after bypassing warbyparker.com I wasn’t getting the nonce. A letter to hostmaster@warbyparker.com got a response but I didn’t need glasses at the time. I honestly didn’t know it was fixed until today when I remembered as a result of this thread.
Maybe I’m just being in too bad of a mood! Most of this comment dates back to when I was working in university IT 6-7 years ago and spent a long time battling blacklisting problems. I don’t think I ever got responses from Google, and while they did start accepting mail from us again each time it was long enough later (a few days) that it was unclear whether someone had read my message or the blocklist entry had just hit an expiration date. The worst problems we had in the other direction were abuse coming from Google (spam and fraud email, calendar invites, and phishing via Google Forms) and I don’t think I ever saw any evidence of their acting on abuse complaints. We started blacklisting certain gmail addresses at the institutional level because they were sending so much mail to us that just had very long address lists pasted into the To/Cc. The irony is that SpamAssassin was picking this stuff up and we had to manually bump the domain reputation of gmail.com to avoid it learning to spamcan every marginal email coming from Google.
I have not seen this kind of problem since I worked there (although I haven’t run as large of a mailserver since), but I would assume Google has tightened up their controls at some point because it was remarkably brazen behavior to continue from a major email provider for, as I recall, around a year. I’d guess this was 2013 or so. This situation actually lead somewhat directly to that institution switching to Google Workspace later because it fostered a perception that the in-house operation was incompetent, which is sort of ironic. It really got at one of those no-win problems in security: we were getting phished via Google Forms on a weekly basis, but whenever we tried taking active measures like dropping email with Google Forms links it turned into a huge upset and we had to back off. When I worked in incident response in 2015-2017 phishing via Google Forms continued to be a top-ten problem, but at that organization we had somewhat better tools available to combat it and ended up using our outbound web filters to inject a redirect to a warning page the user had to click through (we were fortunate enough to have TLS interception). Google now provides some additional measures to mitigate this problem but they of course require that you be a Google Workspaces customer. I assume they’ve also stepped up detection and enforcement a lot because in more recent cases where I’ve seen Google Forms phishing, the link has often been dead by the time I saw it. In any case the whole thing left me with a very negative perception of how Google runs their email operation (which was boosted when I was involved in the pre-sales process on the switch to G-Suite, which was amazingly incompetent).
And I shouldn’t let Google color my perception of the whole industry, my recollection is that Microsoft was pretty easy to get a hold of on hotmail/outlook.com issues, although I think we may have leveraged our existing relationship with them to get a better contact than what they have publicly.
With smaller organizations it’s always been a lot easier, of course being a university much of our interaction was with other universities and postmaster@ almost always got a response from someone helpful, whether we were the source or victim of the problem. Unfortunately this situation has become worse over the years as more and more institutions and businesses are moving over to cloud services that they have limited control over and knowledge of. In my current position, where I don’t even really deal with email, I’ve run into a few cases of “something is blocking email from you but we can’t figure out what.” It’s almost always a cloud email security gateway they use, which have a tendency to be very opaque.
Although we haven’t had any issues so far sending to Google, I don’t doubt for a second that they would be a pain to deal with. We manage several dozen Google Workspaces for clients and it is constantly causing bother.
It’s also a good idea to have an alias for postmaster@ (root too, often aliases tables have that per default). There’s many systems and people that default to sending there.
Even for popular combinations, there are multiple ways to architect the mail delivery, storage, and management process. For example, there are at least 4-5 distinct architectural options for configuring Postfix to make mail available to Dovecot. Different distributions may package these services pre-configured for one approach or the other, or with nothing pre-configured at all. In fact, mail servers are an area where your choice of distribution can matter a great deal..
I’d love to read an outline of these options, along with a map of distributions which provide each option out of the box. Does such an outline exist?
That’s a good question, if I can’t find one I might take it on. That 4-5 is a very rough estimate based on my recollections, but I think it’s more likely to be an underestimate than an overestimate. The biggest difference is whether or not you choose to use Dovecot as MDA: you can either have Postfix “deliver” mail and then Dovecot read the mailboxes created by Postfix, or you can have Postfix transfer email via socket to Dovecot for “delivery.” The latter tends to be preferred these days because it allows Dovecot to do a lot of optimization by having full ownership of the mailbox, including filtering and live indexing of inbound mail and IMAP push with less performance overhead. One reason you might not do it though is if you want to use a totally separate MDA for whatever reason (neither the one built into Postfix nor the one built into Dovecot, maybe procmail for compatibility with existing procmailrcs).
An additional complication is that most MDAs support either “mbox” or “maildir” format for file system storage (maildir tends to be better for a few reasons, but there are catches), and some MDAs and IMAP servers support using a relational database instead (usually your best performance option but you can imagine it gets weird when you have multi-MB email bodies). There’s also a Unix tradition of storing mbox/maildir as the user and in the user home directory, which has benefits but can make troubleshooting a bigger headache. If you NFS mount or otherwise centralize your home directories, users are 100% guaranteed to accidentally delete their whole maildir from time to time, leading to annoying backup recovery tickets. But moving these out of the user homedir tends to violate some expectations on the part of the mail software and has its own pains. Some of the details of these catches lead to the common situation on older mail setups where the user’s inbox is limited in size compared to other folders: it used to be very common in NFS environments to put the inbox, but only the inbox, in the user’s home dir in mbox format (because all of the Linux tooling expects it to be there), but all other folders stored on the mail server in maildir format outside of home directories. Fortunately I have not seen this in a while except for, oddly, Dreamhost, which I assume just hasn’t changed their mail setup for a good while. My memory of what exactly was the main painpoint on moving the inbox is fuzzy now, but I know back when I ran mail we had already put out default mutt/pine configs to use IMAP rather than reading locally, so I think there was some other issue beyond that. I hazily think the details of the UID behavior of Procmail may have been involved.
I thought this was a very long, thoughtful opinion piece on running your own email server(s). The advise at the end of the article was especially helpful. Like:
It’s not a good idea to run a “set-and-forget” mailserver.
I am skeptical of one-click solutions like iRedMail due to how opaque email troubleshooting can be. If you want to run your own mailserver, I would strongly encourage you to take it as a learning experience and get smart on how the MTA, MDA, spam filter, etc actually work.
I run my own mailserver. I go against a couple of the suggestions on both sides of this issue. You can do it, it’s not hard but the learning curve is tedious and long. I built my first internet mailserver for just myself back in 1996 or so. I sent and received mail via hybrid SMTP/UUCP from ripco communications in Chicago. I expanded the solution to my whole domain in about 2000 when I bought a SparcStation 20 and ran my mail on OpenBSD. I also transitioned from UUCP to pure SMTP at that time. My CATV company put a stop to that when they shut down inbound port 25 in like 2001. If I had it all to do over again, today, I’d probably just delegate my mail to someone else.
I commented earlier but I would add this. Today a lot of programmers don’t know email. I occasionally still host things on real linux boxes and one of the steps in my ansible install is to configure postfix to relay mail local mail through an internal smart smtp relay. The tell question is: What email server and user/password should I use for this project. The tell answer is the look on the questioner’s face when I answer: pick one of 127.0.0.1:25, [::1]:25, or localhost:25, no username or password.
Yes, I feel this way. l do not know email, but feel like I should and I am trying to learn it. I really just want to know how to send registration and password reset emails without having to signup for another service. Then I’ll work on learning more from there.
The programming APIs for sending email are usually well document. The questions are: “What server should I use?”, “What port?”, “What username?” and “What password?” If your administrator can and has setup local relaying properly, the answers are: (“localhost”, 25, None, None)
The setup I use has one system setup to relay all mail from a group of systems. I use postfix and there are two elements to this setup:
One system, the relay, knows how to handle mail addressed from your domain to the list of destinations to which you want to send mail. And will handle mail for anything in your fleet by IP address.
The other systems, the clients, will accept mail via localhost only and with use SMTP to handshake inbound mail, not destined for local delivery, through the relay.
To configure the clients, you want to have relayhost = [therelay.example.com]:25 in postfix’s main.cf. Googling for postfix local mail relay smart host should be helpful here.
Configuration of the relay largely depends on who you need to get mail to and what mechanisms you have to deliver mail. Googling for postfix smart host or postfix smart relay should get you started. Note well that any reasonable tutorial will include settings/entries for:
inet_protocols, inet_interfaces: These postfix settings determine how your relay listens for incoming mail to relay or directly deliver.
mynetworks_style, mynetworks, smtpd_relay_restrictions: These settings control the relaying aspects of postfix. By default postfix will reject mail if it doesn’t think that it’s the destination for that mail. By tuning these three settings, you can limit who can use your postfix machine as a mail relay. These settings are important because tuning them the wrong way can make your system an open mail relay which is very bad. Usually these will be setup such that the relay will accept mail from IP addresses its own LAN without question. On the other hand clients will set this up so the only valid source for mail is localhost.
mydestination: This postfix setting determines what domains you consider to be your responsibility for delivery. Ex, a machine whose destination includes example.com will expect to be or to know the delivery port for mail to chip@example.com.
The biggest problem is usually how to get mail out to the internet. I think that my earlier comments trivialized that fact. There are generally two main ways to do it. Either, your ISP will give you the IP address and port of a system which will relay mail to the rest of the internet for you. Sometimes they will require you to use a username/password here also. The postfix tutorials that mention SASL cover this. If you run your own mailserver, and this is a much harder thing to do securely, you’ll need to get permission from your provider, either ISP or Virtual Private Server, to directly contact servers via port 25 on the internet.
I run a small fleet of Linux boxes on an otherwise Microsoft centric network for my company. We handle our mail via O365. I got permission for one of my systems to relay mail through the corporate SMTP gateways . This is my relay. I’ve configured the rest of the fleet to use my relay as a smart host for all messages.
And on that topic, if you’re going to run a mail server and you’re not using a 100% virtual approach with everything stored in a relational database, it will be well worth your time to become an expert on Linux file permissions and the SetUID behavior of your MDA and, possibly, MTA, filter, etc if they interact with file storage. Many of the real-world problems you’ll run into will turn out to be related to file permissions and what UID a specific component of the mailserver is acting as when it takes a specific action. Depending on details of the setup, processing an individual incoming email often includes steps that run as a service user, and steps that run as the user the email is being delivered to.
While I would agree with that, is there something specific that happened here? I can’t remember I had to ever deal with permissions on my own in 15 years and using files. The only thing was when I switched providers and mail server (Debian/postfix to OpenBSD/opensmtpd, as well as some other parts). But that was just backup and chown on the new system. Was very straightforward.
But but it course, running a server, having a grasp of how permissions work is a good idea and if never recommend anyone to do something else. Also obviously understand the software you are running.
I always ran without db (if you don’t count aliases db). For mail going the very Unix approach feels like the right way. After all most Unix like OSs come with some usage for mail being integrated like that (status mails, cron mailing output, etc.), so often you will find mails for users on non mail servers. See OpenBSD’z welcome mail.
One thing I’d like to add: have a good understand of DNS. That can help a lot, also elsewhere, but knowing about DNS PTR is a good idea in general, whenever you set up any kind of server and then knowing how SPF and DKIM work, not just blindly copy pasting of course it’s also a good idea.
Obviously understanding what you set up is a good idea. Since mail doesn’t change a lot you likely won’t touch it much do make sure to comment and document even the parts that seem obvious now. :-)
Main situation where I’ve run into these problems has always been related to storing mail outside of user home directories - I recommend not doing that, but the biggest upside is that it means that pretty much everything “just works” since most mail software is already set up to setuid to the recipient before trying to handle mail on disk, and expects all user config files to be in user home and read via setuid. If you start changing that situation then you become a lot more likely to run into headaches where you have mail processing happening as ‘mail’ sometimes and as the user other times, mail directories and config files owned by different users, etc. The postfix master.cf Mostly this has come up because if you’re putting a lot of functionality in your IMAP server it can become desirable to not allow the user direct access to their own mail on disk, since they bypass whatever the IMAP server is doing and that can create inconsistency and headaches (this was yet much worse in the days of POP or, worse, the days of IMAP except for five users who have never reconfigured their client).
Another adjacent source of headaches is chroot. Many distros distribute Postfix preconfigured to run under a chroot, which I think is a very good idea considering the huge attack surface with file I/O access in Postfix, but it does mean that some of your mail components need to run chrooted and others need to not be chrooted, and if you’re changing the config from defaults you will probably screw that up at least once and have a problem that’s annoying to diagnose.
Strangely, one thing I was surprised about was how it was possible to get in touch with Postmasters when we had issues with blacklisting. It generally took a while to track down the proper way to contact them, but I think there’s only been one mailhost (Apple) who we never heard back from – although we were removed from their blacklist the day after. When blacklisted by Microsoft (Outlook, Hotmail, Live, etc.) I even got to chat with a helpful real person; an actual named person no less!
Suppose it depends on the provider. I’ve had quick and helpful responses from Migadu when needed in the past
This was my experience also. A couple of years ago, I was trying to change my Warby Parker password after a breach on their end. The process failed at the send a nonce to Chris via email step. I could see that the cause of the failure was that I hadn’t added warbyparker.com to my greylister so I wasn’t going to get immediate delivery. Even after bypassing warbyparker.com I wasn’t getting the nonce. A letter to hostmaster@warbyparker.com got a response but I didn’t need glasses at the time. I honestly didn’t know it was fixed until today when I remembered as a result of this thread.
Maybe I’m just being in too bad of a mood! Most of this comment dates back to when I was working in university IT 6-7 years ago and spent a long time battling blacklisting problems. I don’t think I ever got responses from Google, and while they did start accepting mail from us again each time it was long enough later (a few days) that it was unclear whether someone had read my message or the blocklist entry had just hit an expiration date. The worst problems we had in the other direction were abuse coming from Google (spam and fraud email, calendar invites, and phishing via Google Forms) and I don’t think I ever saw any evidence of their acting on abuse complaints. We started blacklisting certain gmail addresses at the institutional level because they were sending so much mail to us that just had very long address lists pasted into the To/Cc. The irony is that SpamAssassin was picking this stuff up and we had to manually bump the domain reputation of gmail.com to avoid it learning to spamcan every marginal email coming from Google.
I have not seen this kind of problem since I worked there (although I haven’t run as large of a mailserver since), but I would assume Google has tightened up their controls at some point because it was remarkably brazen behavior to continue from a major email provider for, as I recall, around a year. I’d guess this was 2013 or so. This situation actually lead somewhat directly to that institution switching to Google Workspace later because it fostered a perception that the in-house operation was incompetent, which is sort of ironic. It really got at one of those no-win problems in security: we were getting phished via Google Forms on a weekly basis, but whenever we tried taking active measures like dropping email with Google Forms links it turned into a huge upset and we had to back off. When I worked in incident response in 2015-2017 phishing via Google Forms continued to be a top-ten problem, but at that organization we had somewhat better tools available to combat it and ended up using our outbound web filters to inject a redirect to a warning page the user had to click through (we were fortunate enough to have TLS interception). Google now provides some additional measures to mitigate this problem but they of course require that you be a Google Workspaces customer. I assume they’ve also stepped up detection and enforcement a lot because in more recent cases where I’ve seen Google Forms phishing, the link has often been dead by the time I saw it. In any case the whole thing left me with a very negative perception of how Google runs their email operation (which was boosted when I was involved in the pre-sales process on the switch to G-Suite, which was amazingly incompetent).
And I shouldn’t let Google color my perception of the whole industry, my recollection is that Microsoft was pretty easy to get a hold of on hotmail/outlook.com issues, although I think we may have leveraged our existing relationship with them to get a better contact than what they have publicly.
With smaller organizations it’s always been a lot easier, of course being a university much of our interaction was with other universities and postmaster@ almost always got a response from someone helpful, whether we were the source or victim of the problem. Unfortunately this situation has become worse over the years as more and more institutions and businesses are moving over to cloud services that they have limited control over and knowledge of. In my current position, where I don’t even really deal with email, I’ve run into a few cases of “something is blocking email from you but we can’t figure out what.” It’s almost always a cloud email security gateway they use, which have a tendency to be very opaque.
Although we haven’t had any issues so far sending to Google, I don’t doubt for a second that they would be a pain to deal with. We manage several dozen Google Workspaces for clients and it is constantly causing bother.
It’s also a good idea to have an alias for postmaster@ (root too, often aliases tables have that per default). There’s many systems and people that default to sending there.
I’d love to read an outline of these options, along with a map of distributions which provide each option out of the box. Does such an outline exist?
That’s a good question, if I can’t find one I might take it on. That 4-5 is a very rough estimate based on my recollections, but I think it’s more likely to be an underestimate than an overestimate. The biggest difference is whether or not you choose to use Dovecot as MDA: you can either have Postfix “deliver” mail and then Dovecot read the mailboxes created by Postfix, or you can have Postfix transfer email via socket to Dovecot for “delivery.” The latter tends to be preferred these days because it allows Dovecot to do a lot of optimization by having full ownership of the mailbox, including filtering and live indexing of inbound mail and IMAP push with less performance overhead. One reason you might not do it though is if you want to use a totally separate MDA for whatever reason (neither the one built into Postfix nor the one built into Dovecot, maybe procmail for compatibility with existing procmailrcs).
An additional complication is that most MDAs support either “mbox” or “maildir” format for file system storage (maildir tends to be better for a few reasons, but there are catches), and some MDAs and IMAP servers support using a relational database instead (usually your best performance option but you can imagine it gets weird when you have multi-MB email bodies). There’s also a Unix tradition of storing mbox/maildir as the user and in the user home directory, which has benefits but can make troubleshooting a bigger headache. If you NFS mount or otherwise centralize your home directories, users are 100% guaranteed to accidentally delete their whole maildir from time to time, leading to annoying backup recovery tickets. But moving these out of the user homedir tends to violate some expectations on the part of the mail software and has its own pains. Some of the details of these catches lead to the common situation on older mail setups where the user’s inbox is limited in size compared to other folders: it used to be very common in NFS environments to put the inbox, but only the inbox, in the user’s home dir in mbox format (because all of the Linux tooling expects it to be there), but all other folders stored on the mail server in maildir format outside of home directories. Fortunately I have not seen this in a while except for, oddly, Dreamhost, which I assume just hasn’t changed their mail setup for a good while. My memory of what exactly was the main painpoint on moving the inbox is fuzzy now, but I know back when I ran mail we had already put out default mutt/pine configs to use IMAP rather than reading locally, so I think there was some other issue beyond that. I hazily think the details of the UID behavior of Procmail may have been involved.
I thought this was a very long, thoughtful opinion piece on running your own email server(s). The advise at the end of the article was especially helpful. Like:
I agree, but those two are probably great ideas for any service you ever run
One thought on one click solutions: study them and their configs!
I run my own mailserver. I go against a couple of the suggestions on both sides of this issue. You can do it, it’s not hard but the learning curve is tedious and long. I built my first internet mailserver for just myself back in 1996 or so. I sent and received mail via hybrid SMTP/UUCP from ripco communications in Chicago. I expanded the solution to my whole domain in about 2000 when I bought a SparcStation 20 and ran my mail on OpenBSD. I also transitioned from UUCP to pure SMTP at that time. My CATV company put a stop to that when they shut down inbound port 25 in like 2001. If I had it all to do over again, today, I’d probably just delegate my mail to someone else.
I commented earlier but I would add this. Today a lot of programmers don’t know email. I occasionally still host things on real linux boxes and one of the steps in my ansible install is to configure postfix to relay mail local mail through an internal smart smtp relay. The tell question is: What email server and user/password should I use for this project. The tell answer is the look on the questioner’s face when I answer: pick one of
127.0.0.1:25,[::1]:25, orlocalhost:25, no username or password.Yes, I feel this way. l do not know email, but feel like I should and I am trying to learn it. I really just want to know how to send registration and password reset emails without having to signup for another service. Then I’ll work on learning more from there.
The programming APIs for sending email are usually well document. The questions are: “What server should I use?”, “What port?”, “What username?” and “What password?” If your administrator can and has setup local relaying properly, the answers are: (“localhost”, 25, None, None)
That is what I want to know how to do. Is there any good documentation on how to do that?
The setup I use has one system setup to relay all mail from a group of systems. I use postfix and there are two elements to this setup:
To configure the clients, you want to have
relayhost = [therelay.example.com]:25in postfix’s main.cf. Googling forpostfix local mail relay smart hostshould be helpful here.Configuration of the relay largely depends on who you need to get mail to and what mechanisms you have to deliver mail. Googling for
postfix smart hostorpostfix smart relayshould get you started. Note well that any reasonable tutorial will include settings/entries for:inet_protocols, inet_interfaces: These postfix settings determine how your relay listens for incoming mail to relay or directly deliver.mynetworks_style, mynetworks, smtpd_relay_restrictions: These settings control the relaying aspects of postfix. By default postfix will reject mail if it doesn’t think that it’s the destination for that mail. By tuning these three settings, you can limit who can use your postfix machine as a mail relay. These settings are important because tuning them the wrong way can make your system an open mail relay which is very bad. Usually these will be setup such that the relay will accept mail from IP addresses its own LAN without question. On the other hand clients will set this up so the only valid source for mail is localhost.mydestination: This postfix setting determines what domains you consider to be your responsibility for delivery. Ex, a machine whose destination includesexample.comwill expect to be or to know the delivery port for mail tochip@example.com.The biggest problem is usually how to get mail out to the internet. I think that my earlier comments trivialized that fact. There are generally two main ways to do it. Either, your ISP will give you the IP address and port of a system which will relay mail to the rest of the internet for you. Sometimes they will require you to use a username/password here also. The postfix tutorials that mention SASL cover this. If you run your own mailserver, and this is a much harder thing to do securely, you’ll need to get permission from your provider, either ISP or Virtual Private Server, to directly contact servers via port 25 on the internet.
I run a small fleet of Linux boxes on an otherwise Microsoft centric network for my company. We handle our mail via O365. I got permission for one of my systems to relay mail through the corporate SMTP gateways . This is my relay. I’ve configured the rest of the fleet to use my relay as a smart host for all messages.
While I would agree with that, is there something specific that happened here? I can’t remember I had to ever deal with permissions on my own in 15 years and using files. The only thing was when I switched providers and mail server (Debian/postfix to OpenBSD/opensmtpd, as well as some other parts). But that was just backup and chown on the new system. Was very straightforward.
But but it course, running a server, having a grasp of how permissions work is a good idea and if never recommend anyone to do something else. Also obviously understand the software you are running.
I always ran without db (if you don’t count aliases db). For mail going the very Unix approach feels like the right way. After all most Unix like OSs come with some usage for mail being integrated like that (status mails, cron mailing output, etc.), so often you will find mails for users on non mail servers. See OpenBSD’z welcome mail.
One thing I’d like to add: have a good understand of DNS. That can help a lot, also elsewhere, but knowing about DNS PTR is a good idea in general, whenever you set up any kind of server and then knowing how SPF and DKIM work, not just blindly copy pasting of course it’s also a good idea.
Obviously understanding what you set up is a good idea. Since mail doesn’t change a lot you likely won’t touch it much do make sure to comment and document even the parts that seem obvious now. :-)
Main situation where I’ve run into these problems has always been related to storing mail outside of user home directories - I recommend not doing that, but the biggest upside is that it means that pretty much everything “just works” since most mail software is already set up to setuid to the recipient before trying to handle mail on disk, and expects all user config files to be in user home and read via setuid. If you start changing that situation then you become a lot more likely to run into headaches where you have mail processing happening as ‘mail’ sometimes and as the user other times, mail directories and config files owned by different users, etc. The postfix master.cf Mostly this has come up because if you’re putting a lot of functionality in your IMAP server it can become desirable to not allow the user direct access to their own mail on disk, since they bypass whatever the IMAP server is doing and that can create inconsistency and headaches (this was yet much worse in the days of POP or, worse, the days of IMAP except for five users who have never reconfigured their client).
Another adjacent source of headaches is chroot. Many distros distribute Postfix preconfigured to run under a chroot, which I think is a very good idea considering the huge attack surface with file I/O access in Postfix, but it does mean that some of your mail components need to run chrooted and others need to not be chrooted, and if you’re changing the config from defaults you will probably screw that up at least once and have a problem that’s annoying to diagnose.