The hardest part about e-mail setup is to not get flagged/rejected as spam. As for this:
I would add a few things to that list but this highlights that some people are already “golden”, just by setting up proper rDNS, SPF and DKIM.
I dispute that this is factually accurate. I have run my mail server since years from the same IP, and had all of these set up properly for years. outlook.com still rejected my mail outright, and gmail delivered my mail to the spam folder.
One thing not mentioned here is signing up for so-called “feeback loops” at the big e-mail providers. But creating an account somewhere at some U.S. company I don’t have direct business with is something I simply don’t want to do.
On a sidenote, I ever only had problems delivering to U.S.-based e-mail providers. Everybody else seems to not have had any problem with my mail.
I also can confirm this point. I frequently have to resend critical emails, using my gmail account, twice if I dont hear back after a day or two. I’ve been slowly unrolling my own email server and going back to gmail full time solely because of this.
I also get a lot more spam than my much more publicly listed gmail address. spamassassin and greylisting are much easier to beat than Google’s dedicated team of paid professionals.
I agree with the “cock-up before conspiracy” principle, but a sufficiently advances conspiracy is indistinguishable from a cock-up. ;)
Google is one of the largest email providers
Small independent servers/providers are disproprortionately affected
No other provider rejects so much legitimate email
Google often rejects them silently, without sending DMARC reports even if DMARC address is configured
It’s profitable for Google if people give up and switch to GMail
Google is known to engage in anti-competitive practices, such as making their services Chrome-only and lowering the search rating of websites that aren’t using AMP
In this situation, possibility that it’s intentional shouldn’t be rejected.
In this situation, possibility that it’s intentional shouldn’t be rejected.
Especially if the incentives and past practices tied to incentives line up to make something likely to happen. You covered that in 5 and 6. I’m adding this for emphasis. You could add Google services on Android to that list, too, given how that plays out on stock vs phones focused on privacy or user freedom.
This is what I did after I was sick of disappearing mail from my nice setup with opensmtpd. I didn’t mind administering it, but email is useless if it’s unreliable.
I also get a lot more spam than my much more publicly listed gmail address. spamassassin and greylisting are much easier to beat than Google’s dedicated team of paid professionals.
I suppose their approach is to better reject a legitimate mail than to allow a spam e-mail.
I’ve been slowly unrolling my own email server and going back to gmail full time solely because of this.
I found a counter-measure, as I mentioned in other threads on the topic. You can use an SMTP relay provider, i.e. relay all your outgoing e-mail through a third party’s SMTP server. You’d then have to pay for the SMTP relay provider, though.
I also get a lot more spam than my much more publicly listed gmail address. spamassassin and greylisting are much easier to beat than Google’s dedicated team of paid professionals.
rspamd is very, very effective at killing spam in my experience. Give it a try if you haven’t already.
I spent a number of years working for an Email Service Provider that shifted enough emails to saturate a gigabit connection; we always had issues with Gmail and Hotmail (sometimes but not often Yahoo, everyone else seemed to play along nicely) even with a direct hotline to the respective engineers at their ends.
Their systems would often purposfully drop incoming email with various error responses that often were seemingly random in nature and nothing to do with anything our end, sometimes they would accept email as deivered while silently dropping it.
Nowadays I use tutanota for simplicity, someone else can manage the maintenance :)
I went through the whole process - reverse DNS records, DKIM, SPF, made sure the IP address of the VPS I hosted it on wasn’t blacklisted anywhere, made sure that the mailserver components were well-configured (not open relays), set up SSL, took all of the mail tests I could find and got perfect scores on all of them.
None of that mattered; important jobsearch-related emails went into peoples’ spam, and I was left constantly wondering if people were just taking a long time to respond or if my email had been sent to spam. AT&T/Yahoo straight up blocked all emails from my IP TWICE; each time I had to send an email to a special address and wait ~48 hours to get my sending restored.
It turned out that since I was using a non-standard TLD for one of my email addresses (.link), I was immediately rejected by many large mail providers just because of that. Of course, there’s no documentation you can refer to; it’s all arbitrary. I guess a lot of bulk spam senders use those kinds of TLDs because they’re cheaper or easier to register or less policed or whatever. Maybe it was some random machine learning model that saw “wow 80% of all messages coming from .link domains have been marked as spam by users; seems like a clear choice to me.” That’s the cost of a highly effective spam filter I guess.
Anyway, all I’m saying is that if you run your own mailserver, you’re at the mercy of the Google/AT&T/Microsofts of the world and whatever random policies they decide to enforce. They’re not going to notice OR care; you’re a single user. They’re always going to optimize their decisions around the other big players since 99.5% of their legitimate mail come from those sources.
So, I sold out and now pay $6/month for GSuite. Email is too important to my life to put at risk. Maybe you’ll have better luck on your own, but the fact is that your entire ability to send/receive email rests on the whims of the large-scale providers. I personally don’t think it’s worth the risk, the effort, or the frustration.
I appreciate this article’s point of view (and not only for the notqmail shoutout ;-). Indeed, for an individual human person who knows their way around Unix, running your own mail server requires little marginal effort or attention beyond running your own server. The initial effort or attention that appears to be warranted, however, may loom large: there are many things that appear to be necessary to know inside and out, and one must at least convince oneself one has sufficiently understood enough of them to feel safe to get started.
It’s not only big mail vendors that got us here. We’re still paying the price for Sendmail having defined the genre. Projects like OpenSMTPD (and hopefully soon notqmail) that offer excellent defaults and just enough configurability in a human-centric way, with polished packages easily available for popular OSes, may not be sufficient to turn the tide. But there’s no question they are necessary.
Completely agree. You get improved privacy and the ability to reject emails during the SMTP conversation. (What do you mean I’m subscribed to the Modbus newsletter? plonk) You also easily control your backup MXs for when your primary server goes down.
Trying to get support for your email in the cloud is not so easy either. I have a customer that uses Big Mailer Corp A for their email and found out of that Big Mailer Corp B marks all their emails as spam. They’ve been trying for two years to fix that one.
Would you mind sharing what your setup looks like (if you haven’t already)? I suspect you may get a far larger volume of mail (and possibly spam too?) than most folks around here, but mainly I’m just curious what you run on what.
I run several mail servers and between them it’s kind of a mishmash. I’ve found postfix, postgrey, and opendkim to be among the most reliable pieces from the bin. For spam I just use postgrey and DNSBLs and a manually-updated blacklist of senders.
Maybe it’s not “hard” in the sense of “swimming across the channel”-hard, but it’s also not “easy” in the sense of “walking to the local Tesco”-easy. It’s more like “doing a 2-month thruhike crossing all of Ireland”-hard. Sure, most people could do it, but it’s quite a bit of effort.
You need quite a lot of knowledge to run a mail server and there are many ways to screw it up, including by not doing something and have stuff appear to work at a glance. It’s all doable, but all things considered, for most purposes, it’s just not a good cost/benefit ratio.
The suggestion that it’s all a myth perpetuated by “Big Mailer Corps” is rather silly.
Back when I ran my own mailserver I had a number of issues: at some point I would be receiving a lot of spam and I had to go in and fix that. Is it “hard”? Guess not, but it’s also non-trivial, and something I didn’t really feel like spending time on. I also had two instances of downtime where something went wrong (I forgot what, it’s been years) and I was no longer receiving mail. Can kind of suck because who knows what ended up getting lost in the aether.
Maybe not telling that self-hosted email is hard directly, but that big-mail-corp hosted email is easy, by opposition to whatever else you might come up with.
I’ve gone back and forth with this idea recently (inspired by another article that was posted here a few weeks back), but it still takes time to maintain yet another system (OS updates, app updates, dealing with quirks from updates, having a backup plan [and testing it periodically], etc). I’ve spent the last few years in the middle ground: paying a small email provider for service, and it’s likely I’ll stay here for a while longer.
Thanks, that’s encouraging. I already run a number of systems for personal services, etc. But email would probably need to be some VPS somewhere that’s more stable than sitting in my garage and sipping residential internet service.
Unlike a web server, you have a few delay before to loose incoming mail, such as one day, in which all mailers will keep trying to send the mail to you again regularly.
After that delay, mailers progressively try less often.
You’re probably referring to my article. Just so you know, my server’s been running perfectly ever since I set it up! I haven’t had any to do any kind of maintenance in the past 3 weeks or so. The initial setup is bit of a pain, I’ll give you that – a whole day’s work, for me anyway. But everything else is a breeze afterwards.
If you’re thinking about setting one up, I say go for it.
I run email, web, DNS and gopher on one (virtual) server. Like tedu, I have not really had to muck with mail. I don’t even have a backup MX record as I found it not worth the hassle. I have reverse DNS and SPF (I haven’t bothered with DKIM yet). I find that it pretty much just works.
This is already happening with one specifically requiring mails to be sent from another Big Mailer Corp to hit the inbox, or requiring that senders be added to the contacts for others. Any other sender will hit spambox unconditionnally for a while before being eventually upgraded to inbox.
Anybody knows which bigcorp player he’s talking about?
My mailserver, for many months, could not send mails to outlook addresses. The outlook server replied “OK” but the mail was transparently discarded. Not inbox, not spam, not trash, nothing. As if the mail had never been sent.
I believe nowadays outlook “only” send my mails to spam.
I have had the same experience. With Gmail it was even more difficult to evade their hyper-aggressive spam filters.
I can’t call any of this “easy” and I had to struggle and learn a lot of new concepts (like DKIM, which is a pain to set up). It’s also very tricky to verify, if it fails it can fail silently; your mail is just dropped or goes to spam. I had that happen when my DNSsec signatures weren’t renewed, for example, and also when I had made a small mistake that made my DKIM invalid or not used (I don’t remember which).
You need to be an expert at mail before this stuff is “easy”. When you get redirected to the spamfolder, those hosts aren’t giving any information about why this happened, so you’re left guessing. Also, you sometimes don’t even know unless you’re in contact with the recipient in some other way than just e-mail (and sometimes people don’t bother to notify you that the mail got flagged as spam). There are tools out there that can help verify your technical setup like rDNS, SPF, DKIM etc. But it’s still annoying as fuck to get it set up. Once you’ve done the work, it basically runs itself though.
So I appreciate the article’s attempt to get more people to try hosting their own mail, I would say it’s quite one-sided and assumes a whole lot of technical sysadmin competency that the author has probably simply become blind to himself.
I had a similar problem and my solution was to route all mail to them via a separate, dedicated IP which didn’t suffer the same problem. A solution possible thanks to the flexibility of Exim. As much as these simpler MTAs seem attractive I wonder how they would cope with such scenarios.
I had this problem sending from my own mail server to Gmail addresses. After a couple of months I just gave up on my own mail server and went to mailbox.org
Food for thought because my default position has been “don’t run your own mail server” until now.
The “proof of work” take is interesting. Although I’ve never bothered with SPF or DKIM and don’t have deliverability problems. I might look at DKIM one day (but SPF is, imho, a total waste of time).
I’m not sure if the “programming” tag is appropriate
For what is worth, my experience is that SPF configured with “-all” does work pretty effectively, and is much more useful these days than it was in the past. It’s also quite easy to set up.
Googling “proof of work email spam” actually revealed that solving this problem originally with HashCash is what has led to the big cryptocurrency revolution!
The hardest part about e-mail setup is to not get flagged/rejected as spam. As for this:
I dispute that this is factually accurate. I have run my mail server since years from the same IP, and had all of these set up properly for years. outlook.com still rejected my mail outright, and gmail delivered my mail to the spam folder.
One thing not mentioned here is signing up for so-called “feeback loops” at the big e-mail providers. But creating an account somewhere at some U.S. company I don’t have direct business with is something I simply don’t want to do.
On a sidenote, I ever only had problems delivering to U.S.-based e-mail providers. Everybody else seems to not have had any problem with my mail.
I also can confirm this point. I frequently have to resend critical emails, using my gmail account, twice if I dont hear back after a day or two. I’ve been slowly unrolling my own email server and going back to gmail full time solely because of this.
I also get a lot more spam than my much more publicly listed gmail address. spamassassin and greylisting are much easier to beat than Google’s dedicated team of paid professionals.
That’s precisely what they want you do to!
I’m going to maintain that it’s an intentional, anti-competitive practice on Google’s side that demands legal scrutiny.
I’m not entirely sure it’s done with malice. Federated just means open to spam and abuse in many cases.
I agree with the “cock-up before conspiracy” principle, but a sufficiently advances conspiracy is indistinguishable from a cock-up. ;)
In this situation, possibility that it’s intentional shouldn’t be rejected.
Especially if the incentives and past practices tied to incentives line up to make something likely to happen. You covered that in 5 and 6. I’m adding this for emphasis. You could add Google services on Android to that list, too, given how that plays out on stock vs phones focused on privacy or user freedom.
You can also use something like FastMail that’s cheap, respects your privacy, and I’m told is faster due to less surveillance tech in their apps.
This is what I did after I was sick of disappearing mail from my nice setup with opensmtpd. I didn’t mind administering it, but email is useless if it’s unreliable.
I suppose their approach is to better reject a legitimate mail than to allow a spam e-mail.
I found a counter-measure, as I mentioned in other threads on the topic. You can use an SMTP relay provider, i.e. relay all your outgoing e-mail through a third party’s SMTP server. You’d then have to pay for the SMTP relay provider, though.
rspamd is very, very effective at killing spam in my experience. Give it a try if you haven’t already.
I spent a number of years working for an Email Service Provider that shifted enough emails to saturate a gigabit connection; we always had issues with Gmail and Hotmail (sometimes but not often Yahoo, everyone else seemed to play along nicely) even with a direct hotline to the respective engineers at their ends.
Their systems would often purposfully drop incoming email with various error responses that often were seemingly random in nature and nothing to do with anything our end, sometimes they would accept email as deivered while silently dropping it.
Nowadays I use tutanota for simplicity, someone else can manage the maintenance :)
I went through the whole process - reverse DNS records, DKIM, SPF, made sure the IP address of the VPS I hosted it on wasn’t blacklisted anywhere, made sure that the mailserver components were well-configured (not open relays), set up SSL, took all of the mail tests I could find and got perfect scores on all of them.
None of that mattered; important jobsearch-related emails went into peoples’ spam, and I was left constantly wondering if people were just taking a long time to respond or if my email had been sent to spam. AT&T/Yahoo straight up blocked all emails from my IP TWICE; each time I had to send an email to a special address and wait ~48 hours to get my sending restored.
It turned out that since I was using a non-standard TLD for one of my email addresses (.link), I was immediately rejected by many large mail providers just because of that. Of course, there’s no documentation you can refer to; it’s all arbitrary. I guess a lot of bulk spam senders use those kinds of TLDs because they’re cheaper or easier to register or less policed or whatever. Maybe it was some random machine learning model that saw “wow 80% of all messages coming from .link domains have been marked as spam by users; seems like a clear choice to me.” That’s the cost of a highly effective spam filter I guess.
Anyway, all I’m saying is that if you run your own mailserver, you’re at the mercy of the Google/AT&T/Microsofts of the world and whatever random policies they decide to enforce. They’re not going to notice OR care; you’re a single user. They’re always going to optimize their decisions around the other big players since 99.5% of their legitimate mail come from those sources.
So, I sold out and now pay $6/month for GSuite. Email is too important to my life to put at risk. Maybe you’ll have better luck on your own, but the fact is that your entire ability to send/receive email rests on the whims of the large-scale providers. I personally don’t think it’s worth the risk, the effort, or the frustration.
I appreciate this article’s point of view (and not only for the notqmail shoutout ;-). Indeed, for an individual human person who knows their way around Unix, running your own mail server requires little marginal effort or attention beyond running your own server. The initial effort or attention that appears to be warranted, however, may loom large: there are many things that appear to be necessary to know inside and out, and one must at least convince oneself one has sufficiently understood enough of them to feel safe to get started.
It’s not only big mail vendors that got us here. We’re still paying the price for Sendmail having defined the genre. Projects like OpenSMTPD (and hopefully soon notqmail) that offer excellent defaults and just enough configurability in a human-centric way, with polished packages easily available for popular OSes, may not be sufficient to turn the tide. But there’s no question they are necessary.
In my experience, a mail server is hard to set up but easy to maintain, and well worth the effort once completed.
Completely agree. You get improved privacy and the ability to reject emails during the SMTP conversation. (What do you mean I’m subscribed to the Modbus newsletter? plonk) You also easily control your backup MXs for when your primary server goes down.
Trying to get support for your email in the cloud is not so easy either. I have a customer that uses Big Mailer Corp A for their email and found out of that Big Mailer Corp B marks all their emails as spam. They’ve been trying for two years to fix that one.
Would you mind sharing what your setup looks like (if you haven’t already)? I suspect you may get a far larger volume of mail (and possibly spam too?) than most folks around here, but mainly I’m just curious what you run on what.
I run several mail servers and between them it’s kind of a mishmash. I’ve found postfix, postgrey, and opendkim to be among the most reliable pieces from the bin. For spam I just use postgrey and DNSBLs and a manually-updated blacklist of senders.
Maybe it’s not “hard” in the sense of “swimming across the channel”-hard, but it’s also not “easy” in the sense of “walking to the local Tesco”-easy. It’s more like “doing a 2-month thruhike crossing all of Ireland”-hard. Sure, most people could do it, but it’s quite a bit of effort.
You need quite a lot of knowledge to run a mail server and there are many ways to screw it up, including by not doing something and have stuff appear to work at a glance. It’s all doable, but all things considered, for most purposes, it’s just not a good cost/benefit ratio.
The suggestion that it’s all a myth perpetuated by “Big Mailer Corps” is rather silly.
Back when I ran my own mailserver I had a number of issues: at some point I would be receiving a lot of spam and I had to go in and fix that. Is it “hard”? Guess not, but it’s also non-trivial, and something I didn’t really feel like spending time on. I also had two instances of downtime where something went wrong (I forgot what, it’s been years) and I was no longer receiving mail. Can kind of suck because who knows what ended up getting lost in the aether.
Maybe not telling that self-hosted email is hard directly, but that big-mail-corp hosted email is easy, by opposition to whatever else you might come up with.
I’ve gone back and forth with this idea recently (inspired by another article that was posted here a few weeks back), but it still takes time to maintain yet another system (OS updates, app updates, dealing with quirks from updates, having a backup plan [and testing it periodically], etc). I’ve spent the last few years in the middle ground: paying a small email provider for service, and it’s likely I’ll stay here for a while longer.
I find update maintenance on a mail server to be rather minimal.
Once you have a server, you can do all sorts of other things with it. For the same mostly fixed cost.
I would estimate 99% of my admin time is spent on something other than smtpd.
Thanks, that’s encouraging. I already run a number of systems for personal services, etc. But email would probably need to be some VPS somewhere that’s more stable than sitting in my garage and sipping residential internet service.
Unlike a web server, you have a few delay before to loose incoming mail, such as one day, in which all mailers will keep trying to send the mail to you again regularly.
After that delay, mailers progressively try less often.
You’re probably referring to my article. Just so you know, my server’s been running perfectly ever since I set it up! I haven’t had any to do any kind of maintenance in the past 3 weeks or so. The initial setup is bit of a pain, I’ll give you that – a whole day’s work, for me anyway. But everything else is a breeze afterwards.
If you’re thinking about setting one up, I say go for it.
I am! Thanks for the update. Yea that article has pushed me closer to giving it a go than anything I’ve read so far :)
I run email, web, DNS and gopher on one (virtual) server. Like tedu, I have not really had to muck with mail. I don’t even have a backup MX record as I found it not worth the hassle. I have reverse DNS and SPF (I haven’t bothered with DKIM yet). I find that it pretty much just works.
Anybody knows which bigcorp player he’s talking about?
My mailserver, for many months, could not send mails to outlook addresses. The outlook server replied “OK” but the mail was transparently discarded. Not inbox, not spam, not trash, nothing. As if the mail had never been sent.
I believe nowadays outlook “only” send my mails to spam.
I have had the same experience. With Gmail it was even more difficult to evade their hyper-aggressive spam filters.
I can’t call any of this “easy” and I had to struggle and learn a lot of new concepts (like DKIM, which is a pain to set up). It’s also very tricky to verify, if it fails it can fail silently; your mail is just dropped or goes to spam. I had that happen when my DNSsec signatures weren’t renewed, for example, and also when I had made a small mistake that made my DKIM invalid or not used (I don’t remember which).
You need to be an expert at mail before this stuff is “easy”. When you get redirected to the spamfolder, those hosts aren’t giving any information about why this happened, so you’re left guessing. Also, you sometimes don’t even know unless you’re in contact with the recipient in some other way than just e-mail (and sometimes people don’t bother to notify you that the mail got flagged as spam). There are tools out there that can help verify your technical setup like rDNS, SPF, DKIM etc. But it’s still annoying as fuck to get it set up. Once you’ve done the work, it basically runs itself though.
So I appreciate the article’s attempt to get more people to try hosting their own mail, I would say it’s quite one-sided and assumes a whole lot of technical sysadmin competency that the author has probably simply become blind to himself.
I had a similar problem and my solution was to route all mail to them via a separate, dedicated IP which didn’t suffer the same problem. A solution possible thanks to the flexibility of Exim. As much as these simpler MTAs seem attractive I wonder how they would cope with such scenarios.
I had this problem sending from my own mail server to Gmail addresses. After a couple of months I just gave up on my own mail server and went to mailbox.org
Food for thought because my default position has been “don’t run your own mail server” until now.
The “proof of work” take is interesting. Although I’ve never bothered with SPF or DKIM and don’t have deliverability problems. I might look at DKIM one day (but SPF is, imho, a total waste of time).
I’m not sure if the “programming” tag is appropriate
For what is worth, my experience is that SPF configured with “-all” does work pretty effectively, and is much more useful these days than it was in the past. It’s also quite easy to set up.
Googling “proof of work email spam” actually revealed that solving this problem originally with HashCash is what has led to the big cryptocurrency revolution!