Ironically most of the spam I receive comes from the big providers, google most of the time. Their outbound ham/spam ratio is abysmal, yet blocking them sounds ridiculous.
Ironically most of the spam I receive comes from the big providers,
google most of the time.
This is my experience as well. The vast majority of the spam that is
not flagged as such by my rspamd comes from Google. I’ve seen it both
from @gmail.com and for email domains they host. I probably get a dozen
of these in my inbox every day.
Email is now an oligopoly, a service gatekept by a few big companies
which does not follow the principles of net neutrality.
Not only is email controlled by an oligopoly, but also many of the
players adopted the Silicon Valley strategy of customer service. There
is no accountability. You know how people used to say “the devil made
me do it”? The modern twist is “the algorithm made me do it”. We
dropped your mail on the floor? Sorry, there’s nobody to complain to,
and even if there were, the algorithm made me do it.
Google lured billions of people with the bait of “free stuff”, much like
McDonalds lures children with the bait of free plastic toys in their
“Happy” Meal. I saw through the McDonalds scam when I was 4 years old.
I didn’t see through the gmail game quite so easily (I used it as my
mail provider for like 3-4 years), but it’s basically the same con. Now
that it has captured so much of the email world, Google can do as it
pleases.
Ever expanding storage limits (remember the figure in KB that would go up in real-time?).
I too used it for quite a long time.
Now I much prefer having my own domain and use a smaller provider, but at the time gmail was the best you could get coming off the back of constantly pressing ‘Send/Receive’ in Outlook.
These days I suspect they have high numbers of users because it comes ‘free’ with a google account, and people seem to care less about communicating via email outside of work. It’s an oligopoly and no enough people give a shit.
Another problem with Google is that with so many people having
@gmail.com addresses, it is quite possible to accidentally receive
somebody else’s email. I still keep a backup @gmail.com address for the
handful of things that need it. One day, I woke up to a notification
forwarded from my @gmail.com address, reminding me that I had an OBGYN
appointment in a city nearly 3000 km away from me. Someone had picked
an @gmail.com address differing from mine by one letter. It was
obvious, because they’d used the same address pattern I did: first
initial + middle initial + surname + digits. Her surname was
spelled correctly in the message. I saw the mistake and called the
clinic to let them know they screwed up.
I also get notifications about this person’s dental appointments. I was
notified when she signed up for electric service at her new home.
This is so, so familiar to me. My first + last name happens to be the equivalent of John Smith in my local language and using first@last.tld email has been lots of fun for me ever since I (finally) snatched the domain name back in 2014. I’ve got over 100 filters to deal with generic/test emails, and every once in a while, a new sender will end up in my inbox.
I used to self host my email for about 6 months before it became too much for me to deal with. These days, I happily pay Fastmail to do it for me. I tend to not rely on Gmail/Apple/Microsoft to deal with my email as much as I can, simply because I don’t want to give them more power than I have to, and luckily, I’m able to afford to pay for my email hosting, something that most of the people around me can’t, unfortunately.
If you don’t want Google handling your email, you’ll need to pay for it, either with your money or your time (or both, if you decide to self host).
I agree wholeheartedly with the post, had been in similar shoes. My major problem wasn’t that my email was delivered to spam, or that it got blackholed. My major problem was that I had to - and still need to - send email to people in organizations (usually close ties to the government), and they don’t use a blacklist. They use a whitelist. You’re not in it? Too bad, you’re not sending us email.
Now, this is a problem when I need to talk with the Kindergarten my kids go to. I can do so on Facebook, or via e-mail. I’m not signing up for the former, so email it is. But their server rejects mail from pretty much anyone but the big players. So I’ve been using a backup gmail account for a good while to communicate with them.
My bank, my electricity service provider, customs, the post office - they all do the same.
However, I did not want to give up self hosting email. Receiving is not a problem, so I just needed a solution for sending. For that, I ended up using a relay, one that - for the time being, at least - is on the whitelist of the organizations I need to communicate with. This way, incoming email is still self-hosted, and the little outgoing email I send, gets delivered reliably. Since setting this up, I had no rejects, and none of my sent email ended up in spam.
It’s a compromise, nevertheless, and is only viable because my outgoing volume is low (<100 email / month). If it would be higher, that’d cost a small fortune to relay. The existing relays are… often quite sketchy too (looking at you, sendgrid, in particular), so finding one that wouldn’t put me in a worse position than I was already in was quite a challenge.
But it works. For now. I fear for a time when it stops.
I’m not here to defend my employer (a search engine company from mountain view), since I am also personally concerned and saddened by the centralisation of email.
But spam is an arms race. If you start allowing spammers to debug why they’re getting blocked, they will start finding ways to workaround or avoid the block, hence the silent black-holing. Unfortunately, the innocent little guy is the biggest victim in all of this.
The state of email is really sad, IMHO. Most big actors are trying to make their proprietary protocols to control the whole market. (e.g. gmail which violate a lot of mail RFCs last time I checked, or Microsoft Exchange) Making it more decentralised will allow for more cheap spam (viagra, etc…) On the other side, most of the spam I receive comes from people scraping my email address from github and sending startup-products-spam to it with Mailchimp/Sendgrid/Mailjet/… Moreover, when you send an email and it’s not received, the blamed is put on the sender. Shouldn’t it be put on the receiver which blocked a false positive in the first place?
We are all experiencing what happened when politicians regulated the web.
[…]
The industry should fix email interoperability before politicians do. We will all win.
This is the result of the almost total absence of politic in the IT sector, or politic driven by this oligopolistic lobby money.
There is nothing new in this imho, the industry is very young, people understand it less at the moment, it is difficult to have a popular opinion on what is good for society, which leads to poor political ownership and lack of interest in regulating it.
We are all experiencing what happened when politicians regulated the web. I hope you are enjoying your cookie modals; browsing the web in 2022 is an absolute hell.
That’s not exactly the politicians’ fault. They’re not serving malware through ads. They DID go and try to regulate things. They just have an off-by-one error, instead of opt-in, they made cookies opt-out.
Actually, the GDPR explicitly requires cookies and all forms of tracking to be opt-in.
It’s just that it takes so long for the courts to punish the websites that they can still keep the old opt-out solution until the courts rule on their specific case.
But at least Google now has cookies and tracking opt-in.
Are you sure? I was certain that everybody was pissed about it.
Also, they said “legitimate requirement” meaning, i need to keep this info _by law _, but the operators took it as “I have a legitimate desire to spy on people in order to satisfy my master the VC”. That’s been challenged recently, I think, but everyone still does it.
If everyone was pissed, I must have missed it. I’ve implemented the GDPR since it passed into law in 2016 (the law had a two year tolerance policy during which non-compliant sites would not be fined, which is why everyone remembers the introduction to be 2018).
And during this entire time it was clearly obvious from the text that all tracking had to be opt-in.
You’re spot on with your analysis of the “legitimate need”, though. There’s only a handful of legitimate needs that are not caused by laws enforcing storage, e.g. storing IPs in logs for a few days to measure DDoS attempts.
But it’s obvious companies are trying to abuse that definition for their own benefit.
I totally agree this is a big issue. I’ve been forwarding email from my domain to Gmail for the past 10 years. Over the past 2 years, a bunch of things started getting marked as spam. The Gmail UI also got extremely slow for me this year.
So I actually decided to go self-hosted. Using the nixos-mailserver project made this pretty easy but as this blog post points out, it can be impossible to figure out why an email is being marked as spam when sending to Gmail, Outlook, etc. Sometimes just luck is involved!
Not sure about outlook since email delivery the always just worked, but I once was helping someone with an ancient setup. Google has a list of things to do to get your emails though
Things like properly setting up DKIM, SPF, etc., but something people tend to not do because it’s rarely mentioned in mail server setup gives is setting the PTR record correctly. It’s easy to forget and how you do that depends on where you host your server. Once you know everything to set it though it’s easy and quick to do.
I would not recommend OVH as their cheap VPS were (are?) often abused by spammers, so it’s clean IP lottery. I got one clean IP some years ago but got tired of self-hosting too (I offloaded that service to Gandi to do more interesting things with my time).
I heard that Hetzner has a decent reputation these days.
Heh, maybe I am one :) I doubt though, I don’t run any services or anything, I only spam people with Azure’s notification emails about failed builds these days :)
I have a tiny Vultr VM that handles outgoing mail for me. Vultr disables outbound SMTP by default. To get it enabled, you have to raise a support ticket that tells them the amount of mail you want to send and why. This took about 2 hours from becoming a customer for the first time and getting a working relay. It probably provides enough friction to prevent scammers from doing it and, importantly, the fact that they shut down outbound spam senders means that their IPs generally don’t end up on block lists (or, if they do, are cleaned quickly).
I think it’s either small providers or just getting lucky. One usually is unlucky with bad IPs, so it’s not like one has search for them. I personally didn’t have that problem, but if you end up with a bad IP I’d honestly just ask the hosting company to give you a new one, because you cannot run the mail service you intend to run.
Of course you can also check black lists, but I’d recommend to just test it, sending emails to different email servers after the initial setup (SMTP server, SPF, DKIM, PTR, DMARC, putting it on dnswl.org) is complete. Judging by the comments here it’s probably enough to just check if the big ones, such as gmail receive your emails correctly, preferably ones that didn’t receive yours yet, but new ones, so you don’t end up having it cleared for just one user. I think there’s also services you can send emails to for testing purposes, that check against black lists and usually also whether DKIM and so on works correctly.
If that works you’re probably fine, if not ask for a new IP and try again. If it didn’t work maybe worthwhile to check if you can find out which blackist it’s on. Both to check whether the whole block is listed and to see if you can’t just get it unlisted. Some of them offer that. Of course that’s only for public black lists.
Of course you can also check black lists, but I’d recommend to just test it, sending emails to different email servers after the initial setup
But not to the big players. You’ll have a lot more luck if the first email that you send to a gmail account is a reply to a message that was sent to your domain. Gmail uses outbound emails to increase reputation of domains and servers. If Google’s customers have been sending emails to your server for a while then they assume that you’re more likely to be real than if you just appear and start sending emails.
I disagree. The goal is to have email working. You are dealing with a remote state machine that has a permanent failure state. If you transition into this state, you have lost. Your goal is to avoid ever reaching that state. Doing things that move you towards that state early will help you catch the problems but will make it impossible to remedy them.
I agree it’s a racket and have had some experience with this at work. We used to have a mail server, but over the years our IP started getting blacklisted and mail wasn’t getting delivered. It was very hard if not impossible to recover from this.
Then we got a new IP because they put fibre in. Nothing ever worked on that, so it’s when we decided to switch to google. Sad really, but now we are bigger and more distributed it probably is a better solution for us.
Personally I don’t host my own but do use a relatively unknown provider, migadu. Never had any issues with mail getting delivered.
Could they be doing anything that a person with a VPS simply cannot?
So I’ve also been doing email for 23 years (or thereabouts; I started in 1999), except in my case its as a a sysadmin/devops/sre/architect (the name changes; the work mostly does not) for organisations on the order of hundreds of thousands to low millions of mailboxes. Since 2012 I’ve been at Fastmail. So I write this as a perspective from the mid-tier of mail providers that mostly do a good job of deliverability and mostly don’t have the power or influence to just arbitrarily bin email.
(All opinions etc my own and not my employer, etc, and also I’m writing this kinda off-the-cuff in my lunch break so please excuse any confusing or contradictory stuff; ask for clarification and I shall attempt to provide!).
I have some sympathy for the author of the article, I really do. I started my life on the internet running my own single-user mail server, and I understand the complexity of the global mail system in 2022, so I’m well aware of just how different it is and how tough it can be.
That said, this article and others like it from recent years almost always underestimate the magnitude of the problem posed by spam, fraud, phishing and other “unwanted stuff” (I’ll just say “spam” from now on). There’s different ways to count it, but there’s capital-B Billions lost to this shit every year. And with that comes small-country-sized amounts of resources to throw at making sure the spam gets through.
The giveaway is statements like “charge a fee; spammers won’t pay” because its straight up not what happens. People sending spam can and do spend good money to try and increase their standing; they’ll pay for the top-tier accounts with legit credit cards; they’ll buy real unsullied cellphone numbers in order to complete SMS-based verification.
So its hard. Does that mean the big players (which I don’t count Fastmail as) just drop small players on the floor? I think that’s too simplistic. Sure, they do have powerful and opaque reputation systems and yes, you can fall afoul of them by unfortunately being in the wrong place (like, an IP address that is near another that is known to be bad), but that’s because those things are very strong markers that whatever you’re sending is going to be spam. You’d be doing a disservice to your customers if you didn’t use it. Its much stronger than content, even, which is why things get rejected at the network edge. It’s nothing to do with content scanning being computationally expensive; that wouldn’t matter if it were more reliable.
(Fastmail does the same stuff around reputation in concept, though perhaps less sophisticated, partly because we’re smaller and that stuff is time-consuming to build and operate, but also because the volumes we deal with a smaller and so allow us to put a human in the loop).
Its just not true to say the big players don’t care about interoperability (though how much they care waxes and wanes with the seasons). The fundamental expectation of email is that anyone can play, and none of them have the market penetration to outright put up walls. Its not just interop between hosting providers, there’s also the various mail services (ESPs, Mailchimp and that sort of thing). Yes, many of them just send through Amazon SES, but not all. The point is, once you get all the significant players together you end up a few dozen companies; a mesh of deliverability agreements isn’t really possible. And that commitment to interop and keeping the system open to anyone that wants to play is where we get, for example, the suite of authentication tech (SPF, DKIM, DMARC, ARC, BIMI, etc), each layered on top of the previous one, none of them being especially great but each trying to file off one sharp edge.
And then, we all attend industry forums like M3AAWG to talk about what we’re currently seeing and share ideas about how to tackle it, and we spend time over at IETF and elsewhere trying to write down what we should do.
There are some “go it alone” elements that are a mixed bag. The most frustrating kinds of these are the shared blacklists out there that have very strange (to us) policies on what to add and when, or don’t provide any information at all about why you’ve been listed, have no (easy) way to get delisted and are used by significant enough sites that we still have to do something when we land on one. These sorts of cases are usual handled by asking a contact for a contact for a contact. Building networks of humans doing the gruntwork at each org is a big part of being able to operate any mail system past a certain size.
And yet, even going to all the events and making lots of friends, as an individual mid-size provider its still not uncommon to bump into things. We semi-regularly get partially blocked by Gmail or Outlook or iCloud or one of the major ISPs that you might not have realised ran a large mail system. We’re currently “warming” a new set of outbound IPs, that is, trying to raise their reputation enough so they can be used reliably for all sending; that’s a task that takes months, slowly trickling out definitely good email, watching responses, backing off when remotes start to push back, and over and over.
There’s a lot to do, which is why we have a whole team dedicated to this work!
So, am I making a point? I don’t know. I’m not defending the system as it currently is; we certainly should improve it somewhat. I think this is all trying to say that this is a global network that open to everyone, everyone involved broadly wants to keep it that way, and taken together that brings global-sized problems which have to be tackled as such. I don’t think its impossible for an individual to succeed in this environment, but I do think it its naive to expect that they should be able to. I don’t even think that’s bad necessarily; its just what happens once things grow past a certain size, and for better or worse, that’s past the size where an individual can easily be involved.
I’ve heard that one possible solution to this is requiring more proof-of-work (hashcash or similar schemes), in order to make bulk-sending annoying/economically infeasible.
If I’m reading you correctly, though, it sounds like the problem is that spammers (and here I include not just v1@gra / 419 scammers but also like Target, newsletter folks, etc.) would be happy just to throw more resources at the problem.
Most spam sending is coming from compromised accounts (webmail, newsletters, VPSes) so the cost is going to be borne by legitimate businesses and their customers. For direct sending (comparatively rare), hash computation is just going to be farmed out to some API (also likely with a compromised account), or in the extreme, they’re gonna build dedicated hashing hardware (I don’t think it goes that far, but if there’s a buck to be made, you can be sure someone will spend the money).
The “legitimate” senders (eg newsletter) meanwhile are not just gonna throw more money it. They’ll do what they have to do, but they don’t like spending more than they have to, so they’re gonna fight such a change pretty hard.
Meanwhile, there’s a lot of underpowered hardware out there, particularly in less-developed parts of the world. Requiring more of it is an additional burden - battery life, delays, etc which make email less usable. So such a change has the chance of adversely affecting already marginalised groups.
The real problem though is all the old software out there that doesn’t support the new scheme. That’s going to be a good chunk of legitimate mail that doesn’t have our new not-spam marker, which just puts us back to looking for other markers to sort the good from the bad.
There’s an old joke response to proposed solutions to spam in the form a checklist: https://craphound.com/spamsolutions.txt. I wouldn’t seriously use it as-is; its just a little too snarky for my tastes. I do refer to it from time to time though, because running through it is a quick way to assess whether a technique might be in or out, but also its a little bit of history of the things we’ve tried that haven’t worked.
From the outside, I think Fastmail is doing absolutely the right thing. The work that you folks have done on JMAP is technically good but, perhaps more importantly, the way that you’ve navigated getting it an IETF-approved standard, rather than just a ‘hey, we have this thing. It’s kind-of documented, good luck creating an interoperable version’ is nothing short of amazing to see. I’d love to see Thunderbird and Dovecot support JMAP.
As soon as Fastmail supports Confidential Computing (so I get strong technical guarantees backed by remote attestation that your administrators can’t see my mail spools), I expect to become one of your customers.
It’s a shame that you’ve dropped the family plans though.
To be honest I had never really considered that mail providers will blackhole stuff to save on processing time. I don’t know how true it is, but it’s interesting!
I definitely would like for e-mail to be “properly” federated again. I would even say that SDF is well positioned to create a layer over e-mail to enforce something like this! Ignoring the “can’t send to Google servers” part, the biggest difficulty to me seems to be the idea that email addresses are printed on paper sometimes. You can autogenerate email addresses to provide to specific people, but you do kind of need an entry point.
An idea I like is to have an address which is purely for “contact requests”. Someone can reach out, and then you can approve this request (and provide a unique address they can use for fuller contacts). This could be somewhat automated via mail clients, and then mean that you have even another “validity token” for inbound stuff. And hopefully in such a world larger mail services would be able to be way less strict about inbound stuff to these unique addresses.
Perhaps that extra layer isn’t needed, and really we just need the big mail providers to do like what Apple and DDG are doing and offer unique addresses (and consider that in their filtration strategies). But I do like the idea of having mail properly be triaged by sender, and all the benefits that could incur.
Usually even with Google you’ll get a bounce that tells you what to do. And if you have an email account with Gmail you can also check what causes the block.
I’ve been running my mail server since circa 2015. That involved switching software, provider (and then IP). Things run smoothly. I use the email server as my main way to communicate, I use it both personally and professionally and have some family accounts as well.
I’ve had a bounces way back when DKIM became a thing. Setting it up fixed these.
Because of my work I frequently send emails to people I’ve never ever talked to. I also tend to prefer email over calls for support, etc. for documentation purposes. So lots of circumstances where I’d even know if if emails were silently dropped (esp. when using it for work as a consultant). I do get my responses though.
So articles like these always baffle me. For a while I beloved that the old IP was the reason but I’ve switched providers in 2019 and I also added domains so I think that reputation systems should not find them to be trusted at least initially.
The only annoying thing that happened was that someone started spamming with my email address causing me to get bounces that aren’t from masks on my server. The oligopoly doesn’t do that cause they check SPF, but huge amounts of tiny servers don’t seem to do that. And seems like that address gets put into many web forms as I receive automatic responses. I’ve had this before with a non self hosted email address which forwards to the self hosted one. It’s rare but annoying cause these of course aren’t spam, just responses. So they look fine in terms of server soup.
I wished the article was a bit more technical. You’d usually get something from the oligopoly about why it bounced. I’ve helped a couple of people debugging their email servers when bounces happen. So far all of those were simple misconfigurations. Oh or using residential IP space which won’t work.
On receiving spam. It’s hugely annoying that a big portion of Spam is sent out from Sendgrid, Mailgun,Mandril, etc, so they have all the things making them look like valid. At times I felt like they should be taken down as professional spammers. Hugely annoying.
On the topic of IPs and reputation. Mailgun, Mandrill and Sendgrid market dedicated IPs to increase delivery rates so I would argue being a well known IP isn’t the thing that will increase your delivery rate.
I have heard stories about being unlucky having received an IP from your hosting company that was previously used to spam. I hadn’t seen such a case personally but I think such a situation can make it look like you can’t host your own email
I’m waiting for the day that self hosted email won’t work anymore. But so far it works without any issues. Given that various smaller websites seem to still be successfully use send mail through php and wonky setups (as I noticed back when using grey listing which would not work properly in these cases) it seems like big providers have to leave a lot of very shady looking things through. Maybe they use user reputation for that (marking as spam, etc.).
Anyways, as mentioned wirh stuff articles I’d always be curious about the technical side. What responses do the oligopoly’s servers send?
On the topic of IPs and reputation. Mailgun, Mandrill and Sendgrid market dedicated IPs to increase delivery rates so I would argue being a well known IP isn’t the thing that will increase your delivery rate.
Indeed, it isn’t. The single reason that lets you deliver email is being on the receiver’s whitelist. That’s it. Nothing else.
You’d usually get something from the oligopoly about why it bounced. I’ve helped a couple of people debugging their email servers when bounces happen. So far all of those were simple misconfigurations.
I ran my self-hosted email for over 20 years now, much like the author of the article. I recently switched to using a relay to send, because fully self hosting that part became impossible. Not because I didn’t configure something:
I had a stable, dedicated, non-residential IP, which I have been the sole user for the past 8+ years. No spam ever left my system during that time. I recently had to change the IP, but that makes no difference, because I had to switch to a relay before that anyway.
I have a stable domain name which I have owned since 2009. I have been the sole sender from this domain.
I have an appropriate PTR record.
SPF, DKIM, DMARC and the rest are set up properly.
90% of the e-mail I send are replies to e-mail sent to me, between known contacts.
Yet, despite all that, my email routinely ended up in spam folders when sent to Google-hosted domains, be them @gmail.com or custom domains. I have no idea why. On the SMTP level, the message is accepted. On the client’s end, they see it is in spam, but the headers tell nothing about why, apart from “we found this message suspicious”, which isn’t very helpful. And that’s the good situation! It’s much worse when Google accepts the email, and then drops it before delivering it to the recipient at all. I have comfirmed that happening with dozens of people: I sent an email, verified my logs that Google’s servers acked it, and even a week later, there was no sign of it in their inbox, neither in Spam, nor anywhere else. It just disappeared without a trace.
In this respect, I found most Microsoft servers better, because those reject the mail at least. That allowed me to switch to a backup @gmail.com account if I desperately needed to send email to such an address. The reject reason: “We do not accept email from this sender at this time, please contact our administrators”. So I did just that, and contacted multiple postmasters of such servers. Turns out they were all using the same deny & allow lists, had no permission to change them, and I would need to contact the administrators of those lists to get my server on the list. No reputation, no SPF, DKIM, DMARC, etc checking. A simple allow list, because the deny list was basically “everything not on the allow list”.
Who are on the allow lists? The big names, and relays that pay them hefty amounts of money.
Why use an allow list? Because a deny list in 2022 does nothing, its an unwinnable uphill battle when IPv4 addresses are frequently reused, when domain names don’t even long enough to update a list, and when IPv6 addresses are a plenty. SPF, DKIM, DMARC all sound good in practice, but they aren’t widespread enough to make a big difference. Not to mention that a large percentage of spam (about 60% of all my incoming spam) comes through the Big Names’ servers, with valid SPF, DKIM and DMARC records. Of the remaining 40% of the spam I receive, half of them also come from places with valid SPF, DKIM and DMARC stuff.
Thing is, those don’t cost much to set up, so spammers do so too. It can be fully automated, and nets them a higher chance of delivery.
Thus, no matter what I did, to be able to email contacts I need to email, without having to keep a separate gmail (or other big-name) account, the only reliable solution was to use a relay, where someone else makes sure they are on the appropriate allow lists. But this way, my email is not fully self hosted anymore. I still have an SMTP server that can send mail. I still have all the things set up for domains I do not relay (because relaying all of them would be pricey, so I keep that to an affordable minimum), but their delivery rate is abyssmal.
On the client’s end, they see it is in spam, but the headers tell nothing about why, apart from “we found this message suspicious”, which isn’t very helpful.
For Gmail I found that if you show press the “Show original” button on an email it tells you more. Based on someone I have helped it appears that Gmail (and probably others) not too long ago switched from not allowing soft-fail in SPF anymore, validating that it is -all instead of ~all.
Sorry, that you were forced to switch. I’ll hang in till it stops working for me as well and then will join the group of people saying email should be abandoned. ;)
Speaking about allow lists. Maybe it also helps to add it to whitelists. Ages ago I added mine to dnswl.org.
As mentioned for my job reasons I do send emails to random people every once in a while. So far that worked, as if it didn’t someone would have let me known, after all the job has to be done. Heh. The same is true when interacting with people in the open source community or making an appointment at a doctor and such things. So far it worked. I hope it will stay that way.
For Gmail I found that if you show press the “Show original” button on an email it tells you more.
Like I said, I checked the headers. There was nothing useful there. I just checked a legit message gmail routed to spam, and there were 0 useful headers. All it told me is what I already knew: how google verified the valid SPF/DKIM/DMARC headers.
There was no header indicating that the message was even considered spam, let alone any that would help me figure out why. All it told me is that both SPF and DKIM passed. Great. I knew that already. Not helpful.
Speaking about allow lists. Maybe it also helps to add it to whitelists. Ages ago I added mine to dnswl.org.
In my case, it would not have made a difference. The allow lists the domains I had to send email to did not make it possible for me to add myself. For one, even discovering which whitelists I’d need to add myself to was a problem, because the lists used were deemed confidential information, or they simply didn’t know, not even their tech people (“we use google/outloook/whatever, ask them”). The one whitelist provider I did manage to contact was asking for upwards of $10k/year/sender domain. Yeah, nope.
So far that worked, as if it didn’t someone would have let me known, after all the job has to be done.
You’re lucky then. For a long while, ‘till about 3-4 years ago, I had no issues either. I had the odd message going to spam here and there, but overall, what I sent was delivered. But then I started to get bounces (while my setup being top notch in every feasible regard, apart from not paying for being on any and all allow lists possible), and even worse, had mail disappear into the void after being accepted by the recipient’s servers.
As for letting me know if they don’t receive email: oh, they did, yes. They called me. Doesn’t help when I need to send scanned attachments and stuff. I can resend and then they’ll call me again a few days later. Doesn’t solve the problem. The caller won’t be able to help me, because they have absolutely no idea how email works, it’s not their job, and I won’t have much luck contacting their tech people either, because they’ll just say “We use gmail/outlook, no clue why your email disappears, talk to them”.
I’m too small for both google and microsoft to care, and the tools available to help figure out what went wrong are totally useless once you have a setup that is supposed to work.
To this day, I have no trouble communicating with most open source projects and people - they usually self host, and have reasonable setups.
The problem is getting mail into the big names (especially into outlook, but google is getting harder and harder fast), and the troubling issue with that is that makes it harder for small businesses to stay out of their clutches. If they want email reliably delivered, being behind Big Names is far easier. It makes email someone else’s problem. They can send email. If they do not receive email, then the problem is clearly at my end as far as they’re concerned. “Tough luck mate, get a gmail account, its free” I was often told.
Usually even with Google you’ll get a bounce that tells you what to do.
What can I say? That’s not my experience.
I have my own domain, with MX records pointing to Fastmail. I had SPF set up but not DKIM, and at some point GMail (and Yahoo, for that matter) started sorting all of my messages into the recipients’ Spam folders. I wasn’t blackholed, but I might as well have been: No bounce, no notification, just quietly hidden out of normal view. It still happens sometimes even now that I have DKIM set up.
Fastmail is not a small company. They’re a sizable player. But smaller than the biggest ones. I’ve been using this domain for over 15 years. And this shit still happens.
Although I agree the situation is unfortunate and extreme, I don’t share the opinion that a closed posse of giant players intentionally chose this path. The whole AMP push by google was a much clearer case of an ill intended big player trying to power grab the web. The email case didn’t quite happen the same way.
The stakes are extremely high. The internet opens a business or any communication initiative, to the whole world. A solo hobby programmer can start an online business with a $100 used laptop and a $5 VPS, from a hut in the third world, as long as they have an internet connection.
While this is amazingly awesome, it also means that any tiny online business, all of the sudden, has the whole set of bad online players in the world as real threats and potential attack sources.
With the amount of important things taking place online, it is just impossible to properly provide a reliable service without a sizeable investment on security. How do you this as a tiny business owner? How do you run a web server with your tiny business and keep it secure?
It won’t likely get attacked if you are a local restaurant and have a booking system for you 5 tables, but what if you made a small SaaS and have ambition to grow? How do you prevent being DDOSed? Of course, the solution is paying for such security for companies that provide it as a commodity. It’s not just email. It’s any federated online service. No one hosts a webpage on their own home internet connection anymore. The threat model is just not compatible with it. Just like in the email case.
If it is easy to set up an email endpoint, how do we sort out which ones are malicious and which ones are legitimate? This is not even specific to online realms. The same is valid for example for entering large buildings or other public areas with valuable resources that could be prone to be stolen or destroyed.
How do you enter a large office building? You provide an identification at the entrance. How does the building security sorts out legit entries from malicious ones? It relies on an outsourced entity issuing system, such as a corporate registry or a country national identity register.
I too missed the times when things were not so serious on the internet. Loads of opportunity, just put up a service and worry about security later. Make your individual online presence looking like a corporate and so on. For good and bad, those times are gone. I am sure many people in here are glad they have a well payed IT job. Value what you have, your job would probably not exist if we were still at the era where running your own email server was viable and common. It doesn’t mean that we need to reject nostalgia.
Many years ago I worked for an email service provider (much like mailgun), back then it was largely entirely done on IP reputation.
We had a /21 block of ipv4 giving us 2,046 ip addresses to send mail from. It was a full time job for a team of people to maintain IP reputation, it took months to warm up an IP to the point where it could be used for mailings and over a year before an IP was trusted enough to send more than 250k emails a month. IP reputation was vendor specific so we had to balance mailings sent to hotmail/outlook, gmail, yahoo, etc in order to carefully increase reputation score.
IP Reputation took a long time to grow but could be cut to zero in an instant by a few recipients marking emails as spam, too many spam reports and the IP was permanently blocked.
Having operated email servers at massive scale, I prefer to pay someone else to host my email :)
IP Reputation took a long time to grow but could be cut to zero in an instant by a few recipients marking emails as spam, too many spam reports and the IP was permanently blocked.
Interesting - I have a habit of using the “mark as spam” button for companies who silently subscribe me to newsletters because they saw my email address for some other reason. While frustrating for the mailer-in-the-middle, this gives me hope that this signal might find its way back to the sender.
Great post. The inability to host our own email servers is a potentially fatal blow to privacy and democracy itself on the internet, especially in our era of social media censorship. I agree it is not possible to send your own email from your own computer or VPS anymore due to IP range blocklisting policies. BUT you can host your own email server using software like Mailu (for example, hosted on a Raspberry Pi in your home, with Wireguard network routing through a commodity VPS with a static IP) for receiving email, and use an email sending provider (like Mailgun) as an email relay for sending. They say their sending logs are only held for a limited time, so this solution still provides some privacy, since email deliveries still come directly to your home server and all of your received emails are kept there. (This provides some degree of privacy, but only if you are sending to someone else not on Gmail, etc.) Maybe it is time to start replacing email with another distributed solution but one with native end to end encryption and identity verification like the Matrix messaging network.
Isn’t the main issue here that SMTP was a protocol designed back in the day when internet was nice?
The idea of being able to unsolicited, send a message to some random inbox feels outdated. Personally I don’t want any unsolicited calls to my phone, snail mail to my postbox, SMS to my mobile, Whatsapp messages etc. In fact, unless I’ve actively consented to some communication, I don’t want it at all, regardless of medium.
We arrived here by steadily eroding the idea of doing what’s the morally right thing to do. It simply isn’t morally right to push marketing on to people that didn’t ask for it. And there’s sadly no turning back the clock.
In that spirit, I think the concept of “consent” needs to be made into a protocol. Something distributed and technology agnostic that all methods of communication be required to use. No soft opt-ins, no exceptions for b2b.
I don’t think it’s the times, but more the cost. It’s crazy cheap to obtain and send emails. Just like with other ads.
Hashcash, which Bitcoin (and thereby others) took the PoW concept from and unlike with blockchain stuff it’s just per email, so it doesn’t have such an environmental impact. It’s more like how you have a cost for password checks using bcrypt, scrypt, etc. these days. The idea is to severely slow down spammers. I think it could still work well, if widely adopted and required. Might also cut down a bit on annoying newsletters. ;)
Of course that won’t stop spam for good, but I think solving it completely is pretty much impossible. However you can raise the cost and you can educate people thereby working on making it unprofitable even with small margins. Sometimes I wonder whether spam blockers are even a disservice at times. When people are good with handling spam, profitability sinks. And there’s certainly trends in spam, because some approaches don’t work so well anymore.
While I am not saying that it will all get better, especially with Gmail and others not displaying the senders address or having it grey on white and tiny isn’t exactly helping. At the same time I do think that as “digital natives” are going to shrink email and maybe also telephone spam.
I used run my mail server for about 8 years. Then I lost an important email to a bank, because their proprietary SMTPd wasn’t compliant. Switched to Fastmail and I’ve been pretty happy with their service for years.
Ironically most of the spam I receive comes from the big providers, google most of the time. Their outbound ham/spam ratio is abysmal, yet blocking them sounds ridiculous.
This is my experience as well. The vast majority of the spam that is not flagged as such by my rspamd comes from Google. I’ve seen it both from @gmail.com and for email domains they host. I probably get a dozen of these in my inbox every day.
Not only is email controlled by an oligopoly, but also many of the players adopted the Silicon Valley strategy of customer service. There is no accountability. You know how people used to say “the devil made me do it”? The modern twist is “the algorithm made me do it”. We dropped your mail on the floor? Sorry, there’s nobody to complain to, and even if there were, the algorithm made me do it.
Google lured billions of people with the bait of “free stuff”, much like McDonalds lures children with the bait of free plastic toys in their “Happy” Meal. I saw through the McDonalds scam when I was 4 years old. I didn’t see through the gmail game quite so easily (I used it as my mail provider for like 3-4 years), but it’s basically the same con. Now that it has captured so much of the email world, Google can do as it pleases.
They lured people by being better than the rest.
I too used it for quite a long time.
Now I much prefer having my own domain and use a smaller provider, but at the time gmail was the best you could get coming off the back of constantly pressing ‘Send/Receive’ in Outlook.
These days I suspect they have high numbers of users because it comes ‘free’ with a google account, and people seem to care less about communicating via email outside of work. It’s an oligopoly and no enough people give a shit.
Another problem with Google is that with so many people having @gmail.com addresses, it is quite possible to accidentally receive somebody else’s email. I still keep a backup @gmail.com address for the handful of things that need it. One day, I woke up to a notification forwarded from my @gmail.com address, reminding me that I had an OBGYN appointment in a city nearly 3000 km away from me. Someone had picked an @gmail.com address differing from mine by one letter. It was obvious, because they’d used the same address pattern I did: first initial + middle initial + surname + digits. Her surname was spelled correctly in the message. I saw the mistake and called the clinic to let them know they screwed up.
I also get notifications about this person’s dental appointments. I was notified when she signed up for electric service at her new home.
Others have told me similar stories.
Some of our mortgage financials were CC’d to my wife’s email, minus the last two letters. The recipient was also quite annoyed about it.
Not really Google’s fault, but fun conversations to be had all round.
This is so, so familiar to me. My first + last name happens to be the equivalent of John Smith in my local language and using first@last.tld email has been lots of fun for me ever since I (finally) snatched the domain name back in 2014. I’ve got over 100 filters to deal with generic/test emails, and every once in a while, a new sender will end up in my inbox.
I used to self host my email for about 6 months before it became too much for me to deal with. These days, I happily pay Fastmail to do it for me. I tend to not rely on Gmail/Apple/Microsoft to deal with my email as much as I can, simply because I don’t want to give them more power than I have to, and luckily, I’m able to afford to pay for my email hosting, something that most of the people around me can’t, unfortunately.
If you don’t want Google handling your email, you’ll need to pay for it, either with your money or your time (or both, if you decide to self host).
I agree wholeheartedly with the post, had been in similar shoes. My major problem wasn’t that my email was delivered to spam, or that it got blackholed. My major problem was that I had to - and still need to - send email to people in organizations (usually close ties to the government), and they don’t use a blacklist. They use a whitelist. You’re not in it? Too bad, you’re not sending us email.
Now, this is a problem when I need to talk with the Kindergarten my kids go to. I can do so on Facebook, or via e-mail. I’m not signing up for the former, so email it is. But their server rejects mail from pretty much anyone but the big players. So I’ve been using a backup gmail account for a good while to communicate with them.
My bank, my electricity service provider, customs, the post office - they all do the same.
However, I did not want to give up self hosting email. Receiving is not a problem, so I just needed a solution for sending. For that, I ended up using a relay, one that - for the time being, at least - is on the whitelist of the organizations I need to communicate with. This way, incoming email is still self-hosted, and the little outgoing email I send, gets delivered reliably. Since setting this up, I had no rejects, and none of my sent email ended up in spam.
It’s a compromise, nevertheless, and is only viable because my outgoing volume is low (<100 email / month). If it would be higher, that’d cost a small fortune to relay. The existing relays are… often quite sketchy too (looking at you, sendgrid, in particular), so finding one that wouldn’t put me in a worse position than I was already in was quite a challenge.
But it works. For now. I fear for a time when it stops.
I’m not here to defend my employer (a search engine company from mountain view), since I am also personally concerned and saddened by the centralisation of email.
But spam is an arms race. If you start allowing spammers to debug why they’re getting blocked, they will start finding ways to workaround or avoid the block, hence the silent black-holing. Unfortunately, the innocent little guy is the biggest victim in all of this.
The state of email is really sad, IMHO. Most big actors are trying to make their proprietary protocols to control the whole market. (e.g. gmail which violate a lot of mail RFCs last time I checked, or Microsoft Exchange) Making it more decentralised will allow for more cheap spam (viagra, etc…) On the other side, most of the spam I receive comes from people scraping my email address from github and sending startup-products-spam to it with Mailchimp/Sendgrid/Mailjet/… Moreover, when you send an email and it’s not received, the blamed is put on the sender. Shouldn’t it be put on the receiver which blocked a false positive in the first place?
This is the result of the almost total absence of politic in the IT sector, or politic driven by this oligopolistic lobby money.
There is nothing new in this imho, the industry is very young, people understand it less at the moment, it is difficult to have a popular opinion on what is good for society, which leads to poor political ownership and lack of interest in regulating it.
Yes. There’s this tidbit just under that one:
That’s not exactly the politicians’ fault. They’re not serving malware through ads. They DID go and try to regulate things. They just have an off-by-one error, instead of opt-in, they made cookies opt-out.
Actually, the GDPR explicitly requires cookies and all forms of tracking to be opt-in.
It’s just that it takes so long for the courts to punish the websites that they can still keep the old opt-out solution until the courts rule on their specific case.
But at least Google now has cookies and tracking opt-in.
Are you sure? I was certain that everybody was pissed about it.
Also, they said “legitimate requirement” meaning, i need to keep this info _by law _, but the operators took it as “I have a legitimate desire to spy on people in order to satisfy my master the VC”. That’s been challenged recently, I think, but everyone still does it.
If everyone was pissed, I must have missed it. I’ve implemented the GDPR since it passed into law in 2016 (the law had a two year tolerance policy during which non-compliant sites would not be fined, which is why everyone remembers the introduction to be 2018).
And during this entire time it was clearly obvious from the text that all tracking had to be opt-in.
You’re spot on with your analysis of the “legitimate need”, though. There’s only a handful of legitimate needs that are not caused by laws enforcing storage, e.g. storing IPs in logs for a few days to measure DDoS attempts.
But it’s obvious companies are trying to abuse that definition for their own benefit.
I totally agree this is a big issue. I’ve been forwarding email from my domain to Gmail for the past 10 years. Over the past 2 years, a bunch of things started getting marked as spam. The Gmail UI also got extremely slow for me this year.
So I actually decided to go self-hosted. Using the nixos-mailserver project made this pretty easy but as this blog post points out, it can be impossible to figure out why an email is being marked as spam when sending to Gmail, Outlook, etc. Sometimes just luck is involved!
Not sure about outlook since email delivery the always just worked, but I once was helping someone with an ancient setup. Google has a list of things to do to get your emails though
Things like properly setting up DKIM, SPF, etc., but something people tend to not do because it’s rarely mentioned in mail server setup gives is setting the PTR record correctly. It’s easy to forget and how you do that depends on where you host your server. Once you know everything to set it though it’s easy and quick to do.
Where would one pick a VPS these days, if one was after clean IPs?
That’s a question a spammer would ask. ;)
I would not recommend OVH as their cheap VPS were (are?) often abused by spammers, so it’s clean IP lottery. I got one clean IP some years ago but got tired of self-hosting too (I offloaded that service to Gandi to do more interesting things with my time).
I heard that Hetzner has a decent reputation these days.
Heh, maybe I am one :) I doubt though, I don’t run any services or anything, I only spam people with Azure’s notification emails about failed builds these days :)
I have a tiny Vultr VM that handles outgoing mail for me. Vultr disables outbound SMTP by default. To get it enabled, you have to raise a support ticket that tells them the amount of mail you want to send and why. This took about 2 hours from becoming a customer for the first time and getting a working relay. It probably provides enough friction to prevent scammers from doing it and, importantly, the fact that they shut down outbound spam senders means that their IPs generally don’t end up on block lists (or, if they do, are cleaned quickly).
I think it’s either small providers or just getting lucky. One usually is unlucky with bad IPs, so it’s not like one has search for them. I personally didn’t have that problem, but if you end up with a bad IP I’d honestly just ask the hosting company to give you a new one, because you cannot run the mail service you intend to run.
Of course you can also check black lists, but I’d recommend to just test it, sending emails to different email servers after the initial setup (SMTP server, SPF, DKIM, PTR, DMARC, putting it on dnswl.org) is complete. Judging by the comments here it’s probably enough to just check if the big ones, such as gmail receive your emails correctly, preferably ones that didn’t receive yours yet, but new ones, so you don’t end up having it cleared for just one user. I think there’s also services you can send emails to for testing purposes, that check against black lists and usually also whether DKIM and so on works correctly.
If that works you’re probably fine, if not ask for a new IP and try again. If it didn’t work maybe worthwhile to check if you can find out which blackist it’s on. Both to check whether the whole block is listed and to see if you can’t just get it unlisted. Some of them offer that. Of course that’s only for public black lists.
But not to the big players. You’ll have a lot more luck if the first email that you send to a gmail account is a reply to a message that was sent to your domain. Gmail uses outbound emails to increase reputation of domains and servers. If Google’s customers have been sending emails to your server for a while then they assume that you’re more likely to be real than if you just appear and start sending emails.
The goal here was not to be lucky, but the opposite, crafting a worst case scenario so you so catch problems as early as possible.
I disagree. The goal is to have email working. You are dealing with a remote state machine that has a permanent failure state. If you transition into this state, you have lost. Your goal is to avoid ever reaching that state. Doing things that move you towards that state early will help you catch the problems but will make it impossible to remedy them.
How can you disagree on what my goal was?
Also: The question was how to check whether your IP causes issues.
The goal was to figure out whether you need to switch the IP, so if I get into that permanent failure state early on I’ll just start over.
On top of that I have my doubts on that triggering a permanent failure, but since that’s just us guessing I will leave it there.
I agree it’s a racket and have had some experience with this at work. We used to have a mail server, but over the years our IP started getting blacklisted and mail wasn’t getting delivered. It was very hard if not impossible to recover from this.
Then we got a new IP because they put fibre in. Nothing ever worked on that, so it’s when we decided to switch to google. Sad really, but now we are bigger and more distributed it probably is a better solution for us.
Personally I don’t host my own but do use a relatively unknown provider, migadu. Never had any issues with mail getting delivered.
Could they be doing anything that a person with a VPS simply cannot?
So I’ve also been doing email for 23 years (or thereabouts; I started in 1999), except in my case its as a a sysadmin/devops/sre/architect (the name changes; the work mostly does not) for organisations on the order of hundreds of thousands to low millions of mailboxes. Since 2012 I’ve been at Fastmail. So I write this as a perspective from the mid-tier of mail providers that mostly do a good job of deliverability and mostly don’t have the power or influence to just arbitrarily bin email.
(All opinions etc my own and not my employer, etc, and also I’m writing this kinda off-the-cuff in my lunch break so please excuse any confusing or contradictory stuff; ask for clarification and I shall attempt to provide!).
I have some sympathy for the author of the article, I really do. I started my life on the internet running my own single-user mail server, and I understand the complexity of the global mail system in 2022, so I’m well aware of just how different it is and how tough it can be.
That said, this article and others like it from recent years almost always underestimate the magnitude of the problem posed by spam, fraud, phishing and other “unwanted stuff” (I’ll just say “spam” from now on). There’s different ways to count it, but there’s capital-B Billions lost to this shit every year. And with that comes small-country-sized amounts of resources to throw at making sure the spam gets through.
The giveaway is statements like “charge a fee; spammers won’t pay” because its straight up not what happens. People sending spam can and do spend good money to try and increase their standing; they’ll pay for the top-tier accounts with legit credit cards; they’ll buy real unsullied cellphone numbers in order to complete SMS-based verification.
So its hard. Does that mean the big players (which I don’t count Fastmail as) just drop small players on the floor? I think that’s too simplistic. Sure, they do have powerful and opaque reputation systems and yes, you can fall afoul of them by unfortunately being in the wrong place (like, an IP address that is near another that is known to be bad), but that’s because those things are very strong markers that whatever you’re sending is going to be spam. You’d be doing a disservice to your customers if you didn’t use it. Its much stronger than content, even, which is why things get rejected at the network edge. It’s nothing to do with content scanning being computationally expensive; that wouldn’t matter if it were more reliable.
(Fastmail does the same stuff around reputation in concept, though perhaps less sophisticated, partly because we’re smaller and that stuff is time-consuming to build and operate, but also because the volumes we deal with a smaller and so allow us to put a human in the loop).
Its just not true to say the big players don’t care about interoperability (though how much they care waxes and wanes with the seasons). The fundamental expectation of email is that anyone can play, and none of them have the market penetration to outright put up walls. Its not just interop between hosting providers, there’s also the various mail services (ESPs, Mailchimp and that sort of thing). Yes, many of them just send through Amazon SES, but not all. The point is, once you get all the significant players together you end up a few dozen companies; a mesh of deliverability agreements isn’t really possible. And that commitment to interop and keeping the system open to anyone that wants to play is where we get, for example, the suite of authentication tech (SPF, DKIM, DMARC, ARC, BIMI, etc), each layered on top of the previous one, none of them being especially great but each trying to file off one sharp edge.
And then, we all attend industry forums like M3AAWG to talk about what we’re currently seeing and share ideas about how to tackle it, and we spend time over at IETF and elsewhere trying to write down what we should do.
There are some “go it alone” elements that are a mixed bag. The most frustrating kinds of these are the shared blacklists out there that have very strange (to us) policies on what to add and when, or don’t provide any information at all about why you’ve been listed, have no (easy) way to get delisted and are used by significant enough sites that we still have to do something when we land on one. These sorts of cases are usual handled by asking a contact for a contact for a contact. Building networks of humans doing the gruntwork at each org is a big part of being able to operate any mail system past a certain size.
And yet, even going to all the events and making lots of friends, as an individual mid-size provider its still not uncommon to bump into things. We semi-regularly get partially blocked by Gmail or Outlook or iCloud or one of the major ISPs that you might not have realised ran a large mail system. We’re currently “warming” a new set of outbound IPs, that is, trying to raise their reputation enough so they can be used reliably for all sending; that’s a task that takes months, slowly trickling out definitely good email, watching responses, backing off when remotes start to push back, and over and over.
There’s a lot to do, which is why we have a whole team dedicated to this work!
So, am I making a point? I don’t know. I’m not defending the system as it currently is; we certainly should improve it somewhat. I think this is all trying to say that this is a global network that open to everyone, everyone involved broadly wants to keep it that way, and taken together that brings global-sized problems which have to be tackled as such. I don’t think its impossible for an individual to succeed in this environment, but I do think it its naive to expect that they should be able to. I don’t even think that’s bad necessarily; its just what happens once things grow past a certain size, and for better or worse, that’s past the size where an individual can easily be involved.
I’ve heard that one possible solution to this is requiring more proof-of-work (hashcash or similar schemes), in order to make bulk-sending annoying/economically infeasible.
If I’m reading you correctly, though, it sounds like the problem is that spammers (and here I include not just v1@gra / 419 scammers but also like Target, newsletter folks, etc.) would be happy just to throw more resources at the problem.
Most spam sending is coming from compromised accounts (webmail, newsletters, VPSes) so the cost is going to be borne by legitimate businesses and their customers. For direct sending (comparatively rare), hash computation is just going to be farmed out to some API (also likely with a compromised account), or in the extreme, they’re gonna build dedicated hashing hardware (I don’t think it goes that far, but if there’s a buck to be made, you can be sure someone will spend the money).
The “legitimate” senders (eg newsletter) meanwhile are not just gonna throw more money it. They’ll do what they have to do, but they don’t like spending more than they have to, so they’re gonna fight such a change pretty hard.
Meanwhile, there’s a lot of underpowered hardware out there, particularly in less-developed parts of the world. Requiring more of it is an additional burden - battery life, delays, etc which make email less usable. So such a change has the chance of adversely affecting already marginalised groups.
The real problem though is all the old software out there that doesn’t support the new scheme. That’s going to be a good chunk of legitimate mail that doesn’t have our new not-spam marker, which just puts us back to looking for other markers to sort the good from the bad.
There’s an old joke response to proposed solutions to spam in the form a checklist: https://craphound.com/spamsolutions.txt. I wouldn’t seriously use it as-is; its just a little too snarky for my tastes. I do refer to it from time to time though, because running through it is a quick way to assess whether a technique might be in or out, but also its a little bit of history of the things we’ve tried that haven’t worked.
From the outside, I think Fastmail is doing absolutely the right thing. The work that you folks have done on JMAP is technically good but, perhaps more importantly, the way that you’ve navigated getting it an IETF-approved standard, rather than just a ‘hey, we have this thing. It’s kind-of documented, good luck creating an interoperable version’ is nothing short of amazing to see. I’d love to see Thunderbird and Dovecot support JMAP.
As soon as Fastmail supports Confidential Computing (so I get strong technical guarantees backed by remote attestation that your administrators can’t see my mail spools), I expect to become one of your customers.
It’s a shame that you’ve dropped the family plans though.
To be honest I had never really considered that mail providers will blackhole stuff to save on processing time. I don’t know how true it is, but it’s interesting!
I definitely would like for e-mail to be “properly” federated again. I would even say that SDF is well positioned to create a layer over e-mail to enforce something like this! Ignoring the “can’t send to Google servers” part, the biggest difficulty to me seems to be the idea that email addresses are printed on paper sometimes. You can autogenerate email addresses to provide to specific people, but you do kind of need an entry point.
An idea I like is to have an address which is purely for “contact requests”. Someone can reach out, and then you can approve this request (and provide a unique address they can use for fuller contacts). This could be somewhat automated via mail clients, and then mean that you have even another “validity token” for inbound stuff. And hopefully in such a world larger mail services would be able to be way less strict about inbound stuff to these unique addresses.
Perhaps that extra layer isn’t needed, and really we just need the big mail providers to do like what Apple and DDG are doing and offer unique addresses (and consider that in their filtration strategies). But I do like the idea of having mail properly be triaged by sender, and all the benefits that could incur.
Usually even with Google you’ll get a bounce that tells you what to do. And if you have an email account with Gmail you can also check what causes the block.
I’ve been running my mail server since circa 2015. That involved switching software, provider (and then IP). Things run smoothly. I use the email server as my main way to communicate, I use it both personally and professionally and have some family accounts as well.
I’ve had a bounces way back when DKIM became a thing. Setting it up fixed these.
Because of my work I frequently send emails to people I’ve never ever talked to. I also tend to prefer email over calls for support, etc. for documentation purposes. So lots of circumstances where I’d even know if if emails were silently dropped (esp. when using it for work as a consultant). I do get my responses though.
So articles like these always baffle me. For a while I beloved that the old IP was the reason but I’ve switched providers in 2019 and I also added domains so I think that reputation systems should not find them to be trusted at least initially.
The only annoying thing that happened was that someone started spamming with my email address causing me to get bounces that aren’t from masks on my server. The oligopoly doesn’t do that cause they check SPF, but huge amounts of tiny servers don’t seem to do that. And seems like that address gets put into many web forms as I receive automatic responses. I’ve had this before with a non self hosted email address which forwards to the self hosted one. It’s rare but annoying cause these of course aren’t spam, just responses. So they look fine in terms of server soup.
I wished the article was a bit more technical. You’d usually get something from the oligopoly about why it bounced. I’ve helped a couple of people debugging their email servers when bounces happen. So far all of those were simple misconfigurations. Oh or using residential IP space which won’t work.
On receiving spam. It’s hugely annoying that a big portion of Spam is sent out from Sendgrid, Mailgun,Mandril, etc, so they have all the things making them look like valid. At times I felt like they should be taken down as professional spammers. Hugely annoying.
On the topic of IPs and reputation. Mailgun, Mandrill and Sendgrid market dedicated IPs to increase delivery rates so I would argue being a well known IP isn’t the thing that will increase your delivery rate.
I have heard stories about being unlucky having received an IP from your hosting company that was previously used to spam. I hadn’t seen such a case personally but I think such a situation can make it look like you can’t host your own email
I’m waiting for the day that self hosted email won’t work anymore. But so far it works without any issues. Given that various smaller websites seem to still be successfully use send mail through php and wonky setups (as I noticed back when using grey listing which would not work properly in these cases) it seems like big providers have to leave a lot of very shady looking things through. Maybe they use user reputation for that (marking as spam, etc.).
Anyways, as mentioned wirh stuff articles I’d always be curious about the technical side. What responses do the oligopoly’s servers send?
Indeed, it isn’t. The single reason that lets you deliver email is being on the receiver’s whitelist. That’s it. Nothing else.
I ran my self-hosted email for over 20 years now, much like the author of the article. I recently switched to using a relay to send, because fully self hosting that part became impossible. Not because I didn’t configure something:
Yet, despite all that, my email routinely ended up in spam folders when sent to Google-hosted domains, be them @gmail.com or custom domains. I have no idea why. On the SMTP level, the message is accepted. On the client’s end, they see it is in spam, but the headers tell nothing about why, apart from “we found this message suspicious”, which isn’t very helpful. And that’s the good situation! It’s much worse when Google accepts the email, and then drops it before delivering it to the recipient at all. I have comfirmed that happening with dozens of people: I sent an email, verified my logs that Google’s servers acked it, and even a week later, there was no sign of it in their inbox, neither in Spam, nor anywhere else. It just disappeared without a trace.
In this respect, I found most Microsoft servers better, because those reject the mail at least. That allowed me to switch to a backup @gmail.com account if I desperately needed to send email to such an address. The reject reason: “We do not accept email from this sender at this time, please contact our administrators”. So I did just that, and contacted multiple postmasters of such servers. Turns out they were all using the same deny & allow lists, had no permission to change them, and I would need to contact the administrators of those lists to get my server on the list. No reputation, no SPF, DKIM, DMARC, etc checking. A simple allow list, because the deny list was basically “everything not on the allow list”.
Who are on the allow lists? The big names, and relays that pay them hefty amounts of money.
Why use an allow list? Because a deny list in 2022 does nothing, its an unwinnable uphill battle when IPv4 addresses are frequently reused, when domain names don’t even long enough to update a list, and when IPv6 addresses are a plenty. SPF, DKIM, DMARC all sound good in practice, but they aren’t widespread enough to make a big difference. Not to mention that a large percentage of spam (about 60% of all my incoming spam) comes through the Big Names’ servers, with valid SPF, DKIM and DMARC records. Of the remaining 40% of the spam I receive, half of them also come from places with valid SPF, DKIM and DMARC stuff.
Thing is, those don’t cost much to set up, so spammers do so too. It can be fully automated, and nets them a higher chance of delivery.
Thus, no matter what I did, to be able to email contacts I need to email, without having to keep a separate gmail (or other big-name) account, the only reliable solution was to use a relay, where someone else makes sure they are on the appropriate allow lists. But this way, my email is not fully self hosted anymore. I still have an SMTP server that can send mail. I still have all the things set up for domains I do not relay (because relaying all of them would be pricey, so I keep that to an affordable minimum), but their delivery rate is abyssmal.
For Gmail I found that if you show press the “Show original” button on an email it tells you more. Based on someone I have helped it appears that Gmail (and probably others) not too long ago switched from not allowing soft-fail in SPF anymore, validating that it is
-allinstead of~all.Sorry, that you were forced to switch. I’ll hang in till it stops working for me as well and then will join the group of people saying email should be abandoned. ;)
Speaking about allow lists. Maybe it also helps to add it to whitelists. Ages ago I added mine to dnswl.org.
As mentioned for my job reasons I do send emails to random people every once in a while. So far that worked, as if it didn’t someone would have let me known, after all the job has to be done. Heh. The same is true when interacting with people in the open source community or making an appointment at a doctor and such things. So far it worked. I hope it will stay that way.
Like I said, I checked the headers. There was nothing useful there. I just checked a legit message gmail routed to spam, and there were 0 useful headers. All it told me is what I already knew: how google verified the valid SPF/DKIM/DMARC headers.
There was no header indicating that the message was even considered spam, let alone any that would help me figure out why. All it told me is that both SPF and DKIM passed. Great. I knew that already. Not helpful.
In my case, it would not have made a difference. The allow lists the domains I had to send email to did not make it possible for me to add myself. For one, even discovering which whitelists I’d need to add myself to was a problem, because the lists used were deemed confidential information, or they simply didn’t know, not even their tech people (“we use google/outloook/whatever, ask them”). The one whitelist provider I did manage to contact was asking for upwards of $10k/year/sender domain. Yeah, nope.
You’re lucky then. For a long while, ‘till about 3-4 years ago, I had no issues either. I had the odd message going to spam here and there, but overall, what I sent was delivered. But then I started to get bounces (while my setup being top notch in every feasible regard, apart from not paying for being on any and all allow lists possible), and even worse, had mail disappear into the void after being accepted by the recipient’s servers.
As for letting me know if they don’t receive email: oh, they did, yes. They called me. Doesn’t help when I need to send scanned attachments and stuff. I can resend and then they’ll call me again a few days later. Doesn’t solve the problem. The caller won’t be able to help me, because they have absolutely no idea how email works, it’s not their job, and I won’t have much luck contacting their tech people either, because they’ll just say “We use gmail/outlook, no clue why your email disappears, talk to them”.
I’m too small for both google and microsoft to care, and the tools available to help figure out what went wrong are totally useless once you have a setup that is supposed to work.
To this day, I have no trouble communicating with most open source projects and people - they usually self host, and have reasonable setups.
The problem is getting mail into the big names (especially into outlook, but google is getting harder and harder fast), and the troubling issue with that is that makes it harder for small businesses to stay out of their clutches. If they want email reliably delivered, being behind Big Names is far easier. It makes email someone else’s problem. They can send email. If they do not receive email, then the problem is clearly at my end as far as they’re concerned. “Tough luck mate, get a gmail account, its free” I was often told.
What can I say? That’s not my experience.
I have my own domain, with MX records pointing to Fastmail. I had SPF set up but not DKIM, and at some point GMail (and Yahoo, for that matter) started sorting all of my messages into the recipients’ Spam folders. I wasn’t blackholed, but I might as well have been: No bounce, no notification, just quietly hidden out of normal view. It still happens sometimes even now that I have DKIM set up.
Fastmail is not a small company. They’re a sizable player. But smaller than the biggest ones. I’ve been using this domain for over 15 years. And this shit still happens.
Although I agree the situation is unfortunate and extreme, I don’t share the opinion that a closed posse of giant players intentionally chose this path. The whole AMP push by google was a much clearer case of an ill intended big player trying to power grab the web. The email case didn’t quite happen the same way.
The stakes are extremely high. The internet opens a business or any communication initiative, to the whole world. A solo hobby programmer can start an online business with a $100 used laptop and a $5 VPS, from a hut in the third world, as long as they have an internet connection. While this is amazingly awesome, it also means that any tiny online business, all of the sudden, has the whole set of bad online players in the world as real threats and potential attack sources.
With the amount of important things taking place online, it is just impossible to properly provide a reliable service without a sizeable investment on security. How do you this as a tiny business owner? How do you run a web server with your tiny business and keep it secure? It won’t likely get attacked if you are a local restaurant and have a booking system for you 5 tables, but what if you made a small SaaS and have ambition to grow? How do you prevent being DDOSed? Of course, the solution is paying for such security for companies that provide it as a commodity. It’s not just email. It’s any federated online service. No one hosts a webpage on their own home internet connection anymore. The threat model is just not compatible with it. Just like in the email case.
If it is easy to set up an email endpoint, how do we sort out which ones are malicious and which ones are legitimate? This is not even specific to online realms. The same is valid for example for entering large buildings or other public areas with valuable resources that could be prone to be stolen or destroyed. How do you enter a large office building? You provide an identification at the entrance. How does the building security sorts out legit entries from malicious ones? It relies on an outsourced entity issuing system, such as a corporate registry or a country national identity register.
I too missed the times when things were not so serious on the internet. Loads of opportunity, just put up a service and worry about security later. Make your individual online presence looking like a corporate and so on. For good and bad, those times are gone. I am sure many people in here are glad they have a well payed IT job. Value what you have, your job would probably not exist if we were still at the era where running your own email server was viable and common. It doesn’t mean that we need to reject nostalgia.
Many years ago I worked for an email service provider (much like mailgun), back then it was largely entirely done on IP reputation.
We had a /21 block of ipv4 giving us 2,046 ip addresses to send mail from. It was a full time job for a team of people to maintain IP reputation, it took months to warm up an IP to the point where it could be used for mailings and over a year before an IP was trusted enough to send more than 250k emails a month. IP reputation was vendor specific so we had to balance mailings sent to hotmail/outlook, gmail, yahoo, etc in order to carefully increase reputation score.
IP Reputation took a long time to grow but could be cut to zero in an instant by a few recipients marking emails as spam, too many spam reports and the IP was permanently blocked.
Having operated email servers at massive scale, I prefer to pay someone else to host my email :)
Interesting - I have a habit of using the “mark as spam” button for companies who silently subscribe me to newsletters because they saw my email address for some other reason. While frustrating for the mailer-in-the-middle, this gives me hope that this signal might find its way back to the sender.
Great post. The inability to host our own email servers is a potentially fatal blow to privacy and democracy itself on the internet, especially in our era of social media censorship. I agree it is not possible to send your own email from your own computer or VPS anymore due to IP range blocklisting policies. BUT you can host your own email server using software like Mailu (for example, hosted on a Raspberry Pi in your home, with Wireguard network routing through a commodity VPS with a static IP) for receiving email, and use an email sending provider (like Mailgun) as an email relay for sending. They say their sending logs are only held for a limited time, so this solution still provides some privacy, since email deliveries still come directly to your home server and all of your received emails are kept there. (This provides some degree of privacy, but only if you are sending to someone else not on Gmail, etc.) Maybe it is time to start replacing email with another distributed solution but one with native end to end encryption and identity verification like the Matrix messaging network.
Isn’t the main issue here that SMTP was a protocol designed back in the day when internet was nice?
The idea of being able to unsolicited, send a message to some random inbox feels outdated. Personally I don’t want any unsolicited calls to my phone, snail mail to my postbox, SMS to my mobile, Whatsapp messages etc. In fact, unless I’ve actively consented to some communication, I don’t want it at all, regardless of medium.
We arrived here by steadily eroding the idea of doing what’s the morally right thing to do. It simply isn’t morally right to push marketing on to people that didn’t ask for it. And there’s sadly no turning back the clock.
In that spirit, I think the concept of “consent” needs to be made into a protocol. Something distributed and technology agnostic that all methods of communication be required to use. No soft opt-ins, no exceptions for b2b.
I don’t think it’s the times, but more the cost. It’s crazy cheap to obtain and send emails. Just like with other ads.
Hashcash, which Bitcoin (and thereby others) took the PoW concept from and unlike with blockchain stuff it’s just per email, so it doesn’t have such an environmental impact. It’s more like how you have a cost for password checks using bcrypt, scrypt, etc. these days. The idea is to severely slow down spammers. I think it could still work well, if widely adopted and required. Might also cut down a bit on annoying newsletters. ;)
Of course that won’t stop spam for good, but I think solving it completely is pretty much impossible. However you can raise the cost and you can educate people thereby working on making it unprofitable even with small margins. Sometimes I wonder whether spam blockers are even a disservice at times. When people are good with handling spam, profitability sinks. And there’s certainly trends in spam, because some approaches don’t work so well anymore.
While I am not saying that it will all get better, especially with Gmail and others not displaying the senders address or having it grey on white and tiny isn’t exactly helping. At the same time I do think that as “digital natives” are going to shrink email and maybe also telephone spam.
This should be merged into icdrmv
Yep, thanks. Done.
I used run my mail server for about 8 years. Then I lost an important email to a bank, because their proprietary SMTPd wasn’t compliant. Switched to Fastmail and I’ve been pretty happy with their service for years.
I have setup smtp servers on VPS whose sole purpose was email verification. Never had an issue with it blocking when sending it to my gmail.
Why could his VPS not receive mail though? That’s how I’m currently doing email.