1. 150
  1.  

  2. 43

    I hosted my own e-mail server from 1995 until 2004, when I switched exclusively to GMail. I’m planning on migrating back to a privately hosted server, and this is the exact thing I worry about - important mail quietly disappearing because it isn’t coming from one of the main service providers:

    Google has blocked personal messages to friends and family in multiple languages, as well as business mail.

    This quote is what really got me:

    Unfortunately, email is starting to become synonymous with Google’s mail, and Google’s machines have decided that mail from my server is simply not worth receiving. Being a good administrator and a well-behaved player on the network is no longer enough.

    I can’t agree more with that sentiment, and as time goes on it can be applied to ever more of the Internet.

    I’m sure there is no malicious intent behind this

    I’m not as confident about that. I understand the whole “don’t attribute to malice..” concept, but this seems to fit the Google business model, which is to use hegemony to drive more traffic through Google. Apathy could also be construed as malice in that context.

    1. 16

      I agree with you. Obviously, I don’t think there is any intention to hurt anyone, but they definitely want as many people on their platform as possible; what they’re doing now helps with that.

      The only thing I can do is recommend that people get off GMail. Use another service, like ProtonMail or FastMail. Better yet, people should host their own!

      1. 13

        Better yet, people should host their own!

        I don’t know if that’s a good idea. Email hosting is hard and requires some amount of expertise. It’s all fun and games until SpamAssassin craps out on you, and then you can spend the money from Mr. Mantu, a royal from Nigeria temporarily unable to access his funds, on a penis enlargement.

        I’ve noticed that I’ve been getting a lot more spam with FastMail in the last year or so (I’ve been using it for the last 6 years or so). It’s not a huge amount (a few every week), but consistent and enough to be annoying.

        Going back to the OP’s post, this seems like “standard” spam filtering. People from, say, Russia or China are probably penalized to some degree as well if you’re from the states or EU. It’s unfortunate, but if it shares proprieties with emails that look like spam then it has a chance of being classified as such. It’s not just gmail that’s really opaque about this. We had some problems with Yahoo! last year and they were just as “helpful” as gmail.

        There’s been two other discussions about this in the last few weeks, and the thinking of some people about that is a bit, ehh, out there? Spam filtering is just really hard. I don’t think Google is going out of their way to block anyone like some people are suggesting.

        What we really need is a better solution for spam instead of all this Baysian/ML stuff. It was a great quick fix in 2003, but it hasn’t been for a long time.

        1. 13

          90% of the spam I receive uses the subject line of the week. It’s actually trivial to filter with nothing more than a subject line match with occasional updates. But I’ve learned different people seem to attract different spam types.

          1. 9

            Yes and 10% can still be a lot of spam.

          2. 5

            I don’t know if that’s a good idea. Email hosting is hard and requires some amount of expertise.

            It used to be very hard because of all the reasons you mentioned, but containers have changed the game. Check out Poste.io (lobsters discussion).

            1. 14

              Given the large number of pretty silly bugs in router firmwares, proprietary firewalls, etc. I don’t think these kind of “entire distro in a container” things are somehow inherently more secure. Worst was probably Cisco last month. They fixed a bunch of exploits in their Enterprise Firewalls by … blocking the curl user agent that was used in the proof-of-concepts.

              If anything, pre-built containers make things harder because now you have no idea what’s going on, making it easier to make mistakes. That thing in particular doesn’t even have a source as far as I can see (only issue tracker on BitBucket). You’re supposed to just run a random Docker container … :-/

              1. 1

                doesn’t even have a source

                You can inspect docker containers.

                1. 7

                  That’s not the same. Have you ever taken over a running system from someone else? There are always almost surprises (not that it’s configured “bad”, just different from what you expected/are used to).

              2. 5

                Yes, but don’t use Poste.io (see the discussion for why).

                Good alternatives are mailcow and mailu.

              3. 2

                Using a solution like mailroute helps. It may a little pricey for hobbyists, but it works amazingly keeping spam at bay.

                1. 2

                  Any braindead person including myself can set up mailcow-dockerized which works out of the box. No know-how required. Docs are very good as well. It gives you really cool web interfaces, 2fa, spam aliases and a ton of other features.

                  https://github.com/mailcow/mailcow-dockerized

                2. 4

                  I run my own email server, but with comments like these it’s no wonder everybody is simply giving up and letting Google run the show.

                  1. 3

                    Email has become unreliable for me as a result of hosting my own mail server, for the past 18 years, gmail has not been particularly bad for me that I am aware of, but hotmail and office365 based email is a major problem…

                    Saying host your own mail is not the answer unfortunately.

                  2. 5

                    I ran into similar problems as in the OP, albeit with Microsoft (outlook.com) and not with Google. I found a solution that works for me: I pay for an SMTP relay. This way I can have my own mail server, but all mail I send out is submitted to the SMTP relay, which then does the actual transmission to the target mail server. The persons operating the SMTP relay have done a nice job, so far. I had not yet any mail blocked since I use it (since beginning of this year).

                    1. 5

                      Which relay do you use?

                      My personal email server also gets all mails to outlook.com shadowbanned

                      1. 4

                        Amazon SES is a few cents per month

                        1. 1

                          Do you happen to have any resources about using SES as an SMTP relay?

                          1. 2

                            There’s not much to it. The website gives you some DNS entries/SMTP credentials to use, then the config is just:

                            table ses_credentials file:/etc/mail/ses_credentials
                            action relay_ses relay host smtp+tls://ses@email-smtp.eu-west-1.amazonaws.com auth <ses_credentials>
                            match auth from any for any action relay_ses
                            

                            where ses_credentials contains ses user:pw (man table and search for relay).

                            1. 1

                              Forgive my ignorance, that doesn’t look like sendmail or postfix config, what mail server is that exactly?

                              1. 1

                                OpenSMTPD!

                        2. 3

                          I use mysmtp.com, a Danish relay provider. They contract with consumers as well and advertise with GDPR compliance.

                      2. 2

                        If it were not for this problem, I would see a bright future in hacking the IMAP standard to provide all sorts of value adds. Just one example, you could own an email domain and provide a custom address for every web-site registration or other one-off applications. Then you could track the content sent back to those addresses and learn something about how the original recipient treats your email address. Alas, gmail has a monopoly on value added information via IMAP. Would be great if some smart lawyer found an anti-trust case here.

                      3. 31

                        Google cut us off after an incident where one of our users account was taken over and a lot of spam sent. We have failed to even come in contact with Google. We are a public library with 200 employees and some 20k active users.

                        If a phone operator failed to communicate in such a situation, regulators would fine them. Perhaps we should regulate large email providers as well?

                        EDIT: I also run a small company mail server on Digital Ocean and its mail ends up in spam 100% of the time for GMail recipients. I’ve tried their reporting tool several times, implemented SPF, DKIM and DMARC - all in vain.

                        1. 16

                          E-Mail providers are in Germany regulated under the same law (TKG) as phone companies. So, if you’re in Germany and have problems with Google (or anyone else) blocking legitmate mail, please, by all means, write to the Bundesnetzagentur as the relevant administrative authority at info@bnetza.de (not hosted by GMail I assume; German language probably required). If they get enough complaints, they’re going to look into the problem.

                          1. 2

                            Czechia… Thanks for the tip, though. Might copy the legislation eventually. :-)

                            1. 1

                              And does it really work?

                              1. 5

                                If you refer to the legal framework: as long as Google provides e-mail service in Germany, they have to obey German law, and as such the BNetzA can fine Google if it doesn’t obey.

                                If you refer to personal capabilities: the BNetzA is one of the larger authorities as it regulates the entire German telco market. If you file a complaint with them, it’s usually handled professionally and friendly, albeit (as with all German authorities) probably slowly. I have had contact with them on another topic some years ago.

                                1. 6

                                  Complaint to BNetzA has worked in my case with Telekom. So I second the recommendation to complain about Google there.

                          2. 28

                            This is a great article.

                            On top of all of that, Google is single handedly turning younger people off to E-mail with their utterly craptastic interface that tried to make email into IM which is most certainly isn’t.

                            And even on top of that they’re guilty of some pretty sincere vendor lock-in - just go ahead and try to use a non Google email address for any of their applications or services, you effectively can’t. I had friends practically beg me not to out and out delete my account so they could continue to share docs and the like with me.

                            And if you used to use a Gmail account but switch to another service (Fastmail in my case) everyone who wants to send you mail to the new address will need to defeat Google’s auto-complete EVERY SINGLE TIME because it will default to sending to Gmail.

                            Do no evil? Not so much.

                            1. 3

                              Keep your Google email and set up forwarding without storage. As you reply to emails to your @gmail from your @fastmail, the autocomplete will eventually learn.

                            2. 21

                              Gmail selectively blocks emails too (depending on who’s receiving them). Sends them into a blackhole without an error. I was able to reproduce it with a contractor who worked with us. For some reason email sent to him would simply disappear. We tried it multiple times over a call and watched in astonishment as Gmail’s server confirmed receipt of email on my end, and the emails appeared nowhere on his end (not even in spam). He said this wasn’t the first time that happened to him.

                              1. 3

                                I’ve noticed they recently implemented a 4-minute delay on some emails. I was very surprised; but my emails still made it to my Gmail mailbox about exactly 4 minutes and 0 seconds after acceptance by Gmail. I initially thought they got lost.

                                I would presume that not all emails do, otherwise, what’d be the purpose of the delay? But I routinely have certain automatically-generated emails delayed by Gmail for exactly 4 minutes now.

                                1. 5

                                  Email is technically an asynchronous collection of services, acceptance and delivery aren’t the same. Maybe it’s some kind of spam detection? Like if their filter is on the fence about a message, it waits to see if more similar / identical messages come in. Only 1 message after 4 minutes -> deliver, 1000 identical messages to different recipients after 4 minutes -> drop all. I don’t actually know, just a wild-ass guess.

                                  1. 3

                                    FTR, this wasn’t an issue with delayed emails. They just never made it.

                                2. 6

                                  I was recently looking into adopting fdm after the author of procmail asked people to stop using his software. I looked at the issue tracker, where now the author of fdm also admitted he doesn’t use his own email software anymore, either; asking further revealed that he mostly switched to Gmail.

                                  What do other people that authored our mail stack use these days? This is the real problem. Even some big public companies with over 1k employees use Gmail nowadays (e.g., Arista Networks, of all else), which I find surprising, TBH, but that’s still the reality.

                                  1. 1

                                    There are bolts of hope here and there.

                                    Il like mblaze for moving things around and dma for sending to a smarthost (my registrar that provides low-volume email services).

                                    We do have some common debian mail servers at work too to provide email to clients. We often have issues with the larger mail providers, and Outlook is not so easy to reach either…

                                  2. 6

                                    I think I noted it elsewhere at some point, but I believe that aggressive spam policies of the kind “you’re spam if you’re not a large provider” are illegal towards small mail providers and could be legally charged, at least in Germany under UWG (German competition law). A small e-mail company would have to test this out in court, though.

                                    1. 1

                                      I’ve discussed this with the Czech anti-trust body but they believe that since customers can always use another email provider if they want to communicate with us and since Google does not have large enough market share (except our client DB says over 40% just GMail) that there is only a small chance to fine them.

                                      But I have not yet filed the official complaint.

                                      1. 2

                                        they believe that since customers can always use another email provider if they want to communicate with us

                                        But they literally can’t, that’s the whole point. If they do, their mail gets swallowed by gmail…

                                        1. 1

                                          Google has only about 20% of Czech email market share and about 40% among young people. Everyone can use Seznam (@post.cz, @email.cz) instead. Google won’t ignore largest Czech provider. Sigh.

                                          We have run our own mail server for at least two decades.

                                    2. 5

                                      From the comments:

                                      We run small hosting company, if any domain blocks our clients mails we block theirs and problem gets solved magically.

                                      1. 3

                                        That’s what many big countries (e.g., China and Russia) do in regards to their visa regime. Precedent is certainly set.

                                        Not sure it’ll really work with Gmail, though…

                                      2. 4

                                        Google worries me as well, which is why I’m thinking about buying a domain with a reasonably sized registrar that provides email services with the domain (my current preference is Gandi) and moving my ‘main’ mail account there. It seems to me the only thing I would really lose is 2FA support for my mail account.

                                        1. 4

                                          I have hosted my own small mail server for years, after being told it would be hard and a lot of work I decided to try anyway. It has been neither hard nor a lot of work. Deliverability to gmail is perhaps less perfect than from gmail itself, but I have only had any kind of real issue maybe once or twice in all my years hosting. Perhaps my contacts are just more willing than many to check their spam box? Or perhaps most of my contacts have me in their address book. I’m certainly not doing everything right, but so far so good.

                                          1. 4

                                            Great article that so well articulates what been bothering me for quite some time. Just weeks ago I got again burned by this, as I sent out an email notifying my friends about a baby, it ended up in Gmail’s spam folder. No matter that some of my friends that knew what to look for, clicked on “Not spam”, it’s still sitting in other’s spam folder.

                                            1. 2

                                              Gmail should white-list any email with a good PGP signature at the SMTP level. In fact all SMTP servers should do this. This seems not difficult to implement - certainly no more difficult than a massive RL spam algorithm - and would mostly solve the problem of blocking individual hackers running their own servers since those people (myself included) tend to sign all their emails anyway. This is how the “Web of Trust” is supposed to function, right?

                                              1. 12

                                                Devil’s advocate: what prevents a spammer from PGP signing their messages?

                                                1. 1

                                                  The same as “blocking misconfigured SPF prevents spam” : spammers are too lazy to setup the full stack… Until they do.

                                                  But this gives you an indicator of what mail can be trusted, and the user would add up email to its “confidence list” and get to a special all-whitelisted inbox : Mail from people we know (whitelist them), automated notifications (whitelist them), mailing lists (whitelist them) or spammers (in greylist).

                                                  SMTP would then never block any whitelisted email.

                                                  The problem is to make a difference between the not-yet-known-sender mail and the spam.

                                                  This is not how it is imenented though… From the user point of view, just a blacklist rather than white/grey/black-list.

                                                  No control over what’s in, just to add more things among what’s out.

                                              2. 2

                                                From my experience usually the only way to get through is by replying to someone’s email, and even if you do that it is not certain that the email will go through. I tried emailing my own gmail account from my personal email hosted on my own server many times and its just impossible to get through for some reason.