1. 17
    Interoperability Issue of Google Mail and G Suite email networking mailman.nanog.org
  1.  

  2. 10

    I struggle with the same predicament: I run my own email server and have seen emails received from my email server fluctuate over the years between the Inbox and Spam folder in my recipient’s email, irrespective of what rules I follow:

    • SPF, DMARC, and other greater than zero length letter email-related acronyms
    • asking people to mark my emails as “Not Spam”
    • registering my IP and domain with Google Webmasters
    • registering my IP and domain with DNSWL.org

    I have gone to lengths trying to solve this problem but gave up a few years back. But let’s take this issue and internalize it within our community:

    What would you do if you were tasked with handling the spam problem at GMail?

    1. 3

      What would you do if you were tasked with handling the spam problem at GMail?

      I’ve been following this debate for a long time, and suffered myself from the problem (albeit it’s outlook.com that’s blocking me usually, not gmail, and outlook.com has quite some marketshare in Germany). Personally, to circumvent it, I pay for an SMTP relay. This means that my mail server directs all outgoing mail to that SMTP relay server, which then does the real delivery. Since many people (mostly businesses) use that SMTP relay server, it has a high volume and mail usually goes directly to INBOX.

      I acknowledge this is a workaround. The other workaround is to pay Google and have Google’s staff fix the problem. But I think there’s a possibility for a real solution (apart from the best solution, which is to fix the SMTP protocol itself). What I see is that it’s almost always low-volume senders that are self-hosting who have the delivery problems. Based on that, I suggest the following:

      People who self-host mail should form groups and create an SMTP relay server with a clean IP reputation. Every member of that group then has his/her mail server direct all the outgoing mail to the group’s SMTP relay, which does the real delivery. As a consequence, the group SMTP relay has the cumulated volume of all group members and has a much better stand in the delivery. The bigger the group, the better.

      It’s required that someone administers that mail relay, most notably to prevent any member from sending spam (e.g. compromised account). Maybe try rotating the administrative duty annually or so. Finally, from a German perspective, an association (Verein) would make a candidate for the legal framework of such a collective SMTP relay. It’d be the legal entity to address in case of problems. It’d also allow some kind of charity “business” model: assocation members may use the SMTP relay for free, but non-members have to pay. In any case, only individuals are accepted as members. German Ehrenamt at work.

      The individual’s fight against GMail and similar companies is not winnable. My suggestion can give the self-hosters a better stand, and still allow real self-hosting: it’s your SMTP server that accepts all the incoming mail, does the spam filtering, the sieve scripts, and whatever you wish. It’s also your server that sends out the outgoing mail, with just one little twist: all outgoing mail is submitted to the group’s SMTP relay server, thereby joining the group SMTP server’s volume and reputation.

      1. 1

        People who self-host mail should form groups and create an SMTP relay server with a clean IP reputation. Every member of that group then has his/her mail server direct all the outgoing mail to the group’s SMTP relay, which does the real delivery. As a consequence, the group SMTP relay has the cumulated volume of all group members and has a much better stand in the delivery. The bigger the group, the better.

        Sorry, but this is just really silly. I don’t want someone else to be running my mail server. If you do, you can already use one of the vapourware products for sending bulk email or whatnot (be careful, though, because a lot of it doesn’t actually implement SMTP correctly, and cannot do greylisting, for example, so, you’d probably only want to use it to send mail to Gmail and Outlook, still using your own IP address to send mail to anyone else).

        OTOH, I certainly wouldn’t mind to contribute to a legal fund to sue Google and Google Mail, due to them having a monopoly position on the corporate email front, and refusing to accept my mail under false pretences of low domain reputation, which was acquired merely because I’m myself a Gmail user, and have been sending mail to my own account that they misclassified as Spam, and extended such misclassification to the rest of their userbase, too.

      2. 1

        What would you do if you were tasked with handling the spam problem at GMail?

        Maybe this, right?

        I mean, the current situation is kind of the worst of both worlds. Not only do we have an ossified protocol nobody can change, but it isn’t even giving us interoperability!

        However, I can only assume there’s been a slowly-escalating cat-and-mouse game, and as more and more people use email powered by one of the big providers it’s at least plausible to me that more spam than legitimate mail would be coming from self-hosted mail. Without working there I’m not sure how I could estimate what kind of spam they’re getting, and without knowing what kind of spam they’re getting I don’t know that they’re actually doing the wrong thing. Maybe the proper application of bayes rule really should lead you to conclude that self-hosted mail is probably spam.

        I would hate it if that were true, but I don’t know how I would be able to tell if it wasn’t true.

        1. 1

          The biggest problem is that the whole Google Mail thing is a black box, that there’s no way to know why exactly your mail is being rejected (e.g., it tells me that my domain has a low reputation, but doesn’t tell me why), and that there’s no recourse on fixing the situation (e.g., there’s no way for appealing the misclassification of those cron emails that I sent from my own domain to my own Gmail account that Google, apparently, deems to contribute to the low reputation).

          Google Search already has a solution for picking up new web properties very fast, whereas the rest of the players (e.g., Yandex) take ages to move a new site from a Spam to a non-Spam status and to include it in their main index — there’s little reason this couldn’t be done for email. The fact that it’s the very small players that are most affected leads me to believe that it’s merely a CoI at play — they don’t want you or me to run our own mail servers — they want all of us to simply use Gmail and G Suite instead.