1. 44
  1.  

  2. 13

    I do have to wonder how long until Google decides that only certain clients can connect with Gmail?

    They already did, kinda. Granted, you can still do IMAP, but it’s a pain in the backside, and Google has been pushing that out, so I wouldn’t be surprised if it would be completely gone in a year or two.

    1. 12

      Starting October 1st, we will start to randomly select tenants and disable basic authentication access for… IMAP, …

      IMAP, at least, isn’t going away, just “basic authentication.” You’ll still be able to use it, but you won’t be logging in with a username and password. Instead, you’d need to use a predictably non-standard MS extension to oauth2 called xoauth. It relies on Azure and O365/Exchange and I didn’t find it to be difficult, but whether or not clients will adopt it is another matter.

      I ran into this last month. I had to work on some IMAP integration recently with O365, and about a week after getting something working MS switched off legacy auth in my employer’s tenant. I had to switch over to xoauth2. The only difficult part was finding any information about how to do it. Microsoft’s documentation, given how much of it there is, is usually scattered and often hard to piece together.

      1. 3

        My workplace did the same. Seems like the easiest way is to pretend to be Thunderbird, and use its client ID and secret.

        I don’t know what this adds other than friction, but maybe that’s the point.

        Using OAuth like this just doesn’t work. I thought corporations learned that during the rise of smartphones, when people were plucking OAuth credentials out of apps for fun.

        1. 1

          I used client modules in go and node. Roughly, it consisted of setting up a new azure app, granting that app permission to use imap in the tenant, enabling oauth flows for the app, creating a principle (user)/client I’d/secret, granting O365 permission for the user. Then it was a matter of enabling imap access on the specific O365 user account/mailboxes, and finally granting the principle access to the boxes. There were a few powershell scripts mixed in.

          It wasn’t obvious or documented and took some trial and error. This has been my experience for anything having to do with azure and integration with virtually any of Microsoft’s other cloud products: business central, o365, etc. It’s a time suck.

          After all that, my understanding based solely on my admin’s findings is that IMAP is sticking around, at least for paying (enterprise) customers. However, vanilla, plaintext, basic auth goes away.

        2. 2

          IMAP, at least, isn’t going away, just “basic authentication.”

          That’s not exactly how the mail is phrased:

          [The Enterprise] only supports the use of the approved email client Outlook, Outlook 365 available in our [The Enterprise] tenant, or Outlook for iOS and Android.

          That makes it sound like you can only use Outlook.

          Also: do they even read what they’re writing? This is so user-hostile it isn’t even funny:

          “If you use an alternate email program that relies on POP3, IMAP or SMTP, like native mail programs in iOS and Android, expect your [The Enterprise] email connection for that app to stop functioning when Microsoft chooses to disable these protocols.”

          1. 2

            [The Enterprise] only supports the use of the approved email client Outlook, Outlook 365 available in our [The Enterprise] tenant, or Outlook for iOS and Android.

            That’s the policy of the tenant. A lot of orgs completely disable POP/IMAP in Exchange, and just use OWA/EWS/MAPI. As stated, IMAP is staying if your org uses it, just with OAuth.

            (The funny thing is Outlook doesn’t support the OAuth extensions to IMAP, so it’ll just be using EWS/MAPI.)

            “If you use an alternate email program that relies on POP3, IMAP or SMTP, like native mail programs in iOS and Android, expect your [The Enterprise] email connection for that app to stop functioning when Microsoft chooses to disable these protocols.”

            Mobile apps are usually using EAS instead of IMAP.

            1. 1

              I agree totally agree regarding the user hostility bit. Just to be clear, my employer /is/ using O365 in an enterprise tenant.

              In my case the integration service relied on IMAP with regular authentication. My admin read that they were going to disable IMAP on some tenants in the future, and the following week they did for ours. He re-enabled it, but the integration wasn’t able to authenticate.

              After a day or two of spelunking though the MSDN nightmare we were able to piece together a process to get the oauth client credentials flow, suitable for integration because it doesn’t require manual consent, working against their tenant/O365. Works like a charm. Still IMAP, different auth scheme.

              So you’re right, IMAP may be going away in general, but it’s alive and well in their enterprise environment. “Basic authentication” no longer works, at least for us.

              Edit: I’m not using an outlook /client/. I’m using imap clients in go, node, and at least initially c#.

              1. 1

                Wow! I didn’t expect this to get posted here. Neat.

                Anyway, yes, user-hostile. But by now, I’m used to this attitude from The Enterprise. They brought the company I was working for and by God, we’re going to do things their way (for development, they’re doing The Enterprise Agile method, which is mostly the stuff on the right hand-side of the Agile Manifesto and ignoring the left hand-side).

            2. 17

              IMAP became a legacy protocol around 2010-2012.

              Bron Gondwana and I had dinner and talked about IMAP in 2012. February IIRC. We were both fairly active in the mail sector of the IETF at the time, both saw that IMAP needed updates in order to match modern usage, and discussed what to do. We decided that leaving IMAP was best, and that it would be better to define a new protocol based on JSON+HTTP, with new basics.

              Bron did so, with his team at Fastmall: JMAP.

              IMAP5 has benn defined since, but it takes a more legacy approach. Maintenance rather than evolution.

              Some examples of the kind of thing moden mail readers want to do that are unreasonably hard using IMAP:

              1. Search globally.
              2. Having archived a message on the day before, retrieve an attached PDF from that.
              3. While composing a new message, find some other addresses that might be Cc’d.
              4. Show all the PDF attachments.

              IMAP search is also unreasonably hard to use. For some use cases you want it exact and for others you want fuzzy searching, and the IMAP command doesn’t quite fit either. Gmail and another couple of things made search much more important and IMAP’s facilities weren’t good enough any more.

              There are some things modern servers want too (and clients have to go along with).

              1. 5

                It still surprises me that JMAP is meant to be taken seriously. It’s based on JSON+HTTP and has a name that screams “April fool’s joke”.

                1. 4

                  On the one hand, JSON+HTTP is the perfect redflag for “quickly cobbled together” with little regard for performance or economy. On the other hand, connectivity is often restricted to ports 80 and 443 only, and many tools makes it hard to send anything other than text.

                  For people who don’t want to fight sysadmins (who never block outgoing 443 short of pulling the network plug), this might be exactly what you want… which sucks, to be honest.

                  1. 3

                    Just run IMAP+TLS on 443

                    1. 7

                      That would work indeed. Found their FAQ by the way. They have an answer about JSON + HTTP:

                      The short answer is it’s good enough, it’s widely understood, and it’s by far the easiest thing for developers to adopt. There’s support in basically all OSes and programming languages. And it’s easy to read and debug.

                      That’s the part that screams “quickly cobbled together”.

                      HTTP doesn’t tend to run into firewall issues, and is so commonly used that it has integrations which can help with optimisation (for example, iOS has built-in support for optimising radio usage by batching HTTP calls from different apps where possible, which their mail team have told us they would like to be able to use). This isn’t an innate advantage of HTTP, but rather an advantage of its ubiquity.

                      That’s the part that screams “resistance is futile”. Another instance of HTTP eating the internet like the Borg (except HTTP is winning).

                      With GZIP, JSON data is reasonably compact and fast enough to serialise/parse. However, the encoding/transport part of JMAP is not core to its operation, so future specifications could easily add alternatives (e.g. WebSocket instead of HTTPS, CBOR instead of JSON). For the initial version though, HTTPS+JSON makes the most sense.

                      Yeah, let’s waste CPU cycles on compression. But we can fix the transport layer at a later date, so you can see that it’s not a big deal this was quickly cobbed together!


                      Dammit, you’re basically right: TLS over port 443 should solve most network issues, it doesn’t have to be HTTP unde the hood. A way to resist the Borg I guess.

                      1. 5

                        You have naïve ideas about IMAP. It wastes lots of cycles. The biggest wastes involve computing MSNs.

                        IMAP is designed such that a server can serve everything while keeping open only one mailbox at a time and clients can perform various optimisations using MSN arirthmetic. An implication of that is that servers need to serve MSN data that generally ends up being recomputed expensively by the server and ignored by the clients. Some of it can be cached if a server disallows non-IMAP access, but for various reasons a lot of MSN data is recomputed or regenerated often.

                        MSNs, for those blessed with ignorance of this particular detail, are sequential message numbers. They’re mandatory in IMAP and make it simple for a client to say “give me from/to/subject of the fifth oldest message in this mailbox”, where “fifth oldest” refers to order of insertion into the mailbox and message are only counted if they still are in the mailbox. This is very neat, but almost 100% unused, generates more than its fair share of questions on Stack Overflow, and tricky for a server to implement. Once a server has implemented the rules and taken care of race conditions and concurrent access, there’s not much caching of MSNs left.

                        When a modern client wants to read an attachment from an old archived message, the server frequently has to read a few bytes from disk for each old archived message. That’s O(terrible) and should be O(1).

                        Ignored MSNs are bad, but hardly the only source of waste. Suppose a users types in a search. An IMAP client then issues as many SELECT commands as needed (each of which generates much MSN-related work) andthen UID SEARCH, which in turn causes something like a Lucene search in the backend. So where a search via JMAP would cause one Lucene search of the user’s mail, one via IMAP causes dozen of (often concurrent) searches, which is very much worse for Lucene’s performance. If the server uses per-user Lucene, then it’s horrible while if the server uses per-mailbox Lucene it’s just really bad. (On the other hand, moving messages to another mailbox is hard work for per-mailbox Lucene.)

                        Some clients even try to send their searches such that the client can show some results while the search is executing, which ends up causing even more work for the Lucene-like backend, as Lucene gets to perform the same search over and over again, filtering by different date windows each time. It looks nice on the client but the load spike on the backend is brutal. JMAP avoids all of this waste.

                        1. 2

                          You have naïve ideas about IMAP.

                          I never talked about IMAP. Don’t know it, haven’t any idea about it. My only problem here is JMAP’s choices for its serialization format & transport layer. Did you perhaps respond to the wrong comment?

                          I’m still glad I read your comment though. These MSN & search problems with IMAP are good to know. Seems way worse than a bad choice of serialisation format & transport layer. And now I’m beginning to have ideas about IMAP, and they do not look good. :-)

                          1. 2

                            There are three relevant protocols in this domain, namely IMAP, JMAP and Gmail’s. Did you judge JMAP without any domain knowledge?

                            1. 2

                              Did you judge JMAP

                              Not sure what you’re trying to achieve with this attempt at over-generalisation. To repeat my latest comment I said that “my only problem here is JMAP’s choices for its serialization format & transport layer”. Because I happen to know about serialisation formats & transport layers.

                              I don’t presume to judge the rest of JMAP.

                        2. 1

                          That’s the part that screams “quickly cobbled together”.

                          I’m not sure if you’re meaning to come across as dismissive, but to me at least you are.

                          Why does this seem quickly cobbled together, and why does that have anything to do with it being fit for purpose?

                          1. 3

                            Textual formats are a waste of computational resources, and the only reason we use them for machine-to-machine communications is because text editors are ubiquitous. Remember the time where we used XML everywhere just because we have XML libraries everywhere. That helps you get a working prototype quicker, but also locks you into a “one size hopefully fits all” solution.

                            The most glaring issue about textual formats is that they’re terminator based. This means variable length buffers, harder parsers, more vulnerabilities if you have the audacity to use an unsafe language to process this format. Binary formats however can go a long way with simple TLV encodings that lend themselves to very easy and very fast encoding & decoding.

                            I know nothing about JMAP, and to be honest the thing as a whole looks somewhere between decent and bloody excellent. The serialisation & transport layer however was definitely quickly cobbled together, and they know it: it says right there in the FAQ that we may replace that layer by something else if we so chose.

                            Now it looks like the rest of the protocol (everything except transport & serialisation) is much more important for performance (and I guess functionality as well). I realise that serialisation is a little thing. Still, it’s a thing.

                        3. 0

                          That breaks badly. Modern clients want to do things like search globally and run on devices that change address. It sucks when a nice phone client opens 40 connections, makes the server build significant RAM state for each, issues a bunch of slow UID SEARCH commands, then loses WLAN coverage and repeats the whole thing over 4G, with bonus bugs due to faulty resynchronisation.

                          (IMAP maintenance and extension has largely been done by 10-20 people. None of them think JMAP is a bad idea or cobbled together, AFAICT.)

                          1. 2

                            None of those things are magically fixed by picking HTTP+JSON

                            1. 3

                              They are fixed by JMAP.

                              1. 2

                                I should perhaps elaborate on this.

                                JMAP escapes this problem by not having long-running connections with significant per-connection state. You could reduce IMAP to using a long-running TCP connection without significant state, in which case you’d have something rather like HTTP+JSON except that it would use its own verbs instead of HTTP and Lisp instead of JSON.

                                Given how much trouble we’ve seen with people’s ability to parse and use the existing lispy structures and how easy it is to find JSON tooling, the choice is really simple.

                  2. 4

                    Gmail has been telling me my Thunderbird POP3/SMTP client is insecure for years now.

                    1. 4

                      It’s a yikes from me, but it’s not like Microsoft was ever in the running to host my mail before this. I would venture that anybody that cares about IMAP access feels the same way, and that as dumb as this is, it won’t really lose them any business.

                      1. 6

                        Many people who use Outlook do so because it’s required by their employer.

                        1. 2

                          That’s true but I’m not sure I see your point; do you want me to feel bad for those folks?

                          If a company’s mail is hosted by Microsoft, I think it’s unlikely that they care about IMAP. If the person in charge of a company’s mail cares about IMAP, I think it’s unlikely that that company’s mail would be hosted by Microsoft in the first place.

                          1. 1

                            wanted to make sure the impact on users wasn’t overlooked

                            1. 1

                              You can care about imap but struggle to get emails delivered without being inside one of the big bullies.

                        2. 4

                          When JMAP caught on?

                          I’m ok with standards going away when they’re replaced with better standards.

                          I don’t know enough about outlook’s protocol(s) to know if they’re standardized in any meaningful way (I speculate not) or if they’re better than other standards in a meaningful way (I speculate maybe, but probably for specific use cases not everyone needs).

                          1. 25

                            JMAP hasn’t really caught on yet though, isn’t it still just fastmail and selfhosted servers?

                            1. 3

                              Unless something’s changed, I don’t even think fastmail themselves support external JMAP clients, they’ve just used it as an internal implementation detail.

                              1. 1

                                https://www.fastmail.com/developer/integrating-with-fastmail/ seems pretty clear that they offer it and it’s their preferred way to integrate.

                                1. 1

                                  Sure, for developers writing an integration. Last I looked, they had zero documentation for how to connect/configure/use a JMAP client as an end-user – which seems like a necessity if this is to be taken seriously as a competitor to IMAP and POP.

                            2. 6

                              JMAP standards group is still publishing drafts on Calendar, Contacts, etc specs. The general standard however is out, so is the extension for email!

                              Thunderbird seems to be positive to adding it (they even mention using my own JMAP library I wrote for my email client) but that was 2 years ago.

                            3. 3

                              Just replying to the title spontaneously: I think when Mark Crispin died

                              1. 3

                                As others have stated, the problem isn’t that POP/IMAP are being phased out by Office, but “basic authentication” (i.e. traditional username/password) is being phased out. My employer pushed the necessary button to do this a few weeks ago, causing me some confusion as to why my email setup stopped working!

                                There are several tools that can help make existing email setups work with OAuth e.g. Email OAuth 2.0 Proxy, mailctl, mutt_oauth2.py.

                                None of those quite worked for me, so I’ve written my own called pizauth (https://github.com/ltratt/pizauth) which I now consider to be at the alpha release stage, and ready for somewhat wider testing. I and others have so far only tested it on Office’s OAuth servers, so I can’t yet guarantee it works for other sites. Bug reports are most welcome!

                                1. 3

                                  You can re-enable these for your tenant. Microsoft is turning off access and pushing warnings to say the authentication protocol for IMAP and POP3 are not recommended. They want you to use a more modern authentication protocol to access their mail servers. I do not want to force our teams to migrate from the email apps they like, so I re-enabled the protocols tenant wide.

                                  1. 3

                                    There’s always a proxy.

                                    1. 3

                                      When they turn it off you don’t even get a warning. IMAP just stops working. The new auth method (pdf) isn’t difficult to work with, but whether or not non-MS mail clients will support it more broadly remains to be seen.

                                    2. 3

                                      When nobody invested the engineering effort to get them using Security keys

                                      1. 2

                                        I’ve had good luck using mailctl for wrangling credentials for oauth based imap/smtp with whatever365 and gmail.

                                        1. 1

                                          Outlook also recently disabled email forwarding, which would have been another way to escape this encirclement. It’s already causing loss of productivity, but most of the cost is pushed to the users themselves who spend their own time dealing with an unfamiliar and poorly designed client.

                                          1. -1

                                            1995 or thereabouts. Why?

                                            1. -1

                                              This reminded me to open my mail app, to let it sync up my Gmail.