1. 68
  1.  

  2. 9

    I (along with probably every other technical e-mail user) have had thoughts about a replacement mail protocol layered on modern standards, but having flicked through JMAP, the level of abstraction was.. not encouraging. It reminded me of SOAP.

    Does anyone have experience with implementing it? Be-all protocols have a horrible habit of turning out to be garbage, and JMAP kinda reeks of it. It’s main saving grace is that it comes from a well-regarded industry participant, but that’s no guarantee either. Maybe it’s just an allergy to overly generic specifications

    1. 5

      I can only tell you from user experience that it’s life changing and makes me not hate email. It makes GMail seem incredibly slow

      1. 1

        How are you experiencing this? Through the Fastmail web app?

        1. 2

          Yes open the web app on multiple devices and do things: move lots of messages, tag, send, receive, archive, etc. It’s instant across all web apps and very fast over IMAP.

          Now do the same with GMail and watch how it lags before your changes synchronize stage across the Gmail web apps and even IMAP: it will take anywhere from 15 seconds to 5 minutes. I’ve made changes in GMail that took 5 minutes for it to show up in IMAP and vice versa. It’s crazy how bad GMail really is.

    2. 5

      Any good technical summary of how it’s different from IMAP?

      Could the same thing be done as an IMAP extension, but they wanted to convert everything to JSON?

      1. 4

        A few differences with IMAP are mentioned in https://fastmail.blog/2014/12/23/jmap-a-better-way-to-email/

        Basically this is strongly inspired by IMAP + what GMail does.

        The people behind this know IMAP very well. Chris Newman is a co-author of the spec, and the Fastmail people are maintaining Cyrus which is one of the major IMAP servers.

      2. 4

        Does anyone know of a compatible-client list? I’m guessing that Gnus (for now) isn’t on it.

        1. 9

          There’s an official client list here: https://jmap.io/software.html

          1. 1

            Thanks, this is exactly what I was struggling to find :-)

        2. 5

          The specs directory of the JMAP specification counts 42964 words only for the core and mail part (equivalent of IMAP).

          The RFC3501 specifying the core IMAP counts 28059 words.

          How is JMAP simpler than IMAP?

          1. 5

            As a counterpoint: This implementation found it significantly shorter (LOC-wise) to implement JMAP than IMAP.

            That said, not really sure about the specific details, but someone interested might want to take a look.

            1. 1

              Writing web clients just became easier! We now have one heavy (veeery heavy) client that loads light (web) clients for all purposes. This means you indirectly pull Firefox or Chrome as a dependency to the JMAP protocol:

              We are stacking another 2 layers on top of TCP/TLS, and both have the same roles:

              TCP/TLS/IMAP -> TCP/TLS/HTTP/JSON/JMAP

              Why not TCP/TLS/HTTP/JMAP or TCP/TLS/JSON/JMAP ?

              Wrapping all the specific syntax rules over a general purpose description syntax makes parsing easier, but we now have two nested description syntax.

              Your mail servers needed a mail daemon, it now needs Apache/NGiNX (both company-backed mastodons) and a JMAP server using an HTTP library, a FastCGI library, a JSON library.

              If you consider all these dependencies are fine and are ok with adding more as the industry demands it, maybe go for JMAP.

              Of course PHP — a programming language specific to web development — has all of it built-in.

              In the end, the argument is “we made the protocol more complex because our tools make it easier to deal with such stacks” and mine is “I strive for making the stack simpler”.

              1. 1

                “we made the stack more complex” but also removed a lot of the mess from the protocol regarding parsing.

            2. 4

              Not agreeing or disagreeing with your point, but for what it’s worth, no one uses just rfc3501, but many extensions as well.

              1. 1

                Fair point. And some are even useful (IDLE command).

                I’m ok with replacing IMAP if the successor is “excellent”, because the industry will not accept 2 changes in the same century, and JMAP feels only “quite good” to me.

                1. 2

                  And some are even useful (IDLE command).

                  IDLE works on only the currently selected mailbox. Some mailservers I connect to have a limit to the number of connections I can make, so watching a lot of mailboxes for new mail is not possible with IDLE. How do mail clients handle new email notifications?

                  1. 2

                    [EDIT] NOTIFY corrected that mistake https://tools.ietf.org/html/rfc5465

                    Ack! You are right! Such a PITA. I guess this was designed so that you can IDLE only INBOX, but this terribly fits sieve where mail might arrive already sorted into the mailbox…

                    Otherwise, always IDLE on INBOX, and move the mail to other mailboxes afterward, but this will cause EXPUNGE responses as the mail would already be moved from INBOX to the destination mail box once the client request them…

                    1. 2

                      [EDIT] NOTIFY corrected that mistake https://tools.ietf.org/html/rfc5465

                      It did, but I haven’t yet encountered any server to advertise NOTIFY support :/

                      1. 1

                        A shame !

                        SEARCH and NOTIFY are the 2 reasons IMP is useful. Otherwise, POP or even 9P / NFS / SFTP does the job !

              2. 2

                I’m speculating here, but maybe specification is more precise and describes better all edge cases, thus using more words? Does really the number of words make something simpler or not? JSON specification is shorter than TOML, for example, yet their implementations differ in the opposite directions.

                1. 1

                  JSON being much simpler it gets possible to write streaming parsers, that do not allocate memory and read one byte at a time. Making it possible to validate a JSON feed with few stack memory and no heap (if in C), and have a easily debuggable / testable implementation split into a scanner and a lexer.

                  TOML relative complexity (change the object declaration scope in the middle of the file) makes a streaming parser hardly useful.

                  1. 1

                    Yes, only using the word count is a fallacy.

                    But the IMAP standard had way too much in it, and JMAP it could have been made way way simpler while it only went way simpler.

                2. 3

                  Just from a usability standpoint, this blows IMAP out of the water. The inbox refresh is just about instant, and everything else feels snappy too.

                  1. 3

                    Honest question: is anyone outside of Fastmail (and Fastmail-owned Topicbox) using this?

                    1. 15

                      You’ve got to start somewhere. I’d given up on IMAP ever getting better or being replaced. This is an encouraging step.

                      1. 2

                        Absolutely agree, I’m curious to see if anyone’s implemented it independently. It’s been a thing for a few years now, and I don’t recall seeing anyone else implement it but I know my awareness is hardly comprehensive :-)

                        1. 6

                          Apparently someone has a grant or something to add JMAP support to K-9 mail this fall. And Dovecot have been talking about adding support for a couple of years, but I don’t see it as having gotten anywhere.

                            1. 1

                              Can’t wait for K-9 to support JMAP! Since better performance over mobile networks is a core goal of JMAP [1], I except to see a fair improvement when I pull new emails over a LTE or 3G connection.

                              [1] https://jmap.io/#why-is-this-needed

                        2. 1

                          Also, I’m curious why there hasn’t been any move from Fastmail to make it available externally. AFAIK, it’s pretty much an implementation detail of their web client, and there’s no documented way to use an external client connecting via JMAP, nor even mention of it coming in the future.

                          1. 1

                            Looks fine. It’s just a JSON-y and more powerful version of IMAP though. Evolutionary, not revolutionary. I like the support for tags and such though.

                            Can we fix SMTP yet?

                            1. 4

                              As explained at jmap.io, it’s a whole new protocol, not just an expanded JSON version of IMAP. I hope it catches on.

                              1. 1

                                in what way is it different from an expanded JSON version of IMAP?

                                1. 8

                                  It also supports outbound client mail, replacing SMTP for the clients. That’s pretty different. It also doesn’t use a persistent socket, and the API is quite different.

                                  1. 1

                                    So no notification (imap IDLE) support? Polling an HTTP api is what to do?

                                    1. 1

                                      Yes it support notifications:

                                      For desktop clients and webmail, there’s an event source interface. This requires a persistent HTTP connection.

                                      No firewall / Timeout problems?

                                      For mobile, and web integrations, you can set a callback handler, which conforms with the use of a push endpoint by an Application Server as defined in RFC8030.

                                      Be careful with abuses!

                              2. 4

                                … No it’s not?

                                And, in terms of client access, this literally addresses that:

                                JMAP (JSON Meta Application Protocol) is a modern standard for email clients to connect to mail stores. It therefore primarily replaces IMAP + SMTP submission. It does not replace MTA-to-MTA SMTP transmission.

                                1. 0

                                  it’s not? how so?

                              3. 1

                                Obviously written by a Marketing Lead. Where is the technical meat?

                                1. 10

                                  I mean, the RFCs are linked right in the first paragraph.

                                  1. 6

                                    i was expecting this reply, but of course an article like text comparing JMAP to IMAP with perf statistics is what I’m looking for, instead of reading the dreary bare specs and deriving the real world implications by hard thinking.

                                    1. 10

                                      You know, that’s fair. My response was too glib, and I apologize. The JMAP homepage might have better info. As far as I can tell, it’s a more “webby” HTTP API, that doesn’t require a persistent socket, and has changes for use cases that are more common now than when IMAP was standardized. Too, it also handles sending mail from the client.

                                      I’ve thought about implementing it, or at least some of it, just to see. But it’s pretty ugly, and I really mislike the HTTP style, so, I’ll probably just continue free-riding on other people’s work.

                                      1. 1

                                        There is no performance involved with a protocol. Implementations have.

                                        Protocols might be designed with performance in mind though.

                                        1. 1

                                          Anecdotal, but Fastmail web via JMAP is the fastest, most responsive email client I have ever used (other than pure-text ones like mutt).

                                    2. 1

                                      Does it do anything about spamming and phishing?

                                      1. 3

                                        No, it’s a replacement for IMAP + CalDAV.

                                      2. 1

                                        Why not something like this? Is it too late to change?

                                        Handle the session

                                        OPTION  /               - Gives a version + capability string
                                        HEAD   /auth            - HTTP password authentication, delivering a cookie,
                                                                  to avoid authentication on every query (also possible)
                                        GET    /list            - list the mailboxes, 1 per line
                                        

                                        Handle mailboxes

                                        PUT    /box/<path>/     - create mailbox <path>
                                        PATCH  /box/<path>/     - rename mailbox <path> to the new path sent as body
                                        DELETE /box/<path>/     - delete mailbox <path>
                                        GET    /box/<path>/     - get the ID + headers of mail in mailbox <path>
                                        

                                        Handle mails

                                        GET    /mail/<box>/<id> - get the mail <id> (GET attributes might speicify
                                                                  whether to get everything or just a part)
                                        DELETE /mail/<box>/<id> - delete mail <id> from <box>
                                        PATCH  /mail/<box>/<id> - rename mail <id> from <box> to new path sent as body
                                        PUT    /mail/<box>/     - submit a new mail to mailbox <box>
                                        PUT    /mail/           - submit a new mail forsending
                                        

                                        Only HTTP error status are used instead of an error-in-body.

                                        1. 1

                                          No it can’t:

                                          The encoding/transport part of JMAP is not core to its operation, so future specifications could easily add alternatives (e.g. WebSocket instead of HTTPS, CBOR instead of JSON). For the initial version though, HTTPS+JSON makes the most sense.

                                          1. 1

                                            The JSON part could get some simplification as well. The current version feels like going too much down the specifics of each objects like IMAP does.

                                            Why not a single query format? With “query” mapping to the SEARCH command, “parts” to the FETCH, and then some “sort”, “range” and “options” to influence the format in which the query arrives?

                                            {
                                              "query": {
                                                "object": "email",
                                                "action": "query",
                                                "account": "conctact@example.net",
                                                "mailbox": "INBOX",
                                                "fields": {
                                                  "From": "example.net",
                                                  "To": "me@"
                                                },
                                                "or": {
                                                  "dateAfter": 1567370000,
                                                  "dateBefore": 1567300000
                                                }
                                              },
                                            
                                              "parts": {
                                                "body": true,
                                                "mime": [ "1.4" ],
                                                "headers": [ "From", "To", "Subject", "Date" ]
                                              },
                                            
                                              "sort": [ { "date": true }, { "received": false } ],
                                            
                                              "range": [ 0, 10 ],
                                            
                                              "option": {
                                                "collapse": true,
                                                "total": true
                                              }
                                            }
                                            

                                            OTOH, it seems like JMAP is trying very hard to get fast over HTTP/1.1, where doing queries one by one is not acceptable. I’d fix the transport problems at the transport level (HTTP and even TCP), but that was probably not an option for a company that has a market to conquer.

                                            It had to be fast now, without changing anything of the stack, so the only option was to adding one more level, rather than replacing the existing one.

                                            See you soon JMAP, as I believe, given the design oriented on the market, that you will conquer the world of corporate mail servers.

                                          2. 1

                                            If JMAP can achieve higher performances, there might be something interesting going on!

                                            1. 3

                                              I’ve been using JMAP for years via Fastmail and it is truly amazing how fast it is compared to IMAP

                                              1. 1

                                                I think I got it: instead of having to open a connection, enter the login/password, select the mailbox you want to act upon, enter the mailbox command, you directly enter ONE query that has all of the above.

                                                That means there is no direct TCP socket handling in PHP.

                                                One query per operation with full path to the mailbox to interact with every time (instead of walking through all the levels and do parsing every time) with another scheme for notification, and no need for persistent connection. Stateless protocols rocks.

                                                Webmail was a hack, it is now a first class hack^Wcitizen.