I don’t really have this problem; almost everything that ends up in my inbox generally matters, and the unread emails is almost always below a few dozen emails (but usually smaller, 10 at the most). I’m not sure what I’m doing different from you or what kind of unwanted emails you’re getting?
One problem I have is that I receive more emails meant for other people who think their email address is my email address. I get reminders for choir practice in Atlanta, deals at a Ford dealership in West Virginia, dentist appointment reminders, and tons of advertising for stores I don’t know about. It’s a never-ending fight to mark as spam or unsubscribe. In a single day, I get one or two emails meant for me and 20-25 meant for other people.
Maybe so, but if after several decades still no unbroken client emerges then there’s still something fishy going on. It might not be the protocol (though I hear that IMAP isn’t exactly the nicest thing to work with) but maybe we just haven’t figured out what a good email workflow would look like.
Of course that’s coupled with the problem that everyone uses email differently.
Good: Central principle of one command to search taking parameters giving back uids, one command to fetch taking UIDs giving back parts of listed mails you asked: this is simple and nicely designed, You can now search and do something with the result like fetching it.
Bad: IMHO, he most useful feature of IMAP after searching mails is to have notifications of email: that is why there is the effort of keeping a TCP connexion open. But this is brought through extensions.
Ugly: many small quirks, like syntax with many edge cases, a select command that gets in the way (and prevents searching a string in everything for instance), mail IDs are at most int 64: too short to keep them immutable.
Maybe something like Sieve could help with that, if it ever become widespread. It is not too late for email to evolve: autocrypt, chat-over-imap (email) IMAP IDLE and NOTIFY.
We now have all the tools to build a solid experience providing all the features we are looking for from all chat systems. All it takes is findingsomeclientthatsuitsyourneeds.
The first is that email has changed a lot in how it’s used. Most users 20 years ago had one client per mail account. POP3 was common and its usage model was very simple: clients downloaded all messages, the server storage is just a cache of the last n days of emails. IMAP allowed storing the entire mail history on the server (POP3 did in theory, but a bunch of operations were O(n) in terms of the size of the mail box) and started shifting things to the server. Adding in things like Sieve and you’re in a situation where there’s a lot of duplication between the client and server. Most mail clients can do spam filtering but no one really wants this anymore: if Thunderbird does spam filtering, then no spam received since the last time I had my laptop on and now will be filtered when I check the mail on my phone / tablet / any other device. Now, most people check email on multiple devices, so you need to do almost everything on the server for it to be useful. Features that only work on one client are annoying.
The second problem is that the communities that write mail servers and mail clients are quite disconnected. Look at JMAP. It is a big improvement over IMAP, but it’s only really supported by a single client for a proprietary mail service, which uses its own server. They can do this because they have the vertical integration (the same is true for Gmail and Outlook / Exchange): they add features to the client and server at the same time. JMAP is low priority in Dovecot because Thunderbird doesn’t use it, it’s low priority in Thunderbird because Dovecot doesn’t use it.
This is made worse by the fact that supporting good server-side features (e.g. Sieve) reduces the perceived value of the client. I use Sieve to filter all of my mail and use a Thunderbird plugin that gives me syntax highlighting of the script and updates it over the managesieve protocol. As a result, Thunderbird’s complex mail rule features are of zero value to me. Similarly, my mail server does server-side spam tagging and I filter the results with Sieve (with IMAPSIEVE rules to mark messages as spam or not-spam when they are moved to and from the junk mailbox).
These then Supporting a new feature on the server removed any value for me in the client having spam detection and message filtering logic.
I agree with the generally point, and here are some nuances or extrapolations.
JMAP
JMAP is actually more REST-like than most “RESTful” APIs. It is stateless, highly cacheable, supports transparent intermediaries and provides a uniform interface for manipulating different resources. However, it doesn’t use HTTP verbs to implement this. – https://jmap.io/
With the advent of HTTP 3 and presence of HTTP 2, why bother optimizing a few extra round trips – or – why bother with HTTP at all?
We end-up with HTTP 200 with this in it: { "status": 400, ...}.
On the other hand, immutable UIDs, constant syntax without so many exceptions are a good thing.
server-side -vs- client-side
It is a good thing that things are moving toward server-side for some aspects: servers are easier to maintain than each and every user’s machines (that may still literally be between their hands when we need to do the maintenance).
Yes, sure, former/existing mail client look silly now with all its features.
walled garden
Either we embrace (extend, extinguish) the situation and stick our head into the lion’s head with walled garden like whatsapp, signal, telegram and so on.
Either we accept the issue and try to bridge everything together (matrix- or matterbridge-style) and hope it will get better (which will not happen as long as bridges are maintained)
Either we jump back to the lowest common denominator, and keep building from there to improve the situation for everyone at once.of
It looks like a short-, middle-, long-term solution choice, so maybe all three up front? A lot ahead, yet a few years ago, none of these issue did really exist!
With the advent of HTTP 3 and presence of HTTP 2, why bother optimizing a few extra round trips – or – why bother with HTTP at all?
We end-up with HTTP 200 with this in it: { “status”: 400, …}.
For JMAP specifically, its more of a RPC protocol that runs over HTTP (or anything really; a profile for WebSockets and works great).
Why HTTP+JSON? Mostly because its convenient: everyone understands it, and every platform has high-performance libraries for it. The learning curve on IMAP is a big part of why JMAP exists.
As for optimising a few round-trips, not everywhere is nearby. Much of Fastmail’s frontend engineering was done in Melbourne, Australia, 300ms from our servers on the US east coast. A lot of our UI performance comes from packing a lot into each request.
We end-up with HTTP 200 with this in it: { “status”: 400, …}.
The RFC that details with the detailed error responses, RFC7807, specifically calls out that the status code listed in the json object must match the status code returned by the web server. The status member of the response object is considered a “convenience” for the consumer of the response, whatever the heck that means.
One area where this could be false, which is addressed in the RFC itself, is when proxy servers, load balancers, etc. might alter the status codes of a response.
Then I am tempted to think this is a misbehavior of the reverse proxy that changes the semantics. I feel like sliding onto a different topic, but is that not the point of discussions… :)
The second problem is that the communities that write mail servers and mail clients are quite disconnected. Look at JMAP. It is a big improvement over IMAP, but it’s only really supported by a single client for a proprietary mail service, which uses its own server. They can do this because they have the vertical integration (the same is true for Gmail and Outlook / Exchange): they add features to the client and server at the same time. JMAP is low priority in Dovecot because Thunderbird doesn’t use it, it’s low priority in Thunderbird because Dovecot doesn’t use it.
I think fastmail has an inner fork of Cyrus, and they have merged big parts of it with upstream Cyrus (the latest version has JMAP support). The problem with JMAP is that parts of it are still nonstandard yet (unpublished or only drafts), mainly notifications and calendar/contacts. meli has basic JMAP support.
This is made worse by the fact that supporting good server-side features (e.g. Sieve) reduces the perceived value of the client.
This is relevant to your first paragraph: client-side filtering doesn’t make sense anymore (or at least like Thunderbird does it). Spam and filtering in general is part of the “mail stack” you use and clients assuming they can replace the concept of a stack is the wrong approach. The user should always be able to define their stack.
The problem with JMAP is that parts of it are still nonstandard yet (unpublished or only drafts), mainly notifications and calendar/contacts.
While the JMAP standards that are in a status of ‘PROPOSED STANDARD’, it is worth noting the vast majority of RFCs in use on the internet are in that status. The list of RFCs with a ‘Internet Standards’ is pretty darn short.
Notifications are a thorny one, but JMAP’s core RFC does have two different methods for notifications/pushing to clients. The more resource friendly method unfortunately also leads to vendor specific implementations.
As for Calendars & Contacts, I agree. I hope that they submit it for a proposed standards track soon.
I realize in hindsight that my comment could be interpreted as me implying it is being done in private, that was not my intention.
It’s more that I’m itching for it to be published as a Proposed Standard RFC so that I can safely assume it’s what people will be developing with. I have a few ideas I want to explore but I’d rather wait until there would be other implementations in development that I could test interoperability against.
No problem, and fair enough! We’re of course keen to get them finalised too, but the IETF process can be slow. Its worth it though; the JMAP core and mail specs spent three years (I think) at IETF and are very much better for it!
I think fastmail has an inner fork of Cyrus, and they have merged big parts of it with upstream Cyrus (the latest version has JMAP support).
Fastmail has six internal commits that we apply to upstream master on each deploy. They are all either adjustments to handle differences of opinion or historical bugs in our middleware or stored data that require a lot of work to update (eg FM’s idea of “admin” is different to Cyrus’ idea), or they’re minor changes to match the way we like to operate the service (eg we convert all assertions and fatal errors into coredumps, triggering a bunch of analysis tooling).
I’m not saying Cyrus upstream is necessarily easy to operate, but in terms of JMAP support, its complete and correct (bugs notwithstanding) and what we use for both the Fastmail and Topicbox UIs and for the general protocol access we offer to users.
I completely agree that spam filters and blocking techniques should be a more prominent feature in general. For the average user, there is far too much spam in their inbox. Heck, for most users even. I mentioned a few techniques to limit spam in my article:
Email cloaking services protect your personal email being given away when you sign up for something. You can use spamgourmet or a temporary email when signing up for online accounts anywhere that you don’t trust. Similarly, you can use an email client like Thunderbird which has additional spam filters built in. As Kev mentioned, you can also train your spam filter to make it better at identifying what is and isn’t spam.
As you say “this is more of an application-level problem” then a protocol one. We can probably fix most of our problems by changing some of out habits surrounding email. However, it begs the question as to why should we have to? For something that was created in the 1960’s, surely a better solution exists?
https://jmap.io/ - JSON Meta Application Protocol. Developed by the people at Fastmail (I’m biased: I’m a happy paying customer, they’re based here in Melbourne, and are great human beings).
https://www.djcbsoftware.nl/code/mu/mu4e.html - a great mail client for Emacs. All that would be required for mu4e to support JMAP mailboxen would be a command line tool to download JMAP in the same vein as mbsync for IMAP.
. All that would be required for mu4e to support JMAP mailboxen would be a command line tool to download JMAP in the same vein as mbsync for IMAP.
Watch out for future releases of meli, I plan on shipping a “sync” mode that syncs to local maildir without launching a UI, just like mbsync/offlineimap. Although you would be technically using a mail client (meli) to sync mail for your mail client (mu4e) it’d be the intended use of this functionality 🙃
That’s how mu4e is designed to work :) It’s a front-end for mu, the mail indexing utility, which in turn expects to operate on a synchronised maildir. Currently I’m using mbsync to synchronise my Fastmail and Gmail mail via IMAP; it’d be easy enough to switch to meli once it supported JMAP.
Fwiw, I wanted to add that I use Gmail and while my unread-count used to be in the 2k+ range regularly, I’ve since managed to achieve inbox-zero by diligently adding custom email rules (filters) over the course of a year, and now I get almost no unwanted emails in my inbox. While I didn’t use this setup myself, here’s a GH project to help you manage your Gmail filters - https://github.com/mbrt/gmailctl.
And hence why I’m still using GMail as my email client for many years. Not because I’m overly fond of anything else about their mail infrastructure, as Fastmail is being used to do the actual “host my domain’s email” problem, but because it’s a better email client. I’m yet to see anything (paid or free) that beats it, which is annoying as I’d like to move to something self-hosted, but not at the cost of usability.
I use gmail and zohomail and both are really great at filtering out garbage. I only get things I want to see. I think email is generally pretty good. There’s definitely a much better signal-to-noise ratio than regular mail.
I don’t really have this problem; almost everything that ends up in my inbox generally matters, and the unread emails is almost always below a few dozen emails (but usually smaller, 10 at the most). I’m not sure what I’m doing different from you or what kind of unwanted emails you’re getting?
One problem I have is that I receive more emails meant for other people who think their email address is my email address. I get reminders for choir practice in Atlanta, deals at a Ford dealership in West Virginia, dentist appointment reminders, and tons of advertising for stores I don’t know about. It’s a never-ending fight to mark as spam or unsubscribe. In a single day, I get one or two emails meant for me and 20-25 meant for other people.
I never get these. I think it depends on the address how many of these you get.
Maybe so, but if after several decades still no unbroken client emerges then there’s still something fishy going on. It might not be the protocol (though I hear that IMAP isn’t exactly the nicest thing to work with) but maybe we just haven’t figured out what a good email workflow would look like.
Of course that’s coupled with the problem that everyone uses email differently.
I’ve been working on an “unbroken client” for a while now. From my experience, email just has so much stuff going on. Many bugs emerge when different components of a system interact in unforeseen ways, and an email client has to have many components.
IMAP:
I’m no big defender of IMAP, but there’s a lot of work being done to shave off some of the rough edges lately, see https://datatracker.ietf.org/wg/extra/documents/. In particular, the “64-bit mail ids” thing you mention is largely resolved by IMAP ObjectID (RFC8474).
I was wondering about this, looks like more room to use things like filesystem inodes or other methods to avoid a lookup to an index for every mail.
Now it is all about adoption ! :)
Maybe something like Sieve could help with that, if it ever become widespread. It is not too late for email to evolve: autocrypt, chat-over-imap (email) IMAP IDLE and NOTIFY.
We now have all the tools to build a solid experience providing all the features we are looking for from all chat systems. All it takes is finding some client that suits your needs.
I think there are two, related, problems.
The first is that email has changed a lot in how it’s used. Most users 20 years ago had one client per mail account. POP3 was common and its usage model was very simple: clients downloaded all messages, the server storage is just a cache of the last n days of emails. IMAP allowed storing the entire mail history on the server (POP3 did in theory, but a bunch of operations were O(n) in terms of the size of the mail box) and started shifting things to the server. Adding in things like Sieve and you’re in a situation where there’s a lot of duplication between the client and server. Most mail clients can do spam filtering but no one really wants this anymore: if Thunderbird does spam filtering, then no spam received since the last time I had my laptop on and now will be filtered when I check the mail on my phone / tablet / any other device. Now, most people check email on multiple devices, so you need to do almost everything on the server for it to be useful. Features that only work on one client are annoying.
The second problem is that the communities that write mail servers and mail clients are quite disconnected. Look at JMAP. It is a big improvement over IMAP, but it’s only really supported by a single client for a proprietary mail service, which uses its own server. They can do this because they have the vertical integration (the same is true for Gmail and Outlook / Exchange): they add features to the client and server at the same time. JMAP is low priority in Dovecot because Thunderbird doesn’t use it, it’s low priority in Thunderbird because Dovecot doesn’t use it.
This is made worse by the fact that supporting good server-side features (e.g. Sieve) reduces the perceived value of the client. I use Sieve to filter all of my mail and use a Thunderbird plugin that gives me syntax highlighting of the script and updates it over the managesieve protocol. As a result, Thunderbird’s complex mail rule features are of zero value to me. Similarly, my mail server does server-side spam tagging and I filter the results with Sieve (with IMAPSIEVE rules to mark messages as spam or not-spam when they are moved to and from the junk mailbox). These then Supporting a new feature on the server removed any value for me in the client having spam detection and message filtering logic.
Very well put.
I agree with the generally point, and here are some nuances or extrapolations.
JMAPWith the advent of HTTP 3 and presence of HTTP 2, why bother optimizing a few extra round trips – or – why bother with HTTP at all?
We end-up with HTTP 200 with this in it:
{ "status": 400, ...}.On the other hand, immutable UIDs, constant syntax without so many exceptions are a good thing.
server-side -vs- client-sideIt is a good thing that things are moving toward server-side for some aspects: servers are easier to maintain than each and every user’s machines (that may still literally be between their hands when we need to do the maintenance).
Yes, sure, former/existing mail client look silly now with all its features.
walled gardenIt looks like a short-, middle-, long-term solution choice, so maybe all three up front? A lot ahead, yet a few years ago, none of these issue did really exist!
For JMAP specifically, its more of a RPC protocol that runs over HTTP (or anything really; a profile for WebSockets and works great).
Why HTTP+JSON? Mostly because its convenient: everyone understands it, and every platform has high-performance libraries for it. The learning curve on IMAP is a big part of why JMAP exists.
As for optimising a few round-trips, not everywhere is nearby. Much of Fastmail’s frontend engineering was done in Melbourne, Australia, 300ms from our servers on the US east coast. A lot of our UI performance comes from packing a lot into each request.
The RFC that details with the detailed error responses, RFC7807, specifically calls out that the status code listed in the json object must match the status code returned by the web server. The status member of the response object is considered a “convenience” for the consumer of the response, whatever the heck that means.
One area where this could be false, which is addressed in the RFC itself, is when proxy servers, load balancers, etc. might alter the status codes of a response.
Then I am tempted to think this is a misbehavior of the reverse proxy that changes the semantics. I feel like sliding onto a different topic, but is that not the point of discussions… :)
I think fastmail has an inner fork of Cyrus, and they have merged big parts of it with upstream Cyrus (the latest version has JMAP support). The problem with JMAP is that parts of it are still nonstandard yet (unpublished or only drafts), mainly notifications and calendar/contacts. meli has basic JMAP support.
This is relevant to your first paragraph: client-side filtering doesn’t make sense anymore (or at least like Thunderbird does it). Spam and filtering in general is part of the “mail stack” you use and clients assuming they can replace the concept of a stack is the wrong approach. The user should always be able to define their stack.
While the JMAP standards that are in a status of ‘PROPOSED STANDARD’, it is worth noting the vast majority of RFCs in use on the internet are in that status. The list of RFCs with a ‘Internet Standards’ is pretty darn short.
Notifications are a thorny one, but JMAP’s core RFC does have two different methods for notifications/pushing to clients. The more resource friendly method unfortunately also leads to vendor specific implementations.
As for Calendars & Contacts, I agree. I hope that they submit it for a proposed standards track soon.
They’re being worked on in public: https://datatracker.ietf.org/wg/jmap/documents/
I realize in hindsight that my comment could be interpreted as me implying it is being done in private, that was not my intention.
It’s more that I’m itching for it to be published as a Proposed Standard RFC so that I can safely assume it’s what people will be developing with. I have a few ideas I want to explore but I’d rather wait until there would be other implementations in development that I could test interoperability against.
No problem, and fair enough! We’re of course keen to get them finalised too, but the IETF process can be slow. Its worth it though; the JMAP core and mail specs spent three years (I think) at IETF and are very much better for it!
Fastmail has six internal commits that we apply to upstream master on each deploy. They are all either adjustments to handle differences of opinion or historical bugs in our middleware or stored data that require a lot of work to update (eg FM’s idea of “admin” is different to Cyrus’ idea), or they’re minor changes to match the way we like to operate the service (eg we convert all assertions and fatal errors into coredumps, triggering a bunch of analysis tooling).
I’m not saying Cyrus upstream is necessarily easy to operate, but in terms of JMAP support, its complete and correct (bugs notwithstanding) and what we use for both the Fastmail and Topicbox UIs and for the general protocol access we offer to users.
Thanks for linking these other clients, I’ve been meaning to experiment with some and MailPile looks pretty good.
I mostly took it from https://autocrypt.org/dev-status.html
@kev @freddym Thanks for the inspiration!
No problem!
I completely agree that spam filters and blocking techniques should be a more prominent feature in general. For the average user, there is far too much spam in their inbox. Heck, for most users even. I mentioned a few techniques to limit spam in my article:
As you say “this is more of an application-level problem” then a protocol one. We can probably fix most of our problems by changing some of out habits surrounding email. However, it begs the question as to why should we have to? For something that was created in the 1960’s, surely a better solution exists?
Some rays of light:
https://jmap.io/ - JSON Meta Application Protocol. Developed by the people at Fastmail (I’m biased: I’m a happy paying customer, they’re based here in Melbourne, and are great human beings).
https://www.djcbsoftware.nl/code/mu/mu4e.html - a great mail client for Emacs. All that would be required for mu4e to support JMAP mailboxen would be a command line tool to download JMAP in the same vein as mbsync for IMAP.
Watch out for future releases of meli, I plan on shipping a “sync” mode that syncs to local maildir without launching a UI, just like mbsync/offlineimap. Although you would be technically using a mail client (meli) to sync mail for your mail client (mu4e) it’d be the intended use of this functionality 🙃
That’s how mu4e is designed to work :) It’s a front-end for mu, the mail indexing utility, which in turn expects to operate on a synchronised maildir. Currently I’m using mbsync to synchronise my Fastmail and Gmail mail via IMAP; it’d be easy enough to switch to meli once it supported JMAP.
I’ve been using Evolution and so far love it. What could be better?
Fwiw, I wanted to add that I use Gmail and while my unread-count used to be in the 2k+ range regularly, I’ve since managed to achieve inbox-zero by diligently adding custom email rules (filters) over the course of a year, and now I get almost no unwanted emails in my inbox. While I didn’t use this setup myself, here’s a GH project to help you manage your Gmail filters - https://github.com/mbrt/gmailctl.
Filtering is definitely the way for email.
And hence why I’m still using GMail as my email client for many years. Not because I’m overly fond of anything else about their mail infrastructure, as Fastmail is being used to do the actual “host my domain’s email” problem, but because it’s a better email client. I’m yet to see anything (paid or free) that beats it, which is annoying as I’d like to move to something self-hosted, but not at the cost of usability.
I use gmail and zohomail and both are really great at filtering out garbage. I only get things I want to see. I think email is generally pretty good. There’s definitely a much better signal-to-noise ratio than regular mail.
They are really good at reading people’s mail, yes. ;)