For me, having multiple monitors has been a game-changer; it allows me to spread out my tasks visibly and switch between them as needed without losing track.
Interesting to see this perspective from someone else diagnosed with ADHD in adulthood, since I’m pretty much the opposite myself. I’ve found that having more than one monitor can be more of a nuisance than a help; if I make a chat app constantly visible for example, it distracts me more, and it becomes harder to focus on a single thing. I aggressively purge apps from my phone that send too many notifications for a similar reason. macOS features like Stage Manager help me keep frequently updating windows tucked away until I need them.
Heh, interesting. I was also diagnosed in adulthood (just last year), and I’m somewhere in between.
Back when I was moving between home desk, office desk, other locations (coffee shops, libraries) and my commute, I kept one monitor, with the display mirrored to my laptop, so when I changed screens all my windows stayed in the same spot. I’m in a different job now, fully remote (though I still move around a little for a change of scenery) and I still prefer this setup.
The thing I’ve learned is how important visual and spatial consistency is for being able to keep track of where I am and move between tasks. Its not so different to the “leave breadcrumbs in your environment” idea that’s common in the “ADHD tips & tricks” posts (ugh); it just seems that for me into extends to my screen layout as well. So I need things to stay where I put them, and nothing new to appear without some explicit action on my part. An extra monitor would just make me stress about things I might have already lost, and things I might lose in the future if I don’t place this window in the right place right now.
(Animations, surprise notifications, popups, etc are pretty much guaranteed to pull my attention away, so I’m pretty hostile towards anything that won’t at least let me turn them off, which means I’m pretty cranky at most software most of the time 😬)
I’ve never been diagnosed with ADHD myself but I never got accustomed to a multi-monitor setup (and I’m a huge outlier at every company I show up) and I fight all sorts of notifications very aggressively. I’m still evaluating Stage Manager, simply because I’m too accustomed to the “old” window management in macOS, and typically a Do Not Disturb with full screen app is good enough for me when I need to do deep work.
Also, a huge productivity booster for me is a simple Lo-Fi playlist in Apple Music paired with AirPods Max or similar over-ear noise cancelling headphones, no matter if I’m working remotely or in the office surrounded by other folks.
I’d still like to see some root cause analysis on the zvol performance thing here. It makes no sense to me that a file going via the ZPL is slower than the same storage via the ZVOL layer, which does less work. I strongly suspect that there’s a silly mistake somewhere in the ZVOL code that could make this faster.
I asked around, and it’s complicated. It’s around quota checks, and the fact that the at the place where quota enforcement occurs, we don’t know if the writing op is an overwrite or an append. So we stall until the transaftion is flushed, and then we know what the final size for the object should be, and can report back to the user.
Theres a parameter in mainline (and 2.2 when iti lands) that let’s you turn off quota checks for zvols that should resolve this.
Thanks. For VM uses, I don’t really care about dynamic quotas but I want ZVOLs to count towards the quota at their maximum size as soon as they’re created. In particular, I really don’t want a write to fail because it overflowed the quota (that would probably kill the VM, since most guests do not like storage failing) but I don’t want users to be able to create large numbers of ZVOLs to get around their quota restrictions. Is that what the PR does? It sounded more like it just disabled quota checks entirely (which works for me on a single user machine, but not in general).
For regular zvols, disabling the quota check is fine. When a zvol is created, a reservation is also created, which effectively charges the dataset the maximum size of the zvol at that moment, even if its not used.
The difference is for “sparse” volumes (created with zfs create -Vs) is that the reservation isn’t created. That’s where the quota check comes in - if it wasn’t there, there’d be nothing inside ZFS stopping a higher-level dataset going over its quota.
From the PR it looks like the parameter is deliberately a fairly coarse mechanism for controlling this behaviour, leaving room for something a bit more granular in the future.
This is a great question, that I spent far more of my evening on answering than I intended. Alas, the answer seems to be no, its not possible :(
I’ve filed an issue describing what I found from code read and testing and suggesting some new permissions that I think will make all this work: https://github.com/openzfs/zfs/issues/15193
Unfortunately I don’t really have the time to jfdi, but if my proposal is not wildly off-base, it actually shouldn’t be a very big change. Lets see where we get.
If you’re thinking about the ZFS security model then there’s one change I’d love to have:
When you load a channel program, allow it to appear as a file under /dev/zfs. Allow any unprivileged user to launch the channel program as long as they provide a file descriptor pointing to it with the ioctl that launches it.
This would make it possible to have much richer security policies. For example, I want my backup program to be able to create and delete snapshots and bookmarks that match a particular pattern, but not arbitrary snapshots. I could write a channel program that does this quite easily and use devfs rules to grant access to that program’s device node to the backup user, without granting it access to arbitrary creation and deletion. Similarly, I could use devfs rule sets with jails to allow jails to run specific channel programs, without giving them access to load arbitrary channel programs.
So it’s probably worth using zstd, even though like 80% of my data by mass is music/videos/games that are already compressed.
I thought the Zstd implementation in ZFS doesn’t bailout when it can’t compress unlike LZ4. If true, especially if a lot of your data is audio/video in already-compressed formats, this would have useless overhead. My pool is currently LZ4 default for after reading this at one point, but has that chnaged?
I thought the Zstd implementation in ZFS doesn’t bailout when it can’t compress unlike LZ4.
You’re right for 2.1. In master (so 2.2, whenever that happens) there’s early abort for zstd too. Turns out that LZ4 aborting early is a very good predictor of whether or not zstd will get a decent compression ratio, and LZ4 is fast enough that the overhead of compressing twice is barely noticeable (see analyis within).
This is great to see that it’s finally landed. I may need to re-assess some pools and swap over to Zstandard where it makes sense. They were offered by the same guy (whose hobby was programming), so it’s no surprise when we see a lot of correlations between the two.
It’s purely wasteful on say *.mkv and .ogg files because they’re already in a compressed format so the CPU either tries to compress what it can’t or wastes less cycles trying then aborting early. Compression off or zle could be better on these sorts of volumes, with the exception of subtitles/lyrics files would certainly would benefit–but this is why the early abort is important.
It definitely seemed useful when it was introduced, but I wonder how much I was biased by gzip’s speed. My slow Pentium Silver in my NAS can compress with zstd faster than my disks can write and can decompress at over ten times their read speed. As long as the CPU is faster at compressing than the disk is at writing, the early abort just buys you a bit lower power consumption and maybe some spare cycles for something else when in a very write heavy workload (though the CPU is often back-pressured then anyway).
If I understand correctly, you can also lose some compression from the early abort if the start of a block is high entropy but the end is low entropy, so it’s not free. The compression, if it reduces the size usefully, buys you better read speed, better L2ARC usage and better ARC usage now that the ARC can hold compressed data, so the trade off is really unclear to me. I suspect it makes a lot more of a difference on machines that are not predominantly storage servers.
That does all sound about right. My use case is an NVMe on root on a laptop where the storage device and CPU are already pretty quick, but also power efficiency/battery usage is also considered.
1 anecdotal report: At least in my experience running a bunch of VMs whose qcow2 disks were stored on a ZFS file system, ZFS on linux stopped crashing when we disabled compression.
So I’ve also been doing email for 23 years (or thereabouts; I started in 1999), except in my case its as a a sysadmin/devops/sre/architect (the name changes; the work mostly does not) for organisations on the order of hundreds of thousands to low millions of mailboxes. Since 2012 I’ve been at Fastmail. So I write this as a perspective from the mid-tier of mail providers that mostly do a good job of deliverability and mostly don’t have the power or influence to just arbitrarily bin email.
(All opinions etc my own and not my employer, etc, and also I’m writing this kinda off-the-cuff in my lunch break so please excuse any confusing or contradictory stuff; ask for clarification and I shall attempt to provide!).
I have some sympathy for the author of the article, I really do. I started my life on the internet running my own single-user mail server, and I understand the complexity of the global mail system in 2022, so I’m well aware of just how different it is and how tough it can be.
That said, this article and others like it from recent years almost always underestimate the magnitude of the problem posed by spam, fraud, phishing and other “unwanted stuff” (I’ll just say “spam” from now on). There’s different ways to count it, but there’s capital-B Billions lost to this shit every year. And with that comes small-country-sized amounts of resources to throw at making sure the spam gets through.
The giveaway is statements like “charge a fee; spammers won’t pay” because its straight up not what happens. People sending spam can and do spend good money to try and increase their standing; they’ll pay for the top-tier accounts with legit credit cards; they’ll buy real unsullied cellphone numbers in order to complete SMS-based verification.
So its hard. Does that mean the big players (which I don’t count Fastmail as) just drop small players on the floor? I think that’s too simplistic. Sure, they do have powerful and opaque reputation systems and yes, you can fall afoul of them by unfortunately being in the wrong place (like, an IP address that is near another that is known to be bad), but that’s because those things are very strong markers that whatever you’re sending is going to be spam. You’d be doing a disservice to your customers if you didn’t use it. Its much stronger than content, even, which is why things get rejected at the network edge. It’s nothing to do with content scanning being computationally expensive; that wouldn’t matter if it were more reliable.
(Fastmail does the same stuff around reputation in concept, though perhaps less sophisticated, partly because we’re smaller and that stuff is time-consuming to build and operate, but also because the volumes we deal with a smaller and so allow us to put a human in the loop).
Its just not true to say the big players don’t care about interoperability (though how much they care waxes and wanes with the seasons). The fundamental expectation of email is that anyone can play, and none of them have the market penetration to outright put up walls. Its not just interop between hosting providers, there’s also the various mail services (ESPs, Mailchimp and that sort of thing). Yes, many of them just send through Amazon SES, but not all. The point is, once you get all the significant players together you end up a few dozen companies; a mesh of deliverability agreements isn’t really possible. And that commitment to interop and keeping the system open to anyone that wants to play is where we get, for example, the suite of authentication tech (SPF, DKIM, DMARC, ARC, BIMI, etc), each layered on top of the previous one, none of them being especially great but each trying to file off one sharp edge.
And then, we all attend industry forums like M3AAWG to talk about what we’re currently seeing and share ideas about how to tackle it, and we spend time over at IETF and elsewhere trying to write down what we should do.
There are some “go it alone” elements that are a mixed bag. The most frustrating kinds of these are the shared blacklists out there that have very strange (to us) policies on what to add and when, or don’t provide any information at all about why you’ve been listed, have no (easy) way to get delisted and are used by significant enough sites that we still have to do something when we land on one. These sorts of cases are usual handled by asking a contact for a contact for a contact. Building networks of humans doing the gruntwork at each org is a big part of being able to operate any mail system past a certain size.
And yet, even going to all the events and making lots of friends, as an individual mid-size provider its still not uncommon to bump into things. We semi-regularly get partially blocked by Gmail or Outlook or iCloud or one of the major ISPs that you might not have realised ran a large mail system. We’re currently “warming” a new set of outbound IPs, that is, trying to raise their reputation enough so they can be used reliably for all sending; that’s a task that takes months, slowly trickling out definitely good email, watching responses, backing off when remotes start to push back, and over and over.
There’s a lot to do, which is why we have a whole team dedicated to this work!
So, am I making a point? I don’t know. I’m not defending the system as it currently is; we certainly should improve it somewhat. I think this is all trying to say that this is a global network that open to everyone, everyone involved broadly wants to keep it that way, and taken together that brings global-sized problems which have to be tackled as such. I don’t think its impossible for an individual to succeed in this environment, but I do think it its naive to expect that they should be able to. I don’t even think that’s bad necessarily; its just what happens once things grow past a certain size, and for better or worse, that’s past the size where an individual can easily be involved.
I’ve heard that one possible solution to this is requiring more proof-of-work (hashcash or similar schemes), in order to make bulk-sending annoying/economically infeasible.
If I’m reading you correctly, though, it sounds like the problem is that spammers (and here I include not just v1@gra / 419 scammers but also like Target, newsletter folks, etc.) would be happy just to throw more resources at the problem.
Most spam sending is coming from compromised accounts (webmail, newsletters, VPSes) so the cost is going to be borne by legitimate businesses and their customers. For direct sending (comparatively rare), hash computation is just going to be farmed out to some API (also likely with a compromised account), or in the extreme, they’re gonna build dedicated hashing hardware (I don’t think it goes that far, but if there’s a buck to be made, you can be sure someone will spend the money).
The “legitimate” senders (eg newsletter) meanwhile are not just gonna throw more money it. They’ll do what they have to do, but they don’t like spending more than they have to, so they’re gonna fight such a change pretty hard.
Meanwhile, there’s a lot of underpowered hardware out there, particularly in less-developed parts of the world. Requiring more of it is an additional burden - battery life, delays, etc which make email less usable. So such a change has the chance of adversely affecting already marginalised groups.
The real problem though is all the old software out there that doesn’t support the new scheme. That’s going to be a good chunk of legitimate mail that doesn’t have our new not-spam marker, which just puts us back to looking for other markers to sort the good from the bad.
There’s an old joke response to proposed solutions to spam in the form a checklist: https://craphound.com/spamsolutions.txt. I wouldn’t seriously use it as-is; its just a little too snarky for my tastes. I do refer to it from time to time though, because running through it is a quick way to assess whether a technique might be in or out, but also its a little bit of history of the things we’ve tried that haven’t worked.
From the outside, I think Fastmail is doing absolutely the right thing. The work that you folks have done on JMAP is technically good but, perhaps more importantly, the way that you’ve navigated getting it an IETF-approved standard, rather than just a ‘hey, we have this thing. It’s kind-of documented, good luck creating an interoperable version’ is nothing short of amazing to see. I’d love to see Thunderbird and Dovecot support JMAP.
As soon as Fastmail supports Confidential Computing (so I get strong technical guarantees backed by remote attestation that your administrators can’t see my mail spools), I expect to become one of your customers.
It’s a shame that you’ve dropped the family plans though.
Not an expert, but I do run a lot of ZFS RAIDz2 on NVMe on Linux and have done a fair bit of tuning for it. I don’t know which specific thing is given you such “impossible” numbers, but I’m happy to suggest a few things that might be in play, and maybe how to even squeeze more out of it! (btw, don’t mean this to come across as patronising, I’m just writing a few things out for readers that haven’t seen it, or for actual experts to tell me I’m doing it wrong!).
Most of the performance is going to be from the ARC, that is, the memory cache. ZFS will aggressively use RAM for caching (on Linux, by default, the ARC will grow to as much as half the physical RAM). You’ve already seen this in Note #3; reducing the RAM reduces throughput. Incidentally, you can tune how much RAM is used for the ARC with zfs_arc_min and zfs_arc_max (see zfs-module-parameters(5)); you don’t have to reduce the system “physical” RAM (though maybe that was more convenient for you to do).
Compression gets ZFS a huge amount of throughput, because its faster to do a smaller read and decompress it than wait for the IO (turning compression off can actually make things slower, not faster, because it has to hit the disk more). Compression is block level, and as a special case, all-zero blocks are not even written - the block header has a special case that says “this is all-zeroes, length XXX” that ZFS just inflates. Finally, turning off compression doesn’t change the compression state of already-written blocks, so if you’re benchmarking on data that already exists, you’ll need to rewrite it to really “uncompress” it.
In a RAIDzX, data is striped across multiple devices, and reads can be issued in parallel to get bits of the file back and then reassemble them in memory. You have 32 lanes, so you’re probably right in saying you’re not saturating the PCI bandwidth. You’re almost certainly getting stuff as fast as the drives can give it to you.
You’re using 512B blocks. Most current NVMe is running 4K blocks internally. The drive firmware will likely be loading the full 4K block and returning a 512B chunk of it to the system, and keeping the rest of the block cached in its own memory. In sequential reads that’s going to mean almost always 7 out of 8 blocks are going to served from the drive’s own cache memory, before touching the actual flash cells. (This is well worth tuning, by the way - use flashbench to determine the internal block size of your drive, and then find out how to do a low-level format for your device to switch it to its native block size. Along with an appropriate ashift for your pool, it will let ZFS and the Linux block layer deal in the drives native block size all the way through the stack, without ever having to split or join blocks).
ZFS will use a variable blocksize, by default growing blocks as large as 128K. When reading, it will request the entire logical block, composed of multiple physical blocks, from the block layer. If they’re stored sequentially, that can translate to single “range request” on the PCI bus, which may get coalesced into an even larger range, which the drive may be able to service entirely with parallel fetches against multiple flash cells internally.
Not sure which version of ZFS you’re using, but versions before 2.0.6 or 2.1.0 have a serious performance bottleneck on wide low-latency vdevs (GH#12121GH#12212).
In my experience though, and yours too, ZFS performance is pretty good out of the box. Enough that even though my workload does some things that are outright hostile to a CoW filesystem, the gains have been so good that it hasn’t yet been worth changing the software.
Great list, that’s almost surely what’s at play here. I don’t think the drive/file system speeds are actually being measured.
Some other performance tuning things to think about with zfs: if you have a fast slog vdev you can set sync=always but if your zil is slow you can set sync=disabled to gain a lot of speed at the expense of safety. For some use cases that’s okay.
When I ran mechanical disks I used an Optane drive for my slog vdev and it was so fast I couldn’t measure a performance difference when using sync=always.
I am trying out a few things with fio and will post the results here. There was a suggestion on HN that mirrors what you’re suggesting. I’ll update the article if I find that 157 GB/s is a bogus result.
Edit:
OK folks, party is over. 157 GB/s is a misleading number. The FIO library needs separate files for each thread otherwise, it will report incorrect bandwidth numbers. See this post, I am in process of updating the article: https://news.ycombinator.com/item?id=29547346
and then find out how to do a low-level format for your device to switch it to its native block size
What does this mean? I’ve set up aligned partitions and filesystem block sizes (or ashift for ZFS), but I don’t know what a low-level format even means.
All drives (flash and spinners) have a “native” block size. This is the block size that the drive electronics will read from or write to the actual storage media (magnetic platters or flash cells) in a single unit. Sizes vary, but in pretty much every current NVMe SSD the block size is 4KB.
Traditionally though, most drives arrive from the factory set to present a 512B block size to the OS. This is mostly for legacy reasons; back in the mists of time, physical disks blocks were actually 512B, and then the joy of PC backward compatibility means that almost everything ever since starts by pretending to be from 1981, even if that makes no sense anymore.
So, the OS asks the drive what its block size is, and it comes back with 512B. Any upper layers (usually a filesystem, maybe also intermediate layers like cryptoloops) that operate in larger block sizes will eventually submit work to the block layer, and it will then have to split the block into 512B chunks before submitting them to the device.
But, if the device isn’t actually 512B natively, then it has to do more work to get things back into its native block size. Say you write a single 512B block. A drive doing 4K internally will have to fetch the entire 4K block from storage into its memory, update it with the changed 512B, then write it back down. So its a bit slower, and for SSDs, doing more writes, so increasing wear.
So what you can do on many drives is a “low-level format”, which is also an old and now meaningless term for setting up the basic drive structure. Among other things, you can change the block size that is exposed to the OS. If you can make it match the native block size, then the drive never has to do deal in partial blocks. And if you can set the same block size through the entire stack, then you get eliminate partial block overheads from the entire stack.
I should note here that all this talk of extra reads and writes and wear and whatnot makes it sound like every SSD must be a total piece of crap out of the box, running at glacial pace and wearing itself out while its still young. Not so! Drive electronics and firmware are extremely good at minimising the effects of all these, so for most workloads (especially large sequential reads) the difference is barely even measurable.
But if you’re building storage systems that are busy all the time, then there is performance being left on the table, so it can be worth looking at this. My particular workload includes constant I/O of mostly small random reads and writes, so anything extra I can get can help.
I mentioned flashbench before, which is a tool to measure the native block size of a flash drive, since the manufacturer won’t always tell you or might lie about it. It works by reading or writing blocks of different sizes, within and across theoretical block boundaries, and looks at the latency for each operation. For example, you might try to read 4K blocks at 0, 2K, 4K, 6K, etc offsets. If its 4K internally, then the drive only has to load a single block at 0, but will have to load two blocks at 2K to cross the block boundary, and this will be visible because it takes just a little longer to do its work. It’s tough to outsmart the drive electronics (for example, current Intel 3DNAND SSDs will do two 4K fetches in parallel, so a naive read of the latency figures can make it look like it actually has an 8K block size internally), but with some thought and care, you can figure it out. Most of the time it is 4K, so you can use that as a starting point.
On Linux, the nvme list tool can tell you the current block size reported by the drive. Here’s some output for a machine I’m currently right in the middle of reformatting as described above (it was inadvertently introduced to production without having been reformatted, so I’m having to to reformat individual drives then resilver, repeatedly, until its all reformatted. Just another sysadmin adventure!)
[fastmail root(robn)@imap52 ~]# nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- --------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 PHLJ133000RS8P0HGN INTEL SSDPE2KX080T8 1 8.00 TB / 8.00 TB 4 KiB + 0 B VDV10170
/dev/nvme10n1 PHLJ132601GU8P0HGN INTEL SSDPE2KX080T8 1 8.00 TB / 8.00 TB 512 B + 0 B VDV10170
/dev/nvme11n1 PHLJ133000RH8P0HGN INTEL SSDPE2KX080T8 1 8.00 TB / 8.00 TB 4 KiB + 0 B VDV10170
/dev/nvme12n1 PHLJ131000MS8P0HGN INTEL SSDPE2KX080T8 1 8.00 TB / 8.00 TB 4 KiB + 0 B VDV10170
So you can see that nvme10n1 is still on 512B.
And then once you’ve done that, you have to issue a low-level format. I think it might be possible with nvme format, but I use the Intel-specific isdct and intelmas tools. Dunno about other brands, but I expect the info is easily findable especially for high-quality devices.
Do remember though: low-level format destroys all data on the drive. Don’t attempt in-place! And I honestly wouldn’t bother if you’re not sure if you need it, though I guess plenty of people try it “just for fun”. You do you!
Maybe something like Sieve could help with that, if it ever become widespread. It is not too late for email to evolve: autocrypt, chat-over-imap (email) IMAP IDLE and NOTIFY.
We now have all the tools to build a solid experience providing all the features we are looking for from all chat systems. All it takes is findingsomeclientthatsuitsyourneeds.
The first is that email has changed a lot in how it’s used. Most users 20 years ago had one client per mail account. POP3 was common and its usage model was very simple: clients downloaded all messages, the server storage is just a cache of the last n days of emails. IMAP allowed storing the entire mail history on the server (POP3 did in theory, but a bunch of operations were O(n) in terms of the size of the mail box) and started shifting things to the server. Adding in things like Sieve and you’re in a situation where there’s a lot of duplication between the client and server. Most mail clients can do spam filtering but no one really wants this anymore: if Thunderbird does spam filtering, then no spam received since the last time I had my laptop on and now will be filtered when I check the mail on my phone / tablet / any other device. Now, most people check email on multiple devices, so you need to do almost everything on the server for it to be useful. Features that only work on one client are annoying.
The second problem is that the communities that write mail servers and mail clients are quite disconnected. Look at JMAP. It is a big improvement over IMAP, but it’s only really supported by a single client for a proprietary mail service, which uses its own server. They can do this because they have the vertical integration (the same is true for Gmail and Outlook / Exchange): they add features to the client and server at the same time. JMAP is low priority in Dovecot because Thunderbird doesn’t use it, it’s low priority in Thunderbird because Dovecot doesn’t use it.
This is made worse by the fact that supporting good server-side features (e.g. Sieve) reduces the perceived value of the client. I use Sieve to filter all of my mail and use a Thunderbird plugin that gives me syntax highlighting of the script and updates it over the managesieve protocol. As a result, Thunderbird’s complex mail rule features are of zero value to me. Similarly, my mail server does server-side spam tagging and I filter the results with Sieve (with IMAPSIEVE rules to mark messages as spam or not-spam when they are moved to and from the junk mailbox).
These then Supporting a new feature on the server removed any value for me in the client having spam detection and message filtering logic.
I agree with the generally point, and here are some nuances or extrapolations.
JMAP
JMAP is actually more REST-like than most “RESTful” APIs. It is stateless, highly cacheable, supports transparent intermediaries and provides a uniform interface for manipulating different resources. However, it doesn’t use HTTP verbs to implement this. – https://jmap.io/
With the advent of HTTP 3 and presence of HTTP 2, why bother optimizing a few extra round trips – or – why bother with HTTP at all?
We end-up with HTTP 200 with this in it: { "status": 400, ...}.
On the other hand, immutable UIDs, constant syntax without so many exceptions are a good thing.
server-side -vs- client-side
It is a good thing that things are moving toward server-side for some aspects: servers are easier to maintain than each and every user’s machines (that may still literally be between their hands when we need to do the maintenance).
Yes, sure, former/existing mail client look silly now with all its features.
walled garden
Either we embrace (extend, extinguish) the situation and stick our head into the lion’s head with walled garden like whatsapp, signal, telegram and so on.
Either we accept the issue and try to bridge everything together (matrix- or matterbridge-style) and hope it will get better (which will not happen as long as bridges are maintained)
Either we jump back to the lowest common denominator, and keep building from there to improve the situation for everyone at once.of
It looks like a short-, middle-, long-term solution choice, so maybe all three up front? A lot ahead, yet a few years ago, none of these issue did really exist!
With the advent of HTTP 3 and presence of HTTP 2, why bother optimizing a few extra round trips – or – why bother with HTTP at all?
We end-up with HTTP 200 with this in it: { “status”: 400, …}.
For JMAP specifically, its more of a RPC protocol that runs over HTTP (or anything really; a profile for WebSockets and works great).
Why HTTP+JSON? Mostly because its convenient: everyone understands it, and every platform has high-performance libraries for it. The learning curve on IMAP is a big part of why JMAP exists.
As for optimising a few round-trips, not everywhere is nearby. Much of Fastmail’s frontend engineering was done in Melbourne, Australia, 300ms from our servers on the US east coast. A lot of our UI performance comes from packing a lot into each request.
We end-up with HTTP 200 with this in it: { “status”: 400, …}.
The RFC that details with the detailed error responses, RFC7807, specifically calls out that the status code listed in the json object must match the status code returned by the web server. The status member of the response object is considered a “convenience” for the consumer of the response, whatever the heck that means.
One area where this could be false, which is addressed in the RFC itself, is when proxy servers, load balancers, etc. might alter the status codes of a response.
Then I am tempted to think this is a misbehavior of the reverse proxy that changes the semantics. I feel like sliding onto a different topic, but is that not the point of discussions… :)
The second problem is that the communities that write mail servers and mail clients are quite disconnected. Look at JMAP. It is a big improvement over IMAP, but it’s only really supported by a single client for a proprietary mail service, which uses its own server. They can do this because they have the vertical integration (the same is true for Gmail and Outlook / Exchange): they add features to the client and server at the same time. JMAP is low priority in Dovecot because Thunderbird doesn’t use it, it’s low priority in Thunderbird because Dovecot doesn’t use it.
I think fastmail has an inner fork of Cyrus, and they have merged big parts of it with upstream Cyrus (the latest version has JMAP support). The problem with JMAP is that parts of it are still nonstandard yet (unpublished or only drafts), mainly notifications and calendar/contacts. meli has basic JMAP support.
This is made worse by the fact that supporting good server-side features (e.g. Sieve) reduces the perceived value of the client.
This is relevant to your first paragraph: client-side filtering doesn’t make sense anymore (or at least like Thunderbird does it). Spam and filtering in general is part of the “mail stack” you use and clients assuming they can replace the concept of a stack is the wrong approach. The user should always be able to define their stack.
The problem with JMAP is that parts of it are still nonstandard yet (unpublished or only drafts), mainly notifications and calendar/contacts.
While the JMAP standards that are in a status of ‘PROPOSED STANDARD’, it is worth noting the vast majority of RFCs in use on the internet are in that status. The list of RFCs with a ‘Internet Standards’ is pretty darn short.
Notifications are a thorny one, but JMAP’s core RFC does have two different methods for notifications/pushing to clients. The more resource friendly method unfortunately also leads to vendor specific implementations.
As for Calendars & Contacts, I agree. I hope that they submit it for a proposed standards track soon.
I realize in hindsight that my comment could be interpreted as me implying it is being done in private, that was not my intention.
It’s more that I’m itching for it to be published as a Proposed Standard RFC so that I can safely assume it’s what people will be developing with. I have a few ideas I want to explore but I’d rather wait until there would be other implementations in development that I could test interoperability against.
No problem, and fair enough! We’re of course keen to get them finalised too, but the IETF process can be slow. Its worth it though; the JMAP core and mail specs spent three years (I think) at IETF and are very much better for it!
I think fastmail has an inner fork of Cyrus, and they have merged big parts of it with upstream Cyrus (the latest version has JMAP support).
Fastmail has six internal commits that we apply to upstream master on each deploy. They are all either adjustments to handle differences of opinion or historical bugs in our middleware or stored data that require a lot of work to update (eg FM’s idea of “admin” is different to Cyrus’ idea), or they’re minor changes to match the way we like to operate the service (eg we convert all assertions and fatal errors into coredumps, triggering a bunch of analysis tooling).
I’m not saying Cyrus upstream is necessarily easy to operate, but in terms of JMAP support, its complete and correct (bugs notwithstanding) and what we use for both the Fastmail and Topicbox UIs and for the general protocol access we offer to users.
Maybe so, but if after several decades still no unbroken client emerges then there’s still something fishy going on. It might not be the protocol (though I hear that IMAP isn’t exactly the nicest thing to work with) but maybe we just haven’t figured out what a good email workflow would look like.
Of course that’s coupled with the problem that everyone uses email differently.
Good: Central principle of one command to search taking parameters giving back uids, one command to fetch taking UIDs giving back parts of listed mails you asked: this is simple and nicely designed, You can now search and do something with the result like fetching it.
Bad: IMHO, he most useful feature of IMAP after searching mails is to have notifications of email: that is why there is the effort of keeping a TCP connexion open. But this is brought through extensions.
Ugly: many small quirks, like syntax with many edge cases, a select command that gets in the way (and prevents searching a string in everything for instance), mail IDs are at most int 64: too short to keep them immutable.
We’re looking for an Email Client Application developer to join the FastMail team in our Center City Philadelphia office.
Together with the rest of our team, you’ll work on our email services FastMail, Pobox, and Topicbox. When the vast majority of people get email for free, the reason people choose a paid service is because they value both privacy and a great user experience.
You’ll be working on our best-in-breed mail clients, providing super-fast, powerful access to webmail, calendars, contacts and more. You’ll work with our collaborative team of front- and back-end developers, designers and researchers. Improving our customers’ interactive experience, keeping it secure and a delight to use is what we do, every day.
Our interfaces are built upon our own Overture framework and the new email standard JMAP. It gives us full control from top to bottom to build the right solutions to difficult problems. If you’ve worked with other MVC application frameworks, like Apple’s Cocoa, you should find Overture easy to pick up.
You’ll have the opportunity to work at many levels:
building features from scratch with our design and product teams
optimizing and improving existing code
tracking down and fixing elusive bugs reported by customers
Contact: Email us at jobs@fastmail.com to introduce yourself and tell us why you would be a good fit for the job! This job requires clear communication, so that message is part of your application, but please also include a PDF of your resume.
I’ve been interested in looking at gmail alternatives and I’ve heard good things about Fastmail. Good to know there’s an engineer on here, makes me trust it a bit more.
Fastmail. They are trustworthy, quick to respond to service requests, and rock solid. I can count the number of outages in the past ~10 years on one hand.
+1 for Fastmail. I’ve been using them for several years now and they’re very reliable, have a really solid web UI, and from what I can tell a solid position on security. They also contribute to moving the state of internet mail forward by improving Cyrus and contributing to RFCs. All in all I’d highly recommend them.
They also contribute to moving the state of internet mail forward by improving Cyrus and contributing to RFCs.
That’s another good point: they are by all accounts a solid technical citizen, contributing back and moving the state of the art forward. I like to reward good behaviour when I spend my money, and it’s nice to be able to do that and get top of the line service, to boot.
The funny thing is that for the amount of press that Gmail received/receives for being “fast”, once you switch to Fastmail, you realize that Gmail is actually very slow. The amount of bloat and feature-creep they’ve introduced is fascinating.
Love Fastmail. I only wish more tools had first class CalDAV/CardDAV support. When I switched over, I was genuinely surprised how pervasive it’s become to slap on Google account sync and call it a day, even in FOSS. Aside from the built-in macOS/iOS apps, most solutions involve fussing with URLs and 3rd party plugins, if it’s supported at all.
Fastmail has a link generator for CalDAV so it’s super easy to get the right URLs. I do agree for 3rd party plugins, it’s annoying to have to install add-ons for standard and open source protocols…
It was the best one I found, too, overall. I dont know about trustworthy, though, given they’re in a Five Eyes country expanding police and spy authority every year.
Maybe trustworthy from threats other than them, though. I liked them for that.
Yeah, I’m not concerned about state level actors, or more properly, I don’t lose sleep over them because for me and my threat model, there’s simply nothing to be done.
I’m not worried about the state spying on me, I’m worried about the apparatus the state builds to spy on me being misused by service provider employees and random hackers.
I use Fastmail as well, and became a customer by way of pobox.com acquisition.
I’ll have to add, this was about the only time I can ever recall that a service I use was acquired by another company and I was actually fine with it, if not a bit pleased.
My thinking was along the lines of “well, the upstream has purchased one of the biggest users of their tools, can’t be bad.”
I’ve not had any noticeable difference in the level of service provided, technically or socially, except the time difference to Australia is something to keep in mind.
I do hope that no one here in the US lost their jobs because of the acquisition, however.
I do hope that no one here in the US lost their jobs because of the acquisition, however.
Nope! We’ve hired a bunch more people in both offices, and the previous Pobox management are now C-level execs. We’re pretty sure the acquisition has been a win for just about everyone involved :)
I can also recommend it, especially due to their adherence to web standards. After 10+ years of GMail, the only functioning client had been Thunderbird, which too often too large. Since switching to Fastmail, I’ve been having a far better experience with 3rd party clients, and a better mail experience in general (probably also because I left a lot of spam behind me).
I second that. I was searching for a serious e-mail provider for a catch-all email, calendar and contacts.
I had trouble setting up my carddav autodiscovery DNS configuration and they helped me without considering me as a “dumb” client. Serious, clear and direct. The most efficient support I could’ve encountered by far.
It’s paid, and I’m paying the second plan (of 5$/month), and I think it’s perfectly fair, considering that, firstly, e-mail infrastructure is costly, and secondly, that their service is just plain awesome.
They’ve recently added the ability to automatically set up iOS devices with all of their services when you create a new OTP. I didn’t know that I needed this, but it’s a wonderful little bonus. It’s stuff like that that keeps me happily sending them money, and will as long as they keep doing such a good job.
Their help page on the matter isn’t clear, although it does describe a lot of things that seem pretty good. Now you’ve got me wondering. (Happy Fastmail user here, and I even convinced my wife to move to it from GMail!)
edit: It does sound like it’s plain text but you could read it a couple of ways.
All your data is stored on encrypted disk volumes, including backups. We believe this level of protection strikes the correct balance between confidentiality and availability.
Encrypted at rest (encrypted block devices), but cleartext available to the application because we need it for a bunch of things, mostly search, also previews and other bits and pieces. Of course, the applications that hit the on-disk files have their own protections.
using per-mailbox encryption based on keys derived from individual user passwords.
If this is a feature you’re looking for in a hosted solution, Protonmail is probably your best option.
However, even if such claims are made you can’t verify that.
Up to a point you can, Protonmail has released their webmail client as open source. Of course, with today’s JavaScript ecosystem it’ll be very hard to verify that the JavaScript code you are running actually corresponds to that code. Also, you can’t verify they’re not secretly storing a plaintext copy of inbound mails before encryption. But down that path lies madness, or self-hosting.
One of other things I want to highlight is reliability/availability. Making sure I dont miss important emails is even more important than privacy to me. Newer, smaller, and privacy-focused sites might not have as much experience in keeping site up or getting all your mail in reliably.
Fastmail has been around for quite a while with positive feedback from everyone Ive seen. So, they might perform better than others in not missing/losing email and being available. Just speculating here based on what their customers say.
I use FastMail for calendars and contacts. I actually don’t use it for e-mail much since my ISP is pretty ok.
For Android I switched from CalDAV-Sync + CardDAV-Sync to DAVdroid. Both work but the latter is easier to configure (by way of having less config options).
I tried self-hosting Radical for a while but for the time I had to put into it I’d rather pay FastMail $30 per year.
I am also a happy fastmail.com customer since about 2 years now. I used mailbox.org before, a german email provider, which is quite cheap (1€ per month) and allowed to use custom email domains but their spam filter sucked. Fastmail’s spam filter is also not perfect, in fact Gmail has still by far the best filtering, but their service is great and I can use custom email domain’s too. They also develop JMAP a JSON based IMAP replacement.
Is JMAP even supported anywhere? Does anybody use it? Last I checked, not even Fastmail actually used this for anything. Seems like the project started with some energy but is mostly dead now? What a shame, as I’d love to use it somewhere… Please do correct me if I’m wrong.
At FM, we use it internally for some (but not yet all) of our UI-server interactions, and we’re working on converting the UI to use JMAP natively (once the standardisation work has stablised).
Finally, we’re just about to launch a new product that uses JMAP from top to bottom - Cyrus, Ix (a JMAP API generator) and Overture (a UI framework with a JMAP-backed storage layer).
So there’s lots happening on JMAP at FastMail and elsewhere.
That’s really wonderful to hear. Once a year I email FastMail tech support asking them if there’s a JMAP thing, but the answer is always something like “no, and we don’t know when if ever.” And then I’m sad. This here is the first positive confirmation I’ve received, and I’m quite happy to hear it!
Hopefully once you release a fully JMAP designed system, you’ll have auto-exporters from existing tag-based systems like Gmail? Something like this would probably net you a massive user base.
I switched to fastmail last month and I am very happy with it. Before that, I had been self-hosting for 10 years, but I started seeing my emails listed as spam after I switched VPS providers (despite correct SPF etc), and I wasn’t motivated enough to fight for my IP reputation again.
Also Fastmail, moved from Google Apps for domains 2 or 3 years ago. Besides the advantages others mentioned, subdomain addressing is also a cool feature. Some mail providers support plus addressing
makes it easier to write rules and to drop mail when the address is sold to some spammer.
Also their support is pretty good. I had a small feature/refinement request twice, in both cases they had the feature implemented in their beta site in a couple of days.
I went to fastmail two years ago when the server on which I’d hosted my own email for about eight years died. I was happy to give a great company about $60 a year to host my family’s email. I was probably spending $60 a month of my own time just to administer the damn thing.
I’m on Fastmail too, with my own domain, for about ten years. The web UI is focused and fast, and the iOS app is just a webview, but a decent one that’s quick. I use Fastmail aliases and inbox rules to send to multiple external addresses, like a basic private listserve. Tons of advanced features for mail users, DFA, and no advertising or shenanigans with your inbox.
They went through a purchase by Opera a while ago, then a few years later Opera sold the business back to the original Fastmail employees – not a single hiccup or business misstep the whole time. They are laser focused. They contribute back to the open source mail server community.
The only issue on my wishlist is that they still don’t support the full CardDAV protocol, which means I cannot fully sync my Fastmail addressbook with iOS, Mac, Windows, or *nix apps, but they’re working on it, and it’s due soon (early 2018?).
I think it’s cheap for what you get, if you’re into that sort of thing.
Another vote for fastmail. Been a user for several years now. Has by far the best webui out of any provider. Very stable, and quick restoration of backups if you ever need them.
Another +1 for Fastmail. I’ve used them for 3 years and have been pleased with all their services. Their documentation is clear, the system is not hard to use, and they answer questions promptly.
The only thing I’m waiting for is HTTPS support on their web hosting. But if you need serious web hosting, Fastmail probably shouldn’t be yout first choice.
Interesting to see this perspective from someone else diagnosed with ADHD in adulthood, since I’m pretty much the opposite myself. I’ve found that having more than one monitor can be more of a nuisance than a help; if I make a chat app constantly visible for example, it distracts me more, and it becomes harder to focus on a single thing. I aggressively purge apps from my phone that send too many notifications for a similar reason. macOS features like Stage Manager help me keep frequently updating windows tucked away until I need them.
Heh, interesting. I was also diagnosed in adulthood (just last year), and I’m somewhere in between.
Back when I was moving between home desk, office desk, other locations (coffee shops, libraries) and my commute, I kept one monitor, with the display mirrored to my laptop, so when I changed screens all my windows stayed in the same spot. I’m in a different job now, fully remote (though I still move around a little for a change of scenery) and I still prefer this setup.
The thing I’ve learned is how important visual and spatial consistency is for being able to keep track of where I am and move between tasks. Its not so different to the “leave breadcrumbs in your environment” idea that’s common in the “ADHD tips & tricks” posts (ugh); it just seems that for me into extends to my screen layout as well. So I need things to stay where I put them, and nothing new to appear without some explicit action on my part. An extra monitor would just make me stress about things I might have already lost, and things I might lose in the future if I don’t place this window in the right place right now.
(Animations, surprise notifications, popups, etc are pretty much guaranteed to pull my attention away, so I’m pretty hostile towards anything that won’t at least let me turn them off, which means I’m pretty cranky at most software most of the time 😬)
This.
I’ve never been diagnosed with ADHD myself but I never got accustomed to a multi-monitor setup (and I’m a huge outlier at every company I show up) and I fight all sorts of notifications very aggressively. I’m still evaluating Stage Manager, simply because I’m too accustomed to the “old” window management in macOS, and typically a Do Not Disturb with full screen app is good enough for me when I need to do deep work.
Also, a huge productivity booster for me is a simple Lo-Fi playlist in Apple Music paired with AirPods Max or similar over-ear noise cancelling headphones, no matter if I’m working remotely or in the office surrounded by other folks.
I’d still like to see some root cause analysis on the zvol performance thing here. It makes no sense to me that a file going via the ZPL is slower than the same storage via the ZVOL layer, which does less work. I strongly suspect that there’s a silly mistake somewhere in the ZVOL code that could make this faster.
I asked around, and it’s complicated. It’s around quota checks, and the fact that the at the place where quota enforcement occurs, we don’t know if the writing op is an overwrite or an append. So we stall until the transaftion is flushed, and then we know what the final size for the object should be, and can report back to the user.
Theres a parameter in mainline (and 2.2 when iti lands) that let’s you turn off quota checks for zvols that should resolve this.
See https://github.com/openzfs/zfs/pull/13838 for more details.
Thanks. For VM uses, I don’t really care about dynamic quotas but I want ZVOLs to count towards the quota at their maximum size as soon as they’re created. In particular, I really don’t want a write to fail because it overflowed the quota (that would probably kill the VM, since most guests do not like storage failing) but I don’t want users to be able to create large numbers of ZVOLs to get around their quota restrictions. Is that what the PR does? It sounded more like it just disabled quota checks entirely (which works for me on a single user machine, but not in general).
For regular zvols, disabling the quota check is fine. When a zvol is created, a reservation is also created, which effectively charges the dataset the maximum size of the zvol at that moment, even if its not used.
The difference is for “sparse” volumes (created with
zfs create -Vs) is that the reservation isn’t created. That’s where the quota check comes in - if it wasn’t there, there’d be nothing inside ZFS stopping a higher-level dataset going over its quota.From the PR it looks like the parameter is deliberately a fairly coarse mechanism for controlling this behaviour, leaving room for something a bit more granular in the future.
Is it possible with delegated administrative permissions to permit creating ZVOLs but not sparse ones?
This is a great question, that I spent far more of my evening on answering than I intended. Alas, the answer seems to be no, its not possible :(
I’ve filed an issue describing what I found from code read and testing and suggesting some new permissions that I think will make all this work: https://github.com/openzfs/zfs/issues/15193
Unfortunately I don’t really have the time to jfdi, but if my proposal is not wildly off-base, it actually shouldn’t be a very big change. Lets see where we get.
Thanks!
If you’re thinking about the ZFS security model then there’s one change I’d love to have:
When you load a channel program, allow it to appear as a file under /dev/zfs. Allow any unprivileged user to launch the channel program as long as they provide a file descriptor pointing to it with the ioctl that launches it.
This would make it possible to have much richer security policies. For example, I want my backup program to be able to create and delete snapshots and bookmarks that match a particular pattern, but not arbitrary snapshots. I could write a channel program that does this quite easily and use devfs rules to grant access to that program’s device node to the backup user, without granting it access to arbitrary creation and deletion. Similarly, I could use devfs rule sets with jails to allow jails to run specific channel programs, without giving them access to load arbitrary channel programs.
I thought the Zstd implementation in ZFS doesn’t bailout when it can’t compress unlike LZ4. If true, especially if a lot of your data is audio/video in already-compressed formats, this would have useless overhead. My pool is currently LZ4 default for after reading this at one point, but has that chnaged?
You’re right for 2.1. In master (so 2.2, whenever that happens) there’s early abort for zstd too. Turns out that LZ4 aborting early is a very good predictor of whether or not zstd will get a decent compression ratio, and LZ4 is fast enough that the overhead of compressing twice is barely noticeable (see analyis within).
This is great to see that it’s finally landed. I may need to re-assess some pools and swap over to Zstandard where it makes sense. They were offered by the same guy (whose hobby was programming), so it’s no surprise when we see a lot of correlations between the two.
Not that I know of, but the difference still doesn’t matter for me. XD
It’s purely wasteful on say
*.mkvand.oggfiles because they’re already in a compressed format so the CPU either tries to compress what it can’t or wastes less cycles trying then aborting early. Compression off orzlecould be better on these sorts of volumes, with the exception of subtitles/lyrics files would certainly would benefit–but this is why the early abort is important.It definitely seemed useful when it was introduced, but I wonder how much I was biased by gzip’s speed. My slow Pentium Silver in my NAS can compress with zstd faster than my disks can write and can decompress at over ten times their read speed. As long as the CPU is faster at compressing than the disk is at writing, the early abort just buys you a bit lower power consumption and maybe some spare cycles for something else when in a very write heavy workload (though the CPU is often back-pressured then anyway).
If I understand correctly, you can also lose some compression from the early abort if the start of a block is high entropy but the end is low entropy, so it’s not free. The compression, if it reduces the size usefully, buys you better read speed, better L2ARC usage and better ARC usage now that the ARC can hold compressed data, so the trade off is really unclear to me. I suspect it makes a lot more of a difference on machines that are not predominantly storage servers.
That does all sound about right. My use case is an NVMe on root on a laptop where the storage device and CPU are already pretty quick, but also power efficiency/battery usage is also considered.
1 anecdotal report: At least in my experience running a bunch of VMs whose qcow2 disks were stored on a ZFS file system, ZFS on linux stopped crashing when we disabled compression.
So I’ve also been doing email for 23 years (or thereabouts; I started in 1999), except in my case its as a a sysadmin/devops/sre/architect (the name changes; the work mostly does not) for organisations on the order of hundreds of thousands to low millions of mailboxes. Since 2012 I’ve been at Fastmail. So I write this as a perspective from the mid-tier of mail providers that mostly do a good job of deliverability and mostly don’t have the power or influence to just arbitrarily bin email.
(All opinions etc my own and not my employer, etc, and also I’m writing this kinda off-the-cuff in my lunch break so please excuse any confusing or contradictory stuff; ask for clarification and I shall attempt to provide!).
I have some sympathy for the author of the article, I really do. I started my life on the internet running my own single-user mail server, and I understand the complexity of the global mail system in 2022, so I’m well aware of just how different it is and how tough it can be.
That said, this article and others like it from recent years almost always underestimate the magnitude of the problem posed by spam, fraud, phishing and other “unwanted stuff” (I’ll just say “spam” from now on). There’s different ways to count it, but there’s capital-B Billions lost to this shit every year. And with that comes small-country-sized amounts of resources to throw at making sure the spam gets through.
The giveaway is statements like “charge a fee; spammers won’t pay” because its straight up not what happens. People sending spam can and do spend good money to try and increase their standing; they’ll pay for the top-tier accounts with legit credit cards; they’ll buy real unsullied cellphone numbers in order to complete SMS-based verification.
So its hard. Does that mean the big players (which I don’t count Fastmail as) just drop small players on the floor? I think that’s too simplistic. Sure, they do have powerful and opaque reputation systems and yes, you can fall afoul of them by unfortunately being in the wrong place (like, an IP address that is near another that is known to be bad), but that’s because those things are very strong markers that whatever you’re sending is going to be spam. You’d be doing a disservice to your customers if you didn’t use it. Its much stronger than content, even, which is why things get rejected at the network edge. It’s nothing to do with content scanning being computationally expensive; that wouldn’t matter if it were more reliable.
(Fastmail does the same stuff around reputation in concept, though perhaps less sophisticated, partly because we’re smaller and that stuff is time-consuming to build and operate, but also because the volumes we deal with a smaller and so allow us to put a human in the loop).
Its just not true to say the big players don’t care about interoperability (though how much they care waxes and wanes with the seasons). The fundamental expectation of email is that anyone can play, and none of them have the market penetration to outright put up walls. Its not just interop between hosting providers, there’s also the various mail services (ESPs, Mailchimp and that sort of thing). Yes, many of them just send through Amazon SES, but not all. The point is, once you get all the significant players together you end up a few dozen companies; a mesh of deliverability agreements isn’t really possible. And that commitment to interop and keeping the system open to anyone that wants to play is where we get, for example, the suite of authentication tech (SPF, DKIM, DMARC, ARC, BIMI, etc), each layered on top of the previous one, none of them being especially great but each trying to file off one sharp edge.
And then, we all attend industry forums like M3AAWG to talk about what we’re currently seeing and share ideas about how to tackle it, and we spend time over at IETF and elsewhere trying to write down what we should do.
There are some “go it alone” elements that are a mixed bag. The most frustrating kinds of these are the shared blacklists out there that have very strange (to us) policies on what to add and when, or don’t provide any information at all about why you’ve been listed, have no (easy) way to get delisted and are used by significant enough sites that we still have to do something when we land on one. These sorts of cases are usual handled by asking a contact for a contact for a contact. Building networks of humans doing the gruntwork at each org is a big part of being able to operate any mail system past a certain size.
And yet, even going to all the events and making lots of friends, as an individual mid-size provider its still not uncommon to bump into things. We semi-regularly get partially blocked by Gmail or Outlook or iCloud or one of the major ISPs that you might not have realised ran a large mail system. We’re currently “warming” a new set of outbound IPs, that is, trying to raise their reputation enough so they can be used reliably for all sending; that’s a task that takes months, slowly trickling out definitely good email, watching responses, backing off when remotes start to push back, and over and over.
There’s a lot to do, which is why we have a whole team dedicated to this work!
So, am I making a point? I don’t know. I’m not defending the system as it currently is; we certainly should improve it somewhat. I think this is all trying to say that this is a global network that open to everyone, everyone involved broadly wants to keep it that way, and taken together that brings global-sized problems which have to be tackled as such. I don’t think its impossible for an individual to succeed in this environment, but I do think it its naive to expect that they should be able to. I don’t even think that’s bad necessarily; its just what happens once things grow past a certain size, and for better or worse, that’s past the size where an individual can easily be involved.
I’ve heard that one possible solution to this is requiring more proof-of-work (hashcash or similar schemes), in order to make bulk-sending annoying/economically infeasible.
If I’m reading you correctly, though, it sounds like the problem is that spammers (and here I include not just v1@gra / 419 scammers but also like Target, newsletter folks, etc.) would be happy just to throw more resources at the problem.
Most spam sending is coming from compromised accounts (webmail, newsletters, VPSes) so the cost is going to be borne by legitimate businesses and their customers. For direct sending (comparatively rare), hash computation is just going to be farmed out to some API (also likely with a compromised account), or in the extreme, they’re gonna build dedicated hashing hardware (I don’t think it goes that far, but if there’s a buck to be made, you can be sure someone will spend the money).
The “legitimate” senders (eg newsletter) meanwhile are not just gonna throw more money it. They’ll do what they have to do, but they don’t like spending more than they have to, so they’re gonna fight such a change pretty hard.
Meanwhile, there’s a lot of underpowered hardware out there, particularly in less-developed parts of the world. Requiring more of it is an additional burden - battery life, delays, etc which make email less usable. So such a change has the chance of adversely affecting already marginalised groups.
The real problem though is all the old software out there that doesn’t support the new scheme. That’s going to be a good chunk of legitimate mail that doesn’t have our new not-spam marker, which just puts us back to looking for other markers to sort the good from the bad.
There’s an old joke response to proposed solutions to spam in the form a checklist: https://craphound.com/spamsolutions.txt. I wouldn’t seriously use it as-is; its just a little too snarky for my tastes. I do refer to it from time to time though, because running through it is a quick way to assess whether a technique might be in or out, but also its a little bit of history of the things we’ve tried that haven’t worked.
From the outside, I think Fastmail is doing absolutely the right thing. The work that you folks have done on JMAP is technically good but, perhaps more importantly, the way that you’ve navigated getting it an IETF-approved standard, rather than just a ‘hey, we have this thing. It’s kind-of documented, good luck creating an interoperable version’ is nothing short of amazing to see. I’d love to see Thunderbird and Dovecot support JMAP.
As soon as Fastmail supports Confidential Computing (so I get strong technical guarantees backed by remote attestation that your administrators can’t see my mail spools), I expect to become one of your customers.
It’s a shame that you’ve dropped the family plans though.
Not an expert, but I do run a lot of ZFS RAIDz2 on NVMe on Linux and have done a fair bit of tuning for it. I don’t know which specific thing is given you such “impossible” numbers, but I’m happy to suggest a few things that might be in play, and maybe how to even squeeze more out of it! (btw, don’t mean this to come across as patronising, I’m just writing a few things out for readers that haven’t seen it, or for actual experts to tell me I’m doing it wrong!).
Most of the performance is going to be from the ARC, that is, the memory cache. ZFS will aggressively use RAM for caching (on Linux, by default, the ARC will grow to as much as half the physical RAM). You’ve already seen this in Note #3; reducing the RAM reduces throughput. Incidentally, you can tune how much RAM is used for the ARC with
zfs_arc_minandzfs_arc_max(seezfs-module-parameters(5)); you don’t have to reduce the system “physical” RAM (though maybe that was more convenient for you to do).Compression gets ZFS a huge amount of throughput, because its faster to do a smaller read and decompress it than wait for the IO (turning compression off can actually make things slower, not faster, because it has to hit the disk more). Compression is block level, and as a special case, all-zero blocks are not even written - the block header has a special case that says “this is all-zeroes, length XXX” that ZFS just inflates. Finally, turning off compression doesn’t change the compression state of already-written blocks, so if you’re benchmarking on data that already exists, you’ll need to rewrite it to really “uncompress” it.
In a RAIDzX, data is striped across multiple devices, and reads can be issued in parallel to get bits of the file back and then reassemble them in memory. You have 32 lanes, so you’re probably right in saying you’re not saturating the PCI bandwidth. You’re almost certainly getting stuff as fast as the drives can give it to you.
You’re using 512B blocks. Most current NVMe is running 4K blocks internally. The drive firmware will likely be loading the full 4K block and returning a 512B chunk of it to the system, and keeping the rest of the block cached in its own memory. In sequential reads that’s going to mean almost always 7 out of 8 blocks are going to served from the drive’s own cache memory, before touching the actual flash cells. (This is well worth tuning, by the way - use flashbench to determine the internal block size of your drive, and then find out how to do a low-level format for your device to switch it to its native block size. Along with an appropriate
ashiftfor your pool, it will let ZFS and the Linux block layer deal in the drives native block size all the way through the stack, without ever having to split or join blocks).ZFS will use a variable blocksize, by default growing blocks as large as 128K. When reading, it will request the entire logical block, composed of multiple physical blocks, from the block layer. If they’re stored sequentially, that can translate to single “range request” on the PCI bus, which may get coalesced into an even larger range, which the drive may be able to service entirely with parallel fetches against multiple flash cells internally.
Not sure which version of ZFS you’re using, but versions before 2.0.6 or 2.1.0 have a serious performance bottleneck on wide low-latency vdevs (GH#12121 GH#12212).
In my experience though, and yours too, ZFS performance is pretty good out of the box. Enough that even though my workload does some things that are outright hostile to a CoW filesystem, the gains have been so good that it hasn’t yet been worth changing the software.
Great list, that’s almost surely what’s at play here. I don’t think the drive/file system speeds are actually being measured.
Some other performance tuning things to think about with zfs: if you have a fast slog vdev you can set sync=always but if your zil is slow you can set sync=disabled to gain a lot of speed at the expense of safety. For some use cases that’s okay.
When I ran mechanical disks I used an Optane drive for my slog vdev and it was so fast I couldn’t measure a performance difference when using sync=always.
I am trying out a few things with fio and will post the results here. There was a suggestion on HN that mirrors what you’re suggesting. I’ll update the article if I find that 157 GB/s is a bogus result.
Edit: OK folks, party is over. 157 GB/s is a misleading number. The FIO library needs separate files for each thread otherwise, it will report incorrect bandwidth numbers. See this post, I am in process of updating the article: https://news.ycombinator.com/item?id=29547346
Updated, thanks everyone! - https://neil.computer/notes/zfs-raidz2/#note-5
What does this mean? I’ve set up aligned partitions and filesystem block sizes (or ashift for ZFS), but I don’t know what a low-level format even means.
All drives (flash and spinners) have a “native” block size. This is the block size that the drive electronics will read from or write to the actual storage media (magnetic platters or flash cells) in a single unit. Sizes vary, but in pretty much every current NVMe SSD the block size is 4KB.
Traditionally though, most drives arrive from the factory set to present a 512B block size to the OS. This is mostly for legacy reasons; back in the mists of time, physical disks blocks were actually 512B, and then the joy of PC backward compatibility means that almost everything ever since starts by pretending to be from 1981, even if that makes no sense anymore.
So, the OS asks the drive what its block size is, and it comes back with 512B. Any upper layers (usually a filesystem, maybe also intermediate layers like cryptoloops) that operate in larger block sizes will eventually submit work to the block layer, and it will then have to split the block into 512B chunks before submitting them to the device.
But, if the device isn’t actually 512B natively, then it has to do more work to get things back into its native block size. Say you write a single 512B block. A drive doing 4K internally will have to fetch the entire 4K block from storage into its memory, update it with the changed 512B, then write it back down. So its a bit slower, and for SSDs, doing more writes, so increasing wear.
So what you can do on many drives is a “low-level format”, which is also an old and now meaningless term for setting up the basic drive structure. Among other things, you can change the block size that is exposed to the OS. If you can make it match the native block size, then the drive never has to do deal in partial blocks. And if you can set the same block size through the entire stack, then you get eliminate partial block overheads from the entire stack.
I should note here that all this talk of extra reads and writes and wear and whatnot makes it sound like every SSD must be a total piece of crap out of the box, running at glacial pace and wearing itself out while its still young. Not so! Drive electronics and firmware are extremely good at minimising the effects of all these, so for most workloads (especially large sequential reads) the difference is barely even measurable.
But if you’re building storage systems that are busy all the time, then there is performance being left on the table, so it can be worth looking at this. My particular workload includes constant I/O of mostly small random reads and writes, so anything extra I can get can help.
I mentioned flashbench before, which is a tool to measure the native block size of a flash drive, since the manufacturer won’t always tell you or might lie about it. It works by reading or writing blocks of different sizes, within and across theoretical block boundaries, and looks at the latency for each operation. For example, you might try to read 4K blocks at 0, 2K, 4K, 6K, etc offsets. If its 4K internally, then the drive only has to load a single block at 0, but will have to load two blocks at 2K to cross the block boundary, and this will be visible because it takes just a little longer to do its work. It’s tough to outsmart the drive electronics (for example, current Intel 3DNAND SSDs will do two 4K fetches in parallel, so a naive read of the latency figures can make it look like it actually has an 8K block size internally), but with some thought and care, you can figure it out. Most of the time it is 4K, so you can use that as a starting point.
On Linux, the
nvme listtool can tell you the current block size reported by the drive. Here’s some output for a machine I’m currently right in the middle of reformatting as described above (it was inadvertently introduced to production without having been reformatted, so I’m having to to reformat individual drives then resilver, repeatedly, until its all reformatted. Just another sysadmin adventure!)So you can see that nvme10n1 is still on 512B.
And then once you’ve done that, you have to issue a low-level format. I think it might be possible with
nvme format, but I use the Intel-specificisdctandintelmastools. Dunno about other brands, but I expect the info is easily findable especially for high-quality devices.Do remember though: low-level format destroys all data on the drive. Don’t attempt in-place! And I honestly wouldn’t bother if you’re not sure if you need it, though I guess plenty of people try it “just for fun”. You do you!
Maybe something like Sieve could help with that, if it ever become widespread. It is not too late for email to evolve: autocrypt, chat-over-imap (email) IMAP IDLE and NOTIFY.
We now have all the tools to build a solid experience providing all the features we are looking for from all chat systems. All it takes is finding some client that suits your needs.
I think there are two, related, problems.
The first is that email has changed a lot in how it’s used. Most users 20 years ago had one client per mail account. POP3 was common and its usage model was very simple: clients downloaded all messages, the server storage is just a cache of the last n days of emails. IMAP allowed storing the entire mail history on the server (POP3 did in theory, but a bunch of operations were O(n) in terms of the size of the mail box) and started shifting things to the server. Adding in things like Sieve and you’re in a situation where there’s a lot of duplication between the client and server. Most mail clients can do spam filtering but no one really wants this anymore: if Thunderbird does spam filtering, then no spam received since the last time I had my laptop on and now will be filtered when I check the mail on my phone / tablet / any other device. Now, most people check email on multiple devices, so you need to do almost everything on the server for it to be useful. Features that only work on one client are annoying.
The second problem is that the communities that write mail servers and mail clients are quite disconnected. Look at JMAP. It is a big improvement over IMAP, but it’s only really supported by a single client for a proprietary mail service, which uses its own server. They can do this because they have the vertical integration (the same is true for Gmail and Outlook / Exchange): they add features to the client and server at the same time. JMAP is low priority in Dovecot because Thunderbird doesn’t use it, it’s low priority in Thunderbird because Dovecot doesn’t use it.
This is made worse by the fact that supporting good server-side features (e.g. Sieve) reduces the perceived value of the client. I use Sieve to filter all of my mail and use a Thunderbird plugin that gives me syntax highlighting of the script and updates it over the managesieve protocol. As a result, Thunderbird’s complex mail rule features are of zero value to me. Similarly, my mail server does server-side spam tagging and I filter the results with Sieve (with IMAPSIEVE rules to mark messages as spam or not-spam when they are moved to and from the junk mailbox). These then Supporting a new feature on the server removed any value for me in the client having spam detection and message filtering logic.
Very well put.
I agree with the generally point, and here are some nuances or extrapolations.
JMAPWith the advent of HTTP 3 and presence of HTTP 2, why bother optimizing a few extra round trips – or – why bother with HTTP at all?
We end-up with HTTP 200 with this in it:
{ "status": 400, ...}.On the other hand, immutable UIDs, constant syntax without so many exceptions are a good thing.
server-side -vs- client-sideIt is a good thing that things are moving toward server-side for some aspects: servers are easier to maintain than each and every user’s machines (that may still literally be between their hands when we need to do the maintenance).
Yes, sure, former/existing mail client look silly now with all its features.
walled gardenIt looks like a short-, middle-, long-term solution choice, so maybe all three up front? A lot ahead, yet a few years ago, none of these issue did really exist!
For JMAP specifically, its more of a RPC protocol that runs over HTTP (or anything really; a profile for WebSockets and works great).
Why HTTP+JSON? Mostly because its convenient: everyone understands it, and every platform has high-performance libraries for it. The learning curve on IMAP is a big part of why JMAP exists.
As for optimising a few round-trips, not everywhere is nearby. Much of Fastmail’s frontend engineering was done in Melbourne, Australia, 300ms from our servers on the US east coast. A lot of our UI performance comes from packing a lot into each request.
The RFC that details with the detailed error responses, RFC7807, specifically calls out that the status code listed in the json object must match the status code returned by the web server. The status member of the response object is considered a “convenience” for the consumer of the response, whatever the heck that means.
One area where this could be false, which is addressed in the RFC itself, is when proxy servers, load balancers, etc. might alter the status codes of a response.
Then I am tempted to think this is a misbehavior of the reverse proxy that changes the semantics. I feel like sliding onto a different topic, but is that not the point of discussions… :)
I think fastmail has an inner fork of Cyrus, and they have merged big parts of it with upstream Cyrus (the latest version has JMAP support). The problem with JMAP is that parts of it are still nonstandard yet (unpublished or only drafts), mainly notifications and calendar/contacts. meli has basic JMAP support.
This is relevant to your first paragraph: client-side filtering doesn’t make sense anymore (or at least like Thunderbird does it). Spam and filtering in general is part of the “mail stack” you use and clients assuming they can replace the concept of a stack is the wrong approach. The user should always be able to define their stack.
While the JMAP standards that are in a status of ‘PROPOSED STANDARD’, it is worth noting the vast majority of RFCs in use on the internet are in that status. The list of RFCs with a ‘Internet Standards’ is pretty darn short.
Notifications are a thorny one, but JMAP’s core RFC does have two different methods for notifications/pushing to clients. The more resource friendly method unfortunately also leads to vendor specific implementations.
As for Calendars & Contacts, I agree. I hope that they submit it for a proposed standards track soon.
They’re being worked on in public: https://datatracker.ietf.org/wg/jmap/documents/
I realize in hindsight that my comment could be interpreted as me implying it is being done in private, that was not my intention.
It’s more that I’m itching for it to be published as a Proposed Standard RFC so that I can safely assume it’s what people will be developing with. I have a few ideas I want to explore but I’d rather wait until there would be other implementations in development that I could test interoperability against.
No problem, and fair enough! We’re of course keen to get them finalised too, but the IETF process can be slow. Its worth it though; the JMAP core and mail specs spent three years (I think) at IETF and are very much better for it!
Fastmail has six internal commits that we apply to upstream master on each deploy. They are all either adjustments to handle differences of opinion or historical bugs in our middleware or stored data that require a lot of work to update (eg FM’s idea of “admin” is different to Cyrus’ idea), or they’re minor changes to match the way we like to operate the service (eg we convert all assertions and fatal errors into coredumps, triggering a bunch of analysis tooling).
I’m not saying Cyrus upstream is necessarily easy to operate, but in terms of JMAP support, its complete and correct (bugs notwithstanding) and what we use for both the Fastmail and Topicbox UIs and for the general protocol access we offer to users.
Thanks for linking these other clients, I’ve been meaning to experiment with some and MailPile looks pretty good.
I mostly took it from https://autocrypt.org/dev-status.html
Maybe so, but if after several decades still no unbroken client emerges then there’s still something fishy going on. It might not be the protocol (though I hear that IMAP isn’t exactly the nicest thing to work with) but maybe we just haven’t figured out what a good email workflow would look like.
Of course that’s coupled with the problem that everyone uses email differently.
I’ve been working on an “unbroken client” for a while now. From my experience, email just has so much stuff going on. Many bugs emerge when different components of a system interact in unforeseen ways, and an email client has to have many components.
IMAP:
I’m no big defender of IMAP, but there’s a lot of work being done to shave off some of the rough edges lately, see https://datatracker.ietf.org/wg/extra/documents/. In particular, the “64-bit mail ids” thing you mention is largely resolved by IMAP ObjectID (RFC8474).
I was wondering about this, looks like more room to use things like filesystem inodes or other methods to avoid a lookup to an index for every mail.
Now it is all about adoption ! :)
Company: Fastmail
Company site: https://www.fastmail.com/
Position(s): Email Client Application Developer
Location: Philadelphia, PA
Description: https://www.fastmail.com/about/jobs/2019-04-clientdev/
We’re looking for an Email Client Application developer to join the FastMail team in our Center City Philadelphia office.
Together with the rest of our team, you’ll work on our email services FastMail, Pobox, and Topicbox. When the vast majority of people get email for free, the reason people choose a paid service is because they value both privacy and a great user experience.
You’ll be working on our best-in-breed mail clients, providing super-fast, powerful access to webmail, calendars, contacts and more. You’ll work with our collaborative team of front- and back-end developers, designers and researchers. Improving our customers’ interactive experience, keeping it secure and a delight to use is what we do, every day.
Our interfaces are built upon our own Overture framework and the new email standard JMAP. It gives us full control from top to bottom to build the right solutions to difficult problems. If you’ve worked with other MVC application frameworks, like Apple’s Cocoa, you should find Overture easy to pick up.
You’ll have the opportunity to work at many levels:
Contact: Email us at jobs@fastmail.com to introduce yourself and tell us why you would be a good fit for the job! This job requires clear communication, so that message is part of your application, but please also include a PDF of your resume.
I’ve been interested in looking at gmail alternatives and I’ve heard good things about Fastmail. Good to know there’s an engineer on here, makes me trust it a bit more.
Fastmail. They are trustworthy, quick to respond to service requests, and rock solid. I can count the number of outages in the past ~10 years on one hand.
+1 for Fastmail. I’ve been using them for several years now and they’re very reliable, have a really solid web UI, and from what I can tell a solid position on security. They also contribute to moving the state of internet mail forward by improving Cyrus and contributing to RFCs. All in all I’d highly recommend them.
That’s another good point: they are by all accounts a solid technical citizen, contributing back and moving the state of the art forward. I like to reward good behaviour when I spend my money, and it’s nice to be able to do that and get top of the line service, to boot.
I also switched from Gmail to Fastmail.
The funny thing is that for the amount of press that Gmail received/receives for being “fast”, once you switch to Fastmail, you realize that Gmail is actually very slow. The amount of bloat and feature-creep they’ve introduced is fascinating.
You’re talking about the web interface or the speed at which the mail is sent?
The web interface.
I just use thunderbird (and k9 on mobile). I don’t see why you’d ever use a web interface for email when a standalone client is so much nicer to use.
I’m on a desktop client too (Evolution). Just pointing out the advantage of Fastmail over Gmail. :)
Love Fastmail. I only wish more tools had first class CalDAV/CardDAV support. When I switched over, I was genuinely surprised how pervasive it’s become to slap on Google account sync and call it a day, even in FOSS. Aside from the built-in macOS/iOS apps, most solutions involve fussing with URLs and 3rd party plugins, if it’s supported at all.
Fastmail has a link generator for CalDAV so it’s super easy to get the right URLs. I do agree for 3rd party plugins, it’s annoying to have to install add-ons for standard and open source protocols…
It was the best one I found, too, overall. I dont know about trustworthy, though, given they’re in a Five Eyes country expanding police and spy authority every year.
Maybe trustworthy from threats other than them, though. I liked them for that.
Yeah, I’m not concerned about state level actors, or more properly, I don’t lose sleep over them because for me and my threat model, there’s simply nothing to be done.
I’m not worried about the state spying on me, I’m worried about the apparatus the state builds to spy on me being misused by service provider employees and random hackers.
If those are your concerns, using PGP is probably recommended.
That will be most folks, too. Which makes it a really niche concern.
Maybe it oughtn’t be niche, but it’s pretty down my list of practical concerns.
I use Fastmail as well, and became a customer by way of pobox.com acquisition.
I’ll have to add, this was about the only time I can ever recall that a service I use was acquired by another company and I was actually fine with it, if not a bit pleased.
My thinking was along the lines of “well, the upstream has purchased one of the biggest users of their tools, can’t be bad.”
I’ve not had any noticeable difference in the level of service provided, technically or socially, except the time difference to Australia is something to keep in mind.
I do hope that no one here in the US lost their jobs because of the acquisition, however.
Nope! We’ve hired a bunch more people in both offices, and the previous Pobox management are now C-level execs. We’re pretty sure the acquisition has been a win for just about everyone involved :)
I can also recommend it, especially due to their adherence to web standards. After 10+ years of GMail, the only functioning client had been Thunderbird, which too often too large. Since switching to Fastmail, I’ve been having a far better experience with 3rd party clients, and a better mail experience in general (probably also because I left a lot of spam behind me).
I second that. I was searching for a serious e-mail provider for a catch-all email, calendar and contacts.
I had trouble setting up my carddav autodiscovery DNS configuration and they helped me without considering me as a “dumb” client. Serious, clear and direct. The most efficient support I could’ve encountered by far.
It’s paid, and I’m paying the second plan (of 5$/month), and I think it’s perfectly fair, considering that, firstly, e-mail infrastructure is costly, and secondly, that their service is just plain awesome.
They’ve recently added the ability to automatically set up iOS devices with all of their services when you create a new OTP. I didn’t know that I needed this, but it’s a wonderful little bonus. It’s stuff like that that keeps me happily sending them money, and will as long as they keep doing such a good job.
I did not know about such a thing, since I’m not an iOS user, but sure sounds nice !
Do you know if they store the emails in plaintext server-side?
It’s a good question. I don’t know, and would like to. I’ll shoot them a mail.
Their help page on the matter isn’t clear, although it does describe a lot of things that seem pretty good. Now you’ve got me wondering. (Happy Fastmail user here, and I even convinced my wife to move to it from GMail!)
edit: It does sound like it’s plain text but you could read it a couple of ways.
Encrypted at rest (encrypted block devices), but cleartext available to the application because we need it for a bunch of things, mostly search, also previews and other bits and pieces. Of course, the applications that hit the on-disk files have their own protections.
I’d imagine their disks are encrypted as a whole - but not using per-mailbox encryption based on keys derived from individual user passwords.
However, even if such claims are made you can’t verify that and shouldn’t trust a companies word on it. I’d recommend PGP if that is a concern.
If this is a feature you’re looking for in a hosted solution, Protonmail is probably your best option.
Up to a point you can, Protonmail has released their webmail client as open source. Of course, with today’s JavaScript ecosystem it’ll be very hard to verify that the JavaScript code you are running actually corresponds to that code. Also, you can’t verify they’re not secretly storing a plaintext copy of inbound mails before encryption. But down that path lies madness, or self-hosting.
And the desperate hope that your correspondent also is sufficiently paranoid.
+1 for Fastmail. Switched recently after self-hosting (well, the last several years at a friend’s) since the dial-up days and I’m satisfied.
Another Fastmail user here. I switched from GMail and my only regret is that I didn’t switch sooner.
I don’t think there are any workflow advantages, but I appreciate that they don’t track me, and I trust them more than Google.
I have the $30 per year subscription.
One of other things I want to highlight is reliability/availability. Making sure I dont miss important emails is even more important than privacy to me. Newer, smaller, and privacy-focused sites might not have as much experience in keeping site up or getting all your mail in reliably.
Fastmail has been around for quite a while with positive feedback from everyone Ive seen. So, they might perform better than others in not missing/losing email and being available. Just speculating here based on what their customers say.
SMTP actually tolerates outages pretty well… I’ve had my self hosted server down for a couple days, and everyone resent me everything when I fixed it.
Haha. Good to know.
What service do you use for Calendars and such?
I use FastMail for calendars and contacts. I actually don’t use it for e-mail much since my ISP is pretty ok.
For Android I switched from CalDAV-Sync + CardDAV-Sync to DAVdroid. Both work but the latter is easier to configure (by way of having less config options).
I tried self-hosting Radical for a while but for the time I had to put into it I’d rather pay FastMail $30 per year.
Fastmail! We have a family email account and shared calendars and reminders and suchlike, and I have a personal account as well.
I am on fastmail for my domain. Works fine, does everything I need.
I am also a happy fastmail.com customer since about 2 years now. I used mailbox.org before, a german email provider, which is quite cheap (1€ per month) and allowed to use custom email domains but their spam filter sucked. Fastmail’s spam filter is also not perfect, in fact Gmail has still by far the best filtering, but their service is great and I can use custom email domain’s too. They also develop JMAP a JSON based IMAP replacement.
I’d say the fact that JMAP is JSON based is only marginally-relevant; it’s got several significant design improvements over IMAP - e.g:
It’s more than IMAP replacement too, possibly better described as an alternative to Exchange ActiveSync.
I’m with mailbox.org myself, with the 2.5EUR/month plan and a private domain. Mostly happy, I don’t have issues with spam. They seem to be quite opinionated on how to handle spam: https://www.heinlein-support.de/vortrag/spam-quarantaene-und-tagging-der-grosse-irrtum. But it seems classical spam tagging has been added recently, though I haven’t tested it: https://mailbox.org/update-des-webportals-bringt-nuetzliche-zusatzfunktionen-fuer-ihr-e-mail-postfach/
I’m not that happy with the web interface though, it seems to be https://en.wikipedia.org/wiki/Open-Xchange.
Is JMAP even supported anywhere? Does anybody use it? Last I checked, not even Fastmail actually used this for anything. Seems like the project started with some energy but is mostly dead now? What a shame, as I’d love to use it somewhere… Please do correct me if I’m wrong.
Hi, I’m some engineering guy at FastMail.
JMAP is currently going through the standardisation process at the IETF to become an RFC. Several companies have built or are building client and server implementations based on those drafts. We’re putting a lot of work into JMAP support in Cyrus.
At FM, we use it internally for some (but not yet all) of our UI-server interactions, and we’re working on converting the UI to use JMAP natively (once the standardisation work has stablised).
Finally, we’re just about to launch a new product that uses JMAP from top to bottom - Cyrus, Ix (a JMAP API generator) and Overture (a UI framework with a JMAP-backed storage layer).
So there’s lots happening on JMAP at FastMail and elsewhere.
That’s really wonderful to hear. Once a year I email FastMail tech support asking them if there’s a JMAP thing, but the answer is always something like “no, and we don’t know when if ever.” And then I’m sad. This here is the first positive confirmation I’ve received, and I’m quite happy to hear it!
Hopefully once you release a fully JMAP designed system, you’ll have auto-exporters from existing tag-based systems like Gmail? Something like this would probably net you a massive user base.
I switched to fastmail last month and I am very happy with it. Before that, I had been self-hosting for 10 years, but I started seeing my emails listed as spam after I switched VPS providers (despite correct SPF etc), and I wasn’t motivated enough to fight for my IP reputation again.
Also Fastmail, moved from Google Apps for domains 2 or 3 years ago. Besides the advantages others mentioned, subdomain addressing is also a cool feature. Some mail providers support plus addressing
me+foobarbaz@mydomain.com
subdomains addressing is a bit nicer. You can make disposable addresses in the form of:
me@foobarbaz.mydomain.com
makes it easier to write rules and to drop mail when the address is sold to some spammer.
Also their support is pretty good. I had a small feature/refinement request twice, in both cases they had the feature implemented in their beta site in a couple of days.
I went to fastmail two years ago when the server on which I’d hosted my own email for about eight years died. I was happy to give a great company about $60 a year to host my family’s email. I was probably spending $60 a month of my own time just to administer the damn thing.
I’m on Fastmail too, with my own domain, for about ten years. The web UI is focused and fast, and the iOS app is just a webview, but a decent one that’s quick. I use Fastmail aliases and inbox rules to send to multiple external addresses, like a basic private listserve. Tons of advanced features for mail users, DFA, and no advertising or shenanigans with your inbox.
They went through a purchase by Opera a while ago, then a few years later Opera sold the business back to the original Fastmail employees – not a single hiccup or business misstep the whole time. They are laser focused. They contribute back to the open source mail server community.
The only issue on my wishlist is that they still don’t support the full CardDAV protocol, which means I cannot fully sync my Fastmail addressbook with iOS, Mac, Windows, or *nix apps, but they’re working on it, and it’s due soon (early 2018?).
I think it’s cheap for what you get, if you’re into that sort of thing.
What exactly is missing from CardDAV support? I’m happily using it to sync contacts to my iOS/Android devices.
Same here. I use fastmail for every new domain that I need email for and it’s pretty great.
Another vote for fastmail. Been a user for several years now. Has by far the best webui out of any provider. Very stable, and quick restoration of backups if you ever need them.
Another +1 for Fastmail. I’ve used them for 3 years and have been pleased with all their services. Their documentation is clear, the system is not hard to use, and they answer questions promptly.
The only thing I’m waiting for is HTTPS support on their web hosting. But if you need serious web hosting, Fastmail probably shouldn’t be yout first choice.
Yep, fastmail here too, it’s superb.