About three years ago I did a “deep enough” dive (i.e. not a true deep dive) into ZFS vs btrfs for my home and home office use (in Linux). I tend to have relatively small amounts of data, with separate backups, so I have happily run outright RAID1 for years for the sheer simplicity. I intended to use either in a pure mirror setup.
The things that pushed me toward btrfs at that time were:
The lack of native encryption in ZFS, and my uncertainty with combining ecryptfs or LUKS with ZFS. Running btrfs on separate LUKS volumes, and combining those into a RAID setup surely isn’t ideal or efficient, but at least it is simple and I know how each piece works.
Mirror expansion in btrfs works how I (personally) think it should: Take a 2-disk mirror and add a 3rd disk? Just rebalance and going forward each piece of data is on two devices out of the three. Not efficient, but simple. ZFS requires 2 new drives as a separate mirror from the first two. That feels like an unnecessary constraint (and expense) to have to always buy drives in pairs.
A general sense of unease that ZoL wasn’t really a first-class citizen in the ZFS world. That Illumos and the BSDs would always get the advanced, and most stable features, and Linux users might never get this-or-that feature, and that “Just use FreeBSD instead” seemed like an accepted response to questions about ZoL stability (I do not say that in reference to anyone involved in the projects, of course. More from various end-user communities and conversations).
Despite going with LUKS and btrfs (again I emphasize: mirrors), and that setup being great so far, tools like Sanoid have long left me with ZFS envy, although I will always have a soft spot for Bacula (now Bareos). So I was already watching the ZoL releases and the 0.8 milestone, with hopes to try out the encryption.
Now hearing that RAIDZ Expansion is actually on the way and this FreeBSD-and-ZoL news, so close in time, seals the deal. As soon as 0.8 is marked stable, I will be switching to ZFS.
There are definitely other distributions, with active developers and active users. The OpenIndiana folks are still alive and well. The OmniOS community really stepped up to maintain that distribution when OmniTI moved on, and have done an amazing job so far. SmartOS is alive and well, also – I work at Joyent, where we run SmartOS (using ZFS!) on many thousands of production systems for critical tasks.
I have tremendous respect for the FreeBSD community, and at various meetups and through the monthly OpenZFS community leadership call I get to interact with lots of great folks that work on the project. That said, this assertion made by Matt/Kip Macy in the mailing list post is just false:
While it’s true that Delphix are moving their appliance from illumos to Linux, Delphix is just one company that makes use of illumos. Other companies and user communities continue to exist around the software, whether directly consuming illumos-gate or maintaining a downstream fork. I appreciate that Mr Macy is deeply invested in FreeBSD, but this kind of FUD-stoking is deeply unfortunate.
With respect to ZFS, it’s true that the rate of change in the ZOL repository is the highest of all of the open source ZFS implementations. As such, we (in illumos) are also looking at a sustainable way to import and test changes from ZOL, rather than expecting people to push them to us. I don’t think this is an unfortunate or controversial state of affairs: we want (some of) of the changes being made there, but we’re in the best position to import and test those changes on our platform – thus, we’ll do that work!
Hi Joshua, thanks for answering and addressing my concerns. I always considered collaboration between FreeBSD and illumos to be one of a kind, and was bit surprised by this decision.
I didn’t mean to diminish efforts and work others distributions put into illumos, I just found them less known. I am big fan of what you all do at Joyent.
Wow! I’ve run FreeBSD for several years on my home server, primarily for Jails and ZFS. Yesterday we started rebuilding the box on Linux after a hardware failure. We’re moving to Linux mainly because the tooling around Docker is so much more accessible and widely-used than Jails (and Docker) on FreeBSD. Plus, it’s tiring administering a box in our spare time where every task has a bit of extra thinking for the FreeBSD port factor.
The plan was to use mdadm-based RAID6 because we’ve always thought of ZoL as “immature”, but if even FreeBSD is using ZoL going forward, I should definitely take another look!
Having run ZFS (via SmartOS 💖) at a previous place, I was super interested in ZoL as new place was on Debian. Happy to report it works well, the only issue I’ve had the initial installation time (as it has to compile against the headers, due to licensing) which is easily solved with imaging a VM and then starting new ones from there. (We’re only using it for things like MySQL where we have data on disk that’s not re-creatable. Not for the base filesystem on the VM.)
Interesting! I haven’t used ZFS in a few years and mainly remember when there were questions posed about the legality of using it on Linux. If that’s been sorted (I think I’m a good few years behind there) and development is centralising then I’m very happy.
As far as I know, nothing changed about licensing. It is widely agreed binary distribution of ZFS on Linux is a GPL violation, Canonical continues to violate GPL, but no one is willing to sue Canonical over this issue. It seems to me the situation is unlikely to ever change, short of miracle by Oracle.
I guess I still think of Sun as a going concern, so I was surprised that Illumos/Solaris ZFS development was so quiet. I use ZFS on Linux on my server, and it’s never caused me any trouble, but I’m still I guess living in the ’90s, in some ways.
Do you know if the packaged version of zfs outlined in the linked thread is just transitory, or if the plan is for zfs to live in ports in the future as well (much like how pkg itself does)?
About three years ago I did a “deep enough” dive (i.e. not a true deep dive) into ZFS vs btrfs for my home and home office use (in Linux). I tend to have relatively small amounts of data, with separate backups, so I have happily run outright RAID1 for years for the sheer simplicity. I intended to use either in a pure mirror setup.
The things that pushed me toward btrfs at that time were:
Despite going with LUKS and btrfs (again I emphasize: mirrors), and that setup being great so far, tools like Sanoid have long left me with ZFS envy, although I will always have a soft spot for Bacula (now Bareos). So I was already watching the ZoL releases and the 0.8 milestone, with hopes to try out the encryption.
Now hearing that RAIDZ Expansion is actually on the way and this FreeBSD-and-ZoL news, so close in time, seals the deal. As soon as 0.8 is marked stable, I will be switching to ZFS.
So SmartOS is last guardian of illumos as we know? I know there are other distributions, but I consider them less relevant compared to SmartOS.
There are definitely other distributions, with active developers and active users. The OpenIndiana folks are still alive and well. The OmniOS community really stepped up to maintain that distribution when OmniTI moved on, and have done an amazing job so far. SmartOS is alive and well, also – I work at Joyent, where we run SmartOS (using ZFS!) on many thousands of production systems for critical tasks.
I have tremendous respect for the FreeBSD community, and at various meetups and through the monthly OpenZFS community leadership call I get to interact with lots of great folks that work on the project. That said, this assertion made by Matt/Kip Macy in the mailing list post is just false:
While it’s true that Delphix are moving their appliance from illumos to Linux, Delphix is just one company that makes use of illumos. Other companies and user communities continue to exist around the software, whether directly consuming illumos-gate or maintaining a downstream fork. I appreciate that Mr Macy is deeply invested in FreeBSD, but this kind of FUD-stoking is deeply unfortunate.
With respect to ZFS, it’s true that the rate of change in the ZOL repository is the highest of all of the open source ZFS implementations. As such, we (in illumos) are also looking at a sustainable way to import and test changes from ZOL, rather than expecting people to push them to us. I don’t think this is an unfortunate or controversial state of affairs: we want (some of) of the changes being made there, but we’re in the best position to import and test those changes on our platform – thus, we’ll do that work!
Hi Joshua, thanks for answering and addressing my concerns. I always considered collaboration between FreeBSD and illumos to be one of a kind, and was bit surprised by this decision.
I didn’t mean to diminish efforts and work others distributions put into illumos, I just found them less known. I am big fan of what you all do at Joyent.
I was under the impression that all ZFS work was still coming from the OpenZFS joint effort, but Allan Jude explains this in this reply: https://lists.freebsd.org/pipermail/freebsd-current/2018-December/072430.html
It’s good to see that FreeBSD’s and ZFS on Linux’s joint effort will help contribute to what OpenZFS always should have been.
Wow! I’ve run FreeBSD for several years on my home server, primarily for Jails and ZFS. Yesterday we started rebuilding the box on Linux after a hardware failure. We’re moving to Linux mainly because the tooling around Docker is so much more accessible and widely-used than Jails (and Docker) on FreeBSD. Plus, it’s tiring administering a box in our spare time where every task has a bit of extra thinking for the FreeBSD port factor.
The plan was to use mdadm-based RAID6 because we’ve always thought of ZoL as “immature”, but if even FreeBSD is using ZoL going forward, I should definitely take another look!
Very timely news A+++ would read again.
Having run ZFS (via SmartOS 💖) at a previous place, I was super interested in ZoL as new place was on Debian. Happy to report it works well, the only issue I’ve had the initial installation time (as it has to compile against the headers, due to licensing) which is easily solved with imaging a VM and then starting new ones from there. (We’re only using it for things like MySQL where we have data on disk that’s not re-creatable. Not for the base filesystem on the VM.)
Interesting! I haven’t used ZFS in a few years and mainly remember when there were questions posed about the legality of using it on Linux. If that’s been sorted (I think I’m a good few years behind there) and development is centralising then I’m very happy.
As far as I know, nothing changed about licensing. It is widely agreed binary distribution of ZFS on Linux is a GPL violation, Canonical continues to violate GPL, but no one is willing to sue Canonical over this issue. It seems to me the situation is unlikely to ever change, short of miracle by Oracle.
Another nail in Solaris’s coffin perhaps? Or tne end of an era?
Solaris’s final nail was Oracle “no longer distributing the source”. illumos is alive and well, see comment above from someone at Joyent.
I guess I still think of Sun as a going concern, so I was surprised that Illumos/Solaris ZFS development was so quiet. I use ZFS on Linux on my server, and it’s never caused me any trouble, but I’m still I guess living in the ’90s, in some ways.
So does that mean that it will stop supporting TRIM? That’d be unfortunate.
No, I expect that we will still support TRIM.
Do you know if the packaged version of zfs outlined in the linked thread is just transitory, or if the plan is for zfs to live in ports in the future as well (much like how pkg itself does)?
There’s no reasonable way for it to live in ports because we need it in our kernel and in base.