1. 36

    A patch to discourage the use of those codes is actually in review at the moment: https://reviews.freebsd.org/D27176

    1. 1

      Out of curiosity, does anyone use FreeBSD or any BSD in run software in production? Can you share your use case and why BSD was used instead of something else?

      I would be interested in reading the software deployment story.

      1. 1

        Quite a number of companies, actually. For case studies, you may take a look at the FreeBSD Journal from January/February 2021: https://freebsdfoundation.org/past-issues/case-studies/

      1. 2

        Reminds me of FreeBSD’s gnop(8).

        1. 1

          Thanks, didn’t know about it. I made unreliablefs agnostic for operating systems and it has a single dependence - the FUSE library, that exists almost everywhere.

        1. 2

          Poudriere is one of my favorite tools in the FreeBSD ecosystem.

          1. 5

            If only if they’d support Dragonfly, OpenBSD or even NetBSD, I’d go opnsense.

            FreeBSD is, to me, just a boring BSD that lacks leadership and just follows Linux’s decisions. This is why Dragonfly was forked, and as far as I am aware, nothing’s changed.

            1. 17

              Is it a disadvantage to be boring in the world of firewalls?

              1. 10

                I’m sorry to ask, but what are you talking about? FreeBSD’s development seems to be completely unrelated to Linux’s, and why would I want an OS I rely on for infrastructure to be exciting? FreeBSD seems to have some of the best engineering (read, not hacking together random stuff until they feel it’s time for a release) I’ve seen in any OS, the system is consistent, the documentation is excellent, the system is reliable and provides the features its target users want. Th only complaint I have with it as a firewall OS is they never kept up with OpenBSD’s changes to PF, which means having to know two syntaxes if using both.

                1. 2

                  Have you heard of wireguard?

                  1. 4

                    The highly praised cryptokey routing tunnel system that is available for Linux, MacOS, Windows, Android, iOS, OpenBSD and FreeBSD, as well as having a mostly-portable Go implementation?

                    No, tell me more.

                    1. 0

                      I assume you are referring to the developer who implemented, poorly, wiregard in the kernel scheduled to release in FreeBSD 13, only to have it ripped out because there were major concerns brought up?

                      Your point?

                    2. 2

                      what are you talking about?

                      About FreeBSD being an unfortunate choice of an OS for opnsense to run on.

                      FreeBSD’s development seems to be completely unrelated to Linux’s

                      It isn’t, unfortunately. It copied the worst decisions Linux made, such as the fine-grained locks approach to SMP, and the complexity it brings. This is literally the reason Dragonfly exists.

                      For that matter, Dragonfly was relatively recently boasting about the superior performance of its network stack relative to both FreeBSD and Linux. An achievement that has nothing to do with putting more man-hours of effort (Dragonfly has few developers, and FreeBSD is a huge community with significant corporate funding), and everything to do with straight up better engineering.

                      the documentation is excellent

                      Yet FreeBSD is infamously significantly worse than NetBSD and OpenBSD in documentation. It might compete with Dragonfly on that topic, but only because Dragonfly is a small team which effort is focused on actual development.

                      provides the features its target users want

                      Excessively self-fulfilling statement.

                      as a firewall OS is they never kept up with OpenBSD’s changes to PF

                      Is one of many reasons I’d rather use something else as base for a router/firewall.

                      1. 17

                        It isn’t, unfortunately. It copied the worst decisions Linux made, such as the fine-grained locks approach to SMP, and the complexity it brings. This is literally the reason Dragonfly exists.

                        That’s a lot of assertion. FreeBSD uses fine-grained locking because it has good performance and fits well with C. Linux uses a mixture of fine-grained locking and RCU. FreeBSD imported ConcurrencyKit a few years back and so now also has a rich set of lock-free data structures for use in the kernel. Dragonfly pushed in the direction of lightweight message passing, but it’s not clear from any of the concurrency benchmarks that I’ve seen that this was in any way better (in general, I strongly prefer message passing and shared-nothing concurrency, but C is a terrible language for trying to use this kind of model).

                        The reason that Dragonfly exists depends on who you ask. If you ask Matt Dillon, it’s because N:M threading was a terrible idea and 1:1 threading was the right approach. If you ask any of the FreeBSD kernel developers who were around at the time, it’s because Matt Dillon kept committing broken code and shouting at people that they needed to fix the things he’d broken.

                        For that matter, Dragonfly was relatively recently boasting about the superior performance of its network stack relative to both FreeBSD and Linux

                        On what kind of workload? Netflix gets insane performance out of the FreeBSD network stack with large transactions by supporting in-kernel fast-paths for TLS (just the crypto, not the control-plane parts) and things like aio_sendfile. Last time I looked, they were getting around double the per-core performance with TLS that BBC iPlayer was getting from Linux without TLS. At the opposite extreme, Verisign was running half of their root servers on FreeBSD (half on Linux, so a bug in one wouldn’t take out the entire root) and servicing a lot more requests from the FreeBSD ones, particularly the subset of those that were using netmap with an aggressively specialised userspace network stack. Netmap is enabled for pretty much all NICs in FreeBSD, it’s available as a patch set for Intel NICs on Linux. DPDK provides similar abstractions now, but Netmap has been in-tree in FreeBSD for over a decade.

                        FreeBSD was not copying Linux with Jails, it was the first such system to exist in any operating system. Linux now tries to build the same abstractions out of namespaces, cgroups, seccom-bpf, gaffer tape and string. FreeBSD was not copying anyone with Capsicum, which still provides the best set of abstractions for writing compartmentalised applications of any OS. FreeBSD and Linux were both copying the same systems with the TrustedBSD MAC framework and KSMs, but the FreeBSD version is a lot more flexible.

                        1. 3

                          Speaking of RCU style things, SMR was introduced for memory and is now also used by vfs.

                          1. 1

                            Dragonfly pushed in the direction of lightweight message passing, but it’s not clear from any of the concurrency benchmarks that I’ve seen that this was in any way better

                            Message passing does lead to a more structured design, which has a myriad of benefits. When the difference in terms of development manpower is taken into account, there’s no doubt that the approach Dragonfly took was the better one in hindsight.

                            https://www.dragonflybsd.org/performance/

                            On what kind of workload?

                            The network perf data I was thinking about, took a while to find: https://leaf.dragonflybsd.org/~sephe/perf_cmp.pdf

                            Netflix / Verisign

                            Yes, there is no doubt FreeBSD is better than Linux, but that is a low bar to meet.

                            FreeBSD was not copying Linux with…

                            See above. Linux is a mess and should be the textbook example on how not to do software development. FreeBSD has it beat by actually making decisions on where to go and following through.

                            It is just that they will many times blindly go forward with what Linux did. The fine-grained locks over message pasing was a fundamental fuckup I simply can’t pretend didn’t happen and makes me sad to think about every time FreeBSD comes up.

                            I believe the effort overhead associated to it will in time put FreeBSD (and Linux) very clearly behind Dragonfly in performance / scalability. And that wall will actually be insurmountable due to fundamental architectural issues, not something that hacking in some optimizations will get anywhere close to breaking.

                            1. 11

                              It must be hard being a contrarian in the face of people who actually do the work. Or not really addressing any of their points.

                              1. 6

                                Message passing does lead to a more structured design, which has a myriad of benefits.

                                Let me fix that for you:

                                Message passing may lead to a more structured design, which might result in a myriad of benefits.

                                1. 2

                                  I thought message passing used fine grained locks, just down one abstraction layer

                          1. 5

                            I wish there was something better there than freebsd-update(8) which takes hours (!!!!!) to do a major upgrade. It’s so bad, that the FreeBSD infrastructure people rolled their own custom solution.

                            Sadly (?), PkgBase seems dead.

                            1. 4

                              I just updated 2 systems from 12.2 to 13.0 and it took maybe 20 or 30 minutes to do both systems.
                              I don’t disbelieve you, but how is it taking hours for you? Slow internet (I have pretty fast internet)? Slow disks (I have either SSD’s or large zpools that are pretty zippy)?

                              (note: I did have to wait about an hour for my local poudriere to rebuild all packages for 13.0, but that gets done anyway, so I didn’t include it in the time it took).

                              I agree though, freebsd-update is in general pretty slow, and PkgBase seems like it may never arrive, or at the very least won’t happen for quite a while yet.

                              1. 4

                                I hate freebsd-update with a passion. It forks a child process for every single file that it compares and it does so sequentially and so it can be very slow on systems with slow single-threaded performance, platforms where fork is slower than normal (including some virtualised environments). It’s also heavily limited by random disk read latency because everything it does is sequential: If it tried to the comparisons in parallel then it could at least take advantage of things that are in the buffer cache, prefetched, or just cheaper to read out of order with NCQ in one thread while another is blocked on I/O, but it doesn’t.

                                I’ve often seen it take an hour to upgrade a few hundred MiBs of the base system and then had pkg upgrade multiple GiBs of packages in under a minute on exactly the same machine.

                                The logic for detecting changes is also really unreliable. I’ve had freebsd-upgrade leave me with an unbootable system three times. As far as I can tell, it scans the files, detects that they aren’t what it expects, and then patches them anyway.

                                I really wish the FreeBSD Foundation would invest in pkg-base. It’s 80% done (it works great for me, but there are some configurations that it doesn’t work well for and there’s some usability that could be improved) but the rest of the work is tedious and involves the incredible pain of touching the FreeBSD build system, so it’s hard to get volunteers to do it.

                                1. 3

                                  On my 24 core Xeon with ~250Mbps Internet and NVMe disks, upgrading from 12.2 to 13.0 took over two hours.

                                  I also upgraded some more modest 11.4 machines, and those “only” took about half an hour. Unsure about the discrepancy.

                                  Another machine I have got stuck, and I had to restart freebsd-update(8). The second time worked.

                                  freebsd-update(8) on FreeBSD, and syspatch(8) and sysupgrade(8) on OpenBSD are relatively new. Most of my life I have been doing upgrade by building from source, so I guess I shouldn’t complain too much, it’s still progress.

                                  1. 3

                                    freebsd-update(8) on FreeBSD, and syspatch(8) and sysupgrade(8) on OpenBSD are relatively new. Most of my life I have been doing upgrade by building from source, so I guess I shouldn’t complain too much, it’s still progress.

                                    I already used freebsd-update back in the day and I stopped using FreeBSD around 2014. It’s not that new.

                                    Two hours on a Xeon machine seems ridiculous; if you have the time, you should really write a bug report or something for that.

                                    1. 2

                                      I’ve certainly had some systems inexplicably take longer than others, but sounds like you have run into some real bad ones. Yikes!

                                      1. 1

                                        I’ve got a little Celeron J4125 NUC-style machine that I did the 12.2->13 upgrade on. Took about 10 minutes.

                                        I’ll be curious to learn what impacted a beefy Xeon or similar reports I’ve seen crop up occasionally.

                                      2. 3

                                        20/30 minutes still seems pretty long to me for a base system update. freebsd-update seems optimized to reduce network usage with all its binary diff cleverness, which is still useful in various cases, but for many people it’s a lot less useful than it used to be. Even with my regular ADSL connection just doing a download of the ~220M of the full base.tar.xz + kernel.tar.xz will be faster (at 1M/s it’s less than 4 minutes), and the extract/removal of old tools shouldn’t take more than a minute or so.

                                      3. 4

                                        PkgBase is not dead. People are using it as we speak. It’s just hasn’t been enabled by default as there are still some rough edges to be addressed. Here’s a community PkgBase server you may consider using: https://alpha.pkgbase.live/

                                      1. 4

                                        The pot framework for FreeBSD jails has also published some OCI-related bits recently: https://github.com/pizzamig/pot-images

                                        1. 5

                                          Some other helpful links for FreeBSD Boot Environments that I’ve found over time.

                                          1. 7

                                            I wish the normal upgrade process automatically created a new BE for the pre-upgrade state. Nexenta’s version of apt did this and had an apt rollback command. If that works reliably then you can always upgrade to the new shiny whatever, secure in the knowledge that you can revert if it broke things. You can do the same with FreeBSD, but it’s a lot more manual steps.

                                            1. 4

                                              Yes it would be great if at least freebsd-update(8) had a -B flag (example) to do the upgrade within new BE.

                                              At least the additional steps are very few and not that hard.

                                              1. 3

                                                I would be willing to do this, if no one else plans on doing it. Any devs here to comment?

                                                1. 2

                                                  Adding support for that to freebsd-update(8) may be an interesting project. It might be a better idea to add this functionality as a separate program instead, because it could be that freebsd-update will be gone in FreeBSD 14 or FreeBSD 15 because of pkg base.

                                              2. 2

                                                For those building from source, there is beinstall(8) available, which automates installation and boot environment management.

                                                1. 2

                                                  Pity beinstall(8) is only for compiling the sources.

                                                  Its not an issue on a 16-core AMD Ryzen but its PITA on a 2-core low power laptop CPU :)

                                            1. 9

                                              Nomad is cool, because it may work with technologies other than Docker containers. For example, Nomad can be used to orchestrate FreeBSD jails: https://papers.freebsd.org/2020/fosdem/pizzamig-orchestrating_jails_with_nomad_and_pot/

                                              1. 3

                                                And it’s exec command is isolated with a chroot, which makes it super useful when migrating non-containerised workloads too.

                                                1. 2

                                                  Anyone got any experience with that? Seems like a nice way to run plain binaries without having to use docker images (For example when the binaries are compiled with Go).

                                                  1. 4

                                                    I used the java one and the exec ones. It worked great, especially if you don’t require any special libraries already in the system.

                                                    1. 3

                                                      We’ve been using the java driver in production for over 2 years now, we also us the exec driver for smaller tools, basically shell scripts to backup the consul and nomad database.

                                                      1. 2

                                                        I’ve used the exec in presentation demos, where I am running a cluster of nomad VMs, and I have an directory mounted to the host with the apps to run.

                                                        I could of course host a docker registry in the host, but it’s not worth the hassle; I’d rather have simpler demos with less to go wrong!

                                                  1. 4

                                                    It’s already packaged for some operating systems: https://repology.org/project/retro12/versions

                                                    1. 2

                                                      Both pmenu and xclickroot are now available in the FreeBSD ports:

                                                      1. 1

                                                        Thank You.