1. 2

    Doesn’t using proxies address this on top of performance and security boost?

    1. 1

      Using proxies just pushes the problem back one layer; now you need to maintain the proxy’s TLS configuration instead of your web server’s TLS configuration, but it still has to be maintained. Or you outsource maintaining it to Cloudflare. You still can’t walk away from the server entirely and let it keep running quietly.

      (I’m the author of the linked-to entry.)

      1. 1

        “In the era of HTTP, you could have set up a web server in 2000 and it could still be running today, working perfectly well”

        “And now you have to keep reasonably up to date with web server software, TLS libraries, and TLS configurations on an ongoing basis, because I doubt that the deprecation of everything before TLS 1.2 will be the last such deprecation”

        This is what I’m addressing. It seemed like folks you talk about wanted these servers to keep running. These diverse and interesting setups. Then, HTTPS’s varying needs gets in the way. So, we rely on a mature proxy whose developers and/or ecosystem handle all that so HTTP or whatever it proxies to keep working without all that work. Then, the rest can keep focusing on the non-HTTPS stuff that sits behind the proxies. There’s existing tools for that.

        “Another, more relevant side of this is that it’s not going to be possible for people with web servers to just let them sit.”

        This part remains true. Looking at the big picture, it probably was and always will be true for a lot of things in tech and life. Just due to how our environments change constantly whether offline or online. If anything, we should be pleasantly surprised when something we build still works five years later online without changes. Even more as pace of change and extra complexities increase over time.

    1. 8

      I think that coco and others who mentioned it are certainly correct that the package format, by itself, doesn’t improve the security situation. We need better support in Unix distributions for sandboxing of end-user applications. These package formats do help with that, in that by bundling all the dependencies inside the package, you relieve the distribution from the burden of identifying the dependencies and copying them in. Of course, to do it properly you would want the distribution to enforce the details of the sandbox configuration, rather than relying on the package to do it.

      As with containers for server software, this approach has drawbacks. If the distribution isn’t providing the dependencies, it also doesn’t have visibility into whether they’re up-to-date with all security patches. So these package formats as they stand actually take control away from end users… I can’t say I’m a fan. In other words, distributions should want to be involved in identifying the dependencies and copying them into any sandbox that’s being used.

      I do understand the benefit to having a format that works across distributions, but I really think that benefit is almost entirely to software publishers, and not to end users or to distribution maintainers. At least in the server world, system administrators do get some benefits from containers, but I don’t think that applies here.

      Since I do have higher-than-average security needs, I actually don’t ever use Flatpak, since the security trade-off would be unacceptable to me. I regard it as an important security feature to have binaries that are built by my distribution’s tooling, not by the software publisher. That means there are apps I can’t use at all, but that’s just how it is sometimes.

      I will take the opportunity, in passing, to plug NixOS. There is more work to be done wrt sandboxing of graphical apps, but you can already write a Nix package config which will build a Flatpak binary based on a Nix config, rather than on an upstream, publisher-provided recipe. Nix is far better than any other system that exists today, as far as easy configuration of dependencies. I hope that as people think about better solutions to these problems, they’ll keep the Nix tooling in mind and build on top of it when it makes sense to.

      1. 1

        I agree with the view that the first level benefits are primarily to software creators. Software creators get to build one artifact that will work on all Linuxes, and they get to distribute that artifact through a single place (where users can discover it, at least theoretically) instead of many. People like Canonical benefit indirectly through more software being available for Linux and perhaps through running the Snap store which leads to getting to take a cut of any purchases through it. People using Linux theoretically benefit through more software being available for it (and for their distribution), while small Linux distributions might benefit through more software being usable on them.

        (I’m the author of the linked to article.)

        1. 2

          I mean, this trend is specifically taking resources away from efforts to support packaging software properly for distributions. Users who understand their security needs have less software available as a result of it.

      1. 1

        I always thought the short 1U servers were for shallower racks, like the network boxes you often see on walls.

        1. 3

          The normal Dell rails for their short 1U servers are still full length (although perhaps you can get special shorter rails if you ask), so I don’t think they’d fit in short racks. Also, I believe that short telco racks are often only front-mount (with no back set of posts), which doesn’t work for server rails. My best guess about why Dell appears to like shorter 1U servers is ‘less metal for the case’, which presumably translates to cost savings.

          (I’m the author of the linked to entry.)

        1. 2

          I’m currently deep within the third world described in the article, internal client/server TLS. Already being within a private network, it’s unreasonable to purchase a unique certificate for every server host on the network.

          My best two options seem to be:

          1. Dynamic self-signed certificates created at server start up. Publish certificate to centralized & trusted location that clients can read from.
          2. Distributing a single certificate to entire server pool, signed by an implicitly trusted internal CA.
          1. 4

            The standard approach seems to be an internal CA with some sort of automated certificate issuing mechanism (and often trusting only the internal CA, not any public CAs). This does require the automated CA stuff, but I believe there are open source projects for that. If that was too much work, I would be inclined to treat the situation like basic SSH, with a self signed certificate created on startup somehow (either centrally and then distributed, or locally and then published).

            (SSH can also use the ‘internal CA’ route, of course, with server host keys being trusted because they’re signed.)

            1. 1

              We do have an internal CA, so I will probably go that route to get maximum coverage at sites we host. Unfortunately, clients can choose to host themselves and therefore will not trust our internal CA, leaving them to their own devices.

              This service is very core to the company, so failing to form a secure connection means failing to ingest important data. I may end up having to go to a hybrid approach in the end.

              1. 1

                At least for our product at work (cloud-first with on-prem option), the TLS scheme used in “the wild” sometimes meshes badly with internal CA’s used by the on-prem customers. The “stumbling block” is often browsers like Chrome, which can’t easily be convinced to trust an internal CA.

              2. 2

                you want option 3, like @cks mentioned. Each service gets their own cert signed by your internal CA[1]. You would do the same with SSH[2] except obviously it’s by node for ssh instead of by service. Hashicorp Vault[0] will help manage all of this for you.

                0: https://www.vaultproject.io

                1: https://www.vaultproject.io/docs/secrets/pki/

                2: https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-certificates/

              1. 24

                In some cases, I have a great deal of sympathy for the author’s point.

                In the specific case of the software that triggered this post? Not so much. The author IS TALKING ABOUT A SENDMAIL MILTER when they say that

                Python 2 is only legacy through fiat

                No. Not in this case. An unmaintained language/runtime/standard library is an absolute environmental hazard in the case of a sendmail milter that runs on the internet. This is practically the exact use case that it should absolutely be deprecated for, unless you’re prepared to expend the effort to maintain the language, runtime and libraries you use.

                This isn’t some little tool reading sensor data for an experiment in a closed environment. It’s processing arbitrary binary data from untrusted people on the internet. Sticking with this would be dangerous for the ecosystem and I’m glad both python and linux distro maintainers are making it painful for someone who wants to.

                1. 2

                  A milter client doesn’t actually process arbitrary binary data from the Internet in a sensible deployment; it encapsulates somewhat arbitrary binary data (email messages and associated SMTP protocol information that have already passed some inspection from your MTA), passes it to a milter server, and then possibly receives more encapsulated binary data and passes it to the MTA again. The complex binary milter protocol is spoken only between your milter client and your milter server, in a friendly environment. To break security in this usage in any language with safe buffer handling for arbitrary data, there would have to be a deep bug that breaks that fundamental buffer safety (possibly directly, possibly by corrupting buffer contents so that things are then mis-parsed at the protocol level and expose dangerous operations). Such a deep break is very unlikely in practice because safe buffer handling is at the core of all modern languages (not just Python but also eg normal Rust) and it’s very thoroughly tested.

                  (I’m the author of the linked-to blog entry.)

                  1. 2

                    I guess I haven’t thought about one where it would be safe… the last one I worked on was absolutely processing arbitrary binary data from the internet, by necessity. It was used for encrypting/decrypting messages, and on the inbound side, it was getting encrypted message streams forwarded through from arbitrary remote endpoints. The server could do some inspection, but that was very limited. Pinning it to some arbitrary library version for processing the message structures would’ve been a disaster.

                    That’s my default frame of reference when I think of a milter… it processes information either on the way in or way out that sendmail doesn’t know how to and therefore can’t really sanitize.

                    1. 1

                      For us, our (Python) milter client sits between the MTA and a commercial anti-spam system that talks the milter protocol, so it gets a message blob and some metadata from the MTA, passes it off to the milter server, then passes whatever the milter server says about the email’s virus-ness and spam-ness back to the MTA. This is probably a bit unusual; most Sendmail milter clients are embedded directly into an MTA.

                      If our milter client had to parse information out of the message headers and used the Python standard library for it, we would be exposed to any bugs in the email header parsing code there. If we were making security related decisions based on header contents (even things like ‘who gets how much spam and virus checking’), we could have a security issue, not just a correctness or DoS/crash one (and crashes can lead to security issues too).

                      (We may be using ‘milter client’ and ‘milter server’ backward from each other, too. In my usage I think of the milter server as the thing that accepts connections, takes in email, and provides decisions through the protocol; the milter clients are MTAs or whatever that call up that milter server to consult it (and thus may be eg email servers themselves). What I’m calling a milter server has a complicated job involving message parsing and so on, but a standalone client doesn’t necessarily.)

                      1. 2

                        Mine was definitely in-process to the MTA. (I read “milter” and drew no client/server distinction, FWIW. I had to go read up just now to see what that distinction might even be.) Such a distinction definitely wasn’t a thing I had to touch in the late 2000s when I wrote the milter I was thinking about as I responded.

                        The more restricted role makes me think about it a little differently, but it’d still take some more thinking to be comfortable sitting on a parsing stack that was no longer maintained, regardless of whether my distro chose to continue shipping the interpreter and runtime.

                        Good luck to you. I don’t envy your maintenance task here. Doubly so considering that’s most certainly not your “main” job.

                  2. 1

                    Yeah, it’s a good thing they do, it’s not the distro-maintainers fault that Python became deprecated.

                  1. 5

                    It’s got to be a keyword, that’s how it can work with is not, not also being a keyword.

                    Also there’s a quite ubiquitous use of is that justifies its inclusion; compare a is None to a == None to not a… only by testing the identity can you know for certain that None was passed.

                    1. 2

                      You put that very clearly, here and in the other comments; I learned something today.

                      One other ubiquitous use of is: type inspection, like if type(a) is MyClass or if type(a) is str.

                      (Some of the time isinstance(a, MyClass) will also do, but if you want to exclude the possibility of subclass instances, only a type identity check suffices.

                      Ooonn the other hand, one could also argue that is tempts people into checking for identity when a subclass would also suffice; and that this may needlessly prevent interoperation with otherwise-valid subclasses. Hm. I really like the keyword, though, especially compared to ===)

                      1. 5

                        Note that you don’t normally need is for this sort of type inspection, as type(a) == MyClass works fine pretty much always. I think the only time it wouldn’t work correctly is if one of the classes involved had a metaclass that overrode __eq__ and did something perverse with it. I think that the cases where you really need is are uncommon; you need to care about object identity and be in a situation where something is overriding == in a way that affects the comparison you care about. Python definitely should have something that checks object identity in the way that is does, but I don’t know if it needs to be a tempting keyword instead of a built-in function.

                        (I’m the author of the linked to article.)

                        1. 2

                          Every language must have a means of checking both identity and equality … they are fundamentally different things. The choice to make both of them an operator is quite common, the only real distinction Python makes by choosing == and is over, say, == and === is that promotion to also being a keyword.

                          And just because you find it to be uncommon does not mean that it is, in fact, uncommon. I use is all the time, for instance with enum.Enum members, to properly handle default values that need to be mutable, and to identify sentinel values in iterators (usually None, but not if None is a legitimate value of the iterator) … it’s also extremely necessary to things like the Singleton pattern in Python.

                          Moreover you’re throwing out the baby with the bathwater … sure, exactly as you said you can use type(a) == MyClass pretty much anywhere you might use type(a) is MyClass, but why would you? Why necessarily invoke the performance cost of equality comparison? The underlying cost of is is near zero, but by using == you force lookup and execution of MyClass.__eq__ and, if that returns False or NotImplemented you further force lookup and execution of type(a).__eq__ … both of which could be implemented in such a way as to be unbearably time consuming.

                          See the question of identity is, precisely, “are these the same thing” rather than “do these share the same attributes” … they’re fundamentally different questions, and when all you care about is identity, why bother asking for equality?

                      2. 2

                        Arguing that it has to be a keyword because otherwise you would have to write it differently is a weird argument to me. It just means that it would be written not is(a, b) instead, which might not read as nicely, but that’s a different argument.

                        1. 7

                          Perhaps I should have been more specific: is needs to be either a reserved keyword or an operator (in effect it’s both, though it’s not truly an operator because, like and, or, and not it cannot be overloaded) precisely because Guido et all intended it to be used as a syntactic construct… it cannot be a function because it would then have to be used with the call syntax, be able to be assigned to a name, and overloadable in user code. There is no desire for an overloadable identity operation, indeed allowing it to be overloaded would break fundamental language semantics. The same is true of None, which is why it’s also a keyword, though it could just be a globally available variable… in Python the argument for promoting to an operator is “should this bind values around it into an operation” and for promotion to a keyword is “should the meaning of this never even potentially change during runtime” and (optionally) “should this modify other keywords” … in the case of is the answer to all of those is “yes”, so the only option was to make it a keyword.

                          1. 1

                            It seems to me like the same arguments would mean that isinstance should also be a keyword though?

                            1. 3

                              Sure, it could be extended to any number of things — Python 2 didn’t see True or False as keywords, but that protection was extended to them in Python 3 — but would it add any semantic value to do so, by stripping call semantics from it and allowing it to be used as an operator or to directly interact with other keywords?

                              Some keywords have been downgraded (most notably print), but that was because there was added value in doing so, and also a colorable argument for overloading existed. The simple fact is that identity testing is a fundamental operation — much more so than introspecting class phylogeny — and the benefits of being a keyword far outweigh the minor confusion it can cause in relative newbies, while there’d be no advantage to removing it except for people who don’t know why it’s fundamental.

                              In other languages you’ve got equality as == and identity as ===… frankly we’ve got it easy with is.

                          2. 2

                            is being built in means you know it can’t be overridden.

                            It being a function would be ugly

                        1. 2

                          I agree that mutability is often needed (and not only because of evil lawyers). But the reference to Golang seems bit confusing to me. Immutability at a programming language level is completely different kind of immutability than immutability of whole system (e.g. blockchain).

                          Fashion trends and hypes (like „let’s make everything object-oriented/functional/immutable/web/nosql/asynchronous/etc.!“) are usually harmful. There is no one size that would fit all and no holy grail in the software engineering.

                          At programming language or code level both mutable and immutable structures make sense and have its use.

                          Beyond that, things in the real world are almost always mutable and removable because lawyers can show up on your doorstep with a court order to make them so, and the court generally doesn’t care about what problems your choice of technology has created for you in complying.

                          One possible solution of this problem is that you do not own and control that technology and it is running independently of you, it is not your property, like a grown up child that was raised and educated by you, but now it is living independently and you can not stop it or order it something.

                          Whether creating such system is ethical is another question. Sometimes it makes sense…

                          1. 1

                            I apologize; I wrote something that assumed too much context with my link text. It’s referring not to Go the language but to Go’s current design and implementation of a module source code proxy service, which is designed for immutable and potentially permanently cached module source code (even if the upstream vanishes; in fact explicitly partly to cope with the upstream vanishing). The current official proxy is operated by Google with no apparent point of contact to get them to remove something from the proxy’s cache.

                            (I’m the author of the linked to article.)

                          1. 1

                            The hardware required for this meager amount of monitoring makes me want to cry. Why do we keep making things worse?

                            ServiceNow using Xymon has aggregated all their monitoring data into one system that monitors

                            Statistics:
                             Hosts                      : 569869
                             Pages                      : 60421
                             Status messages            : 740185
                            

                            That’s a lot of servers and almost 750,000 services being checked. I remember single server setups on old HP boxes with 8GB of RAM doing 50,000 hosts back in 2008.

                            In the monitoring space we’re building new tools which are considerably less efficient but nobody seems to care.

                            1. 2

                              I’m not sure that 31,000 samples a second is meager, but this server is clearly larger than we need for ingesting samples (as is obvious from the usage level). It needs RAM (and CPU) primarily for complex or long time range queries over the raw metrics data, and doing such queries is both a tradeoff between server resources and response time (Prometheus loads data into memory for query speed), and also between flexibility and pre-computing a limited set of measures and data.

                              The server itself is a basic Dell 1U server (an R230). The actual hardware cost involved is modest and wouldn’t be reduced significantly by making it smaller. Also, system metrics suggest that we rarely go above 6 or 7 GB of RAM these days in active use, so loading it up to 32 GB is probably overkill with current versions of Prometheus. But better safe than sorry with an OOM explosion due to an unexpectedly complex and memory-intensive query.

                              (I’m the author of the linked-to entry.)

                              1. 1

                                I’ve never heard of Xymon but I suppose most people are coming from Nagios/Icinga and I didn’t notice Prometheus being any worse than those setups from the hardware requirements. Not that this is saying a lot, but I see that as the baseline. I heard check_mk was better but I never used it myself.

                                Oh, and one thing - again no idea what Xymon in that setup did, but Prometheus is not primarily or solely used for monitoring but metrics. First of all those will always use a lot more processing power on the receiving end than a simple online/offline check. Not everything can be shoehorned into rrdtool.

                              1. 3

                                I have no idea what “spooky errors at a distance” Chris is talking about.

                                h : α → β
                                g : β → γ
                                f : γ -> δ
                                
                                let v = h foo 
                                let v' = g v
                                let v'' = 
                                

                                Suppose the type of h changes to α → ε. If the type ε is not compatible with β, h foo will fail to typecheck. Coming up with a type that is compatible with everything except γ is hardly easy.

                                1. 4

                                  I think your example is different from what Chris had in mind, but I agree with your point. It’s hard to say that type inference would be problematic without knowing both what the inference algorithm is, and what the code in question is.

                                  I use OCaml a lot and I can’t say I often run into this kind of “spooky errors at a distance”. In practice, even if you don’t use type annotations in the code, it’s good practice to write module interfaces with explicit types. So a change in types in one place will give pretty obvious errors about where to fix it. If you’re not writing interfaces, you’re probably dealing with localized, coupled code, where making the type changes aren’t a big deal and the issues Chris describes aren’t so important.

                                  1. 1

                                    The situation I was thinking of goes something like this. Go has inside a function type inference for return types, so you can write newvar := afunc() and the type of newvar is inferred from afunc()’s return type. Now suppose you extend this to inferring the return type of a function through some syntax (either returning newvar or simply doing return afunc()). This allows you to create a chain of functions with inferred return types; you call afunc() and use its type as your inferred return type, then someone calls you and uses your return type (inferred from afunc()) as their return type, and so on. Then eventually someone at the top of the call chain actually uses a return value in a way that requires it to have a specific type, which is compatible with afunc()’s current return type.

                                    Then afunc() changes its return type. Your function is still type correct, just with a different inferred return type, and so is your caller, and so is your caller’s caller, and so on all the way up to the person who actually uses a return value in a way that is now incompatible with afunc()’s new return type. That person gets a type error.

                                    (To make this easier to happen, you can actually do a lot of generic things with values in Go; for example, as long as a value is some sort of pointer, you can always compare it with nil, and if a type is some sort of a number you can compare it with zero or another constant. This covers a lot of ground in much practical code, right up to the point where someone wants a particular sort of pointer or number for various reasons.)

                                    1. 1

                                      I haven’t used Go but this sounds very similar to languages with “more” type inference (Haskell, for instance) which force you to add type annotations to top-level forms so these errors cannot propagate outside the source file anyway.

                                      I see a distinction between type systems with “proper inference” and the ones that only propagate types “downwards”[*] – can Go infer the type of a variable by its usage later on in the function? (Does it ever need to?)

                                      [*] I think var x = y in C# is an example of this, where it’s “inference” in that it lets you omit declaring the type of x when both you and the compiler know exactly what its type is (it’s the type of y).

                                      1. 2

                                        can Go infer the type of a variable by its usage later on in the function?

                                        No, it can’t. I’ve seen it referred to as “type deduction” instead, although the spec uses no such language (nor does it use the word “inference”): https://golang.org/ref/spec#Variable_declarations

                                        I think “limited inference,” as used by the OP, is an okay phrase to use to describe this. But there is certainly a distinction between the inference used by Go and the inference used, by, say, Haskell.

                                  1. 1

                                    Is uMatrix really worth it over just using uBlock Origin’s medium or hard mode?

                                    1. 2

                                      The one clear area that uBlock Origin’s medium and hard modes don’t seem to cover is cookies (for successful requests). You can clean them up after the fact with Cookie AutoDelete, but they’ll still be sent back to sites you visit before you close things down (or manually delete them).

                                      1. 1

                                        Good to know. Thank you very much.

                                    1. 1

                                      Do you also run NoScript, or is that functionality covered by uBlock Origin?

                                      1. 3

                                        It can be covered by uMatrix, either globally, 1st party, or individual host level.

                                        1. 3

                                          I don’t use NoScript; I used to, but for various reasons replaced it with uMatrix (which I now prefer). uBlock Origin can also block Javascript if you want it to but with less fine-grained control than uMatrix gives you. My opinion is that people who just want to block (most) Javascript and who already use uBlock Origin should probably use it for that, because it’s simpler than uMatrix. I use uMatrix because I want to block things besides Javascript (especially cookies), and I like the fine-grained control.

                                        1. 2

                                          Isn’t uMatrix an advanced version of uBlock Origin? What’s the rationale behind using both of them?

                                          1. 6

                                            The relationship between the two is not quite that straightforward. uMatrix gives you much more fine grained control over blocking types of things, but uBlock Origin has much better built-in ad blocking and filtering than uMatrix does in practice (including many more lists available standard) and will probably do it more efficiently, simply because matching blocklists isn’t uMatrix’s focus. uBlock Origin also has directly built in support for blocking page elements on the fly either permanently or temporarily. You can to some extent control Javascript in uBlock Origin’s dynamic filtering instead of resorting to uMatrix, but I don’t think uO gives you much or any control over cookies and other things.

                                            I use both because I care about both ad-blocking (uBlock Origin’s role, along with zapping away unwanted page elements) and having fine grained control over Javascript, cookies, and other annoyances of the modern web (uMatrix’s role). Although I haven’t tried it, I’m pretty sure my web experience would be significantly worse and more annoying if I only used uMatrix; I don’t think that taking out only Javascript (and cookies) stops enough of the annoyances of the modern web these days. And uBlock Origin blocks things in situations where you have to enable Javascript, such as Youtube.

                                            (I’m the author of the linked-to blog entry.)

                                          1. 1

                                            Perhaps moving to Thanos would be a simpler solution? but I am not sure introducing k8s into the stack…

                                            1. 5

                                              Adding k8s into an environment that doesn’t use it would be a significant increase in complexity, and as far as I know we’d still need to set up a storage backend for Thanos as well. Although I didn’t mention it in the entry, we’re operating at a low enough metrics volume that 13 months of metrics is currently only 758 GB, which is easy enough to hold on regular storage without downsampling or other Thanos features.

                                              (I’m the author of the original article.)

                                            1. 17

                                              Other people are not happy about Go making direct system calls; for example, on Solaris and Illumos, the only officially supported method of making system calls is also through the C library (although Go does it itself on those operating systems).

                                              This is false. Go uses libc on those systems. I wrote the port.

                                              1. 9

                                                I’ve updated my article with a correction (and the error is my fault for relying on a faulty memory instead of trying to check). Thank you for noting this.

                                                (I’m the author of the original article.)

                                              1. 2

                                                Requiring all languages to go through the C library’s normal Unix API for system calls means constraining all languages to live with C’s historical baggage and limits. You could invent a new C library API for all of the system calls that directly wrote the error number to a spot the caller provided, which would make life much simpler, but no major Unix or C library is so far proposing to do this.

                                                IMHO cleaner and proper solution would be a reusable abstraction layer (Go developers probably already had to write such code) that provides nicer API for the syscalls and that could be used also by other languages, not only Go. But of course, nobody can force the Go developers to do such work for others.

                                                1. 4

                                                  The core problem is that this new API needs to be supported by the platform in order to be efficient (and to deal with things like the stack size issue). Otherwise you’re adding a layer of indirection over all C library functions you export without fixing all of the problems and without being able to make strong guarantees, and for at least some functions this slowdown matters.

                                                  (I’m the author of the original article.)

                                                1. 3

                                                  Actually, how does Go handle this on Windows? As far as I know the only kernel API that Windows presents is through C functions or COM.

                                                  1. 8

                                                    On Windows and some other platforms, Go does call through the system libraries. Go can make calls into platform libraries, it just doesn’t like doing it.

                                                    (I’m the author of the linked-to entry.)

                                                    1. 1

                                                      Cool, thanks!

                                                  1. 4

                                                    One of the major issues with XHTML in practice was its requirement for strict error handling, which it inherited from XML and which was generally observed by the browsers. If an XHTML web page didn’t validate, the browser showed you only an error message. As reported in other comments, most ‘XHTML’ web pages didn’t (and were saved from this draconian fate only because they were served in a way that caused browsers to interpret them as HTML instead of XHTML).

                                                    The direct problem with this is that it is both a terrible user experience and directed at the wrong person; it is directly punishing you (the browser user), while only the page authors and site owners have the power to correct the problem. If their entire site is broken, they may get punished indirectly by a traffic volume drop, but if it’s only some pages, well, you get screwed.

                                                    The indirect problem is the implications for this. Because the consequences of invalid XHTML are so severe, the W3C was essentially demanding that everyone change how they created web pages so that they only created valid XHTML. In a world where web pages are uncommon and mostly hand written, perhaps this looked viable. In a world where a very large number of web pages are dynamically generated on the fly, it is not. Major sites with dynamically generated pages were never going to rewrite their page generation systems just to produce assured-valid XHTML, when XHTML gave them essentially nothing in practice except a worse user experience for visitors if something ever went wrong. And even by the mid 00s, the web was far more like the latter than the former.

                                                    (How well people do even today at creating valid XML can be seen by observing how frequently Atom format syndication feeds in the wild are not fully valid XML. Every feed reader that wants to do a really good job of handling feeds does non-strict parsing.)

                                                    1. 5

                                                      If an XHTML web page didn’t validate, the browser showed you only an error message. As reported in other comments, most ‘XHTML’ web pages didn’t (and were saved from this draconian fate only because they were served in a way that caused browsers to interpret them as HTML instead of XHTML).

                                                      This is worth elaborating on a bit. People now mostly think of it in terms of obvious errors, like you forgot to close a tag or quote an attribute. But XHTML had some truly nasty hidden failure modes:

                                                      • Using named character entities, like © for a copyright symbol? You’re now at the mercy of whatever parses your site; a tag-soup HTML parser or a validating XML parser will load and understand the extra named entities in XHTML, but a non-validating XML parser isn’t required to and can error on you (and remember, every error is a fatal error in XML) for using any named entity other than the base five defined in XML itself.
                                                      • Using inline JavaScript (which was common back then)? Well, the content of the script element is declared in the XHTML DTD as PCDATA. Which means you now have to wrap your JavaScript in an explicit CDATA block or else risk well-formedness errors if you use any characters with special meanings. You know, like that < in your for loop.
                                                      • Oh, and speaking of JavaScript: the XHTML DOM is not the same as the HTML DOM. Methods you’re used to for manipulating the HTML DOM will not work in an XHTML document parsed as XHTML, and vice-versa. But you still have to support processing as HTML because not all browsers can handle XHTML-as-XML. Good luck!
                                                      • And while we’re on the subject of content types: did you know the Content-Type header can suddenly make your XHTML documents not be well-formed? Turns out, if you serve as text/html or text/xml, and don’t also specify the charset in your Content-Type header, the consumer on the other end is required to parse your document as ASCII. Even if your XML prolog declares UTF-8. Really. So better not have any bytes in your document outside the ASCII range or else you’ll get an error.

                                                      And that’s just some of the stuff I still remember a decade and a half later. XHTML was a mess.

                                                      1. 1

                                                        XHTML DOM is not the same as the HTML DOM. Methods you’re used to for manipulating the HTML DOM

                                                        Personally client side mangling of the DOM was one of those places where the www truly jumped the shark. Then client side mangling and animation of CSS….

                                                        Shudder.

                                                      2. 3

                                                        I drank the cool-aid and really really tried to do it well. I’ve even used XSLT and XML serializers to generate proper markup. But even when I did everything right, it was undone by proxy servers that “optimized” markup or injected ads (those were dark times for HTTPS). First-party ads didn’t work with the XHTML DOM. Hardly anything worked.

                                                        So in the end users were unhappy, stakeholders were unhappy, and I could have used simpler tools.

                                                        1. 1

                                                          it was undone by proxy servers that “optimized” markup or injected ads (those were dark times for HTTPS). First-party ads didn’t work with the XHTML DOM.

                                                          Well, do be honest you weren’t serving up XHTML then so you can’t blame XHTML for that.

                                                          If there was a flaw in the XHTML design was the inability to insert standalone sub-documents. ie. Like the <img src=“foo.png”> and no matter what was inside there, your document rendered, maybe with a “broken image” icon… but your outer document rendered. You needed for what you’re talking about is a <notMyShit src=”…”> tag that would render whatever was at the other end of that url in a hermetically sealed box same as an image. And if the other shit was shit, a borked doc icon would be fine.

                                                          1. 1

                                                            You mean an iframe?

                                                            1. 1

                                                              Ok, my memory is fading about those dark days…. I see it was available from html4 / xhtml1.0 so basically he had no excuse.

                                                              /u/kornel’s problems didn’t arise from xhtml, it arose from his service providers doing hideous things. So don’t blame xhmtl.

                                                              1. 2

                                                                The blame that can be placed squarely on XHTML is, I think, that of being an unrealistic match for its ecosystem. Hideous behavior from service providers may have occasionally been part of the picture, but a small one compared to a lot of what’s been brought up in this thread.

                                                                1. 2

                                                                  It’s clear from your other comments that you view the existence of any type of scriptable interface to an HTML or XHTML document as a mistake, but the simple fact is that it was already a baseline expected feature of the web platform, which consisted of:

                                                                  • Markup language for document authoring (HTML/XHTML)
                                                                  • Style language for document formatting (CSS)
                                                                  • An API for document manipulation (DOM)

                                                                  Ad networks, and many other things, already made use of the DOM API for the features they needed/wanted.

                                                                  And then XHTML came along, and when served as XHTML it had a DOM which was different from and incompatible with the HTML DOM, which meant it was difficult and complex to write third-party code which could be dropped into either an HTML document, an XHTML-served-as-HTML document, or an XHTML-served-as-XHTML document.

                                                                  1. 2

                                                                    Ad networks, and many other things, already made use of the DOM API for the features they needed/wanted.

                                                                    Yup. Ad networks as well as those features were never a thing I have needed or wanted….

                                                                    API for document manipulation

                                                                    Never needed or wanted that except as a rubber crutch given to cripple to overcome the fact that html as a standard had completely stalled and stopped advancing on any front anybody wanted.

                                                                    difficult and complex to write third-party code which could be dropped into either an HTML document

                                                                    And the 3rd party code was written as a hideous kludge to overcome the stalled html standard.

                                                                    It’s amazing gobsmacking what they have achieved with say d3.js …. but that is despite the limitations rather than because. If I look at the code for d3 and look at the insane kludges and hacks they do and compared to other better graphics apis… I literally cry for the wasted time and resources.

                                                              2. 1

                                                                I was serving application/xhtml+xml, but the evil proxies either thought they support it or sniffed content.

                                                                HTML5 actually added <iframe srcdoc="">, but it’s underwhelming due to iframe’s frameness.

                                                            2. 2

                                                              In a world where web pages are uncommon and mostly hand written, perhaps this looked viable. In a world where a very large number of web pages are dynamically generated on the fly, it is not.

                                                              On the contrary, I would expect that any dynamically-generated site should be able to quite easily generate valid XML, while any sufficiently-complex hand-written XML will likely have at least one validation error.

                                                              If it’s really that difficult to generate well-formed XML … maybe we should have just dumped it and stuck with S-expressions?

                                                              Correctness matters, particularly with computers which handle people’s sensitive information.

                                                              1. 3

                                                                People who hand write web pages do so in small volume, and can reasonably be pushed to run validators after they save or switch to an XHTML-aware editing mode or web page editor. Or at least that is or was the theory, and somewhat the practice of people who did produce valid XHTML.

                                                                Software that produces HTML through templates, which is extremely common, must generally be majorly rewritten to restructure its generation process to always produce valid XHTML. At scale, text templating is not compatible with always valid XHTML; the chance for mistakes, both in markup and in character sets, is too great. You need software that simply doesn’t allow invalid XHTML to be created no matter what, and that means a complete change in template systems and template APIs. Even if you can get away without that, you likely need to do major rewrites inside the template engine itself. Major rewrites are not popular, especially when they get you nothing in practice.

                                                                1. 2

                                                                  Correctness matters

                                                                  It does, but the presentation layer is not the right place to enforce it.

                                                                2. 1

                                                                  It was ahead of its time. I think strict validation will maybe be an option in… let’s say 2025 or 2030. We still have a long way to go before people consistently use software that produces HTML the same way as JSON—which is to say, via serialization of a data structure.

                                                                  We’re slowly, slowly getting there.

                                                                  1. 4

                                                                    I don’t think it’s ever going to happen for HTML, because there’s no benefit. At all.

                                                                    XML was meant to solve the problem of unspecified and wildly different error handling HTML clients, but HTML5 solved it by specifying how to parse every kind of garbage instead (including what exactly happens on 511th nested <font> tag).

                                                                    XML parsers were supposed to be simpler than parsers that handle all the garbage precisely, but we’ve paid that cost already. Now it’s actually easier to run HTML5ever parser than to fiddle with DTD catalogs to avoid XML choking on &nbsp;.

                                                                    We have some validation is some template engines and JSX, but that’s where it should be — in developer tooling, not in browsers.

                                                                  1. 2

                                                                    Thank you for this. I’m not in a position to watch videos, so having slides to go through (transcripts are even better) is just fantastic. Pity that there are several FreeBSD submissions, all behind video tag.

                                                                  1. 2

                                                                    The thing that scares me about SSH certificates for user authentication is the same thing that scares me about all local CAs, which is having unknown signed certificates floating around out there if it’s ever compromised (whether or not you detect that). With properly controlled user keys you can at least get an authoritative list, and if you want to you can constrain it to keys held on hardware tokens.

                                                                    (There are ways to watch for problems with certificates, such as scanning your logs and checking that all accepted certificates are known ones.)

                                                                    1. 8

                                                                      I’ve said it before and I’ll say it again: ZFS should be the default on all Linux distros. It’s in a league of its own, and makes all other existing Linux filesystems irrelevant, bizarre licensing issues be damned.

                                                                      1. 7

                                                                        I use ZFS and love it. But I disagree that ZFS should be the default as-is. It requires a fair bit of tuning. For non-server workloads, the ARC in particular. ZFS does not use Linux’ buffer cache and while ARC size adapts, I have often seen on lower memory machines that the ARC takes too much memory at a given point, leaving too little memory for the OS and applications. So, most users would want to tune zfs_arc_max for their particular workload.

                                                                        I do think ZFS should be available as an option in all Linux distributions. It is simply better than the filesystems that are currently provided in the kernel. (Maybe bcachefs will be a competent alternative in the future.)

                                                                        1. 2

                                                                          I agree.

                                                                          I remember installing FreeBSD 11 once (with root on ZFS) because I needed a machine remotely accessible via SSH to handle files on an existing disk with ZFS.

                                                                          No shizzle, FreeBSD defaults, the machine had 16G of RAM, and during an hours long scp run, ARC decided to eat up all the memory, triggering the kernel into killing processes… including SSH.

                                                                          So I lost access, had to restart scp again (no resume, remember), etc. This is a huge show stopper and it should never happen.

                                                                          1. 1

                                                                            That seems like a bug that should be fixed. Don’t see any reason why that should prevent it from being the default though.

                                                                          2. 1

                                                                            That’s definitely something to consider, however, Apple has made APFS (ZFS inspired) the default on macOS, so there’s got to be a way to make it work for ZFS + Linux Desktop too. ZFS is all about making things work without you having to give it much thought. Desktop distros can pick reasonable defaults for desktop use, and ZFS could possibly make the parameter smarter somehow.

                                                                          3. 2

                                                                            I think the licensing issue is the primary problem for Linux distros.

                                                                            1. 1

                                                                              I agree on technical superiority. What about the Oracle threat given its owner pulled off that API trick? Should we take the risk of all owing Oracle’s lawyers money in some future case? Or rush to implement something different that they don’t control with most of its strengths? I think the latter makes the most sense in the long-term.

                                                                              1. 3

                                                                                Oracle is not a problem, as the ZFS license is not being violated – it is the Linux license.

                                                                                1. 1

                                                                                  “Oracle is not a problem, as the ZFS license is not being violated”

                                                                                  That’s a big claim to make in the event large sums of money are ever involved. Oracle threw massive amounts of lawyers at Google with the result being API’s were suddenly a thing they could copyright. Nobody knew that before. With enough money and malicious intent, it became a thing that could affect FOSS developers or anyone building on proprietary platforms. What will they do next?

                                                                                  I don’t know. Given they’re malicious, the safest thing is to not use anything they own or might have patents on. Just stay as far away from every sue-happy party in patent and copyright spaces. Oracle is a big one that seeks big damages for its targets on top of trying to rewrite the law in cases. I steer clear of their stuff. We don’t even need it, either. It’s just more convenient than alternatives.

                                                                                  1. 8

                                                                                    The CDDL, an OSI-approved open source licensed, includes both a copyright and patent grant for all of the code released by Sun (now Oracle). Oracle have sued a lot of people for a lot of things, but they haven’t come after illumos or OpenZFS and there are definitely companies using both of those bodies of software to make real money.

                                                                                    1. 2

                                                                                      I think you’re missing the implications of they effectively rewrote the law in the case I referenced. If they can do that, it might not matter what their agreements say if it’s their property. The risk might be low enough that it never plays out. One just can’t ever know if they depend on legal provisions with a malicious party that tries to rewrite laws in its favor with lobbyists and lawyers.

                                                                                      And sometimes succeeds unlike basically everyone doing open source and free software. Those seem to barely enforce their agreements and/or be vulnerable to patent suits in case of the permissive licenses. Plus, could the defenders even afford a trial at the current rates?

                                                                                      I bet 10 years ago you wouldn’t have guessed a mobile supplier using an open-ish platform would be fighting to avoid giving over $8 billion dollars to an enterprise-focused, database company. Yet, untrustworthy dependencies let that happen. And we got lucky it was a rich company that depended on OSS/FOSS stuff defending. The rulings could’ve been worse for us if it wasn’t Google.

                                                                                      1. 6

                                                                                        Seeing as Sun gave ZFS away before Oracle bought it, Oracle would have a LOT of legal wackiness to get the CDDL license revoked somehow. But for the safe of argument, let’s assume they do manage somehow to make it invalidated, and went nuts and decided to try and charge everyone currently using ZFS pay bajillions of dollars for “their” tech. Laws would have to change significantly for that to happen, and with such a significant change in current law, there is basically zero chance it would be retro-active from the moment you started using ZFS, so worst case you’d have to pay from the time of the law change. That is if you didn’t just move off of ZFS after the law changed and be out zero dollars.

                                                                                        Also, the OSS version of ZFS is significantly different from Oracle’s version that they are sort of kissing cousins at best anymore. ZFS has been CDDL licensed since 2005, so a long history of divergence from the Oracle version. I think Oracle would have a VERY hard time getting the OSS version back under the Oracle banner(s). Even with very hypothetical significant law changes.

                                                                                        I’m in favour of things competing against ZFS, but currently nothing really does.. BTRFS tries, but their stability record is pretty miserable for anything besides the simplest workloads. ZFS has had wide production usage since 2001. Maybe in another 5 or 10 years we will have a decent stable competitor to some of ZFS’s feature-sets.

                                                                                        But regardless if you are a large company with something to lose, your lawyers will be the ones advising you about using ZFS or not, and Canonical’s lawyers clearly decided there was nothing to worry about, Along with Samsung(who own Joyent, the people behind Illumos). There are also many other large companies that have bet big on Oracle having basically zero legal leg to stand on.

                                                                                        Of course the other side of the coin is the ZFS <-> Linux marriage, but that’s easy just don’t run ZFS under Linux, or use the Canonical shipped version and let Canonical take all the legal heat.

                                                                                        1. 2

                                                                                          Best counterpoints so far. I’ll note this part might not be as strong as you think:

                                                                                          “and Canonical’s lawyers clearly decided there was nothing to worry about, Along with Samsung(who own Joyent, the people behind Illumos)”

                                                                                          The main way companies dodge suits is to have tons of money and patents themselves to make the process expensive as hell for anyone that tries. Linux companies almost got patent sued by Microsoft. IBM, a huge patent holder, stepped up saying they’d deal with anyone that threatened it. They claimed they were putting a billion dollars into Linux. Microsoft backed off. That GPL companies aren’t getting sued made Canonical’s lawyers comfortable but not an actual assurance. Samsung is another giant, patent holder with big lawyers. It takes an Apple-sized company to want to sue them.

                                                                                          So, big, patent holders or projects they protect are outliers. That might work to ZFS’s advantage here. Especially if IBM used it. They don’t prove what will happen with smaller companies, though.

                                                                                          1. 2

                                                                                            I agree with you in theory, but not in practice because of the CDDL (which ZFS is licensed under). This license explicitly grants a “patent peace” see: https://en.wikipedia.org/wiki/Common_Development_and_Distribution_License

                                                                                            I know most/many OSS licenses sort of wimp out on patents and ignore the problem, CDDL doesn’t. Perhaps it could have even stronger language, and there might be some wiggle room for some crazy lawyering.. I just don’t really see Oracle being THAT crazy. Oracle, being solely focused on $$$$, would have to see some serious money bags to go shake loose, I doubt they would ever bother going after anyone not the size of a Fortune 500, the money just isn’t there. Google has giant bags full of money they don’t even know what to do with, so Oracle trying to steal a few makes sense. :P

                                                                                            Oracle going after Google makes sense knowing Oracle, and it was , like you said, brand new lawyering, trying to create API’s out of Copyrights. Patents are not remotely new. So some lawyer for Oracle would have to dream up some new way to screw up laws to their advantage. Possible sure, but it would be possible for any other crazy lawyer to go nuts here (wholly unrelated to ZFS or even technology), it’s not an Oracle exclusive idiocy. Trying to avoid unknown lawyering that’s not even theoretical at this point would be sort of stupid I would think… but I’m not a lawyer.

                                                                                            1. 1

                                                                                              “I know most/many OSS licenses sort of wimp out on patents and ignore the problem, CDDL doesn’t.”

                                                                                              That would be re-assuring on patent part.

                                                                                              “Possible sure, but it would be possible for any other crazy lawyer to go nuts here (wholly unrelated to ZFS or even technology), it’s not an Oracle exclusive idiocy. Trying to avoid unknown lawyering”

                                                                                              Oracle was the only one to flip software copyright on its head like this. So, I don’t think it’s an any company thing. Either way, the threat I’m defending against isn’t unknown lawyering in general: it’s unknown lawyering of a malicious company whose private property I may or may not depend on. When you frame it that way, one might wonder why anyone would depend on a malicious company at all. Avoiding that is a good pattern in general. Then, the license negates some amount of that potential malice for a great product with unknown, residual risk.

                                                                                              I agree the residual risk probably won’t affect individuals, though. An Oracle-driven risk might affect small to mid-sized businesses depending on how it plays out. Good news is swapping filesystems isn’t very hard on Linux and BSD’s. ;)

                                                                                    2. 4

                                                                                      AFAIK, it’s the GPL that’s being violated. But I’m really tired and the SFC does mention something about Oracle suing so 🤷.

                                                                                      Suing based on the use of works derived from Oracle’s CDDL sources would be a step further than the dumb Google Java lawsuit because they haven’t gone after anyone for using OpenJDK-based derivatives of Java. Oracle’s lawsuit-happy nature would, however, mean that a reimplementation of ZFS would be a bigger target because it doesn’t have the CDDL patent grant. Of course, any file system that implements one of their dumb patents could be at risk….

                                                                                      I miss Sun!

                                                                                2. 1

                                                                                  What does ZFS have that is so much better than btrfs?

                                                                                  I’m also not sure these types of filesystems are well suited for databases which implement their own transactions and COW, so I’m not sure I would go as far as saying they are all irrelevant.

                                                                                  1. 11

                                                                                    ZFS is extremely stable and battle-tested, while that’s not a reason in itself to make it a better filesystem, it makes it a extremely safe option when what you’re looking for is something stable to keep your data consistent.

                                                                                    It is also one of the most cross-platform file system. Linux, FreeBSD, MacOS, Windows Illumos. It has a huge amount of development behind it, and as of recently the community has come together significantly across the platforms. Being able to export your pool on FreeBSD and import it on Linux or another platform makes it a much better option if you want to avoid lock-in.

                                                                                    Additionally, the ARC

                                                                                    Problems with btrfs that make it not ready:

                                                                                    1. 0

                                                                                      If I don’t use/want to use RAID5 then I don’t see the problem with btrfs.

                                                                                      1. 3

                                                                                        I ran btrfs in production on my home server for ~3-4 years, IIRC. If you want to use btrfs as a better ext4, e.g. just for the compression and checksumming and maybe, maybe snapshotting, then you’re probably fine. If you want to do anything beyond that, I would not trust it with your data. Or at the very least, I wouldn’t trust it with your data that’s not backed up using something that has nothing to do with btrfs (i.e. is not btrfs snapshots and is not btrfs send/receive).

                                                                                        I had three distinct crashes/data corruption problems that damaged the filesystem badly enough that I had to back up and run mkfs.btrfs again. These were mostly caused by interruptions/power failures while I was making changes to the fs, for example removing a device or rebalancing or something. Honestly I’ve forgotten the exact details now, otherwise I’d say something less vague. But the bottom line is that it simply lacks polish. And mind you, this is from the filesystem that is supposed to be explicitly designed to resist this kind of corruption. I know at least the last case of corruption I had (which finally made me move to ZFS) was obviously preventable but that failure handling hadn’t been written yet and so the fs got into a state that the kernel didn’t know how to handle.

                                                                                    2. 3

                                                                                      well, I don’t know about better, but ZFS has the distinct disadvantage of being out of tree filesystem so it can and will break depending completely on the whims of kernel development. How anyone can call this stable and safe for production use is beyond me.

                                                                                      1. 3

                                                                                        I think the biggest argument is mature implementations used by large numbers of people. That catches lots of common and uncommon problems. In reliability-focused filesystems, that the reliability is field-proven then constantly maintained is more important to me than about anything. The only reason I don’t use it is that it came from Oracle with all the legal unknowns that can bring down the line.

                                                                                        1. 3

                                                                                          When you say “Oracle”, are you referring to ZFS or btrfs? ;)

                                                                                          1. 1

                                                                                            Oh shit! I didn’t know they designed both! Glad I wasn’t using btrfs either. Thanks for the tip haha.

                                                                                        2. 2

                                                                                          On a practical level, ZFS is a lot more tested (in Solaris/Illumos, FreeBSD, and now Linux); more different people have put more terabytes of data in and out of ZFS than they seem to have for btrfs. This matters because we seem to be unable to build filesystems that don’t run into corner cases sooner or later, so the more time and data a filesystem has handled, the more corner cases have been turned up and fixed.

                                                                                          On a theoretical level, my personal view is that ZFS picked a better internal structure for how its storage is organized and managed than btrfs did (unless btrfs drastically changed things since I last looked several years ago). To put it simply, ZFS is a volume manager first and then a filesystem manager second (on top of the volumes), while btrfs is (or was) the other way around (you manage filesystems and volumes are a magical side effect). ZFS’s model does more (obvious) violence to Linux IO layering than I think btrfs’s does, but I strongly believe it is the better one and gives you cleaner end results.

                                                                                        3. 0

                                                                                          Why would I want to run ZFS on my laptop?

                                                                                          1. 1

                                                                                            Why wouldn’t you want to run it on your laptop?