1. 7

    Some of the dormant shells are more ‘mostly finished’ than completely dormant. The one I know about for sure is rc; the modern Unix versions are pleasantly usable, although they still don’t have job control. I believe there’s another version of rc in the Plan 9 ports collection.

    1. 2

      For rc, I’d suggest looking here: https://github.com/benavento/rc

      The plan 9 version of rc largely doesn’t need the interactive features, because they’re outsourced to the window system. The unix version needs them, and most ports don’t add them. This one does.

      And yes – it largely is done. There are some changes committed occasionally in 9front, but for the most part, we seem pretty happy with it.

      1. 1

        The version of rc I personally use has a number of changes cherry-picked from Bert Münnich’s features, which are on top of the base Byron Rakitzis version. Bert Münnich has a number of highly useful features (including quite good command completion); the repo is here.

        (Correct command completion for rc is harder than it looks because rc allows command paths like ‘a/b’, which search for that through $PATH. Most people probably don’t use that rc feature, but I do in order to namespace my personal commands and scripts.)

    1. 2

      Why can’t statically linked Go programs look up hostnames? Is it because of how security is set up on OpenBSD?

      1. 10

        In general, full support for looking up hostnames requires calling the platform’s C library functions. Go currently only does this in dynamically linked programs; in statically linked programs it uses a pure Go resolver that supports only a subset of the possible hostname lookup features.

        (I’m the author of the linked-to article.)

        1. 5

          Not sure about OpenBSD specifically, but on a lot of *NIX systems name lookup is handled via NSS, which dynamically loads libraries for libc to use: this allows you to extend name lookup without recompiling. Solaris also uses library loading to be able to get locale support in libc.

          It’s interesting that this doesn’t mention macOS. All Go programs on macOS broke a while ago because the syscall parameters for gettimeofday changed and Go didn’t go via libSystem and so didn’t pick up the change.

        1. 3

          I think this debate needs a bit more data, so I’m considering making a “call to arms for Python OO enthusiasts” at some point, my plan is:

          • make a github repo, people raise PRs with their code (do I require tests?)
          • their code should be the most idiomatic OO, least refactorable into data + functions
          • no operator overloading specific stuff
          • max 500 lines + I’m only going to look at 5 submissions - I’ve not got all the time in the world
          • I attempt to expunge the OO and present my results - the mob decides

          Is this fair or useful or just dumb?

          1. 3

            I’d play. I mostly agree with your article but think there’s a better case to be made for OO with immutable value objects. Meaning: I think I could write object solutions that would be more or less equivalent to any bag-of-functions approach, both in readability and verbosity.

            1. 1

              I don’t think I could fit a readable submission into 500 lines, but a common (and classical) use for OO is polymorphic evaluation of simple abstract syntax trees for small expression/rule languages. Each different meaningful term or syntax element is a class, and every class has an .Eval() method (usually they take a context object as the argument). To evaluate an expression or a rule, you .Eval() the top level object for the entire rule’s AST, and it .Eval()s its children as appropriate, and so on. Nothing has to know anything about what all of the different terms, operators, elements, and so on are, which makes it quite easy to add more, tinker with the internal implementations, and so on.

              The nicest Python non-OO version I can think of would be to have the AST nodes be bag-of-data objects (different data for different types of nodes) with a common ‘type’ field that named their type. Then you would have a big dict mapping types to their evaluation functions; these functions would take the node bag-of-data object and a context object as arguments. You could also implement a giant ‘eval’ function that knew all of the node types and how to evaluate each one, but that gets ugly.

            1. 8

              Tomorrow seems to be a very bad day for all those poor souls, who didn’t have time/resources to switch to py3 yet. Fortunately enough it can be easily fixed with pip<21 but it will definitely add additional grey hairs to some heads.

              1. 7

                As one of those poor souls, thanks. We have eight years of legacy code that Just Works and so seldom gets touched, and a major 3rd party framework dependency that hasn’t updated to Python 3 either. We just got permission and funding to form a new engineering sub-group to try to deal with this sort of thing, and upper management is already implicitly co-opting it to chase new shinies.

                1. 9

                  Python 3.0 was released in 2008. I personally find it hard to feel sympathy for anyone who couldn’t find time in the last twelve years to update their code, especially if it’s code they are still using today. Even more so for anyone who intentionally started a Python 2 project after the 3.0 ecosystem had matured.

                  1. 9

                    Python 2.7 was released in 2010. Python 3.3 in 2012. Python 2.6 last release was in 2013. Only from this date people could easily release stuff compatible with both Python 2 and Python 3. You may also want to take into consideration the end of support date of some of the distributions shipping Python 2.6 and not Python 2.7 (like Debian Squeeze, 2016).

                    I am not saying that 8 years is too fast, but Python 3.0 release date is mostly irrelevant as the ecosystem didn’t use it.

                    1. 7

                      Python 3.0 was not something you wanted to use; it took several releases before Python 3 was really ready for people to write programs on. Then it took longer for good versions of Python 3 to propagate into distributions (especially long term distributions), and then it took longer for people to port packages and libraries to Python 3, and so on and so forth. It has definitely not been twelve years since the ecosystem matured.

                      Some people do enough with Python that it’s sensible for them to build and maintain their own Python infrastructure, so always had the latest Python 3. Many people do not and so used supplied Python versions, and may well have stable Python code that just works and they haven’t touched in years (perhaps because they are script-level infrastructure that just sits there working, instead of production frontend things that are under constant evolution because business needs keep changing).

                      1. 4

                        Some of our toolchain broke in the last few weeks. We ported to python3 ages ago, but chunks of infrastructure still support both, and some even still default to 2. The virtualenv binary in Ubuntu 18.02 does that; and that’s a still-supported Ubuntu version, and the default runner for GitHub CI.

                        I think python2-related pain will continue for years to come even for people who have done the due diligence on their own code.

                        1. 4

                          Small tip regarding virtualenv: Since python 3.3 virtualenv comes bundled as the venv module in python, so you can just use python -m venv instead of virtualenv, then you are certain it matches the python version you are using.

                          1. 1

                            virtualenv has some nice features which do not exist for venv. One of the examples is activate_this.py script, which can be used for configuration of remote environment, similar to what pytest_cloud does.

                            1. 1

                              virtualenv has some nice features which do not exist for venv

                              Huh, thanks for pointing that out. I haven’t been writing so much Python in the last few years, and I totally thought venv and virtualenv were the same thing.

                        2. 4

                          Consider, at a minimum, the existence of PyPy; PyPy’s own position is that PyPy will support Python 2.7 forever because PyPy is written in RPython, a strict subset of Python 2.7.

                          Sympathy is not required; what you’re missing out on is an understanding that Python is not wholly under control of the Python Software Foundation. By repeatedly neglecting PyPy, the PSF has effectively forced them to create their own parallel Python 2 infrastructure; when PyPI finally makes changes which prevent Python 2 code from deploying, then we may see PyPy grow even more tooling and possibly even services to compensate.

                          It is easy for me to recognize in your words an inkling of contempt for Python 2 users.

                          1. 21

                            Every time you hop into one of these threads, you frame it in a way which implies you think various entities are obligated to maintain a Python 2 interpreter, infrastructure for supporting Python 2 interpreters, and versions of third-party packages which stay compatible with Python 2, for all of eternity.

                            Judging from that last thread, you seem to think I am one of the people who has that obligation. Could you please, clearly, state to me the nature of this obligation – is its basis legal? moral? something else? – along with its origin and the means by which you assume the right to impose it on me.

                            I ask because I cannot begin to fathom where such an obligation would come from, nor do I understand why you insist on labeling it “contempt” when other people choose not to maintain software for you, in the exact form you personally prefer, for free, forever, anymore.

                            1. 2

                              Your sympathy, including any effort or obligation that you might imagine, is not required. I don’t know how to put it any more clearly to you: You have ended up on the winning side of a political contest within the PSF, and you are antagonizing members of the community who lost for no other reason than that you want the political divide to deepen.

                              Maybe, to get some perspective, try replacing “Python 2” with “Perl 5” and “Python 3” with “Raku”; that particular community resolved their political divide recently and stopped trying to replace each other. Another option for perspective: You talk about “these threads”; what are these threads for, exactly? I didn’t leave a top-level comment on this comment thread; I didn’t summon you for the explicit purpose of flamewar.

                              Finally, why not reread the linked thread? I not only was clearly the loser in that discussion, but I also explained that I personally am not permanently tied to Python 2, and that I’m trying to leave the ecosystem altogether in order to avoid these political problems. Your proposed idea of obligation towards me is completely imagined and meant to make you seem like a victim.

                              Here are some quotes which I think display contempt towards Python 2 and its users, from the previous thread (including your original post) and also the thread before that one:

                              If PyPy wants to internally maintain the interpreter they use to bootstrap, I don’t care one way or another. But if PyPy wants that to also turn into broad advertisement of a supported Python 2 interpreter for general use, I hope they’d consider the effect it will have on other people.

                              Want to keep python 2 alive? Step up and do it.

                              What do you propose they do then? Extend Python 2 support forever and let Python 2 slow down Python 3 development for all time?

                              That’s them choosing and forever staying on a specific dependency. … Is it really that difficult for Python programmers to rewrite one Python program in the newer version of Python? … Seems more fair for the project that wants the dependency to be the one reworking it.

                              The PyPy project, for example, is currently dependent on a Python 2 interpreter to bootstrap and so will be maintaining their own either for as long as PyPy exists, or for as long as it takes to migrate to bootstrapping on Python 3 (which they seem to think is either not feasible, or not something they want to do).

                              He’s having a tantrum. … If you’re not on 3, it’s either a big ball of mud that should’ve been incrementally rewritten/rearchitected (thus exposing bad design) or you expected an ecosystem to stay in stasis forever.

                              I’m not going to even bother with your “mother loved you best” vis a vis PyPy.

                              You’re so wrapped up in inventing enemies that heap contempt on you, but it’s just fellow engineers raising their eyebrows at someone being overly dramatic. Lol contempt. 😂😂😂

                              If I didn’t already have a long history of knowing other PyPy people, for example, I’d be coming away with a pretty negative view of the project from my interactions with you.

                              What emotional word would you use to describe the timbre of these attitudes? None of this has to do with maintainership; I don’t think that you maintain any packages which I directly require. I’m not asking for any programming effort from you. Indeed, if you’re not a CPython core developer either, then you don’t have the ability to work on this; you are also a bystander. I don’t want sympathy; I want empathy.

                              1. 6

                                You have ended up on the winning side of a political contest within the PSF, and you are antagonizing members of the community who lost for no other reason than that you want the political divide to deepen.

                                And this is where the problem lies. Your behavior in the previous thread, and here, makes clear that your approach is to insult, attack, or otherwise insinuate evil motives to anyone who disagrees with you.

                                Here are some quotes which I think display contempt towards Python 2 and its users

                                First of all, it’s not exactly courteous to mix and match quotes from multiple users without sourcing them to who said each one. If anyone wants to click through to the actual thread, they’ll find a rather different picture of, say, my engagement with you. But let’s be clear about this “contempt”.

                                In the original post, I said:

                                The PyPy project, for example, is currently dependent on a Python 2 interpreter to bootstrap and so will be maintaining their own either for as long as PyPy exists, or for as long as it takes to migrate to bootstrapping on Python 3 (which they seem to think is either not feasible, or not something they want to do).

                                You quoted this and replied:

                                This quote is emblematic of the contempt that you display towards Python users.

                                I remain confused as to what was contemptuous about that. You yourself have confirmed that PyPy is in fact dependent on a Python 2 interpreter, and your own comments seem to indicate there is no plan to migrate away from that dependency. It’s simply a statement of fact. And the context of the quote you pulled was a section exploring the difference between “Python 2” the interpreter, and “Python 2” the ecosystem of third-party packages. Here’s the full context:

                                Unfortunately for that argument, Python 2 was much more than just the interpreter. It was also a large ecosystem of packages people used with the interpreter, and a community of people who maintained and contributed to those packages. I don’t doubt the PyPy team are willing to maintain a Python 2 interpreter, and that people who don’t want to port to Python 3 could switch to the PyPy project’s interpreter in order to have a supported Python 2 interpreter. But a lot of those people would continue to use other packages, too, and as far as I’m aware the PyPy team hasn’t also volunteered to maintain Python 2 versions of all those packages.

                                So there’s a sense in which I want to push back against that messaging from PyPy folks and other groups who say they’ll maintain “Python 2” for years to come, but really just mean they’ll maintain an interpreter. If they keep loudly announcing “don’t listen to the Python core team, Python 2 is still supported”, they’ll be creating additional burdens for a lot of other people: end users are going to go file bug reports and other support requests to third-party projects that no longer support Python 2, because they heard “Python 2 is still supported”, and thus will feel entitled to have their favorite packages still work.

                                Even if all those requests get immediately closed with “this project doesn’t support Python 2 anymore”, it’s still going to take up the time of maintainers, and it’s going to make the people who file the requests angry because now they’ll feel someone must be lying to them — either Python 2 is dead or it isn’t! — and they’ll probably take that anger out on whatever target happens to be handy. Which is not going to be good.

                                This is why I made comments asking you to consider the effect of your preferred stance on other people (i.e., on package maintainers). This is why I repeated my point in the comments of the previous thread, that an interpreter is a necessary but not sufficient condition for saying “Python 2 is still supported”. I don’t think these are controversial statements, but apparently you do. I don’t understand why.

                                I also still don’t understand comments of yours like this one:

                                Frankly, I think that you show your hand when you say “really important packages like NumPy/SciPy.” That’s the direction that you want Python to go in.

                                Again, this is just a statement of fact. There are a lot of people using Python for a lot of use cases, and many of those use cases are dependent on certain domain-specific libraries. As I said in full:

                                So regardless of whether I use them or not, NumPy and SciPy are important packages. Just as Jupyter (née IPython) notebooks are important, even though I don’t personally use them. Just as the ML/AI packages are important even though I don’t use them. Just as Flask and SQLAlchemy are important packages, even though I don’t use them. Python’s continued success as a language comes from the large community of people using it for different things. The fact that there are large numbers of people using Python for not-my-use-case with not-the-libraries-I-use is a really good thing!

                                Your words certainly imply you think it’s a bad thing that there are, for example, people using NumPy and SciPy, or at least that you think that’s a bad direction for Python to go in. I do not understand why, and you’ve offered no explanation other than to hand-wave it as “contempt” and “denigration”.

                                But really the thing I do not understand is this:

                                You have ended up on the winning side of a political contest within the PSF

                                You seem to think that “the PSF” and/or some other group of people or entities in the Python world are your enemy, because they chose to move to Python 3 and to stop dedicating their own time and resources to maintaining compatibility with and support for Python 2. The only way that this would make any sense is if those entities had some sort of obligation, to you or to others, to continue maintaining compatibility with and support for Python 2. Hence I have asked you for an explanation of the nature and origin of that obligation so that I can try to understand the real root of why you seem to be so angry about this.

                                Admittedly I don’t have high hopes for getting such an explanation, given what happened last time around, but maybe this time?

                                1. 4

                                  Your behavior in the previous thread, and here, makes clear that your approach is to insult, attack, or otherwise insinuate evil motives to anyone who disagrees with you.

                                  As Corbin has said themselves multiple times, they are not a nice person. So unfortunately you can’t really expect anything better than this.

                        3. 2

                          Why will tomorrow be a bad day? pip will continue to work. They’re just stopping releasing updates.

                          1. 1

                            From my OpenStack experience – many automated gates could go south, because they could do something like: pip install pip --upgrade hence dropping support for py2. I know, that whomever is involved in this conundrum, should know better and should introduce some checks. But I also know, that we’re all humans, hence prone to make errors.

                            1. 2

                              pip install pip --upgrade should still work, unless the pip team screwed something up.

                              When you upload something to PyPI, you can specify a minimal support Python version. So Python 2.7 will get the latest version that still supports Python 2.

                              And indeed, if you go to https://pypi.org/project/pip/ you will see “Requires: Python >= 3.6”, so I expect things will Just Work for most Python 2 users.

                        1. 4

                          It’s worth mentioning that the history of the MIPS ISA is more complex than this article makes it out. MIPS (and the company involved) was originally an independent ISA and CPU maker that scored design wins in both DEC and SGI machines, but had problems keeping up in the early 1990s. DEC developed its own Alpha architecture and SGI bought MIPS to secure the CPUs it was using. One view of the spin-out of MIPS from SGI in the late 1990s was as a tacit admission by SGI that MIPS CPUs could no longer compete as things stood and so controlling MIPS was no longer essential or even useful for SGI.

                          1. 9

                            Interesting tangental thought:

                            I saw this article and the URL and was like, “oh its the guy who hates Go again!” But when I looked at the actual website this author actually writes a significant amount about Go, the vast majority not negative. It seems like that the users of lobsters seem to be only interested in posting and upvoting anti-Go articles, and it is we who are biased.

                            1. 4

                              I’m the author of the linked-to article, and as you sort of noticed I’m pretty fond of Go (and use it fairly frequently). If I wasn’t, I wouldn’t write about it much or at all. I suspect that part of the way my writing about Go has come out the way it has is that I think it’s a lot easier to write about problems (including mistakes that you can make) than about things that are good and just work.

                              (We have one core program in our fileserver environment that’s written in Go, for example, but it’s pretty boring; it sits there and just works, and Go made writing it straightforward, with simple patterns.)

                              1. 3

                                I quite fond of Go myself, and I enjoy reading your articles even when I thought you only talked about the problems! 😂

                                I think negative language news (especially regarding “newer” languages) has more success on this site, so there’s a secondary filtering happening as well.

                              2. 0

                                who’s the guy who hates Go?

                              1. 2

                                Is there no mode that would share the physical network port but tag all IPMI traffic with a VLAN you configure?

                                1. 6

                                  Many HPE servers have a dedicated network ports for the iLO card but can also optionally share one of the regular network ports if needed. When in shared mode, you can indeed configure a VLAN tag for the management traffic, which can be different to the VLAN tag used by the host operating system normally.

                                  1. 1

                                    Unfortunately, in the same way that chris explained that a any compromised host might be able to switch the device IPMI mode from dedicated to shared, using a VLAN for segregation can have a similar problem. If the compromised host adds a sub-interface with the tagged VLAN to their networking stack they now can gain network access to the entire IPMI VLAN.

                                    1. 2

                                      In addition there are other annoyance with using a shared interface. Because the OS has control of the NIC it can reset the PHY. If the PHY is interrupted while, for example, you’re connected over Serial over LAN or a virtual KVM, you lose access. If you’re lucky, that’s temporary. If you’re really unlucky the OS can continually reset the PHY making IPMI access unusable. A malicious actor could abuse this to lock out someone from remote management.

                                      That can’t happen when you use a dedicated interface for IPMI (other than explicit IPMI commands sent over /dev/ipmi0). Generally switching a BMC from dedicated mode to shared mode requires a BIOS/UEFI configuration change and a server reset.

                                      (Speaking from experience with shared mode and the OS resetting the NIC. The malicious actor is merely a scenario I just dreamt up.)

                                      1. 1

                                        Indeed, although I suspect in many cases these IPMI modules are already accessible from the compromised host over SMBus/SMIC or direct serial interfaces anyway - possibly even with more privileged access than over the network. That’s how iLOs and DRACs can have their network and user/group settings configured from the operating system.

                                        1. 4

                                          The increased risk mostly isn’t to the compromised host’s own IPMI; as you note, that’s more or less under the control of the attacker once they compromise the host (although network access might allow password extraction attacks and so on). The big risk is to all of the other IPMIs on the IPMI VLAN, which would let an attacker compromise their hosts in turn. Even if an attacker doesn’t compromise the hosts, network access to an IPMI often allows all sorts of things you won’t like, such as discovering your IPMI management passwords and accounts (which are probably common across your fleet).

                                          (I’m the author of the linked to article.)

                                          1. 3

                                            The L2 feature you are looking for is called a protected port. This should be available on any managed switch, but I’ll link to the cisco documentation:


                                            1. 1

                                              In a previous life at a large hosting we used this feature on switch ports that were connected to servers for the purposes of using our managed backup services.

                                  1. 2

                                    Doesn’t using proxies address this on top of performance and security boost?

                                    1. 1

                                      Using proxies just pushes the problem back one layer; now you need to maintain the proxy’s TLS configuration instead of your web server’s TLS configuration, but it still has to be maintained. Or you outsource maintaining it to Cloudflare. You still can’t walk away from the server entirely and let it keep running quietly.

                                      (I’m the author of the linked-to entry.)

                                      1. 1

                                        “In the era of HTTP, you could have set up a web server in 2000 and it could still be running today, working perfectly well”

                                        “And now you have to keep reasonably up to date with web server software, TLS libraries, and TLS configurations on an ongoing basis, because I doubt that the deprecation of everything before TLS 1.2 will be the last such deprecation”

                                        This is what I’m addressing. It seemed like folks you talk about wanted these servers to keep running. These diverse and interesting setups. Then, HTTPS’s varying needs gets in the way. So, we rely on a mature proxy whose developers and/or ecosystem handle all that so HTTP or whatever it proxies to keep working without all that work. Then, the rest can keep focusing on the non-HTTPS stuff that sits behind the proxies. There’s existing tools for that.

                                        “Another, more relevant side of this is that it’s not going to be possible for people with web servers to just let them sit.”

                                        This part remains true. Looking at the big picture, it probably was and always will be true for a lot of things in tech and life. Just due to how our environments change constantly whether offline or online. If anything, we should be pleasantly surprised when something we build still works five years later online without changes. Even more as pace of change and extra complexities increase over time.

                                    1. 8

                                      I think that coco and others who mentioned it are certainly correct that the package format, by itself, doesn’t improve the security situation. We need better support in Unix distributions for sandboxing of end-user applications. These package formats do help with that, in that by bundling all the dependencies inside the package, you relieve the distribution from the burden of identifying the dependencies and copying them in. Of course, to do it properly you would want the distribution to enforce the details of the sandbox configuration, rather than relying on the package to do it.

                                      As with containers for server software, this approach has drawbacks. If the distribution isn’t providing the dependencies, it also doesn’t have visibility into whether they’re up-to-date with all security patches. So these package formats as they stand actually take control away from end users… I can’t say I’m a fan. In other words, distributions should want to be involved in identifying the dependencies and copying them into any sandbox that’s being used.

                                      I do understand the benefit to having a format that works across distributions, but I really think that benefit is almost entirely to software publishers, and not to end users or to distribution maintainers. At least in the server world, system administrators do get some benefits from containers, but I don’t think that applies here.

                                      Since I do have higher-than-average security needs, I actually don’t ever use Flatpak, since the security trade-off would be unacceptable to me. I regard it as an important security feature to have binaries that are built by my distribution’s tooling, not by the software publisher. That means there are apps I can’t use at all, but that’s just how it is sometimes.

                                      I will take the opportunity, in passing, to plug NixOS. There is more work to be done wrt sandboxing of graphical apps, but you can already write a Nix package config which will build a Flatpak binary based on a Nix config, rather than on an upstream, publisher-provided recipe. Nix is far better than any other system that exists today, as far as easy configuration of dependencies. I hope that as people think about better solutions to these problems, they’ll keep the Nix tooling in mind and build on top of it when it makes sense to.

                                      1. 1

                                        I agree with the view that the first level benefits are primarily to software creators. Software creators get to build one artifact that will work on all Linuxes, and they get to distribute that artifact through a single place (where users can discover it, at least theoretically) instead of many. People like Canonical benefit indirectly through more software being available for Linux and perhaps through running the Snap store which leads to getting to take a cut of any purchases through it. People using Linux theoretically benefit through more software being available for it (and for their distribution), while small Linux distributions might benefit through more software being usable on them.

                                        (I’m the author of the linked to article.)

                                        1. 2

                                          I mean, this trend is specifically taking resources away from efforts to support packaging software properly for distributions. Users who understand their security needs have less software available as a result of it.

                                      1. 1

                                        I always thought the short 1U servers were for shallower racks, like the network boxes you often see on walls.

                                        1. 3

                                          The normal Dell rails for their short 1U servers are still full length (although perhaps you can get special shorter rails if you ask), so I don’t think they’d fit in short racks. Also, I believe that short telco racks are often only front-mount (with no back set of posts), which doesn’t work for server rails. My best guess about why Dell appears to like shorter 1U servers is ‘less metal for the case’, which presumably translates to cost savings.

                                          (I’m the author of the linked to entry.)

                                        1. 2

                                          I’m currently deep within the third world described in the article, internal client/server TLS. Already being within a private network, it’s unreasonable to purchase a unique certificate for every server host on the network.

                                          My best two options seem to be:

                                          1. Dynamic self-signed certificates created at server start up. Publish certificate to centralized & trusted location that clients can read from.
                                          2. Distributing a single certificate to entire server pool, signed by an implicitly trusted internal CA.
                                          1. 4

                                            The standard approach seems to be an internal CA with some sort of automated certificate issuing mechanism (and often trusting only the internal CA, not any public CAs). This does require the automated CA stuff, but I believe there are open source projects for that. If that was too much work, I would be inclined to treat the situation like basic SSH, with a self signed certificate created on startup somehow (either centrally and then distributed, or locally and then published).

                                            (SSH can also use the ‘internal CA’ route, of course, with server host keys being trusted because they’re signed.)

                                            1. 1

                                              At least for our product at work (cloud-first with on-prem option), the TLS scheme used in “the wild” sometimes meshes badly with internal CA’s used by the on-prem customers. The “stumbling block” is often browsers like Chrome, which can’t easily be convinced to trust an internal CA.

                                              1. 1

                                                We do have an internal CA, so I will probably go that route to get maximum coverage at sites we host. Unfortunately, clients can choose to host themselves and therefore will not trust our internal CA, leaving them to their own devices.

                                                This service is very core to the company, so failing to form a secure connection means failing to ingest important data. I may end up having to go to a hybrid approach in the end.

                                              2. 2

                                                you want option 3, like @cks mentioned. Each service gets their own cert signed by your internal CA[1]. You would do the same with SSH[2] except obviously it’s by node for ssh instead of by service. Hashicorp Vault[0] will help manage all of this for you.

                                                0: https://www.vaultproject.io

                                                1: https://www.vaultproject.io/docs/secrets/pki/

                                                2: https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-certificates/

                                              1. 24

                                                In some cases, I have a great deal of sympathy for the author’s point.

                                                In the specific case of the software that triggered this post? Not so much. The author IS TALKING ABOUT A SENDMAIL MILTER when they say that

                                                Python 2 is only legacy through fiat

                                                No. Not in this case. An unmaintained language/runtime/standard library is an absolute environmental hazard in the case of a sendmail milter that runs on the internet. This is practically the exact use case that it should absolutely be deprecated for, unless you’re prepared to expend the effort to maintain the language, runtime and libraries you use.

                                                This isn’t some little tool reading sensor data for an experiment in a closed environment. It’s processing arbitrary binary data from untrusted people on the internet. Sticking with this would be dangerous for the ecosystem and I’m glad both python and linux distro maintainers are making it painful for someone who wants to.

                                                1. 2

                                                  A milter client doesn’t actually process arbitrary binary data from the Internet in a sensible deployment; it encapsulates somewhat arbitrary binary data (email messages and associated SMTP protocol information that have already passed some inspection from your MTA), passes it to a milter server, and then possibly receives more encapsulated binary data and passes it to the MTA again. The complex binary milter protocol is spoken only between your milter client and your milter server, in a friendly environment. To break security in this usage in any language with safe buffer handling for arbitrary data, there would have to be a deep bug that breaks that fundamental buffer safety (possibly directly, possibly by corrupting buffer contents so that things are then mis-parsed at the protocol level and expose dangerous operations). Such a deep break is very unlikely in practice because safe buffer handling is at the core of all modern languages (not just Python but also eg normal Rust) and it’s very thoroughly tested.

                                                  (I’m the author of the linked-to blog entry.)

                                                  1. 2

                                                    I guess I haven’t thought about one where it would be safe… the last one I worked on was absolutely processing arbitrary binary data from the internet, by necessity. It was used for encrypting/decrypting messages, and on the inbound side, it was getting encrypted message streams forwarded through from arbitrary remote endpoints. The server could do some inspection, but that was very limited. Pinning it to some arbitrary library version for processing the message structures would’ve been a disaster.

                                                    That’s my default frame of reference when I think of a milter… it processes information either on the way in or way out that sendmail doesn’t know how to and therefore can’t really sanitize.

                                                    1. 1

                                                      For us, our (Python) milter client sits between the MTA and a commercial anti-spam system that talks the milter protocol, so it gets a message blob and some metadata from the MTA, passes it off to the milter server, then passes whatever the milter server says about the email’s virus-ness and spam-ness back to the MTA. This is probably a bit unusual; most Sendmail milter clients are embedded directly into an MTA.

                                                      If our milter client had to parse information out of the message headers and used the Python standard library for it, we would be exposed to any bugs in the email header parsing code there. If we were making security related decisions based on header contents (even things like ‘who gets how much spam and virus checking’), we could have a security issue, not just a correctness or DoS/crash one (and crashes can lead to security issues too).

                                                      (We may be using ‘milter client’ and ‘milter server’ backward from each other, too. In my usage I think of the milter server as the thing that accepts connections, takes in email, and provides decisions through the protocol; the milter clients are MTAs or whatever that call up that milter server to consult it (and thus may be eg email servers themselves). What I’m calling a milter server has a complicated job involving message parsing and so on, but a standalone client doesn’t necessarily.)

                                                      1. 2

                                                        Mine was definitely in-process to the MTA. (I read “milter” and drew no client/server distinction, FWIW. I had to go read up just now to see what that distinction might even be.) Such a distinction definitely wasn’t a thing I had to touch in the late 2000s when I wrote the milter I was thinking about as I responded.

                                                        The more restricted role makes me think about it a little differently, but it’d still take some more thinking to be comfortable sitting on a parsing stack that was no longer maintained, regardless of whether my distro chose to continue shipping the interpreter and runtime.

                                                        Good luck to you. I don’t envy your maintenance task here. Doubly so considering that’s most certainly not your “main” job.

                                                  2. 1

                                                    Yeah, it’s a good thing they do, it’s not the distro-maintainers fault that Python became deprecated.

                                                  1. 5

                                                    It’s got to be a keyword, that’s how it can work with is not, not also being a keyword.

                                                    Also there’s a quite ubiquitous use of is that justifies its inclusion; compare a is None to a == None to not a… only by testing the identity can you know for certain that None was passed.

                                                    1. 2

                                                      You put that very clearly, here and in the other comments; I learned something today.

                                                      One other ubiquitous use of is: type inspection, like if type(a) is MyClass or if type(a) is str.

                                                      (Some of the time isinstance(a, MyClass) will also do, but if you want to exclude the possibility of subclass instances, only a type identity check suffices.

                                                      Ooonn the other hand, one could also argue that is tempts people into checking for identity when a subclass would also suffice; and that this may needlessly prevent interoperation with otherwise-valid subclasses. Hm. I really like the keyword, though, especially compared to ===)

                                                      1. 5

                                                        Note that you don’t normally need is for this sort of type inspection, as type(a) == MyClass works fine pretty much always. I think the only time it wouldn’t work correctly is if one of the classes involved had a metaclass that overrode __eq__ and did something perverse with it. I think that the cases where you really need is are uncommon; you need to care about object identity and be in a situation where something is overriding == in a way that affects the comparison you care about. Python definitely should have something that checks object identity in the way that is does, but I don’t know if it needs to be a tempting keyword instead of a built-in function.

                                                        (I’m the author of the linked to article.)

                                                        1. 2

                                                          Every language must have a means of checking both identity and equality … they are fundamentally different things. The choice to make both of them an operator is quite common, the only real distinction Python makes by choosing == and is over, say, == and === is that promotion to also being a keyword.

                                                          And just because you find it to be uncommon does not mean that it is, in fact, uncommon. I use is all the time, for instance with enum.Enum members, to properly handle default values that need to be mutable, and to identify sentinel values in iterators (usually None, but not if None is a legitimate value of the iterator) … it’s also extremely necessary to things like the Singleton pattern in Python.

                                                          Moreover you’re throwing out the baby with the bathwater … sure, exactly as you said you can use type(a) == MyClass pretty much anywhere you might use type(a) is MyClass, but why would you? Why necessarily invoke the performance cost of equality comparison? The underlying cost of is is near zero, but by using == you force lookup and execution of MyClass.__eq__ and, if that returns False or NotImplemented you further force lookup and execution of type(a).__eq__ … both of which could be implemented in such a way as to be unbearably time consuming.

                                                          See the question of identity is, precisely, “are these the same thing” rather than “do these share the same attributes” … they’re fundamentally different questions, and when all you care about is identity, why bother asking for equality?

                                                      2. 2

                                                        Arguing that it has to be a keyword because otherwise you would have to write it differently is a weird argument to me. It just means that it would be written not is(a, b) instead, which might not read as nicely, but that’s a different argument.

                                                        1. 7

                                                          Perhaps I should have been more specific: is needs to be either a reserved keyword or an operator (in effect it’s both, though it’s not truly an operator because, like and, or, and not it cannot be overloaded) precisely because Guido et all intended it to be used as a syntactic construct… it cannot be a function because it would then have to be used with the call syntax, be able to be assigned to a name, and overloadable in user code. There is no desire for an overloadable identity operation, indeed allowing it to be overloaded would break fundamental language semantics. The same is true of None, which is why it’s also a keyword, though it could just be a globally available variable… in Python the argument for promoting to an operator is “should this bind values around it into an operation” and for promotion to a keyword is “should the meaning of this never even potentially change during runtime” and (optionally) “should this modify other keywords” … in the case of is the answer to all of those is “yes”, so the only option was to make it a keyword.

                                                          1. 1

                                                            It seems to me like the same arguments would mean that isinstance should also be a keyword though?

                                                            1. 3

                                                              Sure, it could be extended to any number of things — Python 2 didn’t see True or False as keywords, but that protection was extended to them in Python 3 — but would it add any semantic value to do so, by stripping call semantics from it and allowing it to be used as an operator or to directly interact with other keywords?

                                                              Some keywords have been downgraded (most notably print), but that was because there was added value in doing so, and also a colorable argument for overloading existed. The simple fact is that identity testing is a fundamental operation — much more so than introspecting class phylogeny — and the benefits of being a keyword far outweigh the minor confusion it can cause in relative newbies, while there’d be no advantage to removing it except for people who don’t know why it’s fundamental.

                                                              In other languages you’ve got equality as == and identity as ===… frankly we’ve got it easy with is.

                                                          2. 2

                                                            is being built in means you know it can’t be overridden.

                                                            It being a function would be ugly

                                                        1. 2

                                                          I agree that mutability is often needed (and not only because of evil lawyers). But the reference to Golang seems bit confusing to me. Immutability at a programming language level is completely different kind of immutability than immutability of whole system (e.g. blockchain).

                                                          Fashion trends and hypes (like „let’s make everything object-oriented/functional/immutable/web/nosql/asynchronous/etc.!“) are usually harmful. There is no one size that would fit all and no holy grail in the software engineering.

                                                          At programming language or code level both mutable and immutable structures make sense and have its use.

                                                          Beyond that, things in the real world are almost always mutable and removable because lawyers can show up on your doorstep with a court order to make them so, and the court generally doesn’t care about what problems your choice of technology has created for you in complying.

                                                          One possible solution of this problem is that you do not own and control that technology and it is running independently of you, it is not your property, like a grown up child that was raised and educated by you, but now it is living independently and you can not stop it or order it something.

                                                          Whether creating such system is ethical is another question. Sometimes it makes sense…

                                                          1. 1

                                                            I apologize; I wrote something that assumed too much context with my link text. It’s referring not to Go the language but to Go’s current design and implementation of a module source code proxy service, which is designed for immutable and potentially permanently cached module source code (even if the upstream vanishes; in fact explicitly partly to cope with the upstream vanishing). The current official proxy is operated by Google with no apparent point of contact to get them to remove something from the proxy’s cache.

                                                            (I’m the author of the linked to article.)

                                                          1. 1

                                                            The hardware required for this meager amount of monitoring makes me want to cry. Why do we keep making things worse?

                                                            ServiceNow using Xymon has aggregated all their monitoring data into one system that monitors

                                                             Hosts                      : 569869
                                                             Pages                      : 60421
                                                             Status messages            : 740185

                                                            That’s a lot of servers and almost 750,000 services being checked. I remember single server setups on old HP boxes with 8GB of RAM doing 50,000 hosts back in 2008.

                                                            In the monitoring space we’re building new tools which are considerably less efficient but nobody seems to care.

                                                            1. 2

                                                              I’m not sure that 31,000 samples a second is meager, but this server is clearly larger than we need for ingesting samples (as is obvious from the usage level). It needs RAM (and CPU) primarily for complex or long time range queries over the raw metrics data, and doing such queries is both a tradeoff between server resources and response time (Prometheus loads data into memory for query speed), and also between flexibility and pre-computing a limited set of measures and data.

                                                              The server itself is a basic Dell 1U server (an R230). The actual hardware cost involved is modest and wouldn’t be reduced significantly by making it smaller. Also, system metrics suggest that we rarely go above 6 or 7 GB of RAM these days in active use, so loading it up to 32 GB is probably overkill with current versions of Prometheus. But better safe than sorry with an OOM explosion due to an unexpectedly complex and memory-intensive query.

                                                              (I’m the author of the linked-to entry.)

                                                              1. 1

                                                                I’ve never heard of Xymon but I suppose most people are coming from Nagios/Icinga and I didn’t notice Prometheus being any worse than those setups from the hardware requirements. Not that this is saying a lot, but I see that as the baseline. I heard check_mk was better but I never used it myself.

                                                                Oh, and one thing - again no idea what Xymon in that setup did, but Prometheus is not primarily or solely used for monitoring but metrics. First of all those will always use a lot more processing power on the receiving end than a simple online/offline check. Not everything can be shoehorned into rrdtool.

                                                              1. 3

                                                                I have no idea what “spooky errors at a distance” Chris is talking about.

                                                                h : α → β
                                                                g : β → γ
                                                                f : γ -> δ
                                                                let v = h foo 
                                                                let v' = g v
                                                                let v'' = 

                                                                Suppose the type of h changes to α → ε. If the type ε is not compatible with β, h foo will fail to typecheck. Coming up with a type that is compatible with everything except γ is hardly easy.

                                                                1. 4

                                                                  I think your example is different from what Chris had in mind, but I agree with your point. It’s hard to say that type inference would be problematic without knowing both what the inference algorithm is, and what the code in question is.

                                                                  I use OCaml a lot and I can’t say I often run into this kind of “spooky errors at a distance”. In practice, even if you don’t use type annotations in the code, it’s good practice to write module interfaces with explicit types. So a change in types in one place will give pretty obvious errors about where to fix it. If you’re not writing interfaces, you’re probably dealing with localized, coupled code, where making the type changes aren’t a big deal and the issues Chris describes aren’t so important.

                                                                  1. 1

                                                                    The situation I was thinking of goes something like this. Go has inside a function type inference for return types, so you can write newvar := afunc() and the type of newvar is inferred from afunc()’s return type. Now suppose you extend this to inferring the return type of a function through some syntax (either returning newvar or simply doing return afunc()). This allows you to create a chain of functions with inferred return types; you call afunc() and use its type as your inferred return type, then someone calls you and uses your return type (inferred from afunc()) as their return type, and so on. Then eventually someone at the top of the call chain actually uses a return value in a way that requires it to have a specific type, which is compatible with afunc()’s current return type.

                                                                    Then afunc() changes its return type. Your function is still type correct, just with a different inferred return type, and so is your caller, and so is your caller’s caller, and so on all the way up to the person who actually uses a return value in a way that is now incompatible with afunc()’s new return type. That person gets a type error.

                                                                    (To make this easier to happen, you can actually do a lot of generic things with values in Go; for example, as long as a value is some sort of pointer, you can always compare it with nil, and if a type is some sort of a number you can compare it with zero or another constant. This covers a lot of ground in much practical code, right up to the point where someone wants a particular sort of pointer or number for various reasons.)

                                                                    1. 1

                                                                      I haven’t used Go but this sounds very similar to languages with “more” type inference (Haskell, for instance) which force you to add type annotations to top-level forms so these errors cannot propagate outside the source file anyway.

                                                                      I see a distinction between type systems with “proper inference” and the ones that only propagate types “downwards”[*] – can Go infer the type of a variable by its usage later on in the function? (Does it ever need to?)

                                                                      [*] I think var x = y in C# is an example of this, where it’s “inference” in that it lets you omit declaring the type of x when both you and the compiler know exactly what its type is (it’s the type of y).

                                                                      1. 2

                                                                        can Go infer the type of a variable by its usage later on in the function?

                                                                        No, it can’t. I’ve seen it referred to as “type deduction” instead, although the spec uses no such language (nor does it use the word “inference”): https://golang.org/ref/spec#Variable_declarations

                                                                        I think “limited inference,” as used by the OP, is an okay phrase to use to describe this. But there is certainly a distinction between the inference used by Go and the inference used, by, say, Haskell.

                                                                  1. 1

                                                                    Is uMatrix really worth it over just using uBlock Origin’s medium or hard mode?

                                                                    1. 2

                                                                      The one clear area that uBlock Origin’s medium and hard modes don’t seem to cover is cookies (for successful requests). You can clean them up after the fact with Cookie AutoDelete, but they’ll still be sent back to sites you visit before you close things down (or manually delete them).

                                                                      1. 1

                                                                        Good to know. Thank you very much.

                                                                    1. 1

                                                                      Do you also run NoScript, or is that functionality covered by uBlock Origin?

                                                                      1. 3

                                                                        It can be covered by uMatrix, either globally, 1st party, or individual host level.

                                                                        1. 3

                                                                          I don’t use NoScript; I used to, but for various reasons replaced it with uMatrix (which I now prefer). uBlock Origin can also block Javascript if you want it to but with less fine-grained control than uMatrix gives you. My opinion is that people who just want to block (most) Javascript and who already use uBlock Origin should probably use it for that, because it’s simpler than uMatrix. I use uMatrix because I want to block things besides Javascript (especially cookies), and I like the fine-grained control.

                                                                        1. 2

                                                                          Isn’t uMatrix an advanced version of uBlock Origin? What’s the rationale behind using both of them?

                                                                          1. 6

                                                                            The relationship between the two is not quite that straightforward. uMatrix gives you much more fine grained control over blocking types of things, but uBlock Origin has much better built-in ad blocking and filtering than uMatrix does in practice (including many more lists available standard) and will probably do it more efficiently, simply because matching blocklists isn’t uMatrix’s focus. uBlock Origin also has directly built in support for blocking page elements on the fly either permanently or temporarily. You can to some extent control Javascript in uBlock Origin’s dynamic filtering instead of resorting to uMatrix, but I don’t think uO gives you much or any control over cookies and other things.

                                                                            I use both because I care about both ad-blocking (uBlock Origin’s role, along with zapping away unwanted page elements) and having fine grained control over Javascript, cookies, and other annoyances of the modern web (uMatrix’s role). Although I haven’t tried it, I’m pretty sure my web experience would be significantly worse and more annoying if I only used uMatrix; I don’t think that taking out only Javascript (and cookies) stops enough of the annoyances of the modern web these days. And uBlock Origin blocks things in situations where you have to enable Javascript, such as Youtube.

                                                                            (I’m the author of the linked-to blog entry.)

                                                                          1. 1

                                                                            Perhaps moving to Thanos would be a simpler solution? but I am not sure introducing k8s into the stack…

                                                                            1. 5

                                                                              Adding k8s into an environment that doesn’t use it would be a significant increase in complexity, and as far as I know we’d still need to set up a storage backend for Thanos as well. Although I didn’t mention it in the entry, we’re operating at a low enough metrics volume that 13 months of metrics is currently only 758 GB, which is easy enough to hold on regular storage without downsampling or other Thanos features.

                                                                              (I’m the author of the original article.)