1. 6

    Clearly a lot of work went into this post, but I found it less of a “state of type hints” and more of a “what and how”, in that I was expecting an overview of adoption, or progress in the type system. Having said that, it’s probably an excellent post for someone wondering “what are type hints in Python and how can I use them?”

    However, I’m pretty sure the final paragraph is incorrect?

    Remember that, similar to unit tests, while it does makes your code base contain an extra number of lines, at the end of the day all the code you add is code that is automatically checked and enforced to be correct. It acts as a safety net to ensure that when you change things around later on things keep working

    In a statically-typed language this would be true, but type hints in Python are just hints. There’s no guarantee from Python that either the hints or what’s passed is actually correct.

    1. 7

      While it’s true that Python’s type-hints aren’t automatically enforced, the same is true of tests: they don’t help either unless you run the test suite. At least for Python, comparing types and tests in this way seems reasonable.

      1.  

        In a statically-typed language this would be true, but type hints in Python are just hints.

        I write a lot of Python and thus far I’ve eschewed type hints, for the simple reason that if I’m gonna go to all that effort I might as well just write Go.

        (I’m being slightly sarcastic, but there’s a kernel of truth in there.)

        1. 5

          I also write a lot of Python, and I type-hint 100% of it. The type hints typically take very little effort to write, and the benefits (IDE help + fewer bugs + easier refactoring) save me a lot of work, so on net it’s almost certainly effort-saving.

          (A bonus that’s possibly orthogonal to effort is that having to think about and write out the types I use encourages me to design cleaner APIs and makes a lot of bad code smells much more obvious.)

          1.  

            Sure, and Python will still support that. Now imagine you’re an organization like Google and you have 100 million lines of Python in your company’s repo. It’s not going to be economically viable to rewrite it but you’d still like to incrementally add type checking to try to lower the rate bugs are found in Python code. That’s where Python’s type checking makes sense at the moment. I’ve still yet to see much adoption in the scientific Python space, for example, although I bet that’s mostly because NumPy doesn’t have type stubs yet.

            1.  

              If wishes were fishes, one of the fish in my river would be to have something like [Elm’s record types], but for specifying Pandas data frame column names instead of record fields. Elm’s record types do not require you to specify all of the input’s fields; instead, you specify the ones the function expects, and the fields that are guaranteed to be in the output. The typechecker keeps track of which fields a record has as it passes through the functions.

              It would be perfect for the domain-specific cleaning and wrangling functions one writes on top of pandas. I realise this is not a trivial thing to wish for, though.

        1. 7

          I’ve been guilty of trash-talking other projects myself in the past

          Well, the blog is titled “Software is Crap” :)

          [Rust’s] designers made the unfortunate choice of having memory allocation failure cause termination – which is perhaps ok for some applications, but not in general for system programs, and certainly not for init

          Rust can help with not allocating at all (e.g. heapless), and try_reserve is in nightly already.

          Zig though is a language oriented exactly at this: it forces you to manually pick an allocator and handle allocation failure. But it is much younger than Rust, so if you’re worried about Rust “mutating” (FYI, Rust 1.x is stable, as in backwards compatible), it’s way too early to consider Zig (0.x).

          non-Linux OSes are always going to be Rust’s second-class citizens

          Yeah, related to that: Rust’s designers made the unfortunate assumption that OSes don’t break userspace from one release to another, just like Linux. The target extension RFC would solve this.

          Other than that… while the core team is indeed focused on the “big three” (Linux/Windows/Mac), Rust does support many “unusual” targets, including Haiku, Fuchsia, Redox, CloudABI.

          Back to inits and service managers/supervisors:

          There are so many of them, many of them are interesting (I’ve been looking at immortal recently), but they all have one big problem: existing service files/scripts on your system are not written for them. So I usually end up just using FreeBSD’s rc for basic pre-packaged daemons + runit for my own custom stuff.

          The Ideal Service Manager™ should:

          • read existing service definitions from system packages (rc scripts, OpenRC scripts, systemd units, daemontools/runit/s6 style bare shell scripts)
          • prevent the services from daemonizing, somehow (injecting -f into $SERVICE_flags? horrible and evil hacks like LD_PRELOADing a library that overrides libc’s daemon() with a no-op? lol)
          • force the services to log to syslog, somehow (redirect stdout/stderr, but what about daemons that open a custom logfile by default? maybe just let them do that)
          • supervise them like runit does

          I guess instead of preventing forking it can support tracking forking services with cgroups on Linux, and… with path=/ ip4=inherit ip6=inherit sysvmsg=inherit ... jails on FreeBSD? I wish there was a 100% reliable way to make sure any service runs in the foreground.

          1. 5

            Well, the blog is titled “Software is Crap” :)

            Yeah, there is that. I had originally wanted to emulate a humorous style I’d seen elsewhere (the long defunct “bileblog”) which badmouthed things in such an over-the-top fashion that you knew it was humorous; I could never quite get that right and it always seemed like I was just being nasty. Now I just try to provide objective criticism; it’s probably not as entertaining to read, but it’s also less likely to upset people. And of course, I also write about Dinit and occasionally write (hopefully) helpful articles on other topics.

            Rust can help with not allocating at all (e.g. heapless), and try_reserve is in nightly already. Zig though is a language oriented exactly at this:

            heapless probably wouldn’t serve my needs, but things like try_reserve are what are sorely needed for Rust to be a serious systems language, so I’m glad that’s happening. There are other reasons (perhaps more subjective) that I don’t like Rust - particular aspects of its syntax and semantics bother me - but in general I think the concept of ownership and lifetime as part of type are worthwhile. I have no doubt that good things will come from Rust.

            As for Zig, I need to look at it again. It certainly also has promise; but you’re right that I’d be worried about its stability and future.

            I guess instead of preventing forking it can support tracking forking services with cgroups on Linux, and… with path=/ ip4=inherit ip6=inherit sysvmsg=inherit … jails on FreeBSD? I wish there was a 100% reliable way to make sure any service runs in the foreground.

            Yeah, that’s a fundamental problem. Linux and DragonFlyBSD both have a simple means to prevent re-parenting past a particular process, which is one potential way to solve it (if you are ok with inserting an intermediate process, and really I don’t think that’s a big deal); cgroups/jails as you mention are another; any other option starts to feel pretty hacky (upstart apparently used ptrace to track forks, but that really feels like abuse of the mechanism to me).

            Thanks for your comments.

            1. 5

              Yeah, there is that. I had originally wanted to emulate a humorous style I’d seen elsewhere (the long defunct “bileblog”) which badmouthed things in such an over-the-top fashion that you knew it was humorous; I could never quite get that right and it always seemed like I was just being nasty. Now I just try to provide objective criticism; it’s probably not as entertaining to read, but it’s also less likely to upset people. And of course, I also write about Dinit and occasionally write (hopefully) helpful articles on other topics.

              The problem with it is: that style of humor is so common in the programming world that even good one is not at all novel. Also, as you say it, it’s also very hard to get right, even for seasoned comedians, which - no offense - most programmer aren’t.

              heapless probably wouldn’t serve my needs, but things like try_reserve are what are sorely needed for Rust to be a serious systems language, so I’m glad that’s happening.

              Everyone attaches their own meaning to “systems language”, and adding “serious” feels a bit like moving goalposts. “Ah, yeah, you got the systems part down, but how about serious”. It might not be convenient at all places and I agree that some things are undone, but we’re up against literally decades old languages. We’re definitely serious about getting that issue solved in a foreseeable timeframe.

              Heapless helps in the sense that you can provide your own stuff on top. Even the basic Box type in Rust is not part of libcore, but libstd.

              Servo takes a middle ground of extending Vec with fallible push. (https://github.com/servo/servo/blob/master/components/fallible/lib.rs)

              The thing here is mostly that stdlibs collection considers allocation failure and unrecoverable error. For ergonomic reasons, that’s a good pick for a standard library.

              So, it’s perfectly feasible to write your own collection library (or, for example extension) even now.

              Also, here’s a list of notes about what’s needed to make fallible stuff in the language proper cool. I can assure you after attending the All Hands that this is definitely a hot topic, but also a hard one.

              This just as a little bit of context, I’m not trying to convince you.

              I’d be very interested in what your semantic issues with Rust are.

              To add to that, I’m happy that you took a look at the language, even if you came away wanting.

              As for Zig, I need to look at it again. It certainly also has promise; but you’re right that I’d be worried about its atability and future.

              I’m definitely hoping for more “new generation” systems programming languages. I think there is quite some space around and I hope that some of these make it.

              1. 4

                I’d be very interested in what your semantic issues with Rust are.

                A proper answer to that would need me to sit down for an hour (or more) and go through again the material on Rust to remember the issues I had. Some of them aren’t very significant, some of them are definitely subjective. I should qualify: I’ve barely actually used Rust, just looked at it a number of times and had second-hand exposure via friends who’ve been using it extensively. The main thing I can remember off the top of my head that I didn’t like is that you get move semantics by default when passing objects to functions, except when the type implements the Copyable trait (in which case you get a copy), so the presence or absence of a trait changes the semantics of an operation. This is subtle and, potentially, confusing (though the error message is pretty direct). I’d rather have a syntactic distinction in the function call syntax to specify “I want this parameter moved” vs copied.

                Other things that bother me are lack of exceptions (I realise this was most likely a design decision, just not one that I agree with) and limited metaprogramming (the “hygienic macro” facility, when I looked at it, appeared a bit half-baked; but then, I’m comparing to C++ which has very extensive metaprogramming facilities, even if they have awful syntax).

                I can assure you after attending the All Hands that this is definitely a hot topic, but also a hard one.

                Yep, understood.

                I’m happy that you took a look at the language, even if you came away wanting.

                I’ll be continuing to watch closely. I’m very interested in Rust. I honestly think that some of the ideas it’s brought to the table will change the way future languages are designed.

                1.  

                  …you get move semantics by default when passing objects to functions, except when the type implements the Copyable trait (in which case you get a copy), so the presence or absence of a trait changes the semantics of an operation.

                  I can definitely understand how that would feel worrying, but in practice it’s not so bad: Rust doesn’t have copy constructors, so the Copytrait means “this type can be safely memcpy()d”. For types that can be cheaply and infinitely duplicated without (heap) allocation, like u32, copy vs. move isn’t that much of a semantic difference.

                  The closest thing to C++’s copy constructor is the Clone trait, whose .clone() method will make a separately-allocated copy of the thing. Clone is never automatically invoked by the compiler, so the difference between moving a String versus copying a String is somefunc(my_string) versus somefunc(my_string.clone()).

                  lack of exceptions

                  As a Python programmer, I’m pretty happy with Rust’s error-handling, especially post-1.0 when then ? early-return operator was added. I feel it’s a very nice balance between C and Go-style error handling, which is explicit to the point of yelling, and Java and Python-style error handling, which is minimal to the point where it’s hard to say what errors might occur where.

                  limited metaprogramming

                  It depends how much you care about getting your hands dirty. Rust doesn’t have full-scale template metaprogramming like C++, but the hygenic macro system (while limited) is a good start. If you want to go further, Rust’s build system includes a standard and cross-compilation-friendly system for running tasks before your code is compiled, so you can run your code through cpp or xsltproc or m4 or a custom Python script or whatever before the Rust compiler sees it. Lastly, “nightly” builds of the compiler will load arbitrary plugins (“procedural macros”) which will let you do all the crazy metaprogramming you like. Since this involves tight integration with the compiler’s internals, this is not a stable, supported feature, but nevertheless some high-profile Rust libraries like the Rocket web framework are built on it.

              2. 2

                Linux and DragonFlyBSD both have a simple means to prevent re-parenting past a particular process

                Hmm?? This sounds very interesting! Please tell me more about it.

                upstart apparently used ptrace to track forks

                Oh, this made me realize that I can actually use DTrace to track forks!

                1.  

                  Hmm?? This sounds very interesting! Please tell me more about it.

                  In linux: prctl(PR_SET_CHILD_SUBREAPER, 1); In DragonFlyBSD (and apparently FreeBSD too, I see): procctl(P_PID, getpid(), PROC_REAP_ACQUIRE, NULL);

                  In both cases this marks the current process as a “reaper” - any child/grandchild process which double-forks or otherwise becomes orphaned will be reparented to this process rather than to init. Dinit uses this already to be able to supervise forking processes, but it still needs to be able to determine the pid (by reading it from a pid file). There’s the possibility though of inserting a per-service supervisor process which can then be used to keep track of all the processes that a particular service generates - although it still doesn’t provide a clean way to terminate them; I think you really do need cgroups or jails for that.

              3.  

                [Rust’s] designers made the unfortunate choice of having memory allocation failure cause termination – which is perhaps ok for some applications, but not in general for system programs, and certainly not for init

                Or just run under Linux and have random processes killed by the OOM killer and random times because that’s so much better letting a program know the allocation didn’t really succeed twenty minutes ago when it could do something about it.

                1.  

                  Agreed, the OOM killer is totally bonkers, but its existence doesn’t justify stopping a program due to a failed allocation.

                  1.  

                    its existence doesn’t justify stopping a program due to a failed allocation.

                    Yes, especially since overcommit can be turned off, which should largely (if not always - I’m not sure) prevent the OOM killer from acting.

                    1.  

                      IIRC overcommit is even off by default in Debian.

                    2.  

                      Right. I was saying just let malloc return NULL and let the program deal with it instead of basically lying about whether the allocation succeeded or not. I disable memory overcommit on most of my systems.

                      1.  

                        For C I totally agree.

                        The Rust equivalent would be:

                        let b = Box::new(...);
                        

                        But Box::new doesn’t return a Result. If allocation fails, the program is terminated.

                        And so far we have only really talked about the heap. As far as I can tell you never know if stack allocation succeeded until you get a crash! Even in C. But I suppose once the stack is hosed, so is your program, which may not be true for the heap.

                1. 10

                  os

                  Upvote and/or comment here if you prefer “os” to be the label of the new taxon.

                  1. 5

                    I see a lot of Plan 9 links in the list, and I don’t think anybody running Plan 9 these days is doing operating systems research or writing papers about it, so “osres” or “os-research” doesn’t seem to fit. Meanwhile, “osdev” makes me expect articles on things like “how to map the VGA framebuffer” and “how to get from 8086 real mode to x86_64 long mode”.

                    “os” is perhaps a little generic, but I guess the alternative would be to tag such posts as “software”, which is even more so.

                    1. 5

                      I do operating system research based on Plan 9.
                      And I think what 9front developers do count as research as much.
                      Also people usually write “papers” when the research produce something interesting. It might take a while. :-)

                      Note that “os” is fine to me, I just think it will require more moderation to keep it focused on less known operating systems.

                  1. 2

                    I’ve definitely worried about “will this test pass for a trivial reason instead of the real reason” before, and often I try and resolve it by (for example) examining the description associated with the exception I caught, a notoriously brittle approach that adds test maintenance burden.

                    This seems a much more sensible and robust solution, and I’m itching to find out whether it’s as practical and useful in large code-bases as it sounds.

                    1. 22

                      So (1) nobody cared to improve the old tools, writing new ones is more fun, and (2) updating the old tools to reflect current reality would break old scripts, so the rational choice is to both let the old tool rot (thus quite possibly breaking anything that relies on it) as well as introduce a new tool that definitely isn’t compatible with the old scripts. Why do I feel like this line of arguing has a problem or two?

                      Pray tell, what happens when the interface provided by the iproute2 utilities no longer reflects reality. Let them rot and write yet another set of new tools? Break them? Introduce subtle lies?

                      Oh by the way, if you’re configuring IPv6 on linux, don’t use the old tools. They’re subtly broken and can waste you a lot of time. I’ve been there. Don’t mention it.

                      Meanwhile, I’m glad that OpenBSD’s ifconfig can show more than one IP per interface. And I can use the same tool to set up my wifi too. It’s a tool meant to work.

                      1. 12

                        The BSDs are maintaining the kernel and the base system in locksteps. This is not the case for Linux distributions. Over the years, Linux developers started to do the same. That’s why we now have iproute2, ethtool, iw and perf, which are userland tools evolving at the same speed as the kernel (and sharing the version number).

                        1. 8

                          nobody cared to improve the old tools

                          The people who want to use the old tools want them to keep working the same way they always have. They already work that way, so the people who want to use the old tools have no motivation to make changes.

                          updating the old tools to reflect current reality would break old scripts

                          It would also piss off the people who want to keep using the old tools, since by definition they would no longer keep working the same way.

                          The names ifconfig and netstat are now claimed and cannot be re-used for a different purpose, in much the same way that filenames like COM1 and AUX are claimed on Windows and cannot be re-used.

                          Meanwhile, I’m glad that OpenBSD’s ifconfig can show more than one IP per interface.

                          My understanding is that OpenBSD reserves the right to change anything at any time, from user-interface details down to the C ABI. “The people who want to use the old tools” are discouraged from using OpenBSD to begin with, so it’s not surprising that OpenBSD doesn’t have to wrestle with these kinds of problems.

                          1. 4

                            the right to change anything at any time

                            While this is true, I think you are taking it a little too literally. You won’t, for example, upgrade to the latest snapshot and find that ls has been replaced by some new tool with a completely different name, or that CVS has been replaced by GIT. And while POSIX doesn’t require (best I can tell) a tool named ifconfig it’s very unlikely you would find it replaced by something else.

                            1. 5

                              Right. And by following the discussions on tech@, I’ve gotten the impression that Theo (as well as many other developers) do deeply care about avoiding unnecessary chance to the user facing parts as tools get replaced or extended. Case in point, daemon configuration. The system got extended with the rcctl tool, but the old way of configuring things via rc.local and rc.conf.local still works as it always did. Nothing like the init system swaps on Linux. Still, extending or changing the behavior of a tool even at the risk of breaking some old script seems to be preferred to making new tools that require everyone to adapt.

                              After a decade of using Linux as well as OpenBSD, I’d say that OpenBSD is way more committed to keeping the user facing parts finger compatible while breaking ABI more freely (“we live in a source code world”). In the Linux world I’ve come to expect ABI compatibility but user compatibility gets rekt all the time.

                        1. 6

                          focuses to solve a single (possibly complex) problem (mosh)

                          But mosh solves two generally unrelated problems: “input prediction” and “automatic reconnection”. It just so happens that a lot of people are using high-latency, low-reliability connections (3G) and find both features useful at the same time, but there definitely exist high-latency, high-reliability connections (like connecting to a data-centre on another continent) and low-latency, low-reliability connections (like connecting from my laptop to my desktop). Sometimes I really wish somebody would split mosh apart so I can use the pieces individually.

                          1. 9

                            I’m not qualified to make any judgment on the technical merits of several dependency management solutions, but as someone working primarily in Go, the churn of solutions is starting to have a real cognitive cost.

                            1. 6

                              Some of the solutions suggested from a couple of the Go devs in that “thread” sound.. almost surreal to me.

                              My favorite one so far:

                              We’ve been discussing some sort of go release command that both makes releases/tagging easy, but also checks API compatibility (like the Go-internal go tool api checker I wrote for Go releases). It might also be able to query godoc.org and find callers of your package and run their tests against your new version too at pre-release time, before any tag is pushed. etc.
                              https://github.com/golang/go/issues/24301#issuecomment-390788506

                              With all the cloud providers starting to offer pay-by-the-second containers-as-a-service, I see no reason we couldn’t provide this as an open source tool that anybody can run and pay the $0.57 or $1.34 they need to to run a bazillion tests over a bunch of hosts for a few minutes. There’s not much Google secret sauce when it comes to running tests.
                              https://github.com/golang/go/issues/24301#issuecomment-390790036

                              That sounds… kind of crazy for anyone that isn’t Google scale or doesn’t have Google money.
                              Are the Go devs just /that/ divorced from the (non Google) reality that the rest of us live in?

                              1. 10

                                Kind of crazy, but not super crazy. As another example, consider Rust’s crater tool. When the Rust team are trying to evaluate the impact of fixing a syntax quirk or behavioural bug, they make a version of the Rust compiler with the change and a version without, and boot up crater to test every publicly available Rust package with both compiler versions to see if anything breaks that wasn’t already broken.

                                crater runs on Mozilla’s batch-job infrastructure (TaskCluster), and Mozilla is much, much smaller than Google scale. On the other hand, they’re still bigger than a lot of organisations, and I believe a crater run can take a few days to complete, so it’s going to be a lot more than “$1.34 … for a few minutes” on public cloud infrastructure.

                                1. 1

                                  I get the spirit of those responses; we’re getting to the point with cloud services were that kind of integration test suite could happen cheaply.

                                  But it is not the answer to the problems that prompted those responses.

                                  Dependency management is hard, and there isn’t a perfect solution, mvs is a cool approach and I’m curious how it shakes out in practice, but to OP’s point, I’m not sure I can do another switch like we’ve done up to now

                                  Manual Vendoring ($GOPATH munging)
                                  govendor
                                  dep
                                  vgo
                                  whatever fixes the problems with vgo

                                  1. 3

                                    Agreed. I have a couple of projects that I have switched solutions at least 4 or 5 times already (manual GOPATH munging, godep, gpm, gb, dep), because each time it was either a somewhat commonly accepted solution, or seemed the least worst alternative (before there was any kind of community consensus).

                                2. 3

                                  I have yet to migrate a project between dependency managers.

                                  The old ones work exactly as well as they always have.

                                  1. 2

                                    I’ve reverted to using govendor for all new projects. I might be able to skip dep if vgo proves to be a good solution.

                                    1. 1

                                      similar story for us; govendor works better with private repos

                                1. 3

                                  At work we use YYYY.MM.NN for internal software (NN being a 0-indexed release number for that month).

                                  I like this for knowing when something was last updated, but it’s not helpful for identifying major changes vs. bugfixes. Perhaps that’s not such a big deal for software that’s on a rapid release cycle.

                                  1. 2

                                    It’s also not a big deal for software that’s too big or complex for “major change” to be meaningful. If a tiny, rarely used Debian package removes support for a command-line flag, that’s a major change (in the SemVer sense) and since Debian includes that package it’s therefore technically a major change in Debian. But if Debian followed SemVer that strictly, its version number would soon leave Chrome and Firefox in the dust, and version numbers would cease being a useful indicator of change.

                                    1. 7

                                      Isn’t Debian’s solution to this to not include major-version changes in updates to an existing release? So it does wait for the next major version of Debian to be included, usually

                                      1. 1

                                        Yep, and this is where the -backports or vendor repos are really useful - newer packages built against the stable release.

                                      2. 2

                                        It’s why we have to make “stable” releases. Otherwise everyone goes crazy. If someone is updating their SemVer too often, they have bad design or do not care for their developers.

                                      3. 2

                                        There’s a discussion of CalVer and breaking changes here: https://github.com/mahmoud/calver/issues/4

                                        Short version, “public” API is a bit of a pipe dream and there’s no replacement for reading (and writing) the docs :)

                                        1. 2

                                          The concept of breaking changes in a public API isn’t really related to ‘read the docs’, except when it comes to compiled in/statically linked libraries.

                                          If you have dependency management at the user end (i.e. via apt/dpkg dependencies that are resolved at install time), you can’t just say “well, install a version that will work, having read the docs and understood what changes when”.

                                          You instead say “I require X major version of package Foo”, because no matter what the developer does, Foo version X.. will always be backwards compatible - new features might be added in a X.Y release, but thats not a problem if theyre added in a backwards compatible manner (either theyre new functionality that has to be opted in for, or e.g. they don’t require extra options to work).

                                          Yes, I know that things like Composer and NPM have a concept of a ‘lock’ file to fix the version to a specific one, but that’s not a solution for anything but internal projects. If you’re installing tools that you aren’t directly developing on yourself using NPM or Composer, you’re doing it wrong.

                                          1. 1

                                            I really don’t see what that has to do with the linked thread. In the very first line, you mention a “public” API. The point is that there’s much less consensus on what constitutes a public API than developers assume. So, you end up having to write/read the docs about what would constitute a “semantic” version change. (Not that docs are a silver bullet, they’re just a necessary part of healthy software development.)

                                            1. 1

                                              The point is that there’s much less consensus on what constitutes a public API than developers assume.

                                              A comment by you making that same claim on GitHub isn’t really evidence of a lack of consensus. What possible definition is there for “public API” besides, “something that will be consumed my someone outside the project”.

                                              So, you end up having to write/read the docs about what would constitute a “semantic” version change.

                                              The decision tree for SemVer is two questions, with 3 possible outcomes. And you’ve still ignored my point. Adherence to semver means you can automatically update dependencies independently of the developer.

                                              So, for instance, if the developer depended on a shared library, that happens to have a security vulnerability, when the library author/project releases a new patch version, end-users can get the fix, regardless of what the tool/app developer is doing that week.

                                              1. 1

                                                The automatic updates work until they don’t. Here is a recent example where Python’s pip broke with many assumptions about public APIs. Your point has not been ignored, I’ve written about it extensively, in the linked thread and in the linked site (and links therein).

                                                As for your closing comment, I’m noticing an important assumption I’m working to uproot: current date is not the only possible date in the release version. manylinux2010 came out in 2018, and is named as much because it’s backwards compatible to 2010.

                                                The Teradata example on the CalVer site also highlights maintaining multiple releases, named after their initial release date. At the consumer level, Windows 98 got updates for years after 2000 came out.

                                                1. 1

                                                  That isn’t a failing of semver, it’s a failing of the developers who didn’t properly identify they had a breaking change.

                                                  The same thing would have happened under calver, they would have marked it as a patch release with compatibility to the previous version, regardless of the date component.

                                                  Expecting people to just forget about the possibility of automatic dependency updates is like suggesting people forget that coffee exists after they’ve had it daily for 10 years.

                                      1. 1

                                        TIL there’s python package manager which creates virtualenvs automatically. Never heard about it before. But PyPA instead of PyPI? Or PyPA is a layer on top of PyPI? Python packaging systems change so fast.

                                        1. 6

                                          PyPA is the Python Packaging Authority, the people who maintain PyPI, tools like pip and setuptools, and make sure they all work nicely together.

                                        1. 14

                                          Raw roff markup is alien and mysterious to modern sensibilities, and a Markdown-family language based on the manpage document model rather than the HTML document model is a cool idea.

                                          Unfortunately, just as Markdown is geared for presentational HTML rather than semantic (there’s no easy way to add a class to a block of markup, for example), it seems scdoc is geared for presentation man(7) output rather than semantic mdoc(7). I can’t really fault the author; I’m not even sure what a semantic Markdown-alike could look like. But it wasn’t too hard for me to learn mdoc(7) myself, and I’ll probably stick with it.

                                          1. 3

                                            These are good points. I deliberately chose not to expose to the user any control over the low-level roff output, and also deliberately chose not to add semantic components because the output is only ever going to be man pages. I’m of the opinion that using several specialized tools (for each facet of your documentation, be it man pages or HTML pages or PDFs) is better than attempting to fit one octopus-shaped peg into several holes, and this principle feeds directly into scdoc’s design.

                                            That being said, it’s totally valid to hold other viewpoints. For some people mdoc may be better. For myself, it’s not.

                                          1. 2

                                            I find literate coding to be one of the best ways to learn. I wish more folks did this. My big takeaways from this are the existence of Tango and static_http_cache.

                                            I wish there was more tooling support for literate coding. What are some IDEs, editor plugins, etc. that facilitate literate coding?

                                            1. 2

                                              For this project, I didn’t have any special editor support, but I did heavily rely on entr to trigger converting Markdown to HTML whenever I saved in my editor, and livereload to trigger my browser to reload whenever the HTML file was updated.

                                              It may seem minor, since neither operation is time-consuming, but before I found those, there were so many times I’d forget to do either or both steps because I was busy thinking about how best to phrase something. It’s been said many times before, but lowering the iteration time of any creative endeavour is always immensely positive.

                                            1. 8

                                              Last year, ST didn’t support ligatures, and I wanted to try them out. So I went off and used first Atom and then VS Code for a month or two. Best thing I can say about either is that they have quite a vibrant plugin ecosystem. When I decided that code ligatures were actually kind of a dumb idea (and definitely not worth the Electron bloat, MS spyware, and assorted rough edges), I came back to ST3 with new appreciation for its relative simplicity, stability and performance. Now, I see ligature support added in 3.1: ain’t that always the way?

                                              Along the way I met my new favorite console editor, vis. Perhaps someday it will replace my GUI editor entirely.

                                              1. 1

                                                Have you looked at Kakoune? I switched from vis to Kakoune a few months ago; I found Kakoune’s editing commands easier to learn than vis’ structural-regular-expression syntax, and I prefer the way Kakoune supports multiple editor windows that I can manage with my normal window manager, over vis’ (and Vim’s) window-splitting system.

                                                In my spare time I’m working on a GTK+ based front-end for Kakoune, at least partially because I wanted code ligature support in my editor. :)

                                                1. 1

                                                  In my spare time I’m working on a GTK+ based front-end for Kakoune, at least partially because I wanted code ligature support in my editor. :)

                                                  That sounds like something I’d be interested in. Is there a repository online?

                                                  I really like Kakoune. My only problem with it is that it’s missing a good solution for hard word wrapping (like Vin’s gwip or Emacs’ M-q).

                                                  1. 2

                                                    It’s still quite a work-in-progress, but feel free take a look.

                                                    Kakoune does have an “autowrap” plugin that will automatically press Enter for you as you type, but it’s a bit quirky. Currently, I have the following in my kakrc:

                                                    # Map the equals key to hard-wrap the selection
                                                    map global normal = '|par -w$kak_opt_autowrap_column<ret>'
                                                    

                                                    Then, re-wrapping the current paragraph is <a-a>p=. With a bit of effort, you could probably wrap that into an insert-mode mapping for <a-q>. par is a fairly smart third-party rewrapping tool, smarter than coreutils’ fmt, but not quite as smart as Vim’s built-in formatting feature.

                                              1. 5

                                                I always thought it interesting how the GNU tools have this split-brain between emacs and vi keybindings. I know that readline (by default) uses emacs, but I believe you’re able to make it use vi-like keybindings instead (or maybe that’s just a feature of Bash?). The two tools I use daily that use vi keybindings are man and less. And then you can see GNU’s influence with info because they use emacs bindings.

                                                All in all, it’s a pain and I’m a vim guy so beyond single-line editing on the command line, emacs bindings are completely foreign to me. As a result info pages are almost useless to me because I have no idea how to correctly navigate them. ¯\_(ツ)_/¯

                                                1. 12

                                                  I believe you’re able to make it use vi-like keybindings instead (or maybe that’s just a feature of Bash?)

                                                  That’s actually mandated by POSIX, as part of the definition of set for the shell. There was originally a proposal to have an “emacs” mode as well, but as the POSIX rationale document states:

                                                  In early proposals, the KornShell-derived emacs mode of command line editing was included, even though the emacs editor itself was not. The community of emacs proponents was adamant that the full emacs editor not be standardized because they were concerned that an attempt to standardize this very powerful environment would encourage vendors to ship strictly conforming versions lacking the extensibility required by the community.

                                                  Gotta love Emacs users ;)

                                                  1. 2

                                                    The original FSF crew were all (most?) emacsphiles, according to Brian Fox.

                                                    Source: many discussions with Brian Fox.

                                                    1. 1

                                                      Aw, man, Brian Fox. What’s up with him these days? I wish I could have been there in the early days of the FSF.

                                                    2. 2

                                                      …I believe you’re able to make it use vi-like keybindings instead…

                                                      In ~/.inputrc:

                                                      set editing-mode vi
                                                      set keymap vi
                                                      

                                                      As a result info pages are almost useless to me because I have no idea how to correctly navigate them.

                                                      That’s why I install pinfo on every Linux machine I use. It’s still not vi-like bindings, but it’s lynx-like bindings, which are at least easy to learn.

                                                      1. 1

                                                        The two tools I use daily that use vi keybindings are man and less.

                                                        If you’re on some reasonably “normal”-ish Linux distro and haven’t gone out of your way to configure things otherwise, man is most likely just displaying its output via less, so those are kind of one and the same as far as keybindings go – and less actually isn’t a GNU program.

                                                        1. 1

                                                          FWIW, less supports both bindings.

                                                          1. 0

                                                            info pages

                                                            Pet peeve. They’re info manuals not pages. Manpages are called that because individually each was supposed to be but a single, one-page cheat sheet of the full Unix manual.

                                                            The whole point of TeXinfo was to generate full manuals all at once, and in multiple formats, with an index, a table of contents, chapters, menus, and hyperlinks. If you don’t like the text-based info reader, there is HTML and PDF output as well. Use those!

                                                            … but I know you’ll tell me next, if it’s not in a text-based terminal, you don’t want to read it. In that case, just read the raw .info[.gz] files. They’re plain text files with a few ASCII control characters.

                                                          1. 16

                                                            When I was very, very young I had an Apple II, and a few scattered magazines. One of the letters-to-the-editor asked for help getting a text-to-speech engine called “SAM” to work, and it stood out in my memory because where the rest of the magazine was interesting and informative, the editors’ reply to that letter was just “Never heard of it, maybe one of our readers knows something?” It was incredibly frustrating to my young mind, because they didn’t know the answer, and I didn’t know the answer, and I had no access to later issues of the magazine so I could never find out if anybody else knew the answer.

                                                            And here we are 30 years later and now I do know the answer, but I’ve forgotten what magazine it was, so I can’t write back and tell them.

                                                            1. 3

                                                              I was curious to see what the author of the PCG random number generator had to say about the recently-released Xoroshiro256** generator, but it’s not listed on that page yet. Luckily, there’s a blog-post about it: A Quick Look at Xoshiro256**.

                                                              1. 8

                                                                At one extreme, Arch Linux and Debian Unstable, where you can do just about anything but there’s only about 98% chance it’ll work on a given day.

                                                                At the other extreme, CentOS/Red Hat Enterprise Linux, where there’s a specific set of things that are guaranteed 100% reliable, and pretty much everything else will break.

                                                                In between are macOS and Windows, where most things work and keep working, but sometimes you reach for that one extra thing and wind up having to upgrade your OS. It’s a pain, but consider it an amortized cost over not spending five minutes a day tracing through shell scripts to find out where a particular thing is failing.

                                                                1. 3

                                                                  To be fair, I haven’t had unstable break on me for years. But I do tend to mostly track testing rather than unstable & keep a close eye on what apt wants to upgrade.

                                                                  1. 2

                                                                    I agree, I’ve had a pretty good experience with Debian unstable in the past 10 years or so (prior to that it used to break more). Contrary to the name, it isn’t really the anything-goes staging area (at least not anymore). To upload to unstable, a package is supposed to have been at least minimally tested, should build cleanly from source, should install cleanly without dependency issues, etc. There is a more anything-goes staging area for packages still working out the bugs, experimental. But you wouldn’t run it as a distribution, only pull in specific packages from there to test.

                                                                  2. 3

                                                                    MacOS itself does not break on every update* but homebrew packages break very often. After each update half of them stop working due to some library having wrong version or for some other reason. Packages often fail to install or install in broken state. Sometimes newer versions of packages has features disabled by default or entirely (i.e. GUI dropped). I know homebrew is mostly hobby project but I had no such bad experience in Gentoo or Arch. And without homebrew, Mac OS is just a glamorous runtime for Photoshop.

                                                                    `* Except starting from 10.13, update from 10.12 failed for me, leaving system in non-booting state, then after fixing that, each minor update started to install twice, now it even offers me to install ancient security update from January. This is literally Mac OS Vista.

                                                                  1. 24

                                                                    I can sympathise with jgm’s desire for a simpler spec; modern Markdown is “more congealed than designed”, to misquote Douglas Adams. However, I’m pretty sure one of Markdown’s original design goals was to make good-looking, readable plain-text documents, ever-so-slightly constraining existing conventions so that they could make good-looking, readable rich-text documents too.

                                                                    To dramatically reduce ambiguities, we can remove the doubled character delimiters for strong emphasis. Instead, use a single _ for regular emphasis, and a single * for strong emphasis.

                                                                    The trouble is that in plain-text documents, people traditionally used * and _ for level-one emphasis (read as “bold” and “underlined” respectively), but typographic tradition is that level-one emphasis is italic text. So “single for level-one emphasis, double for level-two emphasis” is the most natural, semantic translation.

                                                                    Shortcut references like [foo] would have to be disallowed (unless we were willing to force writers to escape all literal bracket characters).

                                                                    I don’t know how I missed it, but until this year I missed that shortcut references were even possible. I started off with long-form [text](url) references, which looked ugly and broke up the text, and eventually twigged to [text][tag] references which still look weird to people who don’t know Markdown (or people who haven’t seen that syntax before). Just being able to write [text] in running prose marks that text as special without overly distracting the reader, and if the next paragraph (or something at the end of the document) says [text]: http://example.com the association should hopefully be plain.

                                                                    Since we have fenced code blocks, we don’t need indented code blocks.

                                                                    Fenced-code blocks are weird and ugly unless you’re already familiar with Markdown, while indenting is a clear visual delimiter.

                                                                    Instead of passing through raw HTML, we should introduce a special syntax that allows passing through raw content of any format.

                                                                    I can appreciate this from a technical standpoint (it’s a simple rule that solves a whole class of problems!) but even without raw HTML support, Markdown is pretty heavily tied to the HTML document model. Consider Asciidoc, which is basically Markdown but for the DocBook XML document model instead. There’s definitely similarities to Markdown, but the differences run much, much deeper than just what kind of raw content pass-through is allowed.

                                                                    We should require a blank line between paragraph text and a list. Always.

                                                                    This also is an excellent technical opinion, but click through to the OP and look at the examples of the new syntax and tell me whether either of them look pleasing.

                                                                    Introduce a syntax for an attribute specification.

                                                                    This definitely makes Markdown more flexible, but doesn’t make it any prettier to read. Also, if anything it ties Markdown even closer to the HTML document model.

                                                                    Overall, these changes would move Markdown further from being a plain-text prettifier, and closer towards being a Huffman-optimized encoding of HTML. That’s not a bad thing, and certainly it seems to be what most people who use Markdown actually want, but it’s quite different from Markdown’s original goals.

                                                                    When the CommonMark first began (as “Standard Markdown”), they tried to get John Gruber involved, but as I recall he refused to take part and told them not to use the name “Markdown”. I felt he was being a jerk, but having thought about it, I wonder if maybe he felt a bit like Chuck Moore about ANSI Forth, that the real value was the idea of a hastily-written script that took something human-friendly and made it computer-friendly, and making a set-in-stone Standard with Conformance Tests would be the exact opposite of Gruber’s idea of Markdown, no matter how compatible it was to his script. I imagine something like ABC notation is much closer, despite being entirely unrelated.

                                                                    1. 8

                                                                      You raise valid points and I can certainly understand where you’re coming from with “[o]verall, these changes would move Markdown further from being a plain-text prettifier” (to which I would sort of concur.) I have some disagreements about fenced code blocks and a blank line between paragraph text and a list but they boil down to personal visual preferences. Code indenting is in my opinion ambiguous and I almost always insert a blank line between paragraph text and a list item. It’s great that original Markdown allows for both.

                                                                      On the other hand, I feel we have to acknowledge that original, i.e., canonical Markdown has evolved from Gruber’s original implementation and scope (simple blog posts?) to the lingua franca of text input in websites—such as the prompt I’m currently writing in—and the de facto plain text format or LaTeX pre-processor. For instance, I do all my writing in plain text Markdown and use Pandoc to produce LaTeX typeset documents, letters, and papers with bibliography. Heck, I even wrote a static blog engine on top of my publishing workflow to guarantee that one text source file produces the same document regardless of output medium (PDF, web, &c.) (The OCD kicked in, lol.) I am a big believer in plain text and separating content from formatting, and Markdown is a great, modern plain text format.

                                                                      Ultimately, I think we have to cater for and balance both realities. Gruber’s original simplicity of the format, which notwithstanding being ambiguous, should be the guiding principle, and the evolved scope of present-day Markdown. And, boy, that’s a tough one.

                                                                      1. 4

                                                                        I definitely agree. Like all successful technologies, Markdown has grown well beyond its creator’s intended limits, and as much as I miss pretty plain-text I much prefer Markdown to BBCode or the rich-text editing features browsers provide. ReStructuredText is a light-weight markup explicitly designed to support multi-target output and the kind of extensibility people want for Markdown, but I find it awkward and pedantic and I’d much rather use Markdown any day.

                                                                        I’m curious, though: you say “Code indenting is in my opinion ambiguous”, how so?

                                                                        1. 2

                                                                          First off, forgot to mention that I always use [link][tag] link references and didn’t know about tagless ones. Neat.

                                                                          Regarding your question: I feel like between tabbed and space-based indentation (which may output the source text differently when cating or writing in $EDITOR) and cases like list-nested blocks (should I indent again or maintain current indentation level?,) that you have to worry how different current and future Markdown interpreters will interpret the document. Instead, I find the fencing solution (with ` backticks, for the record) better and clearer: wherever I am in the document simply fence the code and be done with it—I don’t have to count or check the indentation.

                                                                          1. 1

                                                                            Follow-up: also what @myfreeweb and @Forty-Bot wrote below.

                                                                        2. 4

                                                                          [text][tag] references

                                                                          Ha! I only knew about the shortcut [text] references, TIL on the [text][tag] one :D

                                                                          indenting is a clear visual delimiter

                                                                          Indenting can be a pain to work with though. Especially in web comment fields!

                                                                          1. 4

                                                                            Indenting can be a pain to work with though. Especially in web comment fields!

                                                                            I end up having to paste text in and out of vim to format code correctly. Backticks are a much better solution imo.

                                                                            1. 1

                                                                              One of these days I’ll get around to writing a replacement for the “It’s All Text” extension that works with modern Firefox, then I can write all my comments in a real text editor!

                                                                              1. 3

                                                                                I’ve seen a few around, but haven’t found the time to test any of them to see how well they work.

                                                                          1. 6

                                                                            It almost feels like there’s a wall between the community and the users of what this community produces. Which is weird. We are an open community, with open development, no barriers for new contributors – and yet, there is such a distance between the community of users and the community of developers/designers/outreachers/etc.

                                                                            From the outside, it definitely feels like the GNOME community has its own particular vision and intended user-base, and isn’t much interested in feedback or criticism from outside that target demographic, which (again, from outside) I’d guess is non-technical end-users of Free Software operating systems. I suspect this is a build-it-and-they-will-come type play, and I truly hope GNOME succeeds, but it shouldn’t be a surprise that such a project gets little constructive feedback and involvement from traditional highly-technical users.

                                                                            (disclosure: I’ve been a happy user of GNOME for browsing/email/music/light development work since 3.0, but when I need to Get Stuff Done I have a custom GNOME/i3 hybrid I’d much rather use)

                                                                            1. 1

                                                                              Your quote from the article and comment reminds me of a 2003 rant from Jamie Zawinski: The CADT Model, complaining about Luis Villa closing jwz’s Gnome 1.x unresolved bug reports as a consequence of Gnome 2.x being released.

                                                                              If we rewound the history of open desktop software, after my proverbial pony, I would wish ~GNU had focused on reimplementing the then non-GPL Qt widget library as free software.

                                                                            1. 5

                                                                              The recommended line length in Black is 88 characters.

                                                                              I’ve participated in a few code standardization processes. By and large I’m an enormous fan: I read a lot of code and am helped by the consistency they promote. Python has particularly been kind to me here with pycodestyle and pylint.

                                                                              The thing I’ve learned from them though is that I’m the last person on the planet (or at least the three companies I’ve had a hand in standardizing) that still uses an 80-character terminal to code. It’s the one thing I have a (literal) hard line on. I note the nature of the problem often doesn’t get through and so I’m offered compromises that are as equally bad as to the original. I don’t think I’ve ever been offered “how about 88 characters” but I have worked with folk who themselves tell me they won’t tolerate anything less than 100+ but try to talk us somewhere between 81 and 99 characters.

                                                                              The terminal is 80 characters wide.

                                                                              1. 7

                                                                                The terminal is 80 characters wide.

                                                                                I’m a fan of short lines, but this is a pretty bad argument. What is The Terminal? Even without running a GUI like X11, you can get higher resolutions in framebuffer or whatever it’s called.

                                                                                IMO, the compelling argument for me for shorter lines is readability. I just can’t read long lines, I got lost and confuse by the end. I like reading down than across.

                                                                                1. 2

                                                                                  I had it put to me that code review is significantly easier with an 80 character limit. My worst case is working on my laptop (in terms of screen size), and I get 84 characters across in vimdiff before ‘running out of screen’.

                                                                                  1. 0

                                                                                    The terminal is a VT100, with support for an 80x24 display. The 80 character limit was present before then, however, via the 80-character punched card. Those having 12 lines rather than 24.

                                                                                    It is marvelous that we have a display technology backward compatible with the entire history of computing.

                                                                                    1. 6

                                                                                      I’m not sure I get your argument, though. Because old tech had a width of 80 characters, we should now? Old tech had a clock rate of 100 Mhz and little storage, but I doubt you’d argue we should be using machines with those specs. And the truth is: almost nobody is programming in a VT100 terminal today. So are you making a luddite argument for line length or is there an actual benefit to 80 characters?

                                                                                      1. 6

                                                                                        Long story short, lines are limited to 80 characters because Roman roads were designed for chariots drawn by two horses, side-by-side.

                                                                                        1. 1

                                                                                          I’m saying that if you have to choose a line length, in a standard, that 80 characters is the choice we already made. As a consequence of that choice, 88 characters is two lines. That if your choice is 80 characters or 88 characters, 80 characters has a superior and sufficient claim to be a standard.

                                                                                          Pep 8: Limit all lines to a maximum of 79 characters.

                                                                                          OpenBSD mailing list netiquette: Plain text, 72 characters per line

                                                                                          It is true that terminal emulators will emulate a display larger than 80 characters. I’m a fan of the -S flag to less when I’d like my input file to tell me how wide my display should be. I also routinely work with two terminals side-by-side on a graphical display. Both modes of working are fantastic. The later is enabled by a standard line length of 80 columns. When my terminal has an 81st column, I’m using that column. It’s not available for use by the standard.

                                                                                          1. 4

                                                                                            So is there any way to standardize on lines longer than 80 characters or is this it for eternity/the rest of our natural lives? What would be a compelling argument for longer lines, for you?

                                                                                            1. 2

                                                                                              The other area where the question of line length comes up is typography. There, line length is measured by words rather than characters owing to the proportional spacing of characters in a font. If I was dealing with source code in a typographic context the conventions of that field would also apply–possibly to the exclusion of monospace font conventions, possibly in combination with monospace font conventions, depending on how the code is typeset.

                                                                                              1. 2

                                                                                                Why would we change the standard to longer lines? They are more uncomfortable to read anyway.

                                                                                                1. 3

                                                                                                  You’re missing my point, I’m not saying we should change it just to change it, I’m asking why a decision made 50 years ago still applies today. If long lines are uncomfortable to read, that is at least an argument that would still apply today and not simply the momentum of history.

                                                                                      2. 2

                                                                                        You are not alone! Some of us are still guarding the purity of 79 characters wide code

                                                                                      1. 1

                                                                                        BKED isn’t limited to ASCII 256 characters thanks to having a GUI in disguise.

                                                                                        It’s introduced as a text editor, but if it’s handling and displaying more than 256 characters in a single file, it can’t store them as plain text files, so it must have used some special format. Wouldn’t it have been obvious that something weird was going on when other tools couldn’t read files that BKED had produced?

                                                                                        1. 1

                                                                                          Actually it just displays more than 256 characters and in the context of the article that means graphical chars for the UI. Now that it renders it all on its own, there’s no real need for a special format. I don’t know about the formula and graph examples shown at the end. Those probably had their own format.

                                                                                          But the way it has been written, it can store plain text files. That’s one of the main points of the article IIUC