1. 5

    Probably my five largest pain points:

    • Fix generics (<> is wrong and broken, use []).
    • Fix the broken Eq/PartialEq, Ord/PartialOrd design.
    • Throw out the useless semicolon rules.
    • Replace -> in function declarations with :.
    • Remove ! syntax for macros (creates the wrong incentives).

    Runner ups:

    • Fix/remove the horribly broken Path::join. (Or rather replace Path/PathBuf with an abstraction where you don’t have to decide how to implement the method, because it simply won’t compile.)
    • Fix env::home_dir.
    • Lack of a grown-up deprecation/removal policy.
    1. 12

      Throw out the useless semicolon rules.

      I seem to remember you and I have debated this before, but I don’t think it makes any sense to call the semicolon rules useless. Semicolons convert expressions into statements, and quite apart from the fact that there needs to be some syntax to do that, because Rust is an expression-based language, the visual distinction that you get from a line ending in a semicolon or not allows you to tell at a glance whether something is an expression or a statement.

      1. 1

        The problem with this kind of argument is that languages exist that work without mandatory ; and have none of those made-up issues.

        1. 11

          With semicolons this works fine:

          let foo = 5
            - 3;
          

          But this wouldn’t:

          let foo = 5
            - 3
          

          Because - 3 is a perfectly valid expression on its own. This is just one of the simplest examples I can imagine where “no semicolons” falls apart.

          JavaScript has similar issues when semicolons aren’t used and can lead to unexpected behaviour, so the problem isn’t new. Python side-steps this issue by requiring either being within ( or adding a \ at EOL.

          1. -5

            The problem with this kind of argument is that languages exist that work without mandatory ; and have none of those made-up issues.

            1. 11

              You’re an asshole. I’ve never seen you argue languages with people without acting like an asshole.

              1. 6

                What do you think you’re accomplishing here? Toddlerized argument via repetition pollutes conversations, demonstrates bad faith, and literally makes this site less valuable to others. I’d suggest that you follow the moderators’ advice: step away from the keyboard for a while until you can engage in mutually respectful discussions.

            2. 8

              The issues are not made-up. Please accept that people may disagree with your opinions for other reasons than losing grip on reality.

              Other languages have just chosen different trade offs in complexity of parsing, flexibility of the syntax, and risk of semantic changes caused by line wrapping.

          2. 9

            If 4 out of 5 of your top 5 complaints are just syntax, that’s pretty damn good IMHO.

            1. 2

              That’s what they say about BF too

              1. 2

                I don’t have 5 complaints about Brainfuck. I have one: Lack of semantic richness.

                It’s syntax is actually pretty good, considering what its semantics are. Other languages with the same semantics and different syntax exist, and they also suck.

            2. 4

              Fix generics (<> is wrong and broken, use []).

              I’ve heard this been said generally, but I never found out what the issue is. Isn’t this overloading array/map indexing?

              1. 8

                In Verona, we’re doing the same thing as Pony: [] is for generics, () is call syntax, array access uses call syntax on array objects. It’s a slightly odd decision in C-family languages to have separate syntax for them because C doesn’t have anything that is both indexable and callable (you can’t use array indexing on function pointers), but it’s a necessary hack if you want to write a single-pass compiler that runs on a system with 128 KiB of RAM. A lot of earlier languages used the same syntax because an array is, at the abstract level, just a map from integers to some other value. C doesn’t think of the abstraction like this and thinks of an array as syntactic sugar for pointer arithmetic. C++ keeps this distinction for standard library collection types, but I don’t think I’ve ever seen a C++ class that overloaded both operator() and operator[].

                The big issue with <> for generics (or anything else) is that < and > are both allowed in your program as stand-alone identifiers. This means that you can’t spot a missing angle bracket until you’ve done type checking, whereas you can spot a missing other kind of bracket / brace after tokenisation, before you even parse.

                1. 4

                  it’s not overloading array indexing, because [] for generics is in types, and [] for indexing is in expressions. As far as I can tell, rust grammar is sufficiently well designed that these are never ambiguous.

                  1. 7

                    Counterexample: foo[X](). Is this calling a generic function foo with X as a type argument, or is this calling the Xth element of the foo array?

                    1. 5

                      surely you mean foo::[X]() ? :-)

                      edit: that was snarky. It’s not worse than <>, we’d just have the turbo hammershark or something like that instead of turbofish, but you do have a point. D uses foo!(bar)(baaz) to separate template and regular arguments, for example.

                      1. 3

                        I might be wrong, but I think the main motivation behind using [ rather than < is exactly to get rid of the need to disambiguate expressions vs types via ::

                        If we are fine with replacing turbo-fish with turbo-snail, than yes, just switching the sigils works.

                      2. 1

                        Both!

                        Ok, foo isn’t an array, but indexing works for all sorts of types in rust not just arrays. It’s calling the Xth element of the collection foo, where foo happens to be a generic-function collection type.

                        1. 4

                          Yeah, using the same grammar for type references and expressions and disambiguating on semantic level might work. But that’s a way bigger change than just using [].

                          I wonder if optionality of type arguments would prevent this? Like, foo[()][()] where the first indexing with type () seems like it can be ambiguous. As it, is foo[()] a type argument, or a value argument with an inferred type argument?

                          My gut feeling is that this might not be a huge problem. It seems like “syntax unambiguously classifies the role of expression” property is already lost in Rust in minor way (the prime example would be constant ambiguity in patterns). So perhaps treating the language as a homogeneous tree of things, and classifying then according to semantics later would actually be a clearer model?

                    2. 2

                      The issue is that < and > are already used for less than and greater than signs, so Foo(baz) can either be (Foo less than bar) greater than baz or Foo instantiated with bar applied to baz.

                      Overloading indexing is fine, because it really is a kind of indexing. Indexing is a function (Container, Index) -> Value, and this is a function (GenericType, ConcreteType) -> ConcreteType. I.e. Container = Type and Index, Value = ConcreteType. As a result it doesn’t lead to the same kind of ambiguities.

                      1. 1

                        Wouldn’t that make AST construction dependent on types, and thus a chicken-egg problem similar to what C has with typedefs?

                        Forbidding uses of types in expression contexts world be limiting, e.g. you couldn’t “simply” use static methods or associated constants, because they’re expressions, but start with a type name.

                        1. 1

                          It would [1] if you need to represent Type[Type] and Expr[Expr] as different types of ast nodes. I don’t believe that you actually need to though.

                          [1] You could probably hack around this by checking what symbols are in scope too, but that’s ugly and would probably lead to poor handling of special cases and poor error messages.

                      2. 1

                        The argument I’ve heard is that it makes the grammar more complicated, and leads to some ambiguity issues, which results in things like Rust’s turbofish syntax (::<> iirc) or C++’s .template syntax.

                        A different approach, Ada uses neither <>, nor [], and instead requires associating a new name to a function-call like explicit instantiation of a type or generic function. It seems like it leads to a proliferation of types, but is an approach I haven’t heard much about.

                        1. 0

                          No. A side benefit of using [] for types is that it makes it unlikely that [] will be abused for array/map syntax.

                          1. 13

                            Why do you frame everything in such an antagonistic hyperbole?

                            You may not like [] for array indexing, but calling it “abuse” just brings down the discussion to shit-slinging level.

                            1. -2

                              Do you have an actual argument? (Because tone policing isn’t one.)

                              1. 21

                                No, that’s my job. Take a break from this thread and come back way kinder in the future. Feeling like you’re right doesn’t make it OK to heap scorn on people.

                            2. 2

                              What makes using [] for array/map syntax an “abuse”? What distinguishes it from literally any other syntax choice? I could claim that using () to surround function arguments is an abuse of those symbols, but I have absolutely nothing to back that up.

                          2. 2

                            creates the wrong incentives

                            What are the right incentives?

                            1. 1

                              ! gives macro authors a card blanche to do arbitrary stuff because “the user sees it’s a macro, right?”.

                              Instead, macros should be so well-behaved that users shouldn’t even need to know whether something is a macro or not.

                              1. 4

                                On the flip side, though, if macros aren’t distinguished with a !, then if the author of a macro does do something that isn’t “well-behaved”, the user has even less warning that it’s going to do that.

                                1. 0

                                  See comment above.

                                  1. 5

                                    Which comment? I’ve read all of your replies in this thread and I haven’t seen anything that refutes the assertion that removing ! would just make the developer experience worse if a macro does choose to do crazy things.

                                    1. 1

                                      Instead, macros should be so well-behaved that users shouldn’t even need to know whether something is a macro or not.

                                      Imagine that Rust users would keep filing bugs about stupid macros until the issue is resolved, just like they successfully did with unnecessary unsafe blocks.

                                      1. 4

                                        So your solution to this problem is to make Rust developers waste a bunch of extra time filing issues? Instead of the current state of play, which is that every user of these hypothetical macros has at least a measure of pre-warning that the thing they’re calling is a macro and therefore might perform some crazy AST transformation on its input? I haven’t run a survey, but I would hazard a guess that this “macros do arbitrary stuff” problem is not actually a real problem that Rust developers have, partly because the intention of macros is to allow you to do arbitrary stuff.

                                        1. 7

                                          Rust macro syntax is informed by experience from C where macros have a function-like syntax, and still do surprising arbitrary stuff.

                                          In C this syntax hasn’t stopped people from doing weird stuff with macros, but is stressing users who can’t be sure that everything that looks like a function call is just a function call.

                                          It’s also worth noting that in C it’s considered a good practice to use all-caps for macro names, because even though macros could look just like functions, users don’t want that.

                                2. 2

                                  Yes, except macros do things that functions literally cannot, such as ye olde try!().

                                  1. 0

                                    Worked out great, didn’t it?

                                    1. 5

                                      You’re advocating for try() syntax, not removal of macros.

                            1. 5

                              Do you think it’s reasonable to get mad at me just because I changed the software such that you can’t use it anymore?

                              Of course it isn’t, but I think it is reasonable for people who can’t use the software anymore to discuss what to do, and fork the software so that they can continue to use it, and even advertise to users of your software to switch to the fork in case they want to.

                              1. 5

                                I think this discussion was slightly flawed because it only had author and consumer as the two roles and that relationship was static. When you release something as open source, one of the parts of the social contract is that there’s an expectation that some of the people who find it valuable will contribute. They may contribute money, code, documentation, bug reports, logos, or in other ways. A good bug report with root-cause analysis is a valuable contribution.

                                Successful open source projects tend to be the ones that have a good pipeline for turning users into contributors. The other side of this social contract is that an open source project is that a project is willing for people to make this transition, from consumer to casual contributor to active contributor, and will welcome them.

                                1. 4

                                  Absolutely, but that’s very different from treating the author in a, well, unethical way. The issue is some people both take things way too personally (authors too!) and believe they have certain rights when they actually don’t. As described, the sooner people realize their rights, the better (and discussions should remain civil).

                                1. 11

                                  I can relate to this to some degree. But this is a very good lesson to learn: you don’t owe anything to any of those people. You shouldn’t really feel guilty. If they don’t put the effort to create a proper bug report, why should you put the effort to fix it? More mitigations include: issue templates, labels “can’t reproduce”, “question” or “help wanted”, a more strict policy clearly laid down in the README, or even just… disable the issues section. Just make it clear that it is your project and won’t generally work on other people issues, but may accept pull requests.

                                  I still think it’s worth it to publish the code, even if it only “works for you”. Others, who care, may find it valuable. I don’t know what project’s author talks about, how popular they are, or what community uses them (e.g. the entry barrier for Python is a lot lower than it is for other languages, so expect more poorly written complaints here), but I haven’t encountered this situation to the same degree. The compliments are really nice, even if rare (they wouldn’t feel special otherwise).

                                  1. 3

                                    shouldn’t really feel guilty

                                    I still just do though. This is a personal issue, but I can’t help feeling that way.
                                    I’ll probably just start disabling issues on my GH projects.

                                  1. 4

                                    Aww, that’s cute they use my library. I also attempted a similar tool (backups, but with generic exporters, including HTML so you would get the similar “static website” feel) with a friend. Alas, Telegram changes stuff pretty often, and both the lack of time and interest (on my part) led to the tool being archived. Regarding OP, the output seems really clean from the screenshots. It’s nice to see people build cool things upon my work.

                                    1. 1

                                      Does this also count?:

                                      fizzbuzz = lambda n: [[str(n), 'Buzz'][n % 5 == 0], ['Fizz', 'Fizz Buzz'][n % 5 == 0]][n % 3 == 0]
                                      
                                      1. 1

                                        No, I think that they would consider the dict lookup to be equivalent to a conditional. This is well supported, as historical advice for implementing a swtich-statement in Python has been to use a dict for dispatch by looking up functions for each case.

                                        I think the Python equivalent to the posted code would be…

                                        from itertools import cycle, count, islice
                                        fizzbuzz = lambda ds={3: 'fizz', 5: 'buzz', 7: 'quux'}: zip(count(), zip(*(cycle([[term], *[[]] * (div - 1)]) for div, term in ds.items())))
                                        print(*[*islice(fizzbuzz(), 20)], sep='\n')
                                        

                                        or the equivalent

                                        from itertools import repeat, cycle, repeat, count, islice, chain
                                        from functools import reduce
                                        from operator import add
                                        fizzbuzz = lambda ds={3: 'fizz', 5: 'buzz', 7: 'quux'}: islice(zip(count(), (reduce(add, ts) for ts in zip(*(cycle(chain([[term]], repeat([], div - 1))) for div, term in ds.items())))), 1, None)
                                        print(*[*islice((str(n) if not ts else ''.join(ts) for n, ts in fizzbuzz()), 20)], sep='\n')
                                        

                                        (I’m a little surprised that the posted Haskell version strictly returns strings, rather than some compound type with the original numeric value separate from the matched terms. There’s not much you can do with this output, other than print it to the screen…)

                                        1. 2

                                          I did not use any dictionaries, it’s a list. I’ll be honest, I also didn’t read the OP; I just thought it would be a neat tiny challenge to come up with something if-less in a Python one-liner :P

                                          1. 1

                                            Funny, because this was my Python solution. There is a conditional, yes, but no pattern matching… it’s quite minimal.

                                            f = "fizz"
                                            b = "buzz"
                                            fb = f+b
                                            block = [None, None, f, None, b, f, None, None, f, b, None, f, None, None, fb]
                                            from itertools import cycle
                                            list(map(lambda x, y: x or str(y), cycle(block), range(1, 101)))
                                            
                                        1. 6

                                          Similar concept, although I think this one only uses a single panel and battery? https://solar.lowtechmagazine.com/

                                          1. 3

                                            The Solar Protocol website cites Low Tech Magazine as inspiration. :) I find both facinating.

                                          1. 3

                                            Clicking through to your profile and skim reading a few entries from your blog, IMO lobsters would have loved to have seen all the tech posts there. A bunch of them would likely have been upvoted heavily.

                                            Obviously this isn’t the right forum for things like a guide to WorldEdit, but say “Python Ctypes And Windows”, “An Introduction To Asyncio” and the memory scanning thing would all be 100% on topic.

                                            1. 3

                                              Thank you! That’s nice to hear. I was purposedly holding off from posting the Writing our own Cheat Engine series because I would like to publish it as a show once it’s complete :)

                                              I did not even think about posting the others; I tend to be far too hard on myself when it comes to quality on the things I produce… Maybe you could submit them if you liked them though.

                                              1. 2

                                                would like to publish it as a show once it’s complete :)

                                                Good plan: part 1 probably wouldn’t have done well here if posted before at least part 2 was also written up. But I think it’s happily above the bar now. ^^

                                            1. 2

                                              For what it’s worth, unless you authored more than half of your submitted stories, you won’t be labelled a heavy self promoter: https://github.com/lobsters/lobsters/blob/9c0e2c03e2c14a2fba39298e76258cc460f6f003/app/models/user.rb#L478

                                              1. 1

                                                IIUC, that also includes ask stories. Shouldn’t those be excluded from “heavy self promotion”?

                                                Also, what is this function used for? Moderation?

                                                1. 2

                                                  Also, what is this function used for? Moderation?

                                                  As far as I can see, it’s only a flag shown to moderators, but until one of those chimes in, that ratio is the closest thing to an authoritative answer for what the administration considers acceptable.

                                              1. 4

                                                I feel like it’s hard to say in general, especially when it comes to “spacing” (obviously someone posting a new submission of their own every day would be too much, and nobody would notice if this happens yearly). My guideline for this has been to check how involved a user is in the rest of the community, and see if they only come here to dump their own articles for the sake of exposure, or if they contribute otherwise.

                                                I remember that pushcx once posted some statistics on which user posted submissions from which domain, and there was some suggestion like limiting this percentage to a certain point. So I can’t say for sure what the moderation will do.

                                                Another point I’d like to bring up again, at the risk of being banned, is the general hiding of domains, not just tags. I feel that this would be a good way to counteract this precise issue, because a spammer might use popular tags to avoid people from filtering out their stories, but when they start getting annoying, users can independently start ignoring their domains, thus making the spamming less attractive (at least for logged-in readers).

                                                1. 2

                                                  Such an automated system to limit the submissions for a domain, or any other heuristics, could be tricky to balance. Probably it’s best left to the community through reports. Though I don’t know if other’s methodology for reporting is as thorough as yours.

                                                  1. 1

                                                    One argument I can think of that would be in favour of such heuristics, would be that there is no real harm in having a timeout on when you can post an article from a domain again. Will anything really change is you have to wait a day or two?

                                                    1. 2

                                                      I think ewintr’s comment here puts it nicely: artificial limitations might degrade the overall experience with dumb workarounds.

                                                      1. 1

                                                        That is true, for most complex heuristics a pathological counter-example can be found that breaks its assumptions.

                                                1. 3

                                                  Going to keep working on in my series Writing our Own Cheat Engine, really happy how it’s turning out so far.

                                                  1. 1

                                                    I’m pleased to see how much this list has grown, and it includes my favorites, “the surprising comma” and the “not knot”. This list is probably not surprising to anybody who has had the experience of implementing Python’s grammar, but it is a very good resource for beginning and intermediate Python users.

                                                    1. 1

                                                      Huh, that “surprising comma” is not what I thought it would be. I’ve been bitten by a different kind of “surprising comma”, var = value,. Wondering where the tuple was coming from took its sweet time. I wish tuples required parenthesis, at least for single elements.

                                                    1. 3

                                                      It was only a few months ago that I learned all Python integers are heavyweight, heap-allocated objects. This kind of boggled my mind. This post says that small ints in the 8-bit range are cached, but that’s not much of an optimization.

                                                      Anyone know why Python never started using pointer-tagging for ints, the way tons of other dynamic languages (from LISP to Smalltalk to Lua) do?

                                                      (Fun fact: Smalltalk-80 used pointer-tagging for ints up to +/-16384, and allocated objects for bigger ones. This was really apparent in the GUI: the text view would start to massively slow down as soon as the length of the text exceeded 16KB, because the layout and display would start allocating more and more numbers.)

                                                      1. 6

                                                        Virtually everything in Python is a “heavyweight” object, and neither ultra-low memory use nor blazing-fast bare-metal performance are particularly important to the average Python user. So there’s no real incentive to complicate the implementation with multiple differently-sized or differently-allocated backing types for int. In fact, Python moved away from that. In the old old days, Python had two integer types: one could only handle 32-bit values, and the other was a bignum. You could request the bignum with a suffix on an integer literal – 42 versus 42L – but despite some effort they were never quite interchangeable. The unification to a single (bignum) integer type began in Python 2.2 (released 2001) and now the old L suffix is a syntax error.

                                                        1. 1

                                                          neither ultra-low memory use nor blazing-fast bare-metal performance are particularly important to the average Python user

                                                          They’re not THAT important, but they would sure be nice. It would probably result in me writing more Python and less, say, Rust.

                                                        2. 2

                                                          If you launch an interpreter and use sys.getrefcount, you can see that the first 10 numbers already sum up to over 800 references (Windows, Python 3.9.1). I’m guessing it does help, or it probably wouldn’t be there.

                                                        1. 4

                                                          Wait, that’s the swtich-case from C.

                                                          1. 5

                                                            As far as I know, that only deals with specific values. Python’s let you match on structure, with things like checking for isinstance, hasattr, len, and key in map. It also features guards.

                                                            1. 4

                                                              Not quite, this could cause confusion: https://brennan.io/2021/02/09/so-python/

                                                              See the PEP for some examples: https://www.python.org/dev/peps/pep-0622/

                                                              It’s more like a rust’s match or an elixir’s case than a C’s switch.

                                                              1. 1

                                                                Based off the title I was expecting something like in-language regexp like stuff. Indeed it looks like switch case support (but probably with more matching flexibility than C).

                                                                Python never ceases to amaze me. So complex in some ways, so simple in others.

                                                                1. 4

                                                                  Pattern matching is a well-known and pre-eminent feature in many other languages. In statically-typed languages it even has an exhaustivity check which tells you if you didn’t cover all cases. E.g. check out this function that implements fizzbuzz in OCaml:

                                                                  let fizzbuzz n = match n mod 3, n mod 5 with
                                                                    | 0, 0 -> "FizzBuzz"
                                                                    | 0, _ -> "Fizz"
                                                                    | _, 0 -> "Buzz"
                                                                  

                                                                  It compiles with a warning (which can be set to error):

                                                                  Warning 8: this pattern-matching is not exhaustive.
                                                                  Here is an example of a case that is not matched:
                                                                  (1, 1)
                                                                  

                                                                  If you add a final case:

                                                                    | _ -> string_of_int n
                                                                  

                                                                  It compiles successfully. As it happens, when implementing fizzbuzz we need to handle the case where the number is not a multiple of 3 or 5. And the compiler catches that.

                                                              1. 11

                                                                Why is it so hard to supply a reasonable example in the article?

                                                                1. 22

                                                                  because it’s not an article, but a mailing list post for people with context?

                                                                  1. 20

                                                                    Python package maintainers rarely use semantic versioning and often break backwards compatibility in minor releases. One of several reasons that dependency management is a nightmare in Python world.

                                                                    1. 19

                                                                      I generally consider semantic versioning to be a well-intentioned falsehood. I don’t think that package vendors can have effective insight into which of their changes break compatibility when they can’t have a full bottom-up consumer graph for everyone who uses it.

                                                                      I don’t think that Python gets this any worse than any other language.

                                                                      1. 20

                                                                        I’ve heard this opinion expressed before… I find it to be either dangerously naive or outright dishonest. There’s a world of difference between a) the rare bug fix release or nominally-orthogonal-feature-add release that unintentionally breaks downstream code and b) intentionally changing and deprecating API’s in “minor” releases.

                                                                        In my view, adopting SemVer is a statement of values and intention. It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.

                                                                        1. 18

                                                                          In my view, adopting SemVer is a statement of values and intention. It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.

                                                                          A “statement of values and intention” carries no binding commitment. And the fact that you have to hedge with “as much as is reasonably possible” and “only knowingly break” kind of gives away what the real problem is: every change potentially alters the observable behavior of the software in a way that will break someone’s reliance on the previous behavior, and therefore the only way to truly follow SemVer is to increment major on every commit. Which is the same as declaring the version number to be meaningless, since if every change is a compatibility break, there’s no useful information to be gleaned from seeing the version number increment.

                                                                          And that’s without getting into some of my own direct experience. For example, I’ve been on the Django security team for many years, and from time to time someone has found a security issue in Django that cannot be fixed in a backwards-compatible way. Thankfully fewer of those in recent years since many of them related to weird old functionality dating to Django’s days as a newspaper CMS, but they do happen. Anyway, SemVer’s answer to this is “then either don’t fix it, or do but no matter how you fix it you’ve broken SemVer and people on the internet will scream at you and tell you that you ought to be following SemVer”. Not being a fan of no-win situations, I am content that Django has never and likely never will commit to following SemVer.

                                                                          1. 31

                                                                            A “statement of values and intention” carries no binding commitment.

                                                                            A label on a jar carries no binding commitment to the contents of the jar. I still appreciate that my salt and sugar are labelled differently.

                                                                            1. 2

                                                                              Selling the jar with that label on it in many countries is a binding commitment and puts you under the coverage of food safety laws, though.

                                                                            2. 6

                                                                              Anyway, SemVer’s answer to this is “then either don’t fix it, or do but no matter how you fix it you’ve broken SemVer and people on the internet will scream at you and tell you that you ought to be following SemVer”.

                                                                              What do you mean? SemVer’s answer to “this bug can’t be fixed in a backwards-compatible way” is to increment the major version to indicate a breaking change. You probably also want to get the message across to your users by pushing a new release of the old major version which prints some noisy “this version of blah is deprecated and has security issues” messages to the logs.

                                                                              It’s not perfect, I’m not saying SemVer is a silver bullet. I’m especially worried about the effects of basing automated tooling on the assumption that no package would ever push a minor or patch release with a breaking change; it seems to cause ecosystems like the NPM to be highly fragile. But when taken as a statement of intent rather than a guarantee, I think SemVer has value, and I don’t understand why you think your security issue anecdote requires breaking SemVer.

                                                                              1. 7

                                                                                What do you mean? SemVer’s answer to “this bug can’t be fixed in a backwards-compatible way” is to increment the major version to indicate a breaking change.

                                                                                So, let’s consider Django, because I know that well (as mentioned above). Typically Django does a feature release (minor version bump) every 8 months or so, and every third one bumps the major version and completes a deprecation cycle. So right now Django 3.1 is the latest release; next will be 3.2 (every X.2 is an LTS), then 4.0.

                                                                                And the support matrix consists of the most recent feature release (full bugfix and security support), the one before that (security support only), and usually one LTS (but there’s a period at the end of each where two of them overlap). The policy is that if you run on a given LTS with no deprecation warnings issued from your code, you’re good to upgrade to the next (which will be a major version bump; for example, if you’re on 2.2 LTS right now, your next LTS will be 3.2).

                                                                                But… what happens when a bug is found in an LTS that can’t be fixed in a backwards-compatible way? Especially a security issue? “Support for that LTS is cut off effective immediately, everybody upgrade across a major version right now” is a non-starter, but is what you propose as the correct answer. The only option is to break SemVer and do the backwards-incompatible change as a bugfix release of the LTS. Which then leads to “why don’t you follow SemVer” complaints. Well, because following SemVer would actually be worse for users than this option is.

                                                                                1. 3

                                                                                  But… what happens when a bug is found in an LTS that can’t be fixed in a backwards-compatible way?

                                                                                  Why do people run an LTS version, if not for being able to avoid worrying about it as a dependency? If you’re making incompatible changes: forget about semver, you’re breaking the LTS contract, and you may as well tell drop the LTS tag and people to run the latest.

                                                                                  1. 1

                                                                                    you may as well tell drop the LTS tag and people to run the latest

                                                                                    I can think of only a couple instances in the history of Django where it happened that a security issue couldn’t be fixed in a completely backwards-compatible way. Minimizing the breakage for people – by shipping the fix into supported releases – was the best available option. It’s also completely incompatible with SemVer, and is a great example of why SemVer is at best a “nice in theory, fails in practice” idea.

                                                                                    1. 3

                                                                                      Why not just tell them to upgrade? After all, your argument is essentially that stable APIs are impossible, so why bother with LTS? Every argument against semver also applies against LTS releases.

                                                                                      1. 3

                                                                                        After all, your argument is essentially that stable APIs are impossible

                                                                                        My argument is that absolute perfect 100% binding commitment to never causing a change to observable behavior ever under any circumstance, unless also incrementing the major version at the same time and immediately dropping support for all users of previous versions, is not practicable in the real world, but is what SemVer requires. Not committing to SemVer gives flexibility to do things like long-term support releases, and generally people have been quite happy with them and also accepting of the single-digit number of times something had to change to fix a security issue.

                                                                                  2. 2

                                                                                    “Support for that LTS is cut off effective immediately, everybody upgrade across a major version right now” is a non-starter

                                                                                    If it’s a non-starter then nobody should be getting the critical security patch. You’re upgrading from 2.2 to 3.0 and calling it 2.2.1 instead. That doesn’t change the fact that a breaking change happened and you didn’t bump the major version number.

                                                                                    You can’t issue promises like “2.2.X will have long term support” because that’s akin to knowing the future. Use a codename or something.

                                                                                    1. 7

                                                                                      It’s pretty clear you’re committed to perfect technical adherence to a rule, without really giving consideration to why the rule exists. Especially if you’re at the point of “don’t commit to supporting things, because supporting things leads to breaking SemVer”.

                                                                                      1. 4

                                                                                        They should probably use something like SemVer but with four parts, e.g. Feature.Major.Minor.Patch

                                                                                        • Feature version changes -> We’ve made significant changes / a new release (considered breaking)
                                                                                        • Major version change -> We’ve made breaking changes
                                                                                        • Minor version change -> Non breaking new features
                                                                                        • Patch version change -> Other non-breaking changes

                                                                                        That way 2.*.*.* could be an LTS release, which would only get bug fixes, but if there was an unavoidable breaking change to fix a bug, you’d signal this in the version by e.g. going from 2.0.5.12 to 2.1.0.0. Users will have to deal with the breaking changes required to fix the bug, but they don’t have to deal with all the other major changes which have gone into the next ‘Feature’ release, 3.*.*.*. The promise that 2.*.*.*, as an LTS, will get bug fixes is honored. The promise that the major version must change on a breaking change is also honored.

                                                                                        SemVer doesn’t work if you try to imbue the numbers with additional meanings that can contradict the SemVer meanings.

                                                                                        1. 3

                                                                                          This scheme is very similar to Haskell’s Package Versioning Policy (PVP).

                                                                                        2. 1

                                                                                          I’m saying supporting things and adhering to SemVer should be orthogonal.

                                                                                  3. 5

                                                                                    every change potentially alters the observable behavior of the software

                                                                                    This is trivially false. Adding a new helper function to a module, for example, will never break backwards compatibility.

                                                                                    In contrast, changing a function’s input or output type is always a breaking change.

                                                                                    By failing to even attempt to distinguish between non-breaking and breaking changes, you’re offloading work onto the package’s users.

                                                                                    Optimize for what should be the common case: non-breaking changes.

                                                                                    Edit: to expand on this, examples abound in the Python ecosystem of unnecessary and intentional breaking changes in “minor” releases. Take a look at the numpy release notes for plenty of examples.

                                                                                    1. 7

                                                                                      Python’s dynamic nature makes “adding a helper function” a potentially breaking change. What if someone was querying, say, all definitions of a module and relying on the length somehow? I know this is a bit of a stretch, but it is possible that such a change would break code. I still value semver though.

                                                                                      1. 3

                                                                                        The number of definitions in a module is not a public API. SemVer only applies to public APIs.

                                                                                        1. 4

                                                                                          If you can access it at run-time, then someone will depend on it, and it’s a bit late to call it “not public”. Blame Python for exposing stuff like the call stack to introspection.

                                                                                          1. 2

                                                                                            Eh no? SemVer is very clear about this. Public API is whatever software declares it to be. Undeclared things can’t be public API, by definition.

                                                                                            1. 7

                                                                                              Python has no concept of public vs private. It’s all there all the time. As they say in python land, “We’re all consenting adults here”.

                                                                                              I’m sure, by the way, when Hettinger coined that phrase he didn’t purposely leave out those under the age of 18. Language is hard. :P

                                                                                      2. 1

                                                                                        Adding a new helper function to a module, for example, will never break backwards compatibility.

                                                                                        Does this comic describe a violation of SemVer?

                                                                                        You seriously never know what kinds of things people might be relying on, and a mere definition of compatibility in terms of input and output types is woefully insufficient to capture the things people will expect in terms of backwards compatibility.

                                                                                        1. 6

                                                                                          No, it does not descripbe a violation of SemVer, because spacebar heating is not a public API. SemVer is very clear about this. You are right people will still complain about backward compatibility even if you are keeping 100% correct SemVer.

                                                                                    2. 6

                                                                                      I don’t think I have a naïve view of versioning; putting on my professional hat here, I have a decade of experience dealing with a dependency modeling system that handles the versions of hundreds of thousands of interrelated software artifacts that are versioned more or less independently of each other, across dozens of programming languages and runtimes. So… some experience here.

                                                                                      In all of this time, I’ve seen every single kind of breaking change I could imagine beforehand, and many I could not. They occurred independent of how the vendor of the code thought of it; a vendor of a versioned library might think that their change is minor, or even just a non-impacting patch, but outside of pure README changes, it turns out that they can definitely be wrong. They certainly had good intentions to communicate the nature of the change, but that intention can run hard into reality. In the end, the only way to be sure is to pin your dependencies, all the way down, and to test assiduously. And then upgrade them frequently, intentionally, and on a cadence that you can manage.

                                                                                      1. 1

                                                                                        I don’t think I have a naïve view of versioning; putting on my professional hat here, I have a decade of experience dealing with …

                                                                                        Here here. My experience isn’t exactly like @offby1’s but I can vouch for the rest.

                                                                                      2. 6

                                                                                        I would agree if violations were rare. Every time I’ve tried to solve dependency issues on Python, about 75% of the packages I look into have broken semver on some level. Granted, I probably have a biased sampling technique, but I find it extremely hard to believe that it’s a rare issue.

                                                                                        Backwards compatibility is hard to reason about, and the skill is by no means pervasive. Even having a lot of experience looking for compatibility breaks, I still let things slip, because it can be hard to detect. One of my gripes with semver is that it doesn’t scale. It assumes that tens of thousands of open source devs with no common training program or management structure all understand what a backwards breaking change is, and how to fix it.

                                                                                        Testing for compatibility breaks is rare. I can’t think of any Python frameworks that help here. Nor can I think of any other languages that address this (Erlang might, but I haven’t worked with it first-hand). The most likely projects to test for compatibility between releases are those that manage data on disk or network packets. Even among those, many rely on code & design review to spot issues.

                                                                                        It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.

                                                                                        It’s more likely that current package managers force you into semver regardless if you understand how it’s supposed to be used. The “statement of values” angle is appealing, but without much evidence. Semver is merely popular.

                                                                                        1. 7

                                                                                          I guess this depends on a specific ecosystem? Rust projects use a lot of dependencies, all those deps use semver, and, in practice, issues rarely arise. This I think is a combination of:

                                                                                          • the fact that semver is the only option in Rust
                                                                                          • the combination of guideline to not commit Cargo.lock for libraries + cargo picking maximal versions by default. This way, accidental incompatibilities are quickly discovered & packages are yanked.
                                                                                          • the guideline to commit Cargo.lock for binaries and otherwise final artifacts: that way folks who use Rust and who have the most of deps are shielded from incompatible updates.
                                                                                          • the fact that “library” is a first-class language construct (crate) and not merely a package manager convention + associated visibility rules makes it easier to distinguish between public & private API.
                                                                                          • Built-in support for writing test from the outside, as-if you are consumer of the library, which also catches semver-incompatible changes.

                                                                                          This is not to say that semver issues do not happen, just that they are rare enough. I’ve worked with Rust projects with 200-500 different deps, and didn’t pensive semver breakage being a problem.

                                                                                          1. 5

                                                                                            I would add that the Rust type system is expressive enough that many backwards incompatible changes require type signature changes which are much more obvious than violations of some implicit contract.

                                                                                        2. 4

                                                                                          to be either dangerously naive or outright dishonest

                                                                                          This phrase gets bandied around the internet so much I’m surprised its not a meme.

                                                                                          SemVer is … okay, but you make it sound like lives depend on it. There’s a lot of software running mission critical systems without using SemVer and people aren’t dying everyday because of it. I think we can calm down.

                                                                                      3. 3

                                                                                        Thats the problem of the package management being so old. Back then semantic versioning wasnt that common and it never really caught on. In my opinion the PyPA should make a push to make more packages use semantic versioning. I‘m seeing this trend already, but its too slow…

                                                                                      1. 1

                                                                                        Here’s some additional “patterns” I found recently, in Rust API Guidelines - Future proofing.

                                                                                        1. 1

                                                                                          For Rust, the most amazing CLI parser library you can find is Pest.

                                                                                          1. 2

                                                                                            Any thoughts on why Pest is better that structopt? (which seems to be the default people turn to)?

                                                                                            1. 1

                                                                                              Actually, I cannot edit my answer know, but I meant to say Nom, not Pest, and I wanted to talk about parser combinators. Plenty of options for sure!

                                                                                              1. 4

                                                                                                Neither Pest nor Nom are CLI parsers though… You might use them to implement a CLI parser, but they are not CLI parsers themselves. They are not things I’d recommend to people wanting to write a CLI tool in Rust for handling argv outside of very specialized circumstances.

                                                                                              2. 1

                                                                                                What about argument-specific parsing, like clap or structopt? I haven’t actually used any of these, but I see them being commonly cited.

                                                                                                1. 1

                                                                                                  In my experience there’s lots of good options, with slightly different tradeoffs. Pest isn’t specifically for command lines, it’s just a parser lib. For command line args specifically, there’s two main styles: ones that take a bunch of options and let you query them individually, like clap or argparse, and ones that let you just slap some annotations on a struct and parse your options into that struct, like structopt and argh. They’re all pretty good, just depends on what you want and how you want to get it.

                                                                                                  1. 1

                                                                                                    clap and structopt are both very good, but (and I think this is a big issue) they are both really substantial dependencies that in my own experience deliver a big hit to compile times. Compile times in Rust are an issue, so as powerful these libraries are, at this point I personally prefer not to use either of them.

                                                                                                    1. 1

                                                                                                      Compile times are definitely a pain, but if you plan on compiling the tool once and using it for months or years, I’d argue their flexibility and power pays off (although I’m guilty of avoiding them too, mostly because I haven’t done any serious Rust CLI…). Or even shipping the binaries to users.

                                                                                                      1. 1

                                                                                                        Small clarification, for those not as familiar with Rust (not directed at @alilleybrinker :)). Structopt is a wrapper around clap which adds Custom Derive procedural macros to allow placing attributes on struct definitions and having the argv parser inferred from such definitions. Most of the compile time is spent on the Custom Derive procedural macros and supporting crates. I.e. clap compiled without the optional features is usually very quick and does not add any significant compile time.

                                                                                                        Edit: full disclosure I am the original clap author.

                                                                                                      2. 1

                                                                                                        I’d also checkout Seahorse.

                                                                                                    1. 31

                                                                                                      Good advice, except the one about skipping man pages. Man pages are easier to use and almost always more useful than --help. They’re “rich text”, they automatically open in a pager and they have a pretty much standardized disposition and format.

                                                                                                      The only exception is when developers are too lazy to write them and instead generate them from some other, non-man documentation. These man pages usually turn out terrible. But honestly, mdoc is not that hard. If you care about user experience, you should write a man page for your program.

                                                                                                      They don’t work on Windows? Says who? I’m sure man is ported to Windows. And besides, why should the way Windows works dictate how I write UNIX tools? I use both Windows and UNIX, and I don’t expect them to be the same type of system – that’s why I use both.

                                                                                                      Also, citation needed on the claim that “not enough people” use man pages. How can you know that?

                                                                                                      1. 5

                                                                                                        I think your advice on man pages here is a bit off, or at least, it’s not the advice I would give.

                                                                                                        The only exception is when developers are too lazy to write them and instead generate them from some other, non-man documentation.

                                                                                                        You seem to be using “lazy” as a pejorative here, but I see it as a positive in this case. My main problem with writing the entire man page from scratch is that it duplicates a lot of the documentation that is shown in the --help output. So when the docs change in one place, you have to remember to make the same change in the other place too. This is annoying. So it’s not just a matter of sucking it up and writing the man page in the first place. There’s a question of continued maintenance here.

                                                                                                        I write a portion of my man page in asciidoc while the rest is generated straight from the argv definitions. Overall, I think the resulting man page looks pretty good. Certainly not terrible.

                                                                                                        They don’t work on Windows? Says who? I’m sure man is ported to Windows. And besides, why should the way Windows works dictate how I write UNIX tools? I use both Windows and UNIX, and I don’t expect them to be the same type of system – that’s why I use both.

                                                                                                        I don’t use Windows and I don’t know the state of man tooling on Windows. But I don’t know anyone who uses man on Windows regularly. People who use my CLI tools on Windows might not have cygwin or WSL or whatever else installed. It’s a native Windows program, so they don’t need it. It would be awfully user-hostile of me to only offer the complete docs for the tool in a way that doesn’t let them read it natively on their system. “Oh no sorry, in order to read the docs for this tool, you need to go out and download this other port of a UNIX program. Good luck setting it up.”

                                                                                                        That’s why I try to write man-level documentation in my argv definitions. So that when a user uses --help, they see exactly the same documentation that appears in the man page. And they don’t need a special tool to read it.

                                                                                                        Now obviously, man-level documentation is quite verbose, so the compromise I settled on is that the -h flag will show a much briefer set of docs.

                                                                                                        But the point is this. All the docs for all the flags are located in exactly one place and that place occurs right next to that flag’s definition in the code. So when you need to add a flag or change docs or whatever, everything is right there near each other. Locality is a useful feature.

                                                                                                        So I’d say we agree that “forget about man pages” is bad advice. It makes me sad that it’s being promoted here. But let’s not be blind to their downsides and the realities of maintaining cross platform tools.

                                                                                                        1. 2

                                                                                                          I think you have presented a very responsible way of generating and thinking about man pages here. I like your compromise between -h and --help. I agree with you for the most part!

                                                                                                          However, I think that your generated man page (rg.1), at least as it appears on my system (Alpine Linux), is an example of the risks of such generation. The NAME section is preceded by six empty lines, and the definition lists likewise have too much spacing in comparison to traditional man pages (example). These small details are easy to miss when generating man pages from another format.

                                                                                                          But this is not a fundamental problem with your method, it is probably an easy fix, nor am I against such generation in general, if only done responsibly and consciously. You have clearly put good effort into it. Other than the spacing, rg.1 looks like any other man page.

                                                                                                          1. 1

                                                                                                            Yes, to your point, I used to use Markdown and converted it to a man format using pandoc. But the results were quite terrible. I almost just decided to learn mdoc and figure out how to use that, but noticed a lot of other projects using asciidoc and getting decent results.

                                                                                                        2. 5

                                                                                                          I think man usage is closely tied to what type of programming and what environment/OS is being worked on.

                                                                                                          If you work mostly on JavaScript on windows almost all of the documentation (command or library-wise) you regularly interact with is going to be on the web. If you are doing C on BSD, then you probably lean on man pages a lot. Most of the devs I know at work so Java on Linux or C# on windows, and I doubt most of either group of them are used to man pages.

                                                                                                          And if there’s a windows implementation of man (not cygwin or WSL), I’d love to see it.

                                                                                                          That being said, mandoc/mdoc is delightful, and I wish it would become more popular and cause the authors to change their mind.

                                                                                                          PS powershell help is infuriating - it seems comprehensive, but built for a different thought process than I’ve been able to force myself into. Tips would be welcome.

                                                                                                          1. 7

                                                                                                            If you work mostly on JavaScript

                                                                                                            Indeed, I think this goes for web technologies in general. They don’t use man pages because they are not UNIX tools. The same goes for C++ and Java to some extent. No problem there.

                                                                                                            But when JavaScript developers start building UNIX tools, they should write man pages for them, because the man page is part of what a UNIX tool is. UNIX users expect their programs to have manuals.

                                                                                                            And if there’s a windows implementation of man

                                                                                                            I’m not sure to what extent the man program itself works on Windows, with how it locates man files across the system, but I know mandoc, which man uses to render man pages, is ported. You don’t need Cygwin or WSL to use it.

                                                                                                          2. 4

                                                                                                            To be fair if your only experience with man pages are Linux’s then you probably would almost always skip them.

                                                                                                            It was only when I started using OpenBSD that I realised the true power of the manpage.

                                                                                                            1. 4

                                                                                                              I’ve used both OpenBSD and various distributions of Linux, and I think this is somewhat true, but not entirely. OpenBSD does make a greater effort to document the system as such, which is difficult to do for a Linux distribution, because it uses a bunch of different parts written by different people. But reasonably, program documentation must be pretty much the same on both systems, at least for programs that exist on both. Linux also has a lot of good documentation for C functions.

                                                                                                              1. 4

                                                                                                                ^^ this. I literally have a dedicated vertically oriented “man screen” that synchs from word-at-cursor with a vim keybinding against the OpenBSD man pages first, with a few exceptions (ptrace, mmap, …) even if I am on a Linux machine a the moment.

                                                                                                              2. 3

                                                                                                                I’ve always got the feeling that the man pages themselfs are great but navigating them is not where it could be. Try finding something in man bash. All the information is there but you just can’t find it if you’re not already experienced.

                                                                                                                And ironically understanding less(1) through its man page is horrible. There are just too many not that useful esoteric options, commands, special notations for key combinations etc. pp. for a tool with such a simple and important purpose.

                                                                                                                Using manpages should be dead simple (like markdown for example).

                                                                                                                1. 2

                                                                                                                  My main issue with man is that I often struggle to find exactly the flag I need, as they tend to be rather lengthy. However, on a --help, I can search on my terminal and often find what I need in less steps.

                                                                                                                  As for your point, you should definitely discuss it with them, then the post can be improved for everyone else.

                                                                                                                  1. 12

                                                                                                                    FYI, man usually supports / to search

                                                                                                                    1. 4

                                                                                                                      Yes, but still, there is way more information, so it becomes harder to find exactly what I need. Sometimes such a simple search for a substring doesn’t cut it (if less has more fancy searches, I’m not aware of them).

                                                                                                                      1. 1

                                                                                                                        Yes, man, at least mandoc, supports tags – type ‘:t’ to jump. It also supports full text search across all manpages at once, using either whatis or apropos.

                                                                                                                        For writing manpages, there are also more friendly options than troff these days. For example, scdoc is reasonable. Here’s an example of scdoc documenting its own file format: https://git.sr.ht/~sircmpwn/scdoc/tree/master/scdoc.5.scd

                                                                                                                    2. 3

                                                                                                                      As was said, less (the pager man usually uses) supports / for searching and & for grep-like filtering. Much more convenient, in my opinion, than running --help output through grep.

                                                                                                                      That said, --help is fine for a concise specification of the tool’s options. It is not meant for more descriptive explanations. That’s where man pages are useful.

                                                                                                                    3. 1

                                                                                                                      I use app help all which prints out all help topics, and by piping that to less or your pager of choice, you’ve got essentially the same as you would have with a manpage, including formatting.

                                                                                                                      I could work on also providing a manpage with more or less the same text, but … what’s the point? I suppose that man app is kinda convenient, but it’s a very small convenience and comparatively a lot of work.

                                                                                                                      1. 4

                                                                                                                        The problem is that your x help all interface will always be inferior to man x because it is not a consistent interface that the user can expect every program to work with. I don’t think we appreciate that enough – being able to type man x to read a well-written documentation for every program on the system is a dream come true.

                                                                                                                        By not providing man pages, the developer not only annoys the user; he lowers the user’s expectation that man x will work for any x, which causes the user to look elsewhere for documentation in general, making man less popular, which in turn developers see as an excuse not to write man pages, which causes users not to expect man pages to be available, and so on forever, in a vicious circle. The more UNIX tools that are created without a man page, the less the man ecosystem will prosper.

                                                                                                                        In other words, I think providing man pages is responsible and commendable. For once in the UNIX world, there is a canonical and unified way of doing something. It would be a tremendous loss if we reverted to a thousand different and mutually incompatible help systems.

                                                                                                                        1. 2

                                                                                                                          Manpages aren’t consistent either; conventions differ widely. Here are four manpages from some fairly standard tools (curl, ls, top, xrandr), and the formatting of the flags is all different. I didn’t have to try many manpages to get this either: just load 4 tools from 4 different authors/ecosystems.

                                                                                                                          I’m not supposed to generate these manpages either according to your previous comment, so I’m supposed to endlessly muck about with this unreadable and uneditable troff nonsense for every update, which would be a duplicate of the inline CLI help and a massive waste of time IMO, just so you can type man app. Call it “lazy” if you will, but if we want to sling those sort of adjectives around then not wanting to type app help all – which gives almost identical results to man app – is much lazier, as the amount of effort for that is minimal, whereas the effort for writing non-generated manpages is pretty large.

                                                                                                                          1. 2

                                                                                                                            The formatting in those example is somewhat different, but mostly the same. The differences seem fairly irrelevant because I’ve read all of those man pages before and never noticed that they were slightly different in how they use indentation and bold text. You could say there’s a forest here to be seen beyond the trees.

                                                                                                                            What I said about generation in my original comment only goes for what one might call lazy and ignorant generation, characterized by these things:

                                                                                                                            • The end result does not read like a man page, because it is generated from a type of documentation entirely unlike a man page.
                                                                                                                            • The developer is largely clueless about troff/mandoc and the man page system and trusts a generation system written by somebody else, who might or might not know what they’re doing.

                                                                                                                            This is the type of generation I call lazy. Not generation in general. In fact, troff is a great target for generation – if you know what you’re doing. Your help x/y/z documentation, which I am not saying is bad, might be generated to troff with great success, but it depends on how it is written, and it must be done with care.

                                                                                                                            I realize it might sound a bit demanding: not only must you provide a man page, it must also be well-made. But naturally, if I think man pages should be provided, I also think they should be good. The way I see it, the user is always going to be lazy, but the developer should try not to be, and it is noble if one tries.

                                                                                                                            (Besides, developers regularly work with all kinds of arcane languages, not least the UNIX shell. Troff is just another one, and a relatively simple one at that, especially if you stick to m(an)doc, which doesn’t actually use the troff program per se AFAIK.)

                                                                                                                            (By the way, Git is a good example of a UNIX tool that has man pages like help x/y/z, except they’re man git-x/y/z.)

                                                                                                                            1. 3

                                                                                                                              But the result are exactly the same. Why spend a day (optimistic, will probably be more) writing some troff tool? Just because I know one “arcane” language doesn’t mean I should learn another just to fit the personal preference of some random person on the internet. Life is short. There are a lot of things of value I’d like to do. Dealing with troff is not one of them.

                                                                                                                              Good documentation doesn’t depend on the format; that’s just a boring implementation detail. Docs in HTML or PDF is far from my favourite format, but I’ll choose well-written HTML or PDF docs over some manpage any day, especially if that makes things easier for the author. Good documentation depends on clear writing, which is hard and time-consuming, so it’s probably best to spend time there than on technical implementation stuff. If manpages work for you: great, go for it! If something else works for you: use that.

                                                                                                                              1. 2

                                                                                                                                But the result are exactly the same.

                                                                                                                                What does this refer to? Maybe you can clarify.

                                                                                                                                Regarding format, it is not just an implementation detail. Man pages can be compared with academic papers, which also follow a more or less standardized format. People expect an abstract, an introduction, methodology, background, results and so forth. This format may sometimes feel like a burden for the authors, but ultimately, it helps them. And paradoxically, creativity tends to flourish precisely when it must work under restrictions.

                                                                                                                                I’ll choose well-written HTML or PDF docs over some manpage any day, especially if that makes things easier for the author

                                                                                                                                I think most UNIX users would be find it very annoying if some of their programs were documented in HTML, other in PDF, yet others in mdoc, etc. etc. Man is valuable because it is a universal documentation system. That doesn’t mean it has to be the only documentation system. Very complicated programs, such as Emacs, cannot and should probably not be documented entirely in man.

                                                                                                                                If manpages work for you: great, go for it! If something else works for you: use that.

                                                                                                                                Again, from the perspective I’ve represented here, I don’t think this is a proper way of thinking about it.

                                                                                                                                If man pages are an intrinsic part of what UNIX tools are, which I argue, then not providing a man page for a UNIX program is akin to not providing an installer for a Windows program. Some Windows programs are provided as zip files, which the user must install manually, but that is not in line with what Windows programs should be.

                                                                                                                                I think we should not move towards more fragmentation here. It would surely be easier for developers, because it is always easier (or at least it feels easier) to work without restrictions, but would be a great loss for users and the UNIX ecosystem in general.

                                                                                                                                1. 3

                                                                                                                                  But the result are exactly the same.

                                                                                                                                  What does this refer to? Maybe you can clarify.

                                                                                                                                  That a help command (or -h/--help flag) doesn’t need to be that different from what you get from a manpage; see e.g. this or this. You could argue this or that should be bold or underlined or whatnot, but I’d format it more or less the same in a manpage (not a huge fan of a lot of bold/underline text), and the principle is the same: short, concise plain-text documentation.

                                                                                                                                  The only tool that I still maintain with a manpage essentially just duplicates the --help output. It’s fairly pointless IMO, and just increases the maintenance burden for little reason (the --help output is actually better because it’s easier to align stuff better).


                                                                                                                                  Manpages haven’t been “the standard” for years; loads of CLI tools come without manpages, or essentially just manpage for the sake of it but without the full (or even useful) docs. GNU tools actually do this: the full documentation is usually available in info; for example yesterday I wanted to set the time to the year 2100 to test some potential 2038 bug, and date(1) has just:

                                                                                                                                     -s, --set=STRING
                                                                                                                                            set time described by STRING
                                                                                                                                  

                                                                                                                                  STRING … yeah, useful … what kind of string? It’s not really mentioned anywhere, but info date has much more detailed (actually useful) docs on this. This is common for a lot of CLI tools, not just GNU ones (although only GNU tools use that weird info thing).


                                                                                                                                  Your entire argument is essentially just “I prefer it like this, therefore everyone should be doing it like this”. It’s fine to prefer things and argue the advantages of this – I do this all the time – but I really don’t like it when people go around telling me what I “should” be doing with my spare time, that I’m “lazy”, or that it’s not “the proper way to think about it”. I have no doubt this was not your intention, but it comes off as fairly unpleasant and even aggressive to me. Do with that feedback what you will.

                                                                                                                                  1. 1
                                                                                                                                    1. Thanks for clarifying.

                                                                                                                                    2. I still maintain that man pages are a de facto standard, and I think your example proves that. Otherwise, why would the GNU tools even have man pages?

                                                                                                                                      Further, I would estimate that the majority of CLI tools packaged by popular Linux distributions actually do come with man pages. Sometimes, package maintainers even add them manually. Why would they do this, if they didn’t have an idea that man pages are or at least should be a standard documentation system that their users can rely on being available?

                                                                                                                                      I think your argument is akin to saying that there is no “standard” sense of morality or values within a culture, just because some people within that culture disagree with it. Democracy and liberalism are “standard” values in most Western countries even if some neo-fascists who live there disagree with it.

                                                                                                                                      A couple of exceptions don’t disprove the rule, and even large disobedience to a standard (e.g. the ancient Israelites) does not remove the standard (i.e. their ideal law).

                                                                                                                                    3. No, I argue for man because I think it is a valuable standard. I don’t think it’s perfect, it’s just okay. In fact, some mix between man and info would perhaps be better, in theory. But in practice man is the closest we’ve got.

                                                                                                                                      That’s why I think UNIX developers should put in the effort to write man pages, because I care about the preserving the man system.

                                                                                                                                    Sorry if I came across as arrogant. I tried as much as I could to argue as clearly and convincingly as possible and limit the claims of my own opinions by saying “I think”.

                                                                                                                                    I think this discussion has reached its limits, at least for now, but I’d be glad to discuss this some other time and really try to understand each other. Dank u voor de discussie!

                                                                                                                                    1. 1

                                                                                                                                      By the way, one of the things I’ve been meaning to write for years is a “unified documentation viewer”; typing doc ls will display the info page if it exists, or the man page if it exists, or the output of ls -h and/or ls --help if that works, etc. This would also include native support for rendering asciidoc and/or Markdown, as well as a pager that makes some things a bit easier.

                                                                                                                                      I think something along those lines is probably a more useful path forward than telling people to write manpages, and if done well it probably has the potential to outright replace manpages as the de-facto standard Unix documentation tool. I mean, there’s a reason people aren’t writing them, and addressing these reasons is more helpful (I have similar feelings to the “you should use {IRC,Email,…}” arguments that people use).

                                                                                                                                      1. 1

                                                                                                                                        doc ls

                                                                                                                                        That’s not a bad idea. I’d be very interested in such a tool. It would certainly be very useful, sort of along the lines of tldr.

                                                                                                                                        there’s a reason people aren’t writing them, and addressing these reasons is more helpful

                                                                                                                                        Interestingly, we both seem to take a pragmatic approach to the way we think about these things. In my case, I think that neither e-mail and man are perfect, but nonetheless I care about preserving them because they’re already so widespread, and having a standard, regardless of how imperfect that standard is, is better than fragmentation.

                                                                                                                                        I agree that it is ultimately ineffective to yell at people to use e-mail or man, when they’d be more persuaded if e-mail/man were actually made easier to use, which they certainly can.

                                                                                                                                        Sometimes, though, people just think that man is hard to use. They haven’t actually read the mandoc/mdoc manuals or really tried to find any information on how man pages are created. They’ve ruled out man(doc) beforehand, simply because a lot of people online, who in turn might not have any actual experience with man(doc) either, say that man is not modern or is hard to use or that not enough people read man pages anyway (for example, the page discussed in this thread).

                                                                                                                                        In these cases, while it isn’t helpful to yell at people, it might be helpful to suggest to them that their prejudices about man are incorrect and that they should look into it honestly and reassess it.

                                                                                                                                2. 1

                                                                                                                                  I wonder what @johnaj thinks of help2man or if he has another suggestion. help2man is what I’ve always suggested to cligen users.

                                                                                                                                  It might be “not so hard” to write in a common sub-dialect that winds up formatting well-ish in both help & man formats. Then muscle-memory man cmd folks and --help folks can all get along with minimal extra effort. Maybe sometimes a little auto-postprocessing on the output of the generated troff stuff could help.

                                                                                                                        2. 1

                                                                                                                          This has been one of our more controversial suggestions with reviewers too! This guide is intentionally meant to be a bit opinionated and to open discussion, so I’m enjoying the debate. ;)

                                                                                                                        1. 5

                                                                                                                          Started my new job, gotta figure life out and learn to live alone :)

                                                                                                                          1. 1

                                                                                                                            Congratulations on the new job!

                                                                                                                            1. 2

                                                                                                                              Thank you! I’m not sure if all programmer positions are like this, but I didn’t really look for this job, so I’m guessing I won’t be having trouble finding different ones in the future ^^

                                                                                                                          1. 24

                                                                                                                            I have taken to using the --force-with-lease flag instead of just --force. It gives some protection that the repo you are pushing to hasn’t been changed by someone else in the meantime, and you trash that work.

                                                                                                                            See e.g. http://weiqingtoh.github.io/force-with-lease/

                                                                                                                            1. 7

                                                                                                                              I really wish --force-with-lease was simply --force, and that this latter option was something like --force-for-reals. Saner defaults are always a good thing.

                                                                                                                              1. 1

                                                                                                                                Great tip! Thanks

                                                                                                                                1. 1

                                                                                                                                  Oh this is great! Thanks

                                                                                                                                  1. 1

                                                                                                                                    I learned about this after overwriting a coworker’s commit. Excellent flag, should really be the default for -f