Threads for mjn

  1. 1

    It’s so weird that there’s so much pressure to maintain 4k pages - it was worth the cost to Apple to take the compatibility issues from moving from 4k to 16k pages in hardware, I’m not sure why people are so hell bent on thinking that 4k is still the best option.

    1. 2

      It’s not, but for x86 hardware the only options are 4kb or 2mb, and even today 2mb is just a bit too big to be convenient for small programs. Looks like Aarch64 has more options (16 and 64 kb pages), which I actually didn’t know

      1. 1

        I’m not sure why people are so hell bent on thinking that 4k is still the best option.

        Which people are those?

        1. 1

          Has Apple managed this for macOS? I spoke to some of Apple’s CoreOS team about RISC-V having an 8 KiB default page size at ASPLOS a few years ago and their belief was that it would break large amounts of code. A lot of *NIX software assumes that you can mprotect a 4 KiB granule. For iOS, they’ve generally been willing to break a load of things and require people to put in porting effort but less so for macOS.

          1. 1

            They seem to have, as part of the Intel -> Apple Silicon transition (which I guess requires some porting effort anyway). On the M1/M2, macOS has 16k page sizes, and a quick GitHub search turns up that this did initially break various things that assumed 4k.

            1. 1

              The 4k assumption is actually even worse than just an “assumption” even code that tried to do the right thing: using getpagesize() didn’t work as it was a macro that expanded to 4k on intel machines (at least on Darwin), which made rosetta challenging. The M-series SoCs support some kind of semi-4k page allocation in order to deal with this problem under rosetta.

              For ARM builds on iOS have been 16k for many years preceding that, so a moderate amount of Mac software (which shared code with iOS) had clear source changes to do the right thing, and getpagesize() stopped being a macro so new builds of software got the correct thing.

          1. 4

            I use Python on an almost daily basis professionally and it has enabled me and my peers to get up and running with software remarkably quickly, but it seems that with every bigger release of Python there is syntax added to the language and I’m honestly curious as to what the amount of outcry has to be for something like structural pattern matching to be added. It must be a non-trivial amount of work to add this and maintain it for years to come, so was the amount of requests to support this non-trivial as well?

            Contrasting with a language like Go, which I’ve not used all that much, it seems like the maintainers deliberately eschew many niceties people have come to expect from Python just so the language as a whole is neater with fewer ways to do things. I’m not trying to shit on Python in any respect, I guess I just haven’t come into contact with reasons why any syntactical additions are still being made. To me, just because you can add new language constructs doesn’t mean you necessarily should; that’s what the underlying motivation and rationale PEP 635 essentially comes off as.

            1. 7

              I think what happens is that many Python developers are polyglots, see “good ideas” from other languages, and try to think of ways to incorporate it into Python.

              Structural pattern matching has always been a bit of a “magical feature from those other languages”, but I think that Rust and Scala really paved the way for proving the utility in very imperative code. And it works alright as a way for Python to finally get a switch statement.

              And to really go to bat for the Python developers… for every feature like this we get 5 or 10 things like “Now you can write T | V instead of typing.Union[T, V]”, or new_dictionary = {**old_dictionary, "extra_key": 1}, or merged_dictionary = dict_1 | dict_2.

              There’s lots of discourse around the “harder” feature additions, because so many of the other feature additions are, to be frank, no brainers. Just clear wins for everyone involved, things that do “what you expect”. Dictionaries preserving insertion order happened only fairly recently!

              The sort of summit of expressivity has not been reached in any language in my opinion, and Python is, I think, a very positive contributor to this design space.

              1. 2

                I just feel like this particular addition is

                A) Not applicable in a lot of codebases

                B) Not super-intuitive

                Which is a bad combination. 99% of Python developers will never use this, so when they are looking at piece of code that does use it, they’re going to scratch their head and ask “wait, how does this work?”

                for/else used to be the poster-child for infrequently used, often confused Python features, and probably it is still the worst offender, but I don’t think the language needs more competitors in the space.


                By comparison, f-strings (which I was skeptical of) are useful more or less everywhere, so everyone who isn’t a novice will have used them and know more or less how they work (minus understanding the edge cases around brackets inside an f-string).


                Here’s a quiz. What does this do?

                def f(*, arg1):
                  pass
                
                f()
                

                This feature is sort of borderline to me, but I’m curious if it’s more or less obvious than I think.

                1. 4

                  For me pattern matching feels Pythonic, mostly because it’s already idiomatic to use destructuring in a bunch of other places, like in assignments and looping constructs. So it felt like an odd gap that you couldn’t use destructuring in conditionals (a previous comment of mine with some examples). But there are a bunch of edge cases in this new feature so I’m not 100% sure how much I’ll use it in practice.

                  1. 2

                    Here’s a quiz. What does this do?

                       def f(*, arg1):
                           pass
                    
                       f()
                    

                    This feature is sort of borderline to me, but I’m curious if it’s more or less obvious than I think.

                    This blows up because parameters after * are keyword only. You have to pass, e.g., f(arg1="foo"). (Full disclosure, I guessed the right answer, but I definitely had to check the docs to be sure.)

                    I’m not sure if this is your point, but for me this is a great example of Python providing (way) too many knobs and dials.

                    1. 1

                      I feel like if you showed ~any programmer the switch statement they would be able to say what’s going on. Fancy if statements that look like how you write out case analysis for function definitions in math.

                      There are definitely foot guns! But “non-intuitive” feels off to me. I think it would be interesting to do some real polling on Python programmers for “normal usage”. Like the text adventure seems straightforward to me, and way better than the “first we do a length check then destructure” flow

                      stuff like default arguments being a function property so being a shared reference or … almost every syntactic async thing has a lot higher “this doesn’t do what I thought it does” ratio IMO. And kwarg-only parameters are pretty useful! Subjective though!

                      1. 1

                        I dunno, seems unclear to me when using a name declares a variable vs when it describes a pattern. But maybe I’m overthinking it.

                        1. 1

                          to be honest I was also under the impresison that the binding rules were confusing (mostly from reading examples from people who were arguing it was confusing). But going through the PEP tutorial above all the code was straightforward for me and also expressed patterns for things that I find extremely awkward in Python.

                          I think the biggest gotcha is around constants, since a bare name is always a capture pattern, but dotted name access is allowed. tbh I feel like there should have just been an “Escape symbol” to resolve this question…. but I think that fortunately this is a very easy lint (one that’s present in … every pattern match-included language). It’s not perfect, but I think in practice things will work out through osmosis.

                      2. 1

                        I genuinely did not know for/else was in the language. I’m kind of glad because I’ve used better-suited data structures and programming style because of it, but, wow.

                    2. 2

                      I’ve been writing Python as my go-to language since late 90s and just love how the language has evolved and how the rollout of new features influenced my style and way of thinking. Decorators, generators, comprehensions, context and type annotations all had a major impact on the way I design and write code. I very much like Go’s simplicity, but I find that Python and it’s community has managed to strike a good balance between simplicity (but not minimalism) and progress and keep the language evolving. Ruby’s evolution, for instance, looks a bit sad in comparison. JavaScript, for one, was a surprising success, where the new features really made the language more flowing and expressive, in particular the arrow functions and restructuring.

                      1. 1

                        Oh, it’s OCaml and GPL3. That’s neat.

                        1. 1

                          That’s also what I use! I use it in a star topology where my desktop/laptops/etc. each sync with a cloud VPS.

                          1. 1

                            Unison all the way. Has worked brilliantly for years. 600,000+ files totalling over 100GiB

                          1. 6

                            This is, incidentally, a thing I dislike a lot about Rust stylistically. A lot of Rust is written chaining a bunch of map()s and or_else()s and such.

                            It’s passable if, even as in this article, it’s a straight transform. But it rarely is. There’s often side effects (either unintentional bad ones or good ones that’d make life simpler (eg, a max/min, without using tuples on tuples))… or implicit awaiting in async version of this chaining (how many things are happening in parallel? Hope you did the right buffering call!) and… it’s just a beast to debug. A for loop would be way more obvious in a lot of cases (if for no other reason than to crack open gdb)

                            1. 6

                              In my experience there’s a lot of cases when writing Rust code where you have a simple transform and do want to write it using chained maps and or_else (or even more concisely with the ? operator). When what you’re doing isn’t actually a simple transform, it’s definitely useful to resort to a non-functional construct like a for-loop, but that’s ideally the heavier and less common thing to do.

                              1. 2

                                Why would a for loop be “heavier” than a chain of transforms?

                                If anything, the for loop is easier to optimize by the compiler, and easier to understand by the layperson, right?

                                1. 1

                                  I have no idea whether iterator chains or a for-loop is easier to optimize by the compiler - I’ve seen deep-dives into rustc compiler internals that have argued that iterator chains are actually faster at least sometimes, and I think the broader problem is that it’s difficult for a programmer to actually know which of several semantically-equivalent ways of writing a program will actually result in the most performant code.

                                2. 1

                                  This lines up with my Java intuition as well, although there are so many different Java styles that I don’t claim it’s a universal thing other Java programmers would agree with. If something is doing explicit for loops to transform a collection into another collection, my assumption is either: 1) it’s legacy pre-Java-8 code, written before java.util.stream existed, or 2) it’s doing complex or wonky enough logic that it doesn’t map nicely onto the standard operations, so really does need to drop down to a custom loop.

                                3. 4

                                  A lot of Rust is written chaining a bunch of map()s and or_else()s and such.

                                  I used to do this a lot. The APIs are there. It’s so tempting when there’s a function like map_or_else that looks like it was designed to solve your exact problem. But I think it’s a trap, and as you start writing it often becomes more complicated than anticipated.

                                  These days, I am more skeptical of the ‘functional’ style in Rust. I rely more on language features like match, ?, traits like From/TryFrom, and libraries like anyhow or serde. I couldn’t tell you why, but this is how I feel after using Rust for a couple years.

                                  1. 3

                                    I agree. Chaining in a single expression is usually less readable than a sequence of independent expressions.

                                    1. 2

                                      Yeah, we write a lot of functional heavily chained code in D, but it’s viable (IMO) because all our value types are immutable. No state, pure functions only. There’s a few keywords that are allowed to do mutation (each, sort), but we know to look out for them.

                                    1. 5

                                      Bertrand Meyer seems to be trying to organize and freely re-release his back catalog to the extent possible. A few weeks ago, the re-release of Object-Oriented Software Construction, 2nd edition (1997) had a small discussion here.

                                      1. 1
                                        1. 10

                                          This is a new PDF interpreter for Ghostscript, replacing the previous one that had been in use for decades. The previous interpreter was implemented in Postscript; the new one’s in C. The original one was in Postscript because PDF started life as a Postscript derivative and had a similar enough data model to make it convenient to implement PDF constructs in terms of equivalent Postscript constructs.

                                          They give three reasons for a rewrite:

                                          • Large Postscript programs are difficult to maintain, and it’s increasingly hard to find expert Postscript programmers to maintain it.
                                          • PDF has continued to add features while Postscript hasn’t, so the convenience of using Postscript to implement PDF has lessened over time. Ghostscript has had to extend Postscript with undocumented GS-only Postscript constructs, to mirror new features added to PDF.
                                          • Some readers (notably Adobe Acrobat) are extremely lenient in accepting malformed PDF files and fixing up errors on the fly, which means such files exist in the wild and users expect GS to handle them. But Postscript isn’t a great language for doing non-trivial error handling and recovery.

                                          On a meta level, I’m mostly impressed that a ground-up rewrite of something of this complexity seems to have been completed successfully.

                                          1. 6

                                            Do they have a publicly available document explaining why they chose C for interpreting a language with such a hostile attack surface? i would think any memory-safe language would be a better choice.

                                            1. 1

                                              Your thinking is entirely backwards. Ghostscript is a C project, including its maintainers, and anything different would need very strong reasons. Experience, portability, and control can be named as important factors.

                                          1. 1

                                            I’m moving at the end of May, so getting serious about organizing that. I’m even hiring movers this time and it’s still a huge hassle.

                                            1. 13

                                              Oops, there is a bug! Or at least, Knuth must be leaving something out about how parentheses in the original expression are handled. The example expression he gives in the paper, (X + Y) + W/Z, does have a pair of parentheses, and the scheme works fine on that example. But the parentheses there are in a place that doesn’t actually impact the result – the result would be the same if you wrote X + Y + W/Z.

                                              But Joomy Korkut on twitter pointed out that this scheme can change evaluation order when the original parentheses actually matter: (a+b)*c gets rewritten to ((((a)))+(((b)))*((c))) which is not equivalent!

                                              I’m curious whether there’s a fix for this case while staying in the spirit of quick-and-dirty text substitution. Will think about it a bit and update the post.

                                              Edit: There is in fact a straightforward fix: add even more parentheses! To ensure the original parenthesized expressions continue to “contain” all their original contents without anything escaping, add another replacement rule: replace each ‘(’ in the input with ‘((((’, and each ‘)’ with ‘))))’ (similar to the ‘(((’ and ‘)))’ added around the entire expression). Post updated.

                                              1. 6

                                                I’ve been experimenting a little with using cron jobs to bring back some of this “mildly dynamic” content within a static site generator framework. For example, it’s fairly easy to integrate things like the current weather into a page by just regenerating it once an hour; you don’t need it to be looked up on literally every page load. Other kinds of mildly dynamic content, like anything customized to a specific reader, is admittedly not a good fit, but a decent number of my personal uses of PHP-style stuff would work fine at once-an-hour update speeds.

                                                I’m not sure any of the popular SSG frameworks have real first-class support for this though. As the article notes, the low-friction part of being able to just insert a little code into the template is important.

                                                1. 3

                                                  Author here. I like this idea. Here’s another random example of mildly dynamic functionality: long ago as a child, I remember customising a PHP web forum so that the theme it used would vary based on the time of day. Not something that would work well internationally, but if most users of a forum were in the same time zone it worked.

                                                  The script I use to generate my site actually supports embedded JS, which is evaluated at generation time, PHP style, even if the output is ultimately static:

                                                  <de:script><![CDATA[
                                                    ...
                                                  ]]></de:script>
                                                  

                                                  The output is generated by manipulating a DOM imported from the original XHTML source file of each page to add headers, footers, etc. (Markdown files are automatically converted to XHTML first, then treated the same way.) These scripts can programmatically generate new DOM nodes and edit existing ones. The index on the front page is generated by such an embedded script.

                                                  1. 2

                                                    The output is generated by manipulating a DOM imported from the original XHTML source file of each page to add headers, footers, etc.

                                                    That sounds like the sort of thing that is very hard to understand and change after it’s written. You can do the same thing with JavaScript and have it actually use the user’s real timezone. Ideally it would just be a matter of toggling a class on the body like .morning-theme, etc. and having CSS do the rest.

                                                    It’s sort of incredible the stuff we did back in the day. It definitely had its advantages, but I think the biggest advantage was just the amount of free time I had as a young person. :-)

                                                1. 1

                                                  It was also interesting to see the reaction from open source developers to unsolicited pull requests from what looks like a bot (really a bot found the problem and made the PR, but really a human developer at Code Review Doctor did triage the issue before the PR was raised). After creating 69 pull requests the reaction ranged from:

                                                  I wonder if you’d get better reactions if a human made the PR and didn’t say it came from a bot.

                                                  1. 1

                                                    That’s exactly the quote that prompted me to share the article. I think there was recently a case in the linux kernel community that some university group was submitting (arguably bad) patches that were generated by a tool – it didn’t went well if I recall correctly. Maybe initial reactions would be better, but long term, if the project finds out, it would lead to loss of trust.

                                                    1. 1

                                                      It was the University of Minnesota and they got their entire university banned from submitting anything to the Linux kernel.

                                                      The biggest argument against stuff like this I saw was that the heads of the groups being tested against had not accepted to participate in the study.

                                                    2. 1

                                                      A flipside of that is that you might expect better analysis from a human if it had been filed under a human’s name. These are clearly mostly auto-generated bug reports, and a number of false positives were filed, despite the triaging (from just spot-checking: 1, 2). So filing them under a bot’s name is maybe more honest to manage expectations.

                                                    1. 32

                                                      --version/--verbose along with -v and -V is another set of flags that drives me crazy because it’s not standardized (especially with some programs which only provide the short flags).

                                                      On a side note I do think that -? is an odd translation from Windows land. With Windows command line arguments, /? is more common which is way easier to type.

                                                      EDIT: Updating this to be more constructive - if you’re designing a command line program, please consider including both --version and --verbose. I think -v is generally verbose and -V is generally version. They’re not as consistent, but if you provide short options, I’d recommend those.

                                                      1. 4
                                                        [justin.blank@localhost tms-app]$ java --version
                                                        openjdk 11.0.14 2022-01-18
                                                        OpenJDK Runtime Environment 18.9 (build 11.0.14+9)
                                                        OpenJDK 64-Bit Server VM 18.9 (build 11.0.14+9, mixed mode, sharing)
                                                        [justin.blank@localhost tms-app]$ java -version
                                                        openjdk version "11.0.14" 2022-01-18
                                                        OpenJDK Runtime Environment 18.9 (build 11.0.14+9)
                                                        OpenJDK 64-Bit Server VM 18.9 (build 11.0.14+9, mixed mode, sharing)
                                                        [justin.blank@localhost tms-app]$ java -v
                                                        Unrecognized option: -v
                                                        Error: Could not create the Java Virtual Machine.
                                                        Error: A fatal exception has occurred. Program will exit.
                                                        

                                                        I believe prior to version 8.x.y, --version would fail, but -version would work.

                                                        1. 3

                                                          I believe prior to version 8.x.y, –version would fail, but -version would work.

                                                          Yeah, Java uses single-dash “long” options. Even today I think it was only --version that was special-cased; the rest are still single-dash only. For example, java -classpath works but java --classpath does not.

                                                          I’d be curious about the history behind this. Although the linked post calls the short/long options distinction POSIX, I believe double-dash long options originated as a GNU convention. I could imagine 1990s Sun not being interested in GNU conventions. This is complete speculation though.

                                                          1. 2

                                                            X does too. I suspect people needed longer options only in late 80s/early 90s, and X did it before GNU. Even then, the longer options call back Multics.

                                                        2. 4

                                                          I used docker-compose down -v for years and thing that it gave me verbose output. Turn out that:

                                                          • with docker-compose -v it means prints version and exit
                                                          • with docker-compose down -v it means removed named volumes
                                                          1. 3

                                                            Don’t get me started on tools which need you to run programname help version

                                                          1. 14

                                                            Really surprised fstrings are faster than string.format. I thought it was just syntactic sugar for that.

                                                            1. 18

                                                              From disassembling both versions, it looks like Python added a new bytecode, FORMAT_VALUE, specifically for f-strings, while format is a plain function call. I’d guess most of the savings is skipping the function call (Python function calls are not fast).

                                                              >>> dis.dis('f"{x} {y}"')
                                                                1           0 LOAD_NAME                0 (x)
                                                                            2 FORMAT_VALUE             0
                                                                            4 LOAD_CONST               0 (' ')
                                                                            6 LOAD_NAME                1 (y)
                                                                            8 FORMAT_VALUE             0
                                                                           10 BUILD_STRING             3
                                                                           12 RETURN_VALUE
                                                              
                                                              >>> dis.dis('"{} {}".format(x, y)')
                                                                1           0 LOAD_CONST               0 ('{} {}')
                                                                            2 LOAD_METHOD              0 (format)
                                                                            4 LOAD_NAME                1 (x)
                                                                            6 LOAD_NAME                2 (y)
                                                                            8 CALL_METHOD              2
                                                                           10 RETURN_VALUE
                                                              

                                                              Edit: Digging a bit more, f-strings were initially implemented as syntactic sugar for a call to format, but that was changed due to semantic concerns rather than speed: If a user redefined the format function, the behavior of f-strings would change, which was considered strange/undesirable. So a new opcode was added to avoid f-string behavior depending on the current binding of format, with a side bonus of being faster. I guess format can’t compile to that new opcode precisely because user code is allowed to redefine it.

                                                              1. 5

                                                                I mucked about with this a lot for a silly optimization in a web framework and concluded that it’s not any specific opcode that makes f-strings faster, it’s fundamental to the design (the design it’s settled on now; no idea about the history). (Although, of course, loading format for each value might be bad enough to make the whole thing consistently slower than other options).

                                                                So it’s not exactly hard to be faster than str.formatstr.__mod__ (or "%s%s" % (foo, bar)) is also often signfiicantly faster. There are two reasons for this: special method lookups are faster than named method lookups, and %-formatting is inherently faster than {}-formatting due to its simpler parser and non-generic architecture. Getting back to f-strings, they can be faster than either because not only is there, as you’ve observed, no method call, there’s almost no (runtime) implementation to call either. Processing the f-string happened at compile time. FORMAT_VALUE takes care of formatting values, with a hardcoded shortcut for values that are already strings, while BUILD_STRING just concatenates a bunch of pieces.

                                                                But although f-strings can be faster than the other two, they aren’t always. They are pretty much always faster than str.format, but %-formatting sometimes beats them; the simplest case that I know where this is true is formatting a bunch of ints:

                                                                $ python3 -m timeit -s 'a, b, c, d = 1, 2, 3, 4' '"%d%d%d%d" % (a, b, c, d)'
                                                                200000 loops, best of 5: 1.59 usec per loop
                                                                $ python3 -m timeit -s 'a, b, c, d = 1, 2, 3, 4' 'f"{a}{b}{c}{d}"'
                                                                200000 loops, best of 5: 1.97 usec per loop
                                                                

                                                                This happens because of that non-generic architecture thing. %-formatting has to parse at runtime, but its parser is very fast. On the other hand, since it just innately knows what ints are, it doesn’t have to ask its operands how to format themselves. This also means it can just print the formatted int out directly into what will become the resulting string, while the f-string version needs to put it in a temporary string of its own to be concatenated by BUILD_STRING later.

                                                                1. 1

                                                                  I find it a little surprising that % formatting sometimes wins because it needs an extra memory allocation for the tuple for the arguments.

                                                                  1. 3

                                                                    It does, but the f-string version needs 4 extra allocations (for “1”, “2”, “3”, “4”).

                                                                2. 2

                                                                  Perl probably would have solved a similar problem by having the sugar compile to a call to CORE::GLOBAL::format rather than looking up format in the current scope.

                                                                  Then again, Perl would have solved this specific problem by having an op for string formatting. :)

                                                              1. 28

                                                                I wish the project success- there’s a lot of innovation still to happen in programming languages and I’m always glad to see new ones getting built.

                                                                That said, I’m a little skeptical of the recent trends in optional typing. I don’t dismiss the idea that we might be able to make it work, but my experience so far has been that optional typing ends up being a worst-of-both-worlds scenario. You don’t really get the benefits of a strong static type system, since you have to deal with potentially untyped values, but you still spend some amount of time appeasing the type checker for the code that you did end up trying to make well typed. In the best case scenario you end up with an un-typed language, but more often than not they turn out to have all of the safety and reliability of untyped languages with the quick development time and short iteration cycles of a typed language.

                                                                1. 16

                                                                  Agreed! The following struck me as a pretty typical “pitch” for optional / gradual typing, but it doesn’t make sense (to me):

                                                                  When prototyping, don’t specify any types, and Stanza will behave like a dynamically-typed scripting language. But when confident in your design, gradually add in types to make your code bulletproof.

                                                                  I tend to think about programs as transformations applied to data, so the literal first thing I think about is the types (the data). I don’t write the transformations until I’ve got the types mostly worked out anyway, so there’s no point at which I want to omit the types.

                                                                  1. 9

                                                                    I agree. I see optional typing as a way of adding some benefits of static typing to dynamic languages when you can’t start over from scratch. I don’t see the point of starting a language from scratch with optional typing, unless there is some high-value externality that induces it (Typescript wanting compatibility with Javascript, for instance).

                                                                    1. 7

                                                                      my experience so far has been that optional typing ends up being a worst-of-both-worlds scenario. You don’t really get the benefits of a strong static type system, since you have to deal with potentially untyped values, but you still spend some amount of time appeasing the type checker for the code that you did end up trying to make well typed

                                                                      As far as I can tell, every optionally typed language is simply going about it wrong: The default is that the programmer has to prove something to the type system, otherwise you get a static type error. A better default would be type annotations that cooperate with a runtime contract system for which only guaranteed runtime failures are converted to static errors. For cases where you do in fact want to force static checking, you could signal some assertion as static.

                                                                      Another way to think about it is that type systems produce three-valued logics: proven, unproven, and disproven.

                                                                      What happens when something is unproven is the key design point: Most type systems error. Unsound type systems “assume” the unproven thing is true. A contract system “checks” the unproven things at runtime. We should flip the default form “assume” to “check”, which requires runtime support. Explicit assume, runtime-check, and static-assert syntaxes could provide escape hatches for unsafe eliding of runtime checking, pushing out the boundaries of statically verified code, and requiring static proof respectively.

                                                                      1. 4

                                                                        You’ll like Pytype, which only errors on disproven code: https://github.com/google/pytype#how-is-pytype-different-from-other-type-checkers

                                                                        1. 2

                                                                          Neat!

                                                                          The other big one I’ve heard of (but have not tried) is Erlang’s “Dialyzer” based on “success types”: https://easychair.org/publications/paper/FzsM

                                                                          Unfortunately, I don’t think either of these perform have a runtime checking/contract system.

                                                                          1. 1

                                                                            we even quote and link to stanza in our typing faq :) never actually used stanza beyond a few toy programs, but i’m a big fan of the design and ideas behind it.

                                                                          2. 3

                                                                            A better default would be type annotations that cooperate with a runtime contract system for which only guaranteed runtime failures are converted to static errors.

                                                                            This is how SBCL handles Common Lisp’s optional type annotations by default, “Declarations as Assertions”: “all type declarations that have not been proven to always hold are asserted at runtime.”

                                                                            1. 1

                                                                              Sadly, as far as I can tell from the CL implementations I’ve seen, they only support the declaration of primitive types and arrays thereof. Things get really interesting once you get to user-defined type constructors, higher-kinded types, first-class functions, mutable objects, etc. Especially true if you want to preserve non-functional properties, such as constant space usage or predictable runtime. For example, there are trivial cases where contracts can cause behavior to become accidentally quadratic with naive implementations.

                                                                              1. 2

                                                                                It doesn’t include your entire wishlist, but CL’s type system does support user-defined type constructors, arbitrary unions and intersections of types, etc., which is enough to do algebraic data types and some mildly fancy types like “nonempty list” or “integer outside the range [-500,500]”. There’s an overview here: https://alhassy.github.io/TypedLisp

                                                                          3. 5

                                                                            I think most people who reach for optional typing, based on the pitch they use, really want whole program type inference. Haskell and a few others have this. Most languages with “inference” just have local inference and so you still have to “think about type annotations” but there is no reason to require that.

                                                                            1. 4

                                                                              Speaking as someone who prefers dynamic typing (for live coding, creative coding, generative design, and hobby projects), the benefits I see in using type annotations in a dynamic language are: speeding up performance sensitive code, and getting high quality error messages when I misuse a library function. For the latter, I want error messages to tell me which argument was bad in a call to a library function – I don’t want a stack trace from the interior of a library function that requires me to understand the code.

                                                                              Neither of these benefits requires that all type errors should be reported at compile time. If a statically typed library function runs fast and gives good error messages at runtime, then it meets the requirements.

                                                                              The only time I expect to “fight the type checker” is when I have descended into the language’s statically typed low-level subset to write some high performance code.

                                                                              I think that type annotations and compile time type errors are both excellent additions to a dynamically typed language. By the latter, I mean the compiler is able to deduce and report some run-time errors at compile time. This doesn’t lead to “fighting the type checker” if the static checker doesn’t report false positives (which static type systems will invariably do).

                                                                              My reading of the Stanza docs is that Stanza is not a pure dynamic language. It switches to static type checking rules (inaccurately reporting false-positive type errors on good code) if all of your variables have type annotations. It’s a compromise that attempts to appeal to both static- and dynamic-typing users, but I suspect doesn’t give either side all that they want. So if, by “optional typing”, you mean this kind of compromise, then I may agree.

                                                                              1. 1

                                                                                that’s been my experience with Typescript, for sure.

                                                                                1. 1

                                                                                  Typescript’s an interesting one, in that the strictness is adjustable and can be ramped up as you migrate your JS code over. For new projects you just start with super-strict settings.

                                                                                  1. 2

                                                                                    And yet it’s only so strict. No runtime checks and no runtime references to types.

                                                                                    1. 2

                                                                                      The whole point is to get rid of the need for runtime checks :)

                                                                                      1. 2

                                                                                        That’s just not possible if there’s any IO. Well… maybe it’s not technically not possible but it’s culturally not possible. Maybe if every library in the world used strict TypeScript and didn’t do casts then there wouldn’t need to be runtime checks.

                                                                                        1. 1

                                                                                          Which is why a single any rapidly spreads out like cancer and turns your typedefs into lies.

                                                                                          1. 2

                                                                                            It doesn’t have to be any it can be as too. For example: await fetch().json() as MyAPIResponse

                                                                                      2. 1

                                                                                        That is the theoretical aim, but in practice it winds up just being a half-typed codebase IME. Kind of a shame tbh. But, it’s turned me off of believing this language direction is trustworthy, as opposed to better linter.

                                                                                  1. 3

                                                                                    For me the takeaway is that fork() isn’t a great abstraction for spawning processes on modern OS’s.

                                                                                    1. 3

                                                                                      You might be interested in the paper “A fork() in the road”, from HotOS 2019 (https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf), which argues:

                                                                                      The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog the ways in which fork is a terrible abstraction for the modern programmer to use, describe how it compromises OS implementations, and propose alternatives.

                                                                                      As the designers and implementers of operating systems, we should acknowledge that fork’s continued existence as a first-class OS primitive holds back systems research, and deprecate it. As educators, we should teach fork as a historical artifact, and not the first process creation mechanism students encounter.

                                                                                      1. 1

                                                                                        Yeah, I was vaguely aware that fork()/exec() was not highly regarded these days. This is a kind of wild practical demonstration of the problems with it though.

                                                                                    1. 3

                                                                                      Oof, I never thought about it before, but it makes sense that python can have data races. I’m assuming the GIL locks atomically on each instruction so if you interleave load, change, and store instructions in different threads it’s possible to end up with races. I don’t generally write multithreaded code that way in Python, but I wonder how many bugs I’ve created since I didn’t realizing this :(

                                                                                      1. 4

                                                                                        I’m assuming the GIL locks atomically on each instruction

                                                                                        I just realized an exception to this: a lot of instructions that look primitive can actually call back into Python code (for example BINARY_ADD can call an __add__ method) so those must yield the GIL in the same way a CALL_FUNCTION instruction would. So, whether an instruction is atomic can depend on what data is passed in.

                                                                                        1. 3

                                                                                          I’m assuming the GIL locks atomically on each instruction

                                                                                          That appears to be the case. The official Python FAQ has a question, What kinds of global value mutation are thread-safe?, with the answer:

                                                                                          In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via sys.setswitchinterval(). Each bytecode instruction and therefore all the C implementation code reached from each instruction is therefore atomic from the point of view of a Python program.

                                                                                          For those of us who don’t have the Python bytecode model memorized, the standard package dis can provide some insight into what is actually atomic. Although I don’t believe there are any guarantees that the details of bytecode compilation won’t change between versions.

                                                                                        1. 41

                                                                                          It’s not just one-off custom database schemas, either: you’ll find assumptions about name structure built into various official standards too. OpenID Connect, for example, at least avoids the assumptions about positions of name components (“first” and “last” names) but still breaks names down into given, middle, and family components. That works for names in a lot of cultures, but it discards culturally-relevant information in some cases and forces people to map the parts of their names to concepts that don’t necessarily match reality. And woe unto you if you try to render these names with whitespace between the components: even if you get the order right, Chinese names don’t have whitespace and it’s just flat-out wrong if you insert it.

                                                                                          My experience doing internationalization led me to the conclusion that for a lot of applications, you really want two name fields: “full name” (which is pretty much freeform text, whitespace allowed, name components in any order the user decides is correct) and “abbreviated name you want the software to use to address you.” Do full-text indexing of both, and most of the bases are covered.

                                                                                          1. 16

                                                                                            imo you should have username (for URLs and mentions if applicable, maybe limited character set if you must) and screen name (to show, freeform text, full unicode) and if you really need it, legal name. Don’t use legal name for anything you don’t absolutely need it for.

                                                                                            1. 3

                                                                                              It’s a good rule. Unfortunately the moment your sell anything you need the names and addresses… which legally makes sense, but it means any commercial service will need them.

                                                                                              1. 4

                                                                                                Then ask for a name for those purposes and don’t use it anywhere else.

                                                                                                1. 2

                                                                                                  Yes, but for those you can either pass through this info directly, or if you want to save it for convenience later, just use an address table for that purpose specifically, that takes something looking like a proper address.

                                                                                                2. 2

                                                                                                  Does that imply you just give up on having your software address people as [Salutation] [Family Name] in formal contexts? Mr. Smith, Mx. Xi, Dr. Doom, etc?

                                                                                                  1. 6

                                                                                                    Exactly.

                                                                                                3. 10

                                                                                                  I’ve worked on a site that has been created in Taiwan, and it had only a single field for name. As soon as the UK management took over, they demanded to split it into first_name and last_name. They could not imagine starting emails in any other way than Hello ${first_name}.

                                                                                                  1. 4

                                                                                                    First name: Queen
                                                                                                    Middle name: Elizabeth
                                                                                                    Last name: II

                                                                                                    What culture is that name from?

                                                                                                    1. 9

                                                                                                      I think her actual name is Elizabeth Alexandra Mary Windsor, with Queen (or rather, Queen of the United Kingdom and the other Commonwealth realms) as her title, and the “name” Queen Elizabeth II as a type of shorthand (and to distinguish between her and Queen Elizabeth I, who was actually Elizabeth Tudor). So her first name is “Elizabeth”, last name “Windsor”, title is “Queen” with (I think) the salutation of “Your Royal Highness.”

                                                                                                      I’m not sure where designators like “Jr.” or “IV” fit into the whole naming scheme—part of the last name? A separate part entirely? I’m not sure.

                                                                                                      1. 4

                                                                                                        My (poor) understanding is that most members of the House of Windsor don’t have last names, but use it by convention. Prince William went by William Wales in the military (getting him the cool nickname “Billy Fish”), since his father is Prince of Wales. Prince Philip adopted Mountbatten when he became a UK citizen, but didn’t have one before that (being a Greek prince). Elizabeth is the head of the House of Windsor, so she’d probably use it, though.

                                                                                                        1. 7

                                                                                                          Like the blog post author, QE2 has a fake last name she just made up so there would be something to write on bureaucratic forms.

                                                                                                          1. 4

                                                                                                            She didn’t make up her last name, her great-grandfather did, having switched last names from Saxe-Coburg and Gotha.

                                                                                                            1. 1

                                                                                                              I was surprised to recently learn that St Petersburg became Petrograd during WW1 for much the same reason - anti-German sentiment in Russia.

                                                                                                              1. 2

                                                                                                                The name ‘Mountbatten’ mentioned a few replies up was translated directly from ‘Battenberg’ for the same reason

                                                                                                          2. 4

                                                                                                            Prince Philip adopted Mountbatten when he became a UK citizen, but didn’t have one before that (being a Greek prince).

                                                                                                            Somewhat relatedly, the former King of Greece’s refusal to chose a surname like a regular citizen has been a point of dispute with the Greek government for a few decades, preventing him from getting a Greek passport.

                                                                                                            1. 2

                                                                                                              Fascinating. I know Charles I was referred to as “Charles Stuart” by his prosecutors at the time of his trial, and Louis XVI was referred to as Citoyen Capet after he was deposed as king. No doubt Elizabeth would be referred similarly during the coming revolution…

                                                                                                            2. 2

                                                                                                              I’m not sure where designators like “Jr.” or “IV” fit into the whole naming scheme—part of the last name? A separate part entirely? I’m not sure.

                                                                                                              In the mists of time, X.400 email ORAddresses had a “generational qualifier” field for this purpose, so you’ll still see vestigial remnants of this in LDAP systems (LDAP is/was the “use TCP/IP to access X.500 directories” protocol, with X.500 directories being the “store everyone’s X.400 email addresses and favourite drink preferences as part of a heirarchical global directory”.)

                                                                                                              For military/government procurement purposes, MS Exchange supported X.400/X.500-like behaviour, so you get stuff like:

                                                                                                              https://docs.microsoft.com/en-us/windows/win32/adschema/a-generationqualifier

                                                                                                            3. 6

                                                                                                              It’s a regnal name. I don’t know how the UK laws handle these, as well as noble titles such as “Earl of Whatever”, compared to the actual personal name.

                                                                                                              I imagine the Queen would register as Elizabeth Windsor, and select the title “Queen of the United Kingdom” from the drop-down list.

                                                                                                              1. 1

                                                                                                                UK

                                                                                                            1. 20

                                                                                                              Formal methods. I’ve already committed to this since I convinced my team lead that it’s be a good idea to specify our system in TLA+, so I’m gonna read up on it during the holidays and hopefully not prove myself wrong.

                                                                                                              Prolog, wanted to do a web application with its http library and use it as database.

                                                                                                              Some hardware stuff maybe. My BU at work does a lot of hardware/embedded stuff and I’d love to learn how some of it works.

                                                                                                              Reading/writing assembly, so spending a lot of time on Compiler Explorer probably.

                                                                                                              1. 5

                                                                                                                Prolog, wanted to do a web application with its http library and use it as database.

                                                                                                                I’m curious why you’re interested in Prolog. When I was taught Prolog, it was as an example of a category of programming languages that doesn’t really exist. I’ve actually written a non-trivial amount of Prolog (and enjoy it), but the languages that inherit ideas from Prolog tend to be proof assistants and solvers, rather than general purpose languages. SMT solvers such as Z3 are the closest to descendants of Prolog and are a lot more generally useful (Prolog has SLD derivation and cuts to hack it to try to explore the solution space in other directions, most solvers have pluggable tactics). If I were starting today, I’d learn Z3 rather than Prolog.

                                                                                                                1. 2

                                                                                                                  Well first of all, Z3 doesn’t really fit my purpose of writing a web application, and my plan was more to write a web application and learn Prolog through the process rather than learning Prolog by itself. I’m admittedly more interested in getting the program done than learn everything about Prolog.

                                                                                                                  Second, I don’t really know what I could use Z3 for, or rather, I don’t know what it does exactly and I haven’t found a project that interests me where I think it could be useful. Maybe somebody else could enlighten me on this.

                                                                                                                  1. 2

                                                                                                                    Z3 is an example of an SMT solver. Those take a set of constraints as input, and try to either return an example set of values that satisfy the constraints, or prove that it’s impossible.

                                                                                                                  2. 1

                                                                                                                    SMT solvers such as Z3 are the closest to descendants of Prolog

                                                                                                                    I’d consider Answer Set Programming (ASP) to be the closest descendant. Current practical implementations also have a “solver” flavor (like Z3) but the language uses close to Prolog syntax, and retains semantics that imo capture many of the interesting features of Prolog, like negation-as-failure, ability to straightforwardly define transitive closure, etc. (it comes out of attempts to come up with a less operational semantics for Prolog, of which one proposal, the “stable model semantics”, became the basis for ASP). I use clingo as the implementation, which is pretty solid.

                                                                                                                  3. 4

                                                                                                                    Plus one for learning TLA+ this year. I ordered @hwayne’s book a year or so ago and read it but didn’t complete all the exercises.

                                                                                                                    I found the whole thing frustratingly almost made sense while remaining just out of reach. I had trouble imagining systems that were both motivating to try to model, and also small / simple enough that I would have a chance of succeeding. Systems that would be really helpful to model at work were just way too complicated for me, but problems that I felt I could probably get working seemed trivial and not interesting enough to motivate me to work on them.

                                                                                                                    I’d be very interested if anyone had some suggestions for exercises or problems of various levels of difficulty to practice on. Something like exercism for TLA+ would be awesome.

                                                                                                                    1. 1

                                                                                                                      I think our system is at a sweet spot where it’s not terribly complicated, but also very prone to missing edge cases, and there’s a few very clear invariants that should hold up all throughout, so I think mapping it to what (admittedly little) I know about TLA+ should be possible without too much pain. I’ll probably also try to write a blog post about it when I’m at a good point in specifying it.

                                                                                                                    2. 2

                                                                                                                      TLA+ legitimately changed how I view software. That’s really the best case scenario when learning a new tool - it affects the way you think, whether or not you actually end up using it at your day job.

                                                                                                                      One of the reasons that TLA+ is so much easier to learn than so many other formal methods tools is that Leslie Lamport is, in my opinion, pretty much the best technical writer there is. So many formal methods books and papers are really tough to follow, even with lots of the background knowledge. I found Specifying Systems to be extremely easy to follow along with, and there’s even asides on math concepts when he feels that it’s useful.

                                                                                                                      Can’t recommend going down that path enough.

                                                                                                                    1. 52

                                                                                                                      I’m always relieved when I see a doctor look things up instead of just relying on what they remember from 20 years past.

                                                                                                                      1. 24

                                                                                                                        Doctors also don’t typically use Google/Wikipedia. They use a service called UpToDate*, which is commercial and has its articles written by verified MDs and researchers, but with Google-like search and Wikipedia-like organization. There’s an interesting profile/obituary of the company’s founder here:

                                                                                                                        https://www.statnews.com/2020/04/25/remembering-uptodate-creator-burton-bud-rose/

                                                                                                                        *: I know because my wife is an MD.

                                                                                                                        1. 9

                                                                                                                          I assume this varies by region, but some surveys suggest that a lot of doctors (especially junior doctors) end up using Wikipedia or other free resources in practice. E.g. a 2013 study of European doctors found that:

                                                                                                                          In line with previous research [6], general-purpose search engines (eg, Google), medical research databases (eg, PubMed), and Wikipedia were popular resources, while specialized search engines were unpopular. […] Even though point-of-care databases (eg, UpToDate) provide reliable, evidence-based clinical information, the reported use among physicians was shown to be limited. A possible explanation could be that most physicians are not willing to pay high subscription fees to access medical information.

                                                                                                                          1. 3

                                                                                                                            Yea, perhaps my word choice on “typically” is doing too much work there, or, assuming too much. I revise my statement to say, “Doctors have the option not to use Google/Wikipedia, but still be able to look up quality information with simple search queries.”

                                                                                                                            I’m sure you are right that, as the study suggests, many doctors don’t use a service like UpToDate. It is definitely a cost (although, a trivial cost, compared to other healthcare costs, and ridiculously easy to justify on ROI basis for a practice or hospital). Many of my wife’s friends who did career changes out of medicine even keep their UpToDate subscription (on a personal basis), simply to be able to guide their own (or their family’s) care a little when they are seen by other doctors. IMO, UpToDate is a great service and every MD should have access.

                                                                                                                            Also, I should mention that my wife is quite young, as far as doctors go, and there is a generational divide here. Many doctors who came of age before the information era were forced to “search their brain” for all the answers, so I imagine many of those doctors haven’t adapted to the internet age merely out of habit.

                                                                                                                          2. 9

                                                                                                                            If you have it. I do a lot of Google still. No institutional site license here and I’m not wild about the per cost.

                                                                                                                            (source: also an MD)

                                                                                                                            1. 3

                                                                                                                              I use it to the tune of 100+ CME points a year so to me it’s worth the $52/mo. I know colleagues who split a subscription too.

                                                                                                                            2. 4

                                                                                                                              I’ve observed my GP doctor type stuff into Google and click on a few links. Usually whatever page they land on looks like an official source of some kind, rather than an SEO optimised opinion piece (so, they probably don’t click on the first result).

                                                                                                                              I’m generally fine with that - I trust my Dr to have enough background understanding to gauge whether an article is factual or not.

                                                                                                                          1. 12

                                                                                                                            I don’t program in Python on the regular (I’ve contributed to a couple of projects) so most of my interaction with Python as a language comes from when they wanna EOL their versions, like when Python 2 died and with it all obscure rando throwavay (but vital in production) GNU Image Manipulation plugins I’ve picked up from rando forum posters over the years.

                                                                                                                            (And also whenever trying to install a program written in Python, which is always hairy with venvs and such.)

                                                                                                                            I’m sure my own pet languages are just as brittle and Python is just unusually responsible about it, about EOL announcments etc, but, that’s the main interaction I have with Python qua Python. So whenever people are talking about how easy and painless Python is, I just smile and say “Sure” 🧕

                                                                                                                            All in good fun, I get that this happens with all languages. Just such a hassle every time.

                                                                                                                            1. 6

                                                                                                                              All in good fun, I get that this happens with all languages. Just such a hassle every time.

                                                                                                                              As a counterpoint, AFAIK, most programs that are compiled in Java 1.x will run fine on the latest JVM.

                                                                                                                              1. 7

                                                                                                                                But could you compile it again?

                                                                                                                                The equivalent to that in Python would be using one of the myriad python packaging tools to generate an executable bundle, and those still work. Granted, your choice with an interpreted language is either “bundle your VM with your compilation” or “compile on each run”, but that’s not unique to Python.

                                                                                                                                1. 11

                                                                                                                                  Usually, somewhat to my surprise, yes! The most common case of actually existing Java code breaking that I know of is use of some stuff that was officially never supported (but was once semi-common) from the internal sun.* namespace. That was eventually removed (or closed off somehow, not sure of the details).

                                                                                                                                  I recently resurrected some Java code from my PhD that I wrote in 2009 and it just works! Compiles totally fine on a modern JDK. I’m not a huge Java fan otherwise, but that was very satisfying.

                                                                                                                                  1. 9

                                                                                                                                    What is unique to Python, of the languages I’ve used in anger, is that it is (in my experience) exceedingly eager to make backwards-incompatible deprecations with point releases. This was the case with Python 2, and initially it seemed like the subtext around Python 3 was that it wouldn’t be the case any longer, because the language had learned its lessons. And, yet.

                                                                                                                                    At this point, it feels like so many aspects of how the Python ecosystem works are basically just shouting at you to not use it as the implementation language for any reasonably large-scale project, and at this point I’m inclined to take those shouts at their word.

                                                                                                                                    1. 3

                                                                                                                                      Minor releases I imagine? I think it’s disingenuous to compare the Py2 -> Py 3.0 breakage to anything that has happened since then (source: I had to port a very big project to Python 3, it was a massive pain, even if it did help reveal/clarify loads of bugs). If your contention is that 3.6 -> 3.7 shouldn’t include any breaking changes, just shift the dot to the right once.

                                                                                                                                      I think that the main Python compat issue I’ve seen that hasn’t been like purely “bugfix” has been the async keyword. Python moving forward has made a new parser that allows for “soft keywords” (basically: this will be a keyword only under very specific syntax), to help avoid that stuff.

                                                                                                                                      My general feeling about breakage for a lot of stuff has been that it leans towards fixing initial design mistakes. The classic “str”/“unicode” thing in Python2 was basically wrong. It’s not universally this, but I think that the community has learned this and bends over backwards to support some stuff.

                                                                                                                                      I have found that, for example, Django and related projects have done an extremely good job of offering onboarding ramps for upgrading relative to some stuff I’ve seen (libraries in eternal “we’re design a new entirely incompatible version of the API to make things work better” instead of doing things gradually). But there is definitely a feeling of incompleteness and not really trusting being able to just have an old Python work forever (short of vendoring in all your dependencies I suppose).

                                                                                                                                      Standard libraries really help on this front, and honestly I would wish ecosystems leaned on more, rather than less. That way people can be less reliant on a whole other ecosystem.

                                                                                                                                      1. 3

                                                                                                                                        Python moving forward has made a new parser that allows for “soft keywords” (basically: this will be a keyword only under very specific syntax), to help avoid that stuff.

                                                                                                                                        async was a soft keyword in 3.6. They deliberately moved it to be a hard keyword in 3.7, which broke tons of stuff. :-/

                                                                                                                                    2. 2

                                                                                                                                      You can also instantiate these old classes in the new JVMs, and they work fine. This would be like being able to call Python 2 libraries in Python 3.

                                                                                                                                    3. 2

                                                                                                                                      Oh, wow! I had such a hard time getting java programs to run back in the day before Swing was free but now there’s such a treasure trove of old apps.