Threads for knl

  1. 1

    I’m on mobile, so can’t write that much, appologies. The reason I shared this article is because it articulated quite well what are the pitfalls of matplotlib’s api and how having two different versions will cause so many grievances. That’s something I experienced when starting with matplotlib (that everyone around me uses) - it was so damn hard to find anything out of ordinary. Paired with seaborn it was more headaches than actual pleasure of delivering meaningful graphs. For me, I think the majority of the issues are from the fact that I need to plot something once a week/month, so all interactions are superficial and I easily forget the syntax.

    However, since I found plotnine, Python’s port of ggplot2, many things just fell in place. At least there is high internal consistency and easy facetting.

    1. 6

      Just a little nitpick

      Instead of import file.nix { inherit pkgs; } you can pkgs.callPackage file.nix { }.

      That way you can destructure the same way as people do in nixpkgs and can use the function override if you want later.

      You can also use a recursive attrset instead of that let expression.

      1. 2

        Thanks! I’m new to the ways of Nix.

        1. 1

          What would be the benefit of using rec attrset instead of the let expression?

          1. 2

            DRY

            rec attrset works like both an attrset and a let expression

            1. 2

              I prefer let … in … over rec attrsets, you can always use inherit for the fields to not repeat yourself that much and I had bad experience with rec in the past (the errors were not that helpful).

        1. 2

          Is there more difference between tcpflow and tcpdump then just understanding HTTP?

          1. 6

            Nice to see more projects in this area, I’m happy with atuin right now after having used zsh-histdb for several years.

            1. 1

              What prompted you to switch? I’m currently using zsh-histdb and I’m curious how it compares to atuin and hishtory.

              1. 4

                What prompted me to switch was helping to review/test a pull request that added zsh-histdb importing, that atuin is a more active project with seemingly bigger goals, zsh independence, syncing. What’s not working so well for me with a very big history file is their TUI - I’m still using a non-TUI reverse search for most cases (which isn’t as good as the one of zsh-histdb yet). I think there are lots of possible innovations in shell histories that might allow more exciting features in the future. For example, just imagine you could enable a jupiter notebook style setup where you can access any commands output afterwards easily.

            1. 16

              The original author submitted this to HN with the following description, which I found as a refreshing take:

              Hi HN, I have been making this Cozo database since half a year ago, and now it is ready for public release. My initial motivation is that I want a graph database. Lightweight and easy to use, like SQLite. Powerful and performant, like Postgres. I found none of the existing solutions good enough.

              Deciding to roll my own, I need to choose a query language. I am familiar with Cypher but consider it not much of an improvement over CTE in SQL (Cypher is sometimes notationally more convenient, but not more expressive). I like Gremlin but would prefer something more declarative. Experimentations with Datomic and its clones convinced me that Datalog is the way to go.

              Then I need a data model. I find the property graph model (Neo4j, etc.) over-constraining, and the triple store model (Datomic, etc.) suffering from inherent performance problems. They also lack the most important property of the relational model: being an algebra. Non-algebraic models are not very composable: you may store data as property graphs or triples, but when you do a query, you always get back relations. So I decided to have relational algebra as the data model.

              The end result, I now present to you. Let me know what you think, good or bad, and I’ll do my best to address them. > This is the first time that I use Rust in a significant project, and I love the experience!

              1. 4

                This is the first time that I use Rust in a significant project, and I love the experience!

                After a quick browse through the code, it’s very well organized. Not much to nitpick.

                1. 3

                  Woah. This looks incredibly far along for being less than a year old.

                1. 1

                  I opened the page, I tried to find what this is, and failed… what is FuguIta? How does it help me?

                  1. 5

                    Had to click around a bit:

                    FuguIta is a live system based on OpenBSD operating system.

                  1. 7

                    What can I use this for? Neither this explanation nor the linked pages help me understand where when can I use Jevko (nor how to pronounce it).

                    1. 4

                      What can I use this for?

                      For the same things you could use XML/JSON/TOML for.

                      Neither this explanation nor the linked pages help me understand where when can I use Jevko

                      According to the website:

                      It has no data types, no semantics, no underlying model of cons cells or anything similar. It’s as close to pure generic syntax as it gets.

                      So at the lowest level Jevko is a minimal formal specification for flexible trees of text.

                      For its simplicity it seems quite powerful to me. I don’t see it replacing JSON/markdown or even TOML. But is seems trivial to implement on very low-level, older, less bloated, or less powerful systems.

                      Note: I am not the author of this tool, I merely posted its link because I find it an elegant and versatile solution.

                      1. 3

                        @eterps’ answer points in the right direction.

                        To expand on that, perhaps a better starting point is the home page: https://jevko.org/ Which includes a formal specification: https://jevko.org/spec.html

                        Pronunciation instruction are there as well ;)

                        The name Jevko /ˈdʒef.kɔ/ is derived from Polish drzewko (/ˈdʐɛf.kɔ/), meaning small tree.

                        There is also https://jevko.github.io/ which has a lousy demo that I haven’t updated in a while: https://jevko.github.io/interjevko.bundle.html

                        Finally, the repos on GitHub may give a vague idea on what this can be used for.

                        1. 4

                          So is it sorta like Rebol without a defined execution semantic behind it (e.g. no do/parse dialect)?

                          1. 1

                            The things in common with Rebol are the spirit of Lisp and the square brackets.

                            Important differences are that Jevko is only a syntax, like S-expressions or whatever-the-name-of-Rebol-syntax-is.

                            Indeed there is no semantics attached to it at all.

                            You could make up some semantics and build a language. A toy example: https://github.com/jevko/jevkalk

                            Another important syntactic difference from S-exp/Rebol-syntax is that whitespaces are not a separator in Jevko.

                            The toy language above exploits that to allow spaces in identifiers, so instead of frobCount or frob_count you can write frob count. It’s pretty fun.

                            Another difference is that the syntax is designed in such a way that it should parse into things that look like name-value pairs, which makes it easier to process a lot of things, e.g.:

                            c [d]
                            a [b]
                            

                            would parse to something like

                            [[["c ",[[],"d"]],["\na ",[[],"b"]]],""]
                            

                            which has c paired with [d] and a paired with [b].

                            This structure is useful for further processing in many applications.

                            1. 2

                              I think that for many practical purposes those trailing spaces will become very annoying very fast.

                              1. 2

                                For such purposes, you’d trim them. E.g. https://github.com/jevko/easyjevko.js and https://github.com/jevko/easyjevko.lua implement a very simple format on top of Jevko which takes care of trimming leading and trailing spaces automatically and converts Jevko parse trees into JS/Lua objects/tables/arrays/strings.

                                1. 3

                                  Of course, but for many practical purposes splitting on whitespace is what you want to do anyway and for all those purposes Jevko is at a disadvantage relative to s-expressions. I’d say simplicity includes how simple it is to use for your purposes and an additional trimming step is less simple than a format that doesn’t require that trimming step. So I don’t think Jevko is simpler than s-expressions for many purposes.

                                  1. 2

                                    I phrased my last reply incorrectly, sorry about that.

                                    What I should’ve written is ~ that as a someone who is interested in a specific behavior like the one you are describing, you would want to use not plain Jevko, but a Jevko format that does the right thing for you.

                                    The simplest one right now is the one I linked, although formal spec for that is still TODO.

                                    However it might still turn out to do the wrong thing for you (there is no splitting on spaces, just trimming). In such case you’d indeed need to create your own or wait for one being available.

                                    Or just use S-expressions, of course! ;)

                                    Although there are several reasons why you might still pick Jevko, especially if a formally specified format that meets your constraints shall become available.

                                    Authoritative formal specification being one advantage.

                                    Another might be convenient properties of Jevko, such as being closed under concatenation by design (not true for S-exp).

                              2. 2

                                Important differences are that Jevko is only a syntax, like S-expressions or whatever-the-name-of-Rebol-syntax-is.

                                That’s actually what Rebol is as well; the dialects interpret the syntax, that’s why you can have overloaded values between say Rebol/View, Rebol/Do, Rebol/Secure, Rebol/Load, &c.

                                The toy language above exploits that to allow spaces in identifiers, so instead of frobCount or frob_count you can write frob count. It’s pretty fun.

                                ah, there are a few languages like that, various Algol dialects didn’t treat space as a separator, so you could have identifiers with spaces in them. TCL has some interesting processing sorta like jevkalk has.

                                The processing of structures is quite nice, esp without additional syntax. How do you delineate that from those explicitly wrapped in [ ] pairs?

                                1. 1

                                  That’s actually what Rebol is as well; the dialects interpret the syntax, that’s why you can have overloaded values between say Rebol/View, Rebol/Do, Rebol/Secure, Rebol/Load, &c.

                                  I don’t know Rebol below the very surface, but I presume that there is a central language that binds all these dialects, isn’t there? I.e. the basic semantics. The point is that it is useful to separate Rebol-the-language (syntax+semantics) from the syntax alone in this context.

                                  ah, there are a few languages like that, various Algol dialects didn’t treat space as a separator, so you could have identifiers with spaces in them. TCL has some interesting processing sorta like jevkalk has.

                                  These early Algols and COBOLs didn’t really do the same thing – they simply ignored the whitespace, so abc is the same as ab c or a b c, etc.

                                  The one notable language that did a similar thing is actually very early Lisp. Initially S-expressions had the form (process, frob count, another argument) rather than (process frob-count another-argument), i.e. comma- rather than space-separated.

                                  Tcl is very nice in its minimalism. And it does have a similar syntax to Jevko, namely the Tcl braces. However, it’s still a bit different: https://www.reddit.com/r/sexpr/comments/xplmh7/tcl_braces_are_similar_to_jevko_only_3_special/iq5nt1j/

                                  The processing of structures is quite nice, esp without additional syntax. How do you delineate that from those explicitly wrapped in [ ] pairs?

                                  Not sure if I understand the question. Could you give an example?

                                  1. 2

                                    I don’t know Rebol below the very surface, but I presume that there is a central language that binds all these dialects, isn’t there? I.e. the basic semantics. The point is that it is useful to separate Rebol-the-language (syntax+semantics) from the syntax alone in this context.

                                    No no, that’s not the case; the interpretive context applies what the symbols within the data means, otherwise it’s just data without any other meaning, quite like what you’re doing here really.

                                    These early Algols and COBOLs didn’t really do the same thing – they simply ignored the whitespace, so abc is the same as ab c or a b c, etc.

                                    Algol68 and PL/I actually did support whitespace in identifiers, as did ABC. It’s not very commonly used, but it is different from what Algol60, early Fortran, &c did by being insensitive to whitespace.

                                    Additionally, early Lisp used M-Expressions, similar to Mathematica, but you also had LOGO which could do all sorts of things with it’s quoting conventions.

                                    Not sure if I understand the question. Could you give an example?

                                    I apologize, what I meant was what happens with

                                    c [d]
                                    a [b]
                                    

                                    when we surround them by brackets; is it the same parse tree as:

                                    [
                                    c [d]
                                    a [b]
                                    ]
                                    

                                    or

                                    [c [d]]
                                    [a [b]]
                                    

                                    or

                                    [
                                    [c [d]]
                                    [a [b]]
                                    ]
                                    

                                    ?

                                    1. 1

                                      No no, that’s not the case; the interpretive context applies what the symbols within the data means, otherwise it’s just data without any other meaning, quite like what you’re doing here really.

                                      Interesting. It makes sense: it’s merely a context-dependent evaluation strategy.

                                      But still, for this discussion that’s the irrelevant part.

                                      What is relevant is the uninterpreted data, i.e. the syntax tree, the result of parsing Rebol syntax. I don’t know if it is formally specified somewhere, but if it was, then this is the thing to compare to Jevko. Jevko itself is just a simple syntax.

                                      Just want to be very clear about that. :)

                                      Algol68 and PL/I actually did support whitespace in identifiers, as did ABC. It’s not very commonly used, but it is different from what Algol60, early Fortran, &c did by being insensitive to whitespace.

                                      I did some research on this not that long ago, which involved looking at reference manuals and reports on various old languages.

                                      AFAICT Algol 68 and PL/I didn’t support it in the same way I have implemented here or the way early Lisp did[0].

                                      Algol 68 was space-insensitive, according to this report:

                                      https://www.softwarepreservation.org/projects/ALGOL/report/Algol68_revised_report-AB.pdf

                                      In particular see:

                                      • Page 20, section 1.1.3.1.g).(v)

                                      • Page 117, section 9.4.d)

                                      This means space-insensitivity.

                                      In any case there was a lot of rules involved, stropping, etc.

                                      All this probably made sense when punched cards were still used. I don’t know.

                                      PL/I spec here:

                                      http://www.bitsavers.org/pdf/honeywell/multics/AG94-02_PLI_langSpec_Mar81.pdf

                                      Section 2.6.1. does not allow identifiers in spaces. Section 2.6.4. explicitly defines space as a delimiter.

                                      I can’t find any formal description of ABC, only this: https://homepages.cwi.nl/~steven/abc/programmers/handbook.html

                                      A quick look suggests that it indeed supported identifiers with spaces. That’s interesting! I’m curious about the details.

                                      Additionally, early Lisp used M-Expressions, similar to Mathematica, but you also had LOGO which could do all sorts of things with it’s quoting conventions.

                                      Yes, Jevko is similar to M-expressions in that it also uses square brackets and a Lisp written in it would prefer f[x] over [[f][x]].

                                      I think M-expressions were generally a good idea. S-expressions won likely because they were were clearly enough defined to be implemented and were sufficient to express everything. M-expressions weren’t as clearly defined and used S-expressions as a subset, so there was no point in bothering.

                                      If Lisp was specified only using simplified M-expressions from the beginning, maybe something like Jevko would’ve evolved long ago.

                                      Ad. quoting conventions and similar ways to specify identifiers with spaces (present in some lanugages today) – the problem with them is that it’s extra work and it’s ugly compared to regular identifiers. Identifiers with spaces are kind of tacked on, are not first-class, so to speak. So nobody uses that.

                                      On the other hand if it was natural and simple maybe it would catch on.

                                      Or maybe it wouldn’t. Clearly it didn’t in this timeline. But it’s fun to speculate and design.


                                      Ad. surrounding with brackets.

                                      Ah, now I see, thank you for the examples.

                                      Each of these examples would parse to a different parse tree. Brackets always build a subtree.

                                      In fact nothing is really lost in a Jevko parse tree. The special characters may or may not be implied, but otherwise it’s a concrete syntax tree.

                                      Below are the parse trees for your examples in order, starting with:

                                      c [d]
                                      a [b]
                                      
                                      {"subjevkos":[{"prefix":"c ","jevko":{"subjevkos":[],"suffix":"d"}},{"prefix":"\na ","jevko":{"subjevkos":[],"suffix":"b"}}],"suffix":"","opener":"[","closer":"]","escaper":"`"}
                                      
                                      {"subjevkos":[{"prefix":"","jevko":{"subjevkos":[{"prefix":"\n  c ","jevko":{"subjevkos":[],"suffix":"d"}},{"prefix":"\n  a ","jevko":{"subjevkos":[],"suffix":"b"}}],"suffix":"\n  "}}],"suffix":"","opener":"[","closer":"]","escaper":"`"}
                                      
                                      {"subjevkos":[{"prefix":"","jevko":{"subjevkos":[{"prefix":"\n  c ","jevko":{"subjevkos":[],"suffix":"d"}},{"prefix":"\n  a ","jevko":{"subjevkos":[],"suffix":"b"}}],"suffix":"\n  "}}],"suffix":"","opener":"[","closer":"]","escaper":"`"}
                                      
                                      {"subjevkos":[{"prefix":"","jevko":{"subjevkos":[{"prefix":"\n","jevko":{"subjevkos":[{"prefix":"c ","jevko":{"subjevkos":[],"suffix":"d"}}],"suffix":""}},{"prefix":"\n","jevko":{"subjevkos":[{"prefix":"a ","jevko":{"subjevkos":[],"suffix":"b"}}],"suffix":""}}],"suffix":"\n"}}],"suffix":"","opener":"[","closer":"]","escaper":"`"}
                                      

                                      These were produced by https://github.com/jevko/parsejevko.js

                                      Notice that they all end in:

                                      "opener":"[","closer":"]","escaper":"`"}
                                      

                                      These are the three special characters in Jevko. They are stored at the top level of the parse tree, so you can recover the original text from it, even if you configure the parser to use nonstandard special characters.

                                      [0] In fact early Lisp also didn’t do the exact same thing as my implementation, as it allowed only single embedded blanks. That may be a good idea.

                        1. 2

                          For example, instead of pkgs.git you can also use (pkgs.git.override { guiSupport = true; })

                          Is there a way to find all meaningful overrides without looking at the actual code in nixpkgs? I really liked how homebrew had a list of options with help one could use.

                          1. 6

                            afaik, no. You’ll have to look at the code. Happy to be corrected though.

                            1. 2

                              Yes. A bit manual, but you can wrap it in a script if you want. For example for git package:

                              nix-instantiate --eval --xml -E 'with import <nixpkgs> {}; lib.functionArgs pkgs.git.override'
                              

                              There are no descriptions though. And it will list all standard dependencies because they’re overrideable too.

                              1. 1

                                If you want a fancy version, this splits overrides into packages and other options:

                                #!/bin/sh
                                
                                if [ -z "$1" ]; then
                                  echo 'missing package'
                                  exit 1
                                fi
                                
                                nix-instantiate --eval --json --strict -E 'with import <nixpkgs> {}; let pkgNames = (lib.attrNames pkgs) ++ (lib.attrNames pkgs.darwin.apple_sdk.frameworks); args = lib.functionArgs pkgs.'$1'.override; argNames = lib.attrNames args; includeDefault = x: {name=x; value=lib.getAttr x args;}; in {nonPkgOverrides = lib.listToAttrs (map includeDefault (lib.filter (x: !(lib.elem x pkgNames)) argNames)); pkgOverrides = lib.listToAttrs (map includeDefault (lib.filter (x: lib.elem x pkgNames) argNames));}' | jq
                                
                              2. 1

                                as far as i can tell, the design and development culture in the nix world does not give a flying fuck about any tooling which makes your life easier. the answer is always to just look at the code.

                                1. 1

                                  Yeah, and even the code seems maliciously cryptic. No types, very few comments, abbreviated variable names, and no import statements (packages are functions and to figure out what type of argument it takes, you have to find the call sites and work backwards). Moreover, it’s all thrown in one giant repo with no discernible rhyme or reason as to its organization (at least I could never intuit where a particular package definition ought to live). I don’t like ragging on open source projects, but this stuff is my biggest obstacle to adopting Nix (though there are many others) and it seems entirely avoidable.

                                  1. 1

                                    Yeah, and even the code seems maliciously cryptic

                                    Sure, some of the code can be cryptic, but I really wouldn’t go so far as to say “maliciously.” Do you have any examples, per chance?

                                    no import statements (packages are functions and to figure out what type of argument it takes, you have to find the call sites and work backwards)

                                    I don’t see what you mean by “find the call site.” Within Nixpkgs, for most packages, all you have to do is find the file it’s defined in, and the dependencies/arguments it takes will be at the top. Assuming one has this knowledge, I don’t see what you’d have to work backwards from. Do you mind explaining so I can understand what you mean/try to explain some more? Maybe I’m just confused?

                                    Moreover, it’s all thrown in one giant repo with no discernible rhyme or reason as to its organization (at least I could never intuit where a particular package definition ought to live)

                                    Yeah, this is… known. There’s a recent effort by the Nixpkgs Architecture team to get rid of the organization (or at least, the littered categories, in favor of something simpler (e.g. sorting by alphanumeric order)). Hopefully it’ll make its way into a proposed RFC soon.

                                    I don’t like ragging on open source projects, but this stuff is my biggest obstacle to adopting Nix (though there are many others) and it seems entirely avoidable.

                                    What seems entirely avoidable, other than the organizational issues? Like, would you prefer that callPackage (the mechanism that injects dependencies as function arguments) not exist?

                                    1. 1

                                      Sure, some of the code can be cryptic, but I really wouldn’t go so far as to say “maliciously.” Do you have any examples, per chance?

                                      Mostly I meant “they seem to go out of their way to be cryptic, e.g., by foregoing types, documentation, longer variable names, etc”. It’s probably not actually malice, but that the project is so old and it hasn’t been a priority certainly feels hostile toward users.

                                      don’t see what you mean by “find the call site.” Within Nixpkgs, for most packages, all you have to do is find the file it’s defined in, and the dependencies/arguments it takes will be at the top.

                                      The top just has a list of parameter names, but that tells you close to nothing about what type of data is expected for those parameters. Ideally there are types, but failing that some human-targeted description of the arguments (perhaps with examples or links to other files!) is sort of the bare minimum that I would expect from a project especially one in which users regularly have to dig into nixpkgs (yes I know it’s FOSS but that doesn’t absolve it of criticism).

                                      Yeah, this is… known. There’s a recent effort by the Nixpkgs Architecture team to get rid of the organization (or at least, the littered categories, in favor of something simpler (e.g. sorting by alphanumeric order)). Hopefully it’ll make its way into a proposed RFC soon.

                                      Glad to hear it. Hopefully it addresses this point well, sincerely!

                                      What seems entirely avoidable, other than the organizational issues? Like, would you prefer that callPackage (the mechanism that injects dependencies as function arguments) not exist?

                                      I was referring to things like dynamic typing, cumbersome naming conventions, organizational issues, and the general tendency to minimize documentation.

                                      1. 2

                                        but that the project is so old and it hasn’t been a priority certainly feels hostile toward users.

                                        I really can’t speak for why it’s taken so long, but we have a documentation team now, and their efforts – both long-term and short-term – will hopefully help make Nix more friendly to newcomers and veterans alike.

                                        but failing that some human-targeted description of the arguments (perhaps with examples or links to other files!)

                                        Ideally, an LSP server for Nix could do links between callPackage arguments and their definitions like that. It’s been proposed for one LSP server implementation. Would be nice to see something come of it eventually, if possible!


                                        I know this is a lot of hoping, but I really think that in due time, we’ll be in a good place. I hope.

                                2. 1

                                  for nixpkgs packages themselves, the code is probably your best bet. you can use the nixpkgs search page to avoid spelunking the codebase for what you’re after, though.

                                  outside of nixpkgs, a lot of people do their configuration via NixOS modules, nix-darwin, and home-manager. luckily, all of those provide better API documentation with explanations for most of their options, as well as type annotations.

                                1. 2

                                  Nice writeup, thanks! Would nom be a good option to parse pcap files and various contained network protocols (Ethernet, IP, TCP/UDP, application level messages)? I need to extract some information from large pcap files, but I need to look at various fields of the whole stack.

                                  1. 5

                                    nom not only has support, but was in fact built specifically as a binary parser from the beginning, so it should work excellently for this use case. And it has dedicated support for streaming parsers that’s been steadily improving.

                                    1. 3

                                      Yep, it has specific functionality for binary protocol parsing. It’s been a while since I used it but to my memory its streaming support wasn’t great, but for pcap files in particular I’d expect it to work really well. (And I could be way out of date on the streaming support, they’ve had more than one major version since I used it.)

                                    1. 2

                                      Pretty nice article that came just in time when I was swearing about building a conda environment for our jupyter instance. I just can’t find a set of packages that fit together, and messages I get are not that helpful. I had to switch to pip-compile to get a good overview what versions/packages have conflicting dependencies. And after reading this thread it occured to me that there is another pitfall of python’s packaging tools: if I try to create an environment in one go (a’la pip-compile followed by pip-sync) I will get failures. If I use pip, it will happily install packages with conflicting dependencies – only if I install them one by one, it will warn me. That’s not what I would expect from packaging tools.

                                      1. 27

                                        The naming is a hard problem, and I find these project that inject their own vocabulary pretty hard to understand and use. Homebrew thought it’s cute to name everything after brewing, this one is trying the same with cells/biology.

                                        1. 6

                                          I started saying this at work - it’s cool that we can name everything after StarCraft, but I think “ API Endpoint” (Endpoint for short) is the best name for an API endpoint

                                        1. 3

                                          But when you add full hints for a package used widely in your code base, you might find yourself facing an insurmountable mountain of errors.

                                          I don’t get this. If you add types in your code, and then add types from the third party libraries via stubs, wouldn’t that be a good thing and every error is an actual error?

                                          1. 4

                                            The reality is that a very large number of the errors that mypy finds are not bugs in your code, because mypy essential has its own (made up) type system which is only a subset of what python supports.

                                            Many of the “fixes” you are forced to do are actually different ways to “keep mypy happy”. This can be overwhelming to do all at once (and there is a bigger debate about whether it is ever worth it)

                                            1. 1

                                              I’ve been away from Python since just before typing came around, and I’m really surprised to read this. So are there two competing type-checkers, the official Python one and mypy? If that’s the case wouldn’t the official one (eventually) win out just by virtue of being official? There’s only one set of type annotation syntax, right? So to the extent that mypy doesn’t follow the official spec doesn’t that just make mypy wrong?

                                              Sorry for all the questions, I didn’t realise how much I’d fallen behind in goings-on in Python land.

                                              1. 1

                                                There is no official one! Python is dynamically typed. While type hints have been added to the language, there is very little built-in support for their meaning or functionality, no actual spec for what they mean, and certainly no compile-time or run-time checking in the core language or CPython implementation.

                                                There are several unofficial competing static type checkers, such as mypy and pyre.

                                                1. 1

                                                  Mypy is part of the python GitHub account and I believe it’s maintained by the psf, so it’s the de facto official typechecker.

                                                  1. 2

                                                    I didn’t realise it’s now under the python account, that’s interesting.

                                                    While it certainly seems to be the defacto standard, I’m not sure you could call it “official” in any way. For example, it’s barely mentioned in official Python docs, and it’s always at pains to call mypy a “third party” tool or one of several options. e.g.:

                                                    This is important for things like how seriously you take mypy’s errors, and what is considered “correct” or “incorrect” usage of Python. If mypy tells me some code has an error in it, but it works at runtime, does that in any way imply my code is bad, or deprecated, or likely to fail in some future version of Python? As far as I can see, the answer is always an emphatic “no”, the static type system is a bolt-on which is not official in any sense.

                                                    1. 1

                                                      While it’s under the python org on GitHub, there’s no implication that mypy’s rules are somehow the official semantics of Python or can/should have an impact on those semantics.

                                                      And things like the __class_getitem__ hook mentioned in the other reply were, at the time, controversial because originally the rule for type-hint-related features was that they weren’t supposed to require changes to the Python interpreter or to the language semantics, aside from the initial syntactic support for annotations.

                                                2. -1

                                                  mypy essential has its own (made up) type system

                                                  Unlike the natural, organic, type systems that grow on trees, in the forest?

                                                  1. 3

                                                    Unlike the Python’s type system, which utilizes protocol/trait style a lot, which is difficult to express in mypy type system. It’s possible, but as said, takes work.

                                                3. 1

                                                  While I wouldn’t want describe it as a different system as other comments here, there are some really annoying things you can run into with mypy where mypy is slightly opinionated. One I completely disagree with mypy using specific type’s return rather than the parent one - for example there’s an IO type implementing the context manager contract which normally is expected to return any object (truthy/falsey value), but mypy says this class will always return None, because that’s cpython’s implementation and they don’t know of a usecase where it would differ.

                                                  So mismatches like that can be found both for mypy and between users/providers of a library. Not too often and I still use mypy everywhere, but… there are annoyances.

                                                  1. 4

                                                    As an example of a “made up” type system, mypy accepts this:

                                                    x = [1, "a"]
                                                    

                                                    but rejects this:

                                                    x = [1]
                                                    x.append("a")
                                                    

                                                    In Python lists are heterogenous, in mypy they are implicitly homogenous. You’ll also need to understand concepts like “covariant” and “contravariant” etc. - these concepts don’t exist in Python’s actual type system.

                                                    Plus tons of other things, like “objects only have the attributes defined inside their __init__”, and “functions can’t have arbitrary attributes”, or “__add__ returns an object of the same type as the two inputs” etc,

                                                    These are the kinds of things that you have to invent if you want to add a static type system to something like Python, but I think we do need to be clear that these are just mypy’s ideas about how code should be written and how a type system should work.

                                                    1. 1

                                                      That’s a list[int | str], you just need to annotate it.

                                                      1. 2

                                                        I think you misunderstood the criticism – the issue with the list above is that if mypy sees a single-element list it defaults to assuming the list is homogeneous and all elements must be of the same nominal type as the already-seen element. Which is not an assumption supported by Python’s semantics; mypy has basically decided to impose its own idea of how a list “should” be used in place of the real semantics of how Python’s list type works. The fact that you can annotate your way out of it doesn’t solve the problem with mypy – to see why, imagine if mypy instead assumed Python lists are fixed-length and forced you to annotate your way out of that to use append() on a list mypy thought was “full”. That would be a similar imposition of mypy’s preference over Python semantics.

                                                        1. 1

                                                          Thank you for explaining that more clearly than I did!

                                                          I’m also not saying that mypy is necessarily “wrong” or bad in its decisions, just that we do need to be conscious of the fact that we are potentially facing a defacto change in what is considered correct or idiomatic Python.

                                                1. 9

                                                  for the purpose of this post, it is a way to easily create reproducible development environments

                                                  I don’t understand the advantage here versus all Nix and building a container, if necessary, through Nix. Using the Nix shell is quick/dirty, but it turns a stateless build tool into one that has the state of the shell. Why just have a reproducible development environment when the whole derivation top to bottom can be reproducible? Between packages and apps you can do just about everything and without the overhead of containers.

                                                  I understand Nix is a learning cliff to swallow, but it will, in the long run, simplify this entire setup to just one tool instead of multiple and this one-tool simplicity makes it a lot easier for teams to understand.

                                                  1. 7

                                                    You are right. But it took me literally months to learn enough Nix to be able do everything in Nix only. This setup is easier, it works fine and I figured it out in less than a day. So I stuck to it.

                                                    1. 2

                                                      One of the reasons why I see not going full build in nix is when one needs to collaborate with others that do not use nix. For example, I have to occasionally contribute to internal python project, where each has a different setup (no instructions, use virtualenv with requirements.txt, use conda, use an internal tool). When I’m not a regular contributor, getting into nuances of each setup is a big overhead. That’s why I set up a shell.nix file that bring me almost everything I need to just start coding. Sometimes that works, sometimes it doesn’t, especially when external, non-python dependencies are needed (ironic, right?). That way, others can continue doing work using tools they are comfortable with, while I get my safety net.

                                                      1. 1

                                                        Sure, I bring a Flake to JS projects like this and don’t commit it, but I if it’s my own repo, I’m not going to try suggest folks use 3 tools (Nix, Just, Podman) when I could use 1 (Nix). Depending on the audience you can suggest a devShell or explain how to use the ‘normal’ community tooling, but even then, often the *2nix is overall a better experience and unlocks build sharing like Cachix.

                                                    1. 2

                                                      I’ve stumbled upon this book when I was looking for a review of Great Expectations library. It turns out that this book is pure gold :)

                                                      What I really liked is pretty good overview of existing libraries for data science: https://aeturrell.github.io/coding-for-economists/data-advanced.html

                                                      1. 2

                                                        So the cool thing with PEP302 is that you could in theory just implement a random language in Python that compiles using the ast module. Then when you load your library you can hook into the import system and on-the-fly compile your custom language before giving it to Python.

                                                        Imagine if you had a LISP with macro support! You could just generate a bunch of Python code and hook into it transparently.

                                                        https://youtu.be/1vui-LupKJI?t=978

                                                        :)

                                                        1. 2

                                                          I don’t know about python, but perl has a similar mechanism. Which led to things like: https://metacpan.org/dist/Lingua-Romana-Perligata/view/lib/Lingua/Romana/Perligata.pm

                                                          1. 1

                                                            Author: Damian Conway.

                                                            Figures.

                                                            To simplify the mind-numbingly complex rules of declension and conjugation that govern inflexions in Latin, Perligata treats all user-defined scalar and array variables as neuter nouns of the second declension – singular for scalars, plural for arrays.

                                                            Hashes represent something of a difficulty in Perligata, as Latin lacks an obvious way of distinguishing these “plural” variables from arrays. The solution that has been adopted is to depart from the second declension and represent hashes as masculine plural nouns of the fourth declension.

                                                          2. 1

                                                            I know of one module that utilizes this. pcapnp loads up Cap’n’Proto schema files and creates appropriate Python classes. Pretty neat IMO.

                                                            1. 1

                                                              Neat! I’m curious to find other similar (ab)uses of Python’s import mechanism.

                                                            2. 1

                                                              You can add macros to Python directly with this mechanism.

                                                              (Hy is very cool too.)

                                                            1. 2

                                                              Ah man, for me this is right on the line of too much self-promotion and marketing. Still, nice writeup.

                                                              1. 3

                                                                Really? We upvote basically everything Fly.io puts out. This seems fine to me.

                                                                1. 4

                                                                  Different issue (and I wouldn’t mind less free press for Fly, but they do make decent content). In Fly’s case, it’s existing members dragging in Fly articles in addition to their other stuff; in this case, it’s posting own stuff and basically nothing else.

                                                                2. 2

                                                                  The thing is that OP also posts comments on stories which they’re not the author. So they are involved in the community IMHO, it looks like they’re not here to just post their blog posts and just answer comments about their blog posts.

                                                                  So I’d give it a pass personally. Is this better? (itamarst = pythonspeed.com) the ratio of self-promotion is lower, but the quantity of self-promotion is much higher, by a lot.

                                                                  1. 2

                                                                    Oh I certainly don’t like that either, don’t get me wrong. To your point, that they are commenting on other things is what makes it a borderline case for me. :)

                                                                  2. 2

                                                                    Dunno, seems fair. I like the writeup but also their product, helps me with monitoring my home setup in such an easy way.

                                                                    1. 0

                                                                      Why is this tagged nix?

                                                                      1. 1

                                                                        Because we don’t have an immutable-x tag and chatting with a bunch of Nix folks was fairly instrumental in growing my thinking and this design. Dropped since evidently you object.

                                                                        1. 1

                                                                          I came to ask the very same question :) I don’t think cram uses Nix, as it states:

                                                                          That No previous statefile warning is the secret sauce of Cram. Cram works in terms not just of this log of what changes it will make, but in terms of a persisted log of what changes it has made. This allows Cram to optimize repeated executions to remove installation steps that haven’t changed, while still retaining a precise log of how to get where you are now from an empty slate. This also allows Cram to clean up after itself.
                                                                          

                                                                          Repo doesn’t contain anything nix-related, either. Interestingly, this python project is built using Bazel, that’s something I rarely see.

                                                                          1. 1

                                                                            That’s a story for another time :P

                                                                            1. 1

                                                                              Having tried to use Bazel for Python… ouch, that’s gotta have some very complicated justification because Bazel’s python support is terrible on top of the already pretty terrible user experience of Bazel.

                                                                          1. 5

                                                                            You should really use something like Go where it’s statically compiled into one binary and you don’t have to be a pip wizard to fix weird problems when they happen (not if, when)

                                                                            1. 14

                                                                              I dunno, all this stuff is weird to me since I know that people have been building sysadmin-y scripts and tools and daemons and other things in Python for literal decades and it’s basically just in the past couple years that people seem to have decided that none of it ever happened or worked. And weirdly there’s an inverse relationship between the quality of Python’s packaging ecosystem – which used to be horrible and now is pretty darned good – and the loudness of the complaints or the strenuousness of the people insisting Python packaging is unfit for any purpose.

                                                                              Like, I remember vividly what it was like to try to package and distribute Python in 2008. It was awful! Django literally maintained a “no external dependencies” policy for years because of how awful it was. And now it’s easy! It’s as if all the complaints that ought to have been happening back then have somehow been timeskipped forward to now.

                                                                              1. 7

                                                                                Maybe it’s even the problem that packaging became simpler and people start to include more external packages into their code.

                                                                                When I think about my Python experience at work (with too many badly tracked dependencies) I immediately want to avoid Python for any simple tooling, even though it is my primary language. But on the other hand, if I just use stdlib Python it’s so good for small scripts: Copy to my server, execute, fix bugs on the server, copy back to local PC.

                                                                                1. 3

                                                                                  Jevon’s paradox but for dependencies.

                                                                                  I think part of the issue is that packaging ecosystems keep evolving. Once Python had nothing, but it was okay, because everything had nothing. Then Python got pip and PyPI, which was a big step up! But around the same time Node came out, and while it was a mess with unpinned and nested dependencies, the concept of a node_modules folder was a big step forward and eventually they started pinning all the dependencies and it became pretty easy to use. (Of course, since it was easy to use, people made crap like leftpad and iseven.) Now compared to Go modules or Cargo, Python has fallen pretty badly behind.

                                                                                  It’s interesting to imagine what will happen next to raise the bar again. Probably something inspired by Nix?

                                                                                  1. 1

                                                                                    I have similar experiences. However, I realised that I really fret if I have to package and share any python code if it’s more than a single file (some of this is rooted in how we do things at work). For sharing a single file, even with several dependencies, I have no fears, as I usually wrap it with Nix, just as described here: https://notes.yukiisbo.red/posts/2021/07/Spice_up_with_Nix_Scripts.html

                                                                                  2. 4

                                                                                    It’s cool in some circles to hate on what’s popular.

                                                                                    1. 1

                                                                                      And weirdly there’s an inverse relationship between the quality of Python’s packaging ecosystem – which used to be horrible and now is pretty darned good – and the loudness of the complaints or the strenuousness of the people insisting Python packaging is unfit for any purpose.

                                                                                      That’s probably an indication of how much easier it got for other languages. It seems like Go and Rust are defining the standard language tooling. Even if python got a lot friendlier it’s not as friendly as those languages.

                                                                                      Or maybe it’s an indication of pythons rise in popularity. When you are a niche language you get dedicated developers who care to learn the whole process. When you are popular everyone has to interact with you even if they didn’t take the time to really learn it.

                                                                                      1. 3

                                                                                        Go and Rust are also kind of cheating, because they both, for the moment, are static-linking-only. Which means you only have to worry about having local copies of dependencies in a build directory as you compile. As soon as you introduce dynamic linking, you have to make decisions about where to look for your dynamically-linked dependencies, and you’re back into the thing people specifically are complaining most about in Python, including in this thread (the need for venv or a similar construct to isolate potentially-conflicting sets of dependencies from each other).

                                                                                        Rust at least seems open to the idea of eventually supporting dynamic linking, but it requires committing to an ABI and solving a lot of hard problems that Cargo currently doesn’t have to deal with. Go, I don’t know.

                                                                                    2. 11

                                                                                      Or, just maybe, we should use well-supported languages we’re good at and that we enjoy.

                                                                                      My experience with Python tooling in regards to embedded development is good, especially because in the world of embedded development where you might need to rapidly iterate on how your tooling works you can dive into the source and modify it without much stress. It’s inane to me the author hasn’t heard of poetry or pipenv or such yet, maybe a couple minutes of Google would save him weeks yearly.

                                                                                    1. 4

                                                                                      What are these masked emails? Neither the README nor the attached animation give any hint what are those and why are they useful.

                                                                                      1. 5

                                                                                        It’s a fastmail thing: “A Masked Email address is a unique, automatically generated email address that can be used in place of your real email address. Masked Email addresses are formatted as random words plus a random number, e.g. dog.food3495@domain.tld.”

                                                                                        1. 3

                                                                                          if you use mailbox.org with your own domain, it’s trivial to configure it to accept mails to any @yourdomain email address. no need to use an external tool…

                                                                                          1. 1

                                                                                            I don’t use mailbox.org, but that’s exactly what I do.

                                                                                            1. 1

                                                                                              ya I imagine this isn’t a revolutionary idea, but for some reason it is for fastmail

                                                                                              1. 1

                                                                                                Fastmail is able to do that as well.

                                                                                          2. 2

                                                                                            Apple has a version too.

                                                                                            Someone emailed a nasty note to the reporters at my website from an Apple masked address. I’m not sure how to report abuse with these things. There should be a prominent webpage that lists the address to forward abuse to.

                                                                                            1. 4

                                                                                              I assume this code only works with Fastmail, it sure appears like it. I don’t use Apple’s email addresses, so I’ve never looked, but good to know!

                                                                                              I have a domain with a catch-all address, so I can just invent them for 1 off things, no need to tell my email provider ahead of time.

                                                                                              I assume for abuse, you could always email postmaster@ and reference the masked email address. I imagine it will get routed to /dev/null like every other email problem I’ve ever sent in the last decade, but it might make you feel better! :)

                                                                                              1. 2

                                                                                                I’ve gotten a response from postmaster before! Granted, it was for a ISP in Pennsylvania that had constant delivery problems, but they did at least write back to me. :-)

                                                                                                1. 1

                                                                                                  Congrats on reaching a human!

                                                                                                  I used to be able to reach humans via postmaster@, but I’ve never talked to another human in the past decade emailing that or webmaster@ (which usually just bounces). Seems nobody respects RFC-2142 anymore.

                                                                                            2. 2

                                                                                              Neat, thanks! I’m using fastmail, but didn’t know this is possible. I’ve been using this: https://www.fastmail.help/hc/en-us/articles/360060591053 but seems that masked emails are more versatile — with subdomain addressing anyone can send mails to any random string, losing ability to track who is spamming me.