Threads for hdeshev

  1. 21

    A useful shorthand I find for this kind of thing is to tag shell commands with comments.

    $ my awesome | unix pipe | that I want to --remember # frobnicate the bar
    

    Now the command is searchable with Ctrl-R > frob...

    1. 5

      Been there, done that ;) My .bash_history

      1. 7

        I’m mostly enjoying fish’s completion today. I like the fact that it takes the directory into account. It has its annoyances, but it has never driven me mad to the point that I would start looking for other mechanisms, so far.

        Regarding your public .bash_history, what do you do to prevent sensitive information from accidentally ending up there?

        1. 4

          Crazy you keep yours public. I have mine in a separate private repo from my dotfiles. Too easy to leak a secret.

          1. 5

            It seems like it is not a real history file. For example, it is alphabetically sorted. I assume that they use it as a place to store common commands, searchable by ^R.

            1. 1

              Yeah, otherwise it would murder my productivity if I had to stop after typing an alias- and shortcut-ladden sausage of a line to add a comment. I spam that enter like crazy, the commenting would never work.2

        2. 5

          When I’m in a trolling mood, I show that to people while omitting the space after the # comment and tell them shells support Twitter-style hashtags. :D

          $ long command to do something #k8s-cheat
          
        1. 8

          Spreading the Fossil love. One binary and one file for all your needs: website, source control, wiki, tickets, forum, chat. Deploying changes is just a commit, if you are using the auto-sync mode.

          The downside is that its UI may be too spartan for some people’s tastes and they may need some time getting used to the system’s quirks. The documentation could be better with regards to web customisation too as it focuses on the source control bits way too much IMO.

          1. 6

            i ADORE fossil! (and sqlite for that matter!)

            1. 1

              This is exactly what I do for websites these days, just run fossil-scm as my website. It basically does everything I care about, most content is just a wiki page.

              1. 1

                AS your website? Can you share a link? I’m curious to see that.

                1. 3

                  An example: https://ziefi.com/index

                  These days, I generally buy a domain for each thing I care about and then turn on Fossil and dump content into it.

                  Another example(without a link) I had a friend need lots of help with their house, so I turned up a domain(of their last name) with Fossil on it, and then I documented all the stuff I did, added files to version control of the diff. manuals for the various things I came across, etc. It was super handy, and now they have a documented list complete with manuals for everything I did on their property that will be there until I get tired of paying the domain bill(but they can always run it off the USB I gave them with a checked out version.

                  1. 2

                    this is awesome, i have never seen fossil used like this. :3

                    1. 2

                      Amazing! Thanks for the example site! I stole some ideas from your wiki markup!

                      The lovely thing about these Fossil sites is that you can fossil clone them and tinker on your own. I’m not sure if that was actually your intent and you missed taking away a permission or something haha.

                      1. 3

                        It’s all public data(from my perspective), of course you can fossil clone! Feel free to send fixes to my particular site if you are bored!

                      2. 1

                        Ha! Amazing. That’s pretty much what I expected it to look like, honestly,

                        It probably wouldn’t be that hard to write a few queries and turn fossil into a true static site generator, strip out the stuff you don’t need for the site like the forum, wiki, chat, etc. 🤔

                        1. 2

                          Why bother? I mean the whole point from my perspective, I can get more than I need complete and ready to go in 5m or less, with basically no maintenance work, ever.

                1. 23

                  Is a language good because it has many features? My current thesis is that adding features to languages can open up new ways to encode entire classes of bugs, but adding features cannot remove buggy possibilities.

                  1. 23

                    If you have a foot-gun in your arsenal and you add a new safe-gun, sure, technically that’s just one more way you can shoot yourself in the foot, but that’s missing the point of having a safe-gun.

                    Many features can be used as less bug prone alternatives to old constructs. E.g., match expression instead of a switch statement where you could forget the assignment or forget a break and get unintentional fall-through. Same way features like unique_ptr in C++ can help reduce bugs compared to using bare pointers.

                    1. 12

                      Another thing worth mentioning is that PHP has also grown some good linters that keep you away from the unsafe footguns. I believe it’s gotten really good over the years.

                      1. 7

                        Just to fill this out:

                        Psalm

                        PHPStan

                        EA Inspections Extended

                        Sonar

                        I actually run all of these. Obviously no linter is perfect and you can still have bugs but if you’re passing all of these with strict types enabled, you’re not writing the bad amateur code that got PHP it’s reputation from the “bad old days”. PHP’s not perfect but it’s no more ridiculous than, say, JavaScript, which curiously doesn’t suffer from the same street cred problems.

                        1. 6

                          …JavaScript, which curiously doesn’t suffer from the same street cred problems.

                          I see what you’re saying, but JS actually does kinda have serious street cred problems. I mean, there are a ton of people who basically view JS programmers as second-class or less “talented”. And JS as a language is constantly mocked. I think the difference is that JS just happens to be the built-in language for the most widely deployed application delivery mechanism of all time: the web browser.

                      2. 1

                        It’s not as if match replaced switch; and why did it have default falkthrough to begin with, whilst match doesn’t?

                        1. 2

                          It’s probably just taken verbatim from C. It’s funny because PHP seems to have taken some things from Perl, which curiously does not have this flaw (it does allow a fallthrough with the next keyword, so you get the best of both worlds).

                          1. 1

                            Switch has been in PHP since at least version 3.0 which is from the 1990s. Match doesn’t replace switch in the language but it can replace switch in your own code, making it better.

                        2. 15

                          I disagree. People saying this usually have C++ on their mind, but I’d say C++ is an unusual exception in a class of its own. Every other language I’ve seen evolving has got substantially better over time: Java, C#, PHP, JS, Rust. Apart from Rust, these are old languages, that kept adding features for decades, and still haven’t jumped the shark.

                          PHP has actually completely removed many of its worst footguns like magic quotes or include over HTTP, and established patterns/frameworks that keep people away from the bad parts. They haven’t removed issues like inconsistent naming of functions, because frankly that’s a cosmetic issue that doesn’t get in the way of writing software. It’s very objectionable to people who don’t use PHP. PHP users have higher-priority higher-impact wishes for the language, and PHP keeps addressing these.

                          1. 2

                            removed many of its worst footguns

                            or the infamous mysql API (that was replaced by mysqli)

                            edit: Also I like that the OOP vs functional interfaces keep existing. My old code just runs and I get the choice between OOP and functional stuff (and I can switch as I like)

                            1. 1

                              I liked the original mysql api. Was the easiest to use with proper documentation back then. A footgun is good analogy. A gun can be used in a perfectly safe manner. Of course if you eyeball the barrel or have no regard for basic safety rules about it being loaded or where it is pointed to at any time, then yeah, things are going to go south sooner or later.

                              Likewise, the old functional mysql api was perfectly usable and I never felt any worry about being hacked through sql injection. If you are going to pass numbers as string parameters or rely on things like auto-escape, then just like in the gun example, things are not going to end well. But let’s all be honest, at the point it is expected to be hacked.

                              1. 1

                                I haven’t been around the PHP community in any serious capacity for probably 17 years now, but “with proper documentation” was a double edged sword. The main php.net website was a fantastic documentation reference, except for the part where lots of people posted really terrible solutions to problems on the same page as the official documentation. As I grew as a developer, I learned where a lot of the footguns were, but starting out the easy path was to just grab the solution in the comments on the page and use it, with all of the accompanying downfalls.

                                1. 1

                                  Already back in the day, it baffled me that the site even had comments, let alone people relying on them.nI would never blindly trust anything in the comments.

                          2. 8

                            There is only one way of modifying a language that works in practice: add new features. As one of my colleagues likes to say, you can’t take piss out of a swimming pool. Once a feature is in a language, you can’t remove it without breaking things. You can; however, follow this sequence:

                            1. Add new feature.
                            2. Recommend against using old feature.
                            3. Refactor your codebase to avoid the old feature.
                            4. Add static analysis checks to CI that you aren’t using the old feature.
                            5. Provide compiler options to make use of the old features a hard error.

                            At this point, the old feature technically exists in the language, but not in your codebase and not in new code. I’ve seen this sequence (1-4, at least) used a lot in C++, where unsafe things from C++98 were gradually refactored into modern C++ (C++11 and later), things like the C++ Core Guidelines were written to recommend against the older idioms, then integrated into static analysers and used in CI, so you the old usages gradually fade.

                            If you manage to get to step 5, then you can completely ignore the fact that the language still has the old warts.

                            1. 6

                              I thought I was going crazy. Needed validation as no one would state the obvious.

                              None of these features is a game changer for PHP. And even less so is all the composer and laravel craze that pretty much boils down to a silly explosion of javaesque boilerplate code.

                              Heck, even the introduction of a new object model back in PHP 5 had marginal impact on the language at best.

                              PHP’s killer features were:

                              • Place script in location to deploy and map a URL to it
                              • Out of the box support MySQL. Easy to use alternatives were payed back then, and connecting to MySQL or PostgreSQL was a PITA in most languages.
                              • A robust template engine. It still is among the best and most intuitive to use our there. Although alternatives exist for every language.
                              • Affordable availability on shared hosting with proper performance. This blew the options out of the water, with alternatives coating up to three orders of magnitude more for a minimum setup.

                              These things are not killer features anymore. Writing a simple webapp with a Sinatra-like framework it’s easier than setting up PHP. The whole drop file to deploy only made sense in the days of expensive shared servers. It is counterproductive in the $3 vps era.

                              I would prefer if the language would:

                              1. Ship a robust production grade http server to use with the language instead of the whole mess it requires to be used via third party web servers

                              2. Even better. Drop the whole http request and response as default input/output. It makes no sense nowadays. It is just a cute reliq from past decades. Which is more a source of trouble than a nicety.

                              1. 1

                                Place script in location to deploy and map a URL to it

                                Which was possible for years before PHP via CGI and is no longer possible for PHP in many setups. PHP != mod_php

                                1. 6

                                  Which was possible for years before PHP via CGI

                                  mod_php did this better than CGI did at the time.

                                  1. From what I remember from trying out this stuff at the time, the .htaccess boilerplate for mod_cgi was more hassle and harder to understand.
                                  2. CGI got a rep for being slow. fork/exec on every request costs a little, starting a new Perl interpreter or whatever on every request cost a lot. (and CGI in C was a productivity disaster)
                                  3. PHP had features like parsing query strings and form bodies for you right out of the box. No need to even write import cgi.

                                  Overall the barrier to entry to start getting something interactive happening in PHP was much lower.

                                  From what I remember the documentation you could find online was much more tutorial shaped for PHP than what you could find online for CGI.

                                  PHP != mod_php

                                  Sure now, but pm is discussing the past. PHP == mod_php was de facto true during the period of time in which PHP’s ubiquity was skyrocketing. Where pm above describes what PHP’s killer features “were”, this is the time period they are describing.

                                  1. 4

                                    mod_php did this better than CGI did at the time.

                                    It also did it much worse. With CGI, the web browser would fork, setuid to the owner of the public_html directory, and then execve the script. This had some overhead. In contrast, mod_php would run the PHP interpreter in-process. This meant that it had read access to all of the files that the web server had access to. If you had database passwords in your PHP scripts, then you’d better make sure that you trust all of the other users on the system, because they can write a PHP script that reads files from your ~/public_html and sends them to the requesting client. A lot of PHP scripts had vulnerabilities that let them dump the contents of any file that the PHP interpreter could read and this became any file the web server could read when deployed with mod_php. I recall one system I was using being compromised because the web server could read the shadow password file, someone was able to dump it, and then they were able to do an offline attack (back then, passwords were hashed with MD5 and an MD5 rainbow table for a particular salt was something that was plausible to generate) and find the root password. They then had root access on the system.

                                    This is part of where the PHP hate came from: ‘PHP is fast’ was the claim, and the small print was ‘as long as you don’t want any security’.

                                    1. 1

                                      This is completely irrelevant to the onboarding experience.

                                      Either way, empirically, people didn’t actually care all that much about the fact that their php webhosts were getting broken into.

                                      1. 1

                                        This is completely irrelevant to the onboarding experience.

                                        It mattered for the people who had their database credentials stolen because mod_php gave everyone else on their shared host read access to the file containing them. You’re right that it didn’t seem to harm PHP adoption though.

                                  2. 2

                                    Not to the same extent at all. CGI would spawn a process on the operative system per request. It was practically impossible to keep safe. PHP outsourced the request lifecycle out of the developer’s concern. And did so with a huge performance gain compared to CGI. While in theory you could to “the same” with CGI, in practice,.it was just not viable. When PHP4 arrived, CGi was already in a downwards spiral already, with most hosting providers disabling access to it. While Microsoft and Sun microsystems followed PHP philosophy by offering ASP and JSP, which had their own share of popularity.

                                    PHP is, by and large, mod_php and nowadays fpm. The manual introductory tutorial even assumes such usage. Had they packaged it early on as a regular programming language, with its primary default interpreter hooked up to standard streams, it might have been forgotten today. Although personally I think they should have made that switch long ago.

                                2. 4

                                  “Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary.”

                                  https://schemers.org/Documents/Standards/R5RS/HTML/

                                  1. 1

                                    I think it’s the same principle as with source code: you want as little as possible while keeping things readable and correct

                                  1. 121

                                    I used to give the same advice, but I completely changed my opinion over the past 10 years or so. I eventually put in the time and learned shell scripting. These days my recommendation is:

                                    1. Learn to use the shell. It’s a capable language that can take you very far.
                                    2. Use ShellCheck to automatically take care of most of the issues outlined in the article.

                                    I really don’t want to figure out every project’s nodejs/python/ruby/make/procfile abomination of a runner script anymore. Just like wielding regular expressions, knowing shell scripting is a fundamental skill that keeps paying dividends over my entire career.

                                    1. 60

                                      Bingo.

                                      My advice is:

                                      • Always use #!/usr/bin/env bash at the beginning of your scripts (change if you need something else, don’t rely on a particular path to bash though).
                                      • Always add set -eou pipefail after that.
                                      • Always run shellcheck.
                                      • Always run shfmt.
                                      • Always pay attention to what version of bash you need to support, and don’t go crazy with “new” features unless you can get teammates to upgrade (this is particularly annoying because Apple ships an older version of bash without things like associative arrays).
                                      • Always use the local storage qualifier when declaring variables in a function.
                                      • As much as possible, declare things in functions and then at the end of your script kick them all off.
                                      • Don’t use bash for heavy-duty hierarchical data munging…at that point consider switching languages.
                                      • Don’t assume that a bashism is more-broadly acceptable. If you need to support vanilla sh, then do the work.

                                      While some people like the author will cry and piss and moan about how hard bash is to write, it’s really not that bad if you take those steps (which to be fair I wish were more common knowledge).

                                      To the point some folks here have already raised, I’d be okay giving up shell scripting. Unfortunately, in order to do so, a replacement would:

                                      • Have to have relatively reasonable syntax
                                      • Be easily available across all nix-likes
                                      • Be guaranteed to run without additional bullshit (installing deps, configuring stuff, phoning home)
                                      • Be usable with only a single file
                                      • Be optimized for the use case of bodging together other programs and system commands with conditional logic and first-class support for command-line arguments, file descriptors, signals, exit codes, and other nixisms.
                                      • Be free
                                      • Don’t have long compile times

                                      There are basically no programming languages that meet those criteria other than the existing shell languages.

                                      Shell scripting is not the best tool for any given job, but across every job it’ll let you make progress.

                                      (Also, it’s kinda rich having a Python developer tell us to abandon usage of a tool that has been steadily providing the same, albeit imperfect, level of service for decades. The 2 to 3 switch is still a garbage fire in some places, and Python is probably the best single justification for docker that exists.)

                                      1. 26

                                        While some people like the author will cry and piss and moan about how hard bash is to write, it’s really not that bad if you take those steps (which to be fair I wish were more common knowledge).

                                        I think “nine steps” including “always use two third-party tools” and “don’t use any QoL features like associative arrays” does, in fact, make bash hard to write. Maybe Itamar isn’t just “cry and piss and moan”, but actually has experience with bash and still think it has problems?

                                        1. 2

                                          To use any language effectively there are some bits of tribal knowledge…babel/jest/webpack in JS, tokio or whatever in Rust, black and virtualenv in Python, credo and dialyzer in Elixir, and so on and so forth.

                                          Bash has many well-known issues, but maybe clickbait articles by prolific self-pronoters hat don’t offer a path forward also have problems?

                                          1. 15

                                            If your problem with the article is that it’s clickbait by a self-promoter, say that in your post. Don’t use it as a “gotcha!” to me.

                                            1. 2

                                              I think there’s merit here in exploring the criticism, though room for tone softening. Every language has some form of “required” tooling that’s communicated through community consensus. What makes Bash worse than other languages that also require lots of tools?

                                              There’s a number of factors that are at play here and I can see where @friendlysock’s frustration comes from. Languages exist on a spectrum between lots of tooling and little tooling. I think something like SML is on the “little tooling” where just compilation is enough to add high assurance to the codebase. Languages like C are on the low assurance part of this spectrum, where copious use of noisy compiler warnings, analyzers, and sanitizers are used to guide development. Most languages live somewhere on this spectrum. What makes Bash’s particular compromises deleterious or not deleterious?

                                              Something to keep in mind is that (in my experience) the Lobsters userbase seems to strongly prefer low-tooling languages like Rust over high-tooling languages like Go, so that may be biasing the discussion and reactions thereof. I think it’s a good path to explore though because I suspect that enumerating the tradeoffs of high-tooling or low-tooling approaches can illuminate problem domains where one fits better than the other.

                                              1. 2

                                                I felt that I sufficiently commented about the article’s thesis on its own merits, and that bringing up the author’s posting history was inside baseball not terribly relevant. When you brought up motive, it became relevant. Happy to continue in DMs if you want.

                                              2. 6

                                                You’re really quite hostile. This is all over scripting languages? Or are you passive aggressively bringing up old beef?

                                            2. 9

                                              Integrating shellcheck and shfmt to my dev process enabled my shell programs to grow probably larger than they should be. One codebase, in particular, is nearing probably like 3,000 SLOC of Bash 5 and I’m only now thinking about how v2.0 should probably be written in something more testable and reuse some existing libraries instead of reimplementing things myself (e.g., this basically has a half-complete shell+curl implementation of the Apache Knox API). The chief maintenance problem is that so few people know shell well so when I write “good” shell like I’ve learned over the years (and shellcheck --enable=all has taught me A TON), I’m actively finding trouble finding coworkers to help out or to take it over. The rewrite will have to happen before I leave, whenever that may be.

                                              1. 11

                                                I’d be interested in what happens when you run your 3000 lines of Bash 5 under https://www.oilshell.org/ . Oil is the most bash compatible shell – by a mile – and has run thousands of lines of unmodified shell scripts for over 4 years now (e.g. http://www.oilshell.org/blog/2018/01/15.html)

                                                I’ve also made tons of changes in response to use cases just like yours, e.g. https://github.com/oilshell/oil/wiki/The-Biggest-Shell-Programs-in-the-World


                                                Right now your use case is the most compelling one for Oil, although there will be wider appeal in the future. The big caveat now is that it needs to be faster, so I’m actively working on the C++ translation (oil-native passed 156 new tests yesterday).

                                                I would imagine your 3000 lines of bash would be at least 10K lines of Python, and take 6-18 months to rewrite, depending on how much fidelity you need.

                                                (FWIW I actually wrote 10K-15K lines of shell as 30K-40K lines of Python early in my career – it took nearly 3 years LOL.)

                                                So if you don’t have 1 year to burn on a rewrite, Oil should be a compelling option. It’s designed as a “gradual upgrade” from bash. Just running osh myscript.sh will work, or you can change the shebang line, run tests if you have them, etc.

                                                There is an #oil-help channel on Zulip, liked from the home page

                                                1. 2

                                                  Thanks for this nudge. I’ve been following the development of Oil for years but never really had a strong push to try it out. I’ll give it a shot. I’m happy to see that there are oil packages in Alpine testing: we’re deploying the app inside Alpine containers.

                                                  Turns out that I was very wrong about the size of the app. It’s only about 600 SLOC of shell :-/ feels a lot larger when you’re working on it!

                                                  One thing in my initial quick pass: we’re reliant on bats for testing. bats seemingly only uses bash. Have you found a way to make bats use Oil instead?

                                                  1. 1

                                                    OK great looks like Alpine does have the latest version: https://repology.org/project/oil-shell/versions

                                                    I wouldn’t expect this to be a pain-free experience, however I would say should definitely be less effort than rewriting your whole program in another language!

                                                    I have known about bats for a long time, and I think I ran into an obstacle but don’t remember what it was. It’s possible that the obstacle has been removed (e.g. maybe it was extended globs, which we now support)

                                                    https://github.com/oilshell/oil/issues/297

                                                    In any case, if you have time, I would appreciate running your test suite with OSH and letting me know what happens (on Github or Zulip).

                                                    One tricky issue is that shebang lines are often #!/bin/bash, which you can change to be #!/usr/bin/env osh. However one shortcut I added was OSH_HIJACK_SHEBANG=osh

                                                    https://github.com/oilshell/oil/wiki/How-To-Test-OSH

                                                  2. 1

                                                    Moving away from Python? Now it has my interest… in the past I skipped past know it’d probably take perf hits and have some complicaged setup that isn’t a static binary.

                                                    1. 2

                                                      Yes that has always been the plan, mentioned in the very first post on the blog. But it took awhile to figure out the best approach, and that approach still takes time.

                                                      Some FAQs on the status here: http://www.oilshell.org/blog/2021/12/backlog-project.html

                                                      Python is an issue for speed, but it’s not an issue for setup.

                                                      You can just run ./configure && make && make install and it will work without Python.

                                                      Oil does NOT depend on Python; it just reuses some of its code. That has been true for nearly 5 years now – actually since the very first Oil 0.0.0. release. Somehow people still have this idea it’s going to be hard to install, when that’s never been the case. It’s also available on several distros like Nix.

                                                      1. 1

                                                        What is the status of Oil on Windows (apologies if it’s in the docs somewhere, couldn’t find any mentioning of this). A shell that’s written in pure C++ and has Windows as a first class citizen could be appealing (e.g. for cross-platform build recipes).

                                                        1. 1

                                                          It only works on WSL at the moment … I hope it will be like bash, and somebody will contribute the native Windows port :-) The code is much more modular than bash and all the Unix syscalls are confined to a file or two.

                                                          I don’t even know how to use the Windows sycalls – they are quite different than Unix! I’m not sure how you even do fork() on Windows. (I think Cygwin has emulation but there is way to do it without Cygwin)

                                                          https://github.com/oilshell/oil/wiki/Oil-Deployments

                                                2. 4

                                                  To the point some folks here have already raised, I’d be okay giving up shell scripting. Unfortunately, in order to do so, a replacement would: […] There are basically no programming languages that meet those criteria other than the existing shell languages.

                                                  I believe Tcl fits those requirements. It’s what I usually use for medium-sized scripts. Being based on text, it interfaces well with system commands, but does not have most of bash quirks (argument expansion is a big one), and can handle structured data with ease.

                                                  1. 4

                                                    Always use #!/usr/bin/env bash at the beginning of your scripts (change if you need something else, don’t rely on a particular path to bash though).

                                                    I don’t do this. Because all my scripts are POSIX shell (or at least as POSIX complaint as I can make them). My shebang is always #!/bin/sh - is it reasonable to assume this path?

                                                    1. 4

                                                      you will miss out on very useful things like set -o pipefail, and in general you can suffer from plenty of subtle differences between shells and shell versions. sticking to bash is also my preference for this reason.

                                                      note that the /usr/bin/env is important to run bash from wherever it is installed, e.g. the homebrew version on osx instead of the ancient one in /bin (which doesn’t support arrays iirc and acts weirdly when it comes across shell scripts using them)

                                                      1. 4

                                                        My shebang is always #!/bin/sh - is it reasonable to assume this path?

                                                        Reasonable is very arbitrary at this point. That path is explicitly not mandated by POSIX, so if you want to be portable to any POSIX-compliant system you can’t just assume that it will exist. Instead POSIX says that you can’t rely on any path, and that scripts should instead be modified according to the system standard paths at installation time.

                                                        I’d argue that these days POSIX sh isn’t any more portable than bash in any statistically significant sense though.

                                                        1. 2

                                                          Alpine doesn’t have Bash, just a busybox shell. The annoying thing is if the shebang line fails because there is no bash, the error message is terribly inscrutable. I wasted too much time on it.

                                                          1. 2

                                                            nixos has /bin/sh and /usr/bin/env, but not /usr/bin/bash. In fact, those are the only two files in those folders.

                                                          2. 3

                                                            https://mkws.sh/pp.html hardcodes #!/bin/sh. POSIX definitely doesn’t say anything about shs location but I really doubt you won’t find a sh at /bin/sh on any UNIX system. Can anybody name one?

                                                          3. 2

                                                            I would add, prefer POSIX over bash.

                                                          4. 18

                                                            I checked, and shellcheck (at least the version on my computer) only catches issue #5 of the 5 I list.

                                                            1. 14

                                                              That’s because the other ones are options and not errors. Yes, typically they are good hygiene but set -e, for example, is not an unalloyed good, and at least some experts argue against using it.

                                                              1. 3

                                                                Not for lack of trying: https://github.com/koalaman/shellcheck/search?q=set+-e&type=issues

                                                                There are tons of pedants holding us back IMO. Yes, “set -e” and other options aren’t perfect, but if you even know what those situations are, you aren’t the target audience of the default settings.

                                                              2. 17

                                                                I eventually put in the time

                                                                Yup, that’s how you do it, It’s a good idea to put in the the time to understand shell scripting. Most of the common misconceptions come out of misunderstanding. The shell is neither fragile (it’s been in use for decades, so it’s very stable) nor ugly (I came from JavaScript to learning shell script, and it seemed ugly indeed at first, now I find it very elegant). Keeping things small and simple is the way to do it. When things get complex, create another script, that’s the UNIX way.

                                                                It’s the best tool for automating OS tasks. That’s what it was made for.

                                                                +1 to using ShellCheck, I usually run it locally as

                                                                shellcheck -s sh
                                                                

                                                                for POSIX compliance.

                                                                I even went as far as generating my static sites with it https://mkws.sh/. You’re using the shell daily for displaying data in the terminal, it’s a great tool for that, why not use the same tool for displaying data publicly.

                                                                1. 6

                                                                  No, it really is ugly. But I’m not sure why that matters

                                                                  1. 13

                                                                    I believe arguing if beauty is subjective or not is off topic. 😛

                                                                2. 16

                                                                  I went the opposite direction - I was a shell evangelist during the time that I was learning it, but once I started pushing its limits (e.g. CSV parsing), and seeing how easy it was for other members of my team to write bugs, we immediately switched to Python for writing dev tooling.

                                                                  There was a small learning curve at first, in terms of teaching idiomatic Python to the rest of the team, but after that we had much fewer bugs (of the type mentioned in the article), much more informative failures, and much more confidence that the scripts were doing things correctly.

                                                                  I didn’t want to have to deal with package management, so we had a policy of only using the Python stdlib. The only place that caused us minor pain was when we had to interact with AWS services, and the solution we ended up using was just to execute the aws CLI as a subprocess and ask for JSON output. Fine!

                                                                  1. 15

                                                                    I tend to take what is, perhaps, a middle road. I write Python or Go for anything that needs to do “real” work, e.g. process data in some well-known format. But then I tie things together with shell scripts. So, for example, if I need to run a program, run another program and collect, and then combine the outputs of the two programs somehow, there’s a Python script that does the combining, and a shell script that runs the three other programs and feeds them their inputs.

                                                                    I also use shell scripts to automate common dev tasks, but most of these are literally one-ish line, so I don’t think that counts.

                                                                    1. 2

                                                                      This makes sense to me

                                                                    2. 8

                                                                      we immediately switched to Python for writing dev tooling.

                                                                      FWIW when shell runs out of steam for me, I call Python scripts from shell. I would say MOST of my shell scripts call a Python script I wrote.

                                                                      I don’t understand the “switching” mentality – Shell is designed to be extended with other languages. “Unix philosophy” and all that.

                                                                      I guess I need to do a blog post about this ? (Ah I remember I have a draft and came up with a title – The Worst Amounts of Shell Are 0% or 100%https://oilshell.zulipchat.com/#narrow/stream/266575-blog-ideas/topic/The.20Worst.20Amount.20of.20Shell.20is.200.25.20or.20100.25 (requires login)

                                                                      (Although I will agree that it’s annoying that shell has impoverished flag parsing … So I actually write all the flag parsers in Python, and use the “task file” pattern in shell.)

                                                                      1. 2

                                                                        What is the “task file” pattern?

                                                                        1. 5

                                                                          It’s basically a shell script (or set of scripts) you put in your repo to automate common things like building, testing, deployment, metrics, etc.

                                                                          Each shell function corresponds to a task..

                                                                          I sketched it in this post, calling it “semi-automation”:

                                                                          http://www.oilshell.org/blog/2020/02/good-parts-sketch.html

                                                                          and just added a link to:

                                                                          https://lobste.rs/s/lob0rw/replacing_make_with_shell_script_for

                                                                          (many code examples from others in that post, also almost every shell script in https://github.com/oilshell/oil is essentially that pattern)

                                                                          There are a lot of names for it, but many people seem to have converged on the same idea.

                                                                          I don’t have a link handy not but Github had a standard like this in the early days. All their repos would have a uniform shell interface so that you could get started hacking on it quickly.

                                                                    3. 5

                                                                      You should investigate just for task running. It’s simple like make but none of the pitfalls of it for task running.

                                                                    1. 8

                                                                      The way we handle this problem is to wrap the Firefox binary so that we can setup the necessary environment to make it find flash in the nix store.

                                                                      See, I just can’t get excited about Nix because of stuff like this. Purely functional packages? Nice! But wait, you say I have to patch and / or wrap my binaries? Uh, no, hard pass.

                                                                      This is exactly what happened the one time I decided to jump in and use Nix. I realized that a package I wanted to upgrade was using patchelf in its script and it just struck me that this was way more thought than I ever wanted to put into how to install software on my computer.

                                                                      I guess I just don’t care that much about “purity” for its own sake.

                                                                      1. 17

                                                                        I don’t care about purity - it’s about practical benefits to me. I like to view the patching and wrapping shenanigans as the price of being able to run multiple versions of a library or binary on your system. Most Linux systems don’t let you do that without creating a separate distribution with something like chroot or Docker. And those come with their own complexity.

                                                                        Maybe one day the ecosystem will have matured and we wouldn’t have to do all that.

                                                                        1. 5

                                                                          This does get me excited because once one maintainer does the work of parching the library, it won’t ever break again because it can’t find flash or finds an incompatible version of flash because I did an unrelated system update.

                                                                          1. 3

                                                                            Does that happen to you often? I can’t remember the last time something like that occurred. Probably the early 2000s? I mean, I guess on Linux things like that go wrong more often, but that’s part of the reason I don’t use desktop Linux…

                                                                            Side note: does Flash still exist? I thought it was EOL’d years ago!

                                                                            1. 2

                                                                              Yeah, which means if you want to re-live 2004-era Newgrounds, one of the things you need to do to make this happen is get a browser that is capable of executing Flash.

                                                                              Breaking a userspace program because a library it depends on updated doesn’t happen super-often on desktop linux, but it does happen. And if you don’t use linux as a desktop operating system, you’re forced to use non-free desktop operating systems that make it very difficult for you to use your system without creating a real-name account (Windows), creating an app store account with a credit card (Mac OS), or see political messaging creating by the people who work at Microsoft when logging in.

                                                                          2. 3

                                                                            See, I just can’t get excited about Nix because of stuff like this. Purely functional packages? Nice! But wait, you say I have to patch and / or wrap my binaries? Uh, no, hard pass.

                                                                            Not only that, but if I want to package my stuff for it, I have to learn a new language that’s not very much like any of the ones I already use?

                                                                            I think I’m more hopeful for rpm+ostree (as seen here)… it seems likely to carry most of the benefits Nix is advertising (some of which I find appealing) with less disruption. Maybe if I didn’t already know how to make and use RPMs, the math would be different.

                                                                            1. 4

                                                                              Fortunately unless you’re using some very special or custom build system, the correct wrappers are in place. So in most cases you copy the basic sample file, change the name, change the version and you’re good to go. There are exceptions of course… but those require special handling in RPM specs in similar ways.

                                                                          1. 2

                                                                            Nix with types! Love it! :)

                                                                            For the time being, we haven’t yet worked out one robust, practical and powerful solution to use Nickel as a front-end for Nix development. However, we have been actively thinking about it. And now, Nix integration is the very next step on the roadmap

                                                                            1. 1

                                                                              Learning Docker (finally? am I the las one). I’m reading Learn Docker in a Month of Lunches, and any tips or recommendations are welcome.

                                                                              1. 2

                                                                                If you love poking around at random things… Here’s the easiest way to do it to docker images that somebody else built: https://github.com/wagoodman/dive

                                                                                1. 1

                                                                                  Thanks for the tip.

                                                                                2. 2

                                                                                  I found learning Docker much easier when I was actually trying to build and deploy something. Pick a project (preferably a popular well supported one) from https://www.linuxserver.io/ and deploy it, either on an old machine or in the cloud.

                                                                                  You’ll learn a bunch about Docker getting it up and running, and then you get to play with a new open source project as a bonus :)

                                                                                  (FWIW I personally find docker compose invaluable for this stuff. Not great for k8s or whatever but for small self hosting projects? Priceless.)

                                                                                  1. 2

                                                                                    Thanks for the suggestion. What finally pushed me to look into Docker was this sort of thing. I wanted to use percollate, but I didn’t want to install (deal with, think about…) node or npm. There are a couple of Docker images available (e.g., this one) but I couldn’t get them to work. So (yak shaver that I am), I figure I’ll learn Docker and get something working in the process.

                                                                                    I’ll look into docker compose too.

                                                                                    1. 2

                                                                                      Cool! I was initially thinking your chances of finding a pre-packaged Docker container were slim since the project doesn’t list it as a supported install path, but in this Github issue I see someone built one!

                                                                                      You may have to do some debugging, but just remember that a Dockerfile is ultimately nothing more than a shell script :)

                                                                                      GHLF! Report back with how it goes :)

                                                                                1. 1

                                                                                  Back to libp2p, trying to restart my DHT playground.

                                                                                  1. 15

                                                                                    trying to keep myself and my family safe In the middle of an occupation of the capital of a G7 country. They protested at my son’s school yesterday.

                                                                                    Maybe trying to set up the gaming laptop into a full-fledged setup with dual monitors and an actual keyboard and mouse when I’m not doing that.

                                                                                    1. 3

                                                                                      How is the situation down there conflict wise? The article shows kids playing soccer on the streets. I hope it’s safe!

                                                                                      1. 1

                                                                                        Everyone is safe, but the situation is very uneasy. Many of the occupiers have brought their kids, so I’m not surprised they’re playing soccer, but I wouldn’t want any of my family in the middle like that.

                                                                                      2. 3

                                                                                        Yikes, try and stay safe. I’m over in Orleans myself but this stuff is frankly terrifying.

                                                                                        1. 3

                                                                                          Aw geez. And just a couple of months ago I was thinking of Canada as being a potential refuge place if things go even more crazy in Europe. Best of luck and endurance to you all.

                                                                                          Mars it is then, I guess.

                                                                                          1. 2

                                                                                            If it makes you feel better, there’s likely nowhere on the planet that is truly a refuge if things go super sour.

                                                                                            1. 1

                                                                                              I’d be fine with a small unknown island. But then somebody asks me where my low-latency internet and power come from.

                                                                                          2. 2

                                                                                            I’m really sorry to hear. This is a super scary time and having the threat of violence looming so close has got to be nervous making.

                                                                                            1. 1

                                                                                              Hey, me too! I’m over in Gatineau in the ’burbs but I generally spend a lot of time in Ottawa proper.

                                                                                              1. 0

                                                                                                My sympathies; I’m flying into rat-licker central (Alberta :/ ) for my weekend; I expect I’ll see some of the dumbasses myself along the way, although not as bad as your part of the country. Good luck!

                                                                                                1. 1

                                                                                                  Hey now, we are rat free!

                                                                                              1. 21

                                                                                                Installing Slackware from a heap of floppies on a second hand piece of crap Packard Bell in 1994 is a very big part of why I’m where I’m at today.

                                                                                                1. 8

                                                                                                  I got a Slackware CD-ROM with a magazine in 1993. Unfortunately, the 2 or 3 MB RAM we had in our home PC was too little. So I traded the CD with my uncle for some Sherlock Holmes game.

                                                                                                  In 1994 we had 5 MB RAM (we wanted to play Doom) and I had another Slackware CD-ROM and I was hooked. Lots of discussions with my brother followed about the 40 MB hard disk should be partitioned between Linux and MS-DOS (though I also used loadlin + UMSDOS for a while).

                                                                                                  Patrick Volkerding bootstrapped my Unix education.

                                                                                                  Edit: I even wrote a Slackware Book when I was a student: https://rlworkman.net/howtos/slackbasics.pdf , though I never completely finished it.

                                                                                                  1. 6

                                                                                                    1995 for me. On a Gateway 2000 486SX33 desktop that I picked up for $30 when a local business sold it off as surplus. The floppies were nearly all AOL freebies with scotch tape on the write protect notch.

                                                                                                    And I let the smoke out of a 14” monitor that cost at least as much as that PC when I screwed up my XFree86 modeline.

                                                                                                    The things I learned exploring that system gave me an entirely different view of what I was studying and influenced my tech choices for a long time after.

                                                                                                    The last time I daily drove Slackware was 2002. I’m half tempted to do it again for a while now, just for nostalgia’s sake.

                                                                                                    1. 5

                                                                                                      logging into darkstar was pretty cool.

                                                                                                      1. 4

                                                                                                        Same. But from a magazine CD 😂

                                                                                                        1. 3

                                                                                                          Same here, but in ’97. I downloaded the release from an FTP server on a 33.6 modem. The damn thing took a week!

                                                                                                          1. 3

                                                                                                            Whew, I feel young again! Same in 2001, on a 90 MHz Pentium someone at my dad’s office found in a closet. It had a whole 8 MB of RAM and Slackware was the only thing that would install on it. Never did get X11 working on it, but I sure learned a lot about living in the command line.

                                                                                                            1. 3

                                                                                                              I don’t recall if I first downloaded Slackware via a modem or what. I bought my first copy of SLS Linux (SoftLanding Systems, which predated Slackware) from a Usenet post from some random guy. He mailed me a set of 5.25in floppies.

                                                                                                              After that I was using Slackware for a while.

                                                                                                              By the mid-1990’s I had a CD-ROM drive and was buying CDs with the latest releases. Sometimes in stores. Which had shelves of software in boxes. I am so old.

                                                                                                              1. 2

                                                                                                                Same here!

                                                                                                                1. 2

                                                                                                                  But my experience was on a Toshiba Portege 610CT: https://www.youtube.com/watch?v=Ram4Faoo9t8

                                                                                                              1. 1

                                                                                                                Playing with libp2p

                                                                                                                1. 1

                                                                                                                  After being lied to by source code comments for 20+ years I quite often do not register them at all. Your site’s color scheme contributes to that too – my eyes quickly jumped over the grey blocks.

                                                                                                                  To me, it would be better to have regular paragraphs interspersed with “this is your terminal running commands” blocks. Of course, that could be just me – maybe most folks actually read through comments.

                                                                                                                  1. 1

                                                                                                                    It actually was my intention to make the comments stand back as much as possible. I even thought about only displaying them when you hover a command with the mouse.

                                                                                                                    The reason is that I usually prefer to read code and only read text if I am not sure what the code does. Because most of the time code is self explanatory. As Linus Torvalds puts it “Words are cheap, show me code”.

                                                                                                                  1. 5

                                                                                                                    I’ll echo the “what’s the difference between this and Nix” sentiment, but what’s really interesting to me is if it handles the hard cases. All I see is “easy” examples with Rust and Go.

                                                                                                                    Let’s be mean and go for the ugliest one - python 3.8.6 (or whatever specific version) along with numpy, matplotlib along with pytorch and the CUDA system libraries (of course, all of them pinned to specific versions).

                                                                                                                    1. 2

                                                                                                                      There is no claim that this is competing with Nix. The opposite in fact; we suggest using Nix if you need a full package manager.

                                                                                                                      Though since you bring it up, Python is supported, and both matplotlib and numpy work fine. Hermit doesn’t install libraries, so naturally the CUDA libraries are not supported.

                                                                                                                      1. 2

                                                                                                                        Thanks for your informative response! I was trying to figure out where the tool fit in (my mind’s view) of the whole distro packaging/docker/nix/language packages mess.

                                                                                                                        To be fair, Nix/NixOS doesn’t handle that python/torch/CUDA dumpster fire of a situation too well either. It makes it possible, but you have to work for it…

                                                                                                                    1. 11

                                                                                                                      Python is oriented towards productivity

                                                                                                                      I think this is not invalid, but if you are looking for productive languages in 2021 you could do much better than python. The place where python does still have a competitive edge, of course, is hiring. But then again, you may rue the easiness of hiring because there quite a few footguns and being easy to hire increases the likelihood that you wind up getting someone that is amateurish (even though they are ‘seniors’), or if you get a junior, there aren’t guardrails around those footguns.

                                                                                                                      1. 7

                                                                                                                        I think this is not invalid, but if you are looking for productive languages in 2021 you could do much better than python.

                                                                                                                        This seems incredibly subjective to me. It’s great that you feel that way, but when you word it this way you make it sound like an absolute which it most assuredly is not.

                                                                                                                        1. 6

                                                                                                                          If you don’t agree that Python is optimizing for developer speed (and I certainly don’t) then the whole article falls apart.

                                                                                                                          You say hiring might be easier, but I decline anything from recruiters if it’s Python. I’m so done trying to make sense of code bases with no typing. Mypy helps, but real static types it is not.

                                                                                                                          As far as I’ve seen, Python is picked because it’s the language everyone on the team knows.

                                                                                                                          1. 4

                                                                                                                            Python productivity

                                                                                                                            There are two types of programmer productivity, addressing different problems:

                                                                                                                            1. I don’t know much about the underlying tech. Can I get this script running in 30 minutes?
                                                                                                                            2. I don’t know much about the problem domain. Can I change this multi-million line behemoth without breaking everything and getting fired?

                                                                                                                            Python excels at (1) and fails miserably at (2). Yes, I’m aware of mypy and its attempts at solving the second problem. It’s not there yet.

                                                                                                                            1. 2

                                                                                                                              think this is not invalid, but if you are looking for productive languages in 2021 you could do much better than python

                                                                                                                              Such as? And why?

                                                                                                                              1. 4

                                                                                                                                I’m a plain old C/asm (or Lisp) guy, but it seems to me that if you’re thinking about using Python then modern Javascript (e.g. TypeScript if you like type annotations, as I do) running in node.js does pretty much everything Python does but fifty times faster if you’re actually writing algorithms not just gluing together other people’s C libraries. There’s a similar extensive library of modules ready to use. There’s a slightly different set of footguns. but are they worse? Probably not.

                                                                                                                                1. 5

                                                                                                                                  GP was claiming that there were probably “much better than python” out there, not “roughly comparable”.

                                                                                                                                  But your specific claim is not even true. Node.js doesn’t do everything python does, at all. It doesn’t do sync io nor does it have support for multi threading or multi processing. It does allow you to start your application in multiple processes but doesn’t offer you a way to control them like you do in C for example. This is a huge deal. You have no way to control basic scalability of your application. It will queue up all io calls until it exausts the resources. And has no other concept of concurrency than essentially making everything in paralell. It’s a memory leak by design.

                                                                                                                                  It has an old fashioned and less ergonomic syntax with many more corner cases and qwirks than python. And to my knowledge, it has no well established (or probably not any at all)? GPU library.

                                                                                                                                  The point is that python is versatile. Node js is not even primarily a programming language implementation. It is a single thread event machine that ships with a JavaScript API, provided by an implementation extracted from a browser. Still puzzles me that people don’t find this weird. A library is the main piece and the programming language comes as an addition.

                                                                                                                                  1. 1

                                                                                                                                    No two things that are distinct do all the same things as each other. You choose the features that matter to you.

                                                                                                                                    “It doesn’t do sync io nor does it have support for multi threading”

                                                                                                                                    That’s a contradiction. If you’re programming in plain old JS then it doesn’t do sync io because while you’re waiting it goes off and does things for other threads. But if you’re programming in Iced Coffescript or TypeScript then you can write your program as if the IO was synchronous, without explicitly writing the underlying callback yourself – exactly the same as happens in C where when you do synch io your thread library (or the OS) saves all your registers and loads someone else’s registers and runs another thread until your IO is done.

                                                                                                                                    1. 1

                                                                                                                                      If you’re programming in plain old JS then it doesn’t do sync io because while you’re waiting it goes off and does things for other threads.

                                                                                                                                      What makes you think that is how it works? It is not. Browser JavaScript and nodejs are single threaded. In fact, the whole reason nodejs was created was to provide a build of a single threaded js implementation with async io. Everything is performed in the same thread. Perform an operation that takes perceptible time and your whole application freezes. Your browser will get unresponsive. Put a for cycle with a few million iterations inside a callback and see what happens.

                                                                                                                                      Nodejs is standalone build of V8, which is chrome’s JavaScript engine. Early nodejs builds for Windows did execute IO by relying on worker threads, but that is not the point as you don’t have access to them. It was a workaround not intended for production usage but rather to provide a solution for people to use their windows machines for development. In the end you only have access to one thread and you have to do everything there, by design. You have to trust it to use whatever resources it needs to complete io (essentially starving the machine resources if you push it) and have no API to control what gets done simultaneously or when to wait for what. Notice that this is a perfectly acceptable value proposition for a script that is executed in in the context of a webpage. But it is absurd in simple cases such as writting on a text file line by line sequentially. You have no way of doing it without letting nodejs opening as many IO descriptors as quick as possible. Sort of like fork bombing your machine. Nodejs did include synchronous io, but they deprecated because people put it inside callbacks and flooded GitHub with tickets claiming their application would ‘randomly’ freeze.

                                                                                                                                      To this day, I still haven’t heard the reason why one would choose to write a server application or even a command line utility using this IO model. From what I gathered, the rational includes people not being familiar with multi threading and synchronisation APIs and creating race conditions.

                                                                                                                                      I am not sure what you are referring to when you mention typescript. Last time I checked, it was simply a compiler targeting JavaScript with no functionality whatsoever. But that was a few years ago, I don’t know about presents day. Iced coffescript does provide alternative IO APIs though.

                                                                                                                            1. 37

                                                                                                                              I hate to say it, but while the substance of the article is useful, it disproves the title, in my opinion.

                                                                                                                              The title says that Nix is close to perfect, and then lays out a lot of shortcomings that take it far away from perfect.

                                                                                                                              I personally wish that there was a Nix that was well-documented, not so academic and elitist, and in general, had some empathy for users, especially for new users. In fact, lacking empathy for users alone makes something not close to perfect, in my opinion.

                                                                                                                              Also, the soft split that is mentioned makes me nervous. Why such a split? What makes flakes better enough that some people use them, but not better enough that others don’t?

                                                                                                                              This all might sound very negative, and if so, I apologize. I want Nix’s ideas to take off, so I actually feel discouraged about the whole thing.

                                                                                                                              1. 17

                                                                                                                                Unpopular opinion here. The Nix docs are weird, but they are mostly fine. I usually don’t have any issue with them. The thing that usually gets me is the holes in my knowledge about how certain packaging technologies (both distro- and language-level ones) work and taking some of the things other distros do automatically for granted.

                                                                                                                                Here’s an example. You are playing in a Ubuntu-based distro, and you are writing some Python. You pip install some-dependency, import it, and everything is easy, right. Well, it felt easy because two months ago you apt install-ed a C dependency you forgot about, and that brought in a shared lib that your Python package uses. Or your pip install fetches a pre-built wheel that “just runs” (only on Ubuntu and few other distros, of course).

                                                                                                                                Nix is brutally honest and makes this shit obvious. Unfortunately, dealing with it is hard. [1] Fortunately, once you deal with all that, it tends to stay dealt with and doesn’t randomly break on other people’s computers.

                                                                                                                                Learning Nix has helped me learn Linux in ways I never originally suspected. It’s tons of fun (most of the time)!


                                                                                                                                [1] The rude awakening that a Python library can require a Fortran compiler is always fun to watch from the side. :)

                                                                                                                                1. 10

                                                                                                                                  The Nix docs are weird because they’re written for the wrong audience: people who want to learn Nix. I don’t care about Nix. Every moment I spend learning about Nix is just an inconvenience. Most Nix users are probably like that too. Users just want a packaging system that works, all of this discussion about Nix fundamentals is anti-documentation, things we need to skip to get to what we want: simple recipes for common tasks.

                                                                                                                                  But Nix also has at least two fundamental technical issues and one practical issue that exacerbate the doc situation. The practical issue has to do with the name: it’s just a search disaster that Nix is three things (a distro, a language, and a package manager). On to the technical issues.

                                                                                                                                  1. I can’t explore Nix because of the choice of a lazy language. Forcing values by printing them with builtins.trace is a minefield in Nix. Sometimes printing an object will result in it trying to create thousands of .drv files. Other times you find yourself printing one of the many circular objects that Nix uses.

                                                                                                                                    In Haskell and C++ I get to look at types to figure out what kind of object I’ve got, in addition to the docs. In Scheme and Python I get to print values to explore any object. In Nix? I can do neither. I don’t get types and I don’t get to print objects at runtime easily. At least you can print .drv files to figure out that say, a package happens to have a lib output, and that’s what you need to depend on instead of the default out output.

                                                                                                                                  2. There are almost no well-defined APIs within the Nix world.

                                                                                                                                    Aside from derivations, it’s all ad-hoc. Different parts of the Nix ecosystem work completely differently to accomplish the same goals. So learning how to do something to C packages, doesn’t help you when you’re dealing with Python packages, and doesn’t help you when you’re dealing with Haskell packages (where there are two completely different ecosystems that are very easy for novices to confuse). Flakes add a bit of structure, but they’ve been unstable for 3 years now with no stability on the horizon.

                                                                                                                                  1. 2

                                                                                                                                    I agree on both technical issues. Static types and type signatures in Nix would be especially amazing. I spend so much time wondering “what type does this have” when looking at nixpkgs code. :(

                                                                                                                                    As for the fundamentals and anti-documentation features, I am not so sure. I think Nix is such a fundamentally different way of doing things, that you need to start somewhere. For example, I can’t give users a packaging script sprinkled with content SHA’s without explaining what those are and why we need them in the first place (It’s especially baffling when they sit right next to git commit SHA’s). The Nix pills guide has a good way of introducing the important concepts and maybe it can be shortened, so that people can go through most of the stuff they need in half an hour. I don’t know…

                                                                                                                                2. 10

                                                                                                                                  not so academic and elitist

                                                                                                                                  For a language that is the bastard child of Bash and ML, I would not consider it “academic”. The ugliness of the language is due in no small part to the affordances for real-world work.

                                                                                                                                  As far as elitism…well, it’s a hard tool to use. It’s getting easier. It’s strange to me to expect that such powerful magic shouldn’t take some work to learn (if not master).

                                                                                                                                  1. 19

                                                                                                                                    For a language that is the bastard child of Bash and ML, I would not consider it “academic”. The ugliness of the language is due in no small part to the affordances for real-world work.

                                                                                                                                    I never said the language was academic. I don’t think it is. In fact, it’s less that the language is academic and more that the documentation and culture are.

                                                                                                                                    As far as elitism…well, it’s a hard tool to use. It’s getting easier. It’s strange to me to expect that such powerful magic shouldn’t take some work to learn (if not master).

                                                                                                                                    Power does not imply something must be hard to learn. That is a common misconception, but it’s not true.

                                                                                                                                    As an example, consider Python. It’s not that hard to learn, yet it is enormously powerful. Or Ruby.

                                                                                                                                    In fact, Ruby is a great example because even though it’s powerful, it is mostly approachable because of the care taken in helping it to be approachable, epitomized by _why’s Poignant Guide.

                                                                                                                                    _why’s efforts worked because he made Ruby approachable with humor and by tying concepts of Ruby to what people already knew, even if he had to carefully lay out subtle differences. Those techniques, applied with care, would work for Nix too.

                                                                                                                                    So the problem with Nix is that they use the power as an excuse to not put more effort into making it approachable, like _why did for Ruby. This, I think, is a side effect of the culture I mentioned above.

                                                                                                                                    If someone wrote a Nix equivalent of _why’s Poignant Guide, playing to their own strengths as writers and not just trying to copy _why, I think Nix would have a massive uptake not long after.

                                                                                                                                    In fact, please do write that guide, if you would like to.

                                                                                                                                    1. 12

                                                                                                                                      If I had more spoons I’d definitely do that

                                                                                                                                  2. 9

                                                                                                                                    I agree. It is some of the coolest Linux technology that is out there, but it is so hard to use. Both because its poor documentation and because of how different it is. When I used it, it felt like once a week I would try to do something not allowed/possible and then would have to go multiple pages deep on a thread somewhere to find multiple competing tools that claim to solve that problem best. I think I will try Nix the language on the next personal project I make that involves more than one language, but I haven’t had a chance to do that in a while.

                                                                                                                                    I would love to try NixOS again sometime. Hopefully they will come out with better documentation and/or a “let me cheat just this once so I can continue working” feature.

                                                                                                                                    Edit: I forgot to say, great article though! I enjoyed your perspective.

                                                                                                                                    1. 6

                                                                                                                                      I have found the Guix documentation quite good, and the community very welcoming, for what it’s worth.

                                                                                                                                      1. 6

                                                                                                                                        Keep in mind that the more familiar someone is with the subject the more issues they can talk about. I could go on for ages about problems with python, even though it is a perfect language for most of my use cases - It’s not a contradiction.

                                                                                                                                        The post just concentrated on the negatives rather than positives - and there are some really cool things about nix. Especially if you have use cases where everything else seems to be worse (looking at you chef/puppet/ansible).

                                                                                                                                        1. 1

                                                                                                                                          I wouldn’t feel discouraged if I were you. Nix’s community is nothing but growing. Most of these issues are warts, not dealbreakers.

                                                                                                                                          1. 1

                                                                                                                                            makes me miss WORLDofPEACE. Didn’t know them, but they’re a good example of someone that can make anyone feel welcome IMO.

                                                                                                                                          1. 2

                                                                                                                                            Is there any reason for randomizing, or even rotating, the CA? I don’t understand the reasoning for it. It seems entirely unrelated to the “let’s encrypt can go down” scenario.

                                                                                                                                            1. 12

                                                                                                                                              If you always use LetsEncrypt, that means you won’t ever see if your ssl.com setup is still working. So if and when LetsEncrypt stops working, that’s the first time in years you’ve tested your ssl.com configuration.

                                                                                                                                              If you rotate between them, you verify that each setup is working all the time. If one setup has broken, the other one was tested recently, so it’s vastly more likely to still be working.

                                                                                                                                              1. 2

                                                                                                                                                when LetsEncrypt stops working

                                                                                                                                                That’s how I switched to ZeroSSL. I was tweaking my staging deployment relying on a lua/openresty ACME lib running in nginx and Let’sEncrypt decided to rate limit me for something ridiculous like several cert request attempts. I’ve had zero issues with ZeroSSL (pun intended). Unpopular opinion - Let’s Encrypt sucks!

                                                                                                                                                1. 5

                                                                                                                                                  LE does have pretty firm limits; they’re very reasonable (imo) once you’ve got things up and running, but I’ve definitely been burned by “Oops I misconfigured this and it took a few tries to fix it” too. Can’t entirely be mad – being the default for ACME, no doubt they’d manage to get a hilariously high amount of misconfigured re-issue certs if they didn’t add a limit on there, but between hitting limits and ZeroSSL having a REALLY convenient dashboard, I’ve been moving over to ZeroSSL for a lot of my infra.

                                                                                                                                                2. 2

                                                                                                                                                  But he’s shuffling during the request-phase. Wouldn’t it make more sense to request from multiple CAs directly and have more than one cert per each domain instead of ending up with half your servers working?

                                                                                                                                                  I could see detecting specific errors and recovering from them, but this doesn’t seem to make sense to me :)

                                                                                                                                                3. 6

                                                                                                                                                  It’s probably not a good idea. If you have set up a CAA record for your domain for Let’s Encrypt and have DNSSEC configured then any client that bothers to check will reject any TLS certificate from a provider that isn’t Let’s Encrypt. An attacker would need to compromise the Let’s Encrypt infrastructure to be able to mount a valid MITM attack (without a CAA record, they need to compromise any CA, which is quite easy for some attackers, given how dubious some of the ‘trusted’ CAs are). If you add ssl.com, then now an attacker who can compromise either Let’s Encrypt or ssl.com can create a fake cert for your system. Your security is as strong as the weakest CA that is allowed to generate certificates for your domain.

                                                                                                                                                  If you’re using ssl.com as fall-back for when Let’s Encrypt is unavailable and generate the CAA records only for the cert that you use, then all an attacker who has compromised ssl.com has to do is drop packets from your system to Let’s Encrypt and now you’ll fall back to the one that they’ve compromised (if they compromised Let’s Encrypt then they don’t need to do anything). The fail-over case is actually really hard to get right: you probably need to set the CAA record to allow both, wait for the length of the old record’s TTL, and then update it to allow only the new one.

                                                                                                                                                  This matters a bit less if you’re setting up TLSA records as well (and your clients use DANE), but then the value of the CA is significantly reduced. Your DNS provider (which my be you, if you run your own authoritative server) and the owner of the SOA record for your domain are your trust anchors.

                                                                                                                                                  1. 3

                                                                                                                                                    There isn’t any reason. The author says they did it only because they can.

                                                                                                                                                    1. 2

                                                                                                                                                      I think so. A monoculture is bad in this case. LE never wanted to be the stewards of ACME itself, instead just pushing the idea of automated certificates forward. Easiest way to prove it works is to do it, so they did. Getting more parties involved means the standard outlives the organization, and sysadmins everywhere continue to reap the benefits.

                                                                                                                                                      1. 2

                                                                                                                                                        To collect expiration notification emails from all the CAs! :D

                                                                                                                                                        1. 2

                                                                                                                                                          The article says “Just because I can and just because I’m interested”.

                                                                                                                                                        1. 18

                                                                                                                                                          I love reading these Nix success stories and then last night trying to simply install Grub on a system with zfs filesystem being literally impossible with the way my Nixos system was failing to derive something for no discernible reason, with no documentation anywhere and any reportings of that issue completely ignored :D

                                                                                                                                                          1. 7

                                                                                                                                                            I should just ruin it all and tell my secret. I don’t run NixOS. Plain nixpkgs on an LTS Ubuntu (boomer edition!) all the way.

                                                                                                                                                            I use Nix as a development helper, and, on rare occasions, deploy a service by sticking it in a systemd unit file and nix-copy-closure my way to success. Of course, that’s just for my gear. At $DAYJOB it’s the usual k8s circus.

                                                                                                                                                            1. 6

                                                                                                                                                              Setting up Linux to boot from ZFS root is tricky even under the best of circumstances. What was the reported issue?

                                                                                                                                                              I’m a huge fan of both NixOS and ZFS, but in the future I might aim for a tmpfs root and use ZFS only for persistent data.

                                                                                                                                                              1. 5

                                                                                                                                                                Nix is one of those technologies I think is amazing, but at the same time is practically unusable because of hard semantics and an inscrutable command line interface. It’s kind of like rust: dipping your toes and getting the tutorial working is easy enough, but the first time you are confronted with a real problem, finding the solution requires so much ancillary knowledge of arcane the minutiae of how the system works, it becomes a waste of time and solving it ‘the way I know’ is easier.

                                                                                                                                                              1. 21

                                                                                                                                                                I recently did a similar thing for a friend’s ancient PHP site while upgrading it to a supported version of the runtime.

                                                                                                                                                                That specific PHP 5.6.x release was no longer in nixpkgs. Of course, it being a git repo, allowed me to go back in history, find the last version that had my PHP release in, import it in a nix expression, and BOOM - it just worked.

                                                                                                                                                                I did the same for MySQL. Then branched a version of the site that used newer PHP and MySQL. That allowed me to quickly switch between the two environments and test things.

                                                                                                                                                                10/10 experience. Few people realize that every nixpkgs commit since the beginning of the project is a full Linux distro that you can modify to suit your needs.

                                                                                                                                                                1. 1

                                                                                                                                                                  No.

                                                                                                                                                                  That said, I think the article is surprisingly nuanced, but I fear that the laissez-fair attitude shown in posts like this is a reason many people equate web developers with junior developers.

                                                                                                                                                                  1. 17

                                                                                                                                                                    Did you even read the post? The framework takes 1.4 kB out of 13.9 kB total, and as a result you get a self-contained component you can insert in any page. He also provides a framework-less package for those who already have the Svelte package in their dependencies. That seems a like a well thought out and reasonable approach.

                                                                                                                                                                    Also I would argue that either you use an existing framework or you need to reinvent one, or parts of it. You might end up with less than 1.4 kB overhead but probably with some subtle bugs and performance issues that have already been solved in other frameworks, not to mention the wasted time.

                                                                                                                                                                    1. 5

                                                                                                                                                                      In addition to all the problems other people mentioned already, it’s also a matter of longevity.

                                                                                                                                                                      Do users really want to have a component that invisibly bundles some framework, which they only become fully aware of when the framework ends up being abandoned after one JS hype cycle (~18 months) and the security issues start piling up?

                                                                                                                                                                      1. 1

                                                                                                                                                                        And if don’t you use a framework, the component will remain secure forever? Of course it wouldn’t, it still requires maintenance work and whether it means updating the framework, or switching to a better framework, or updating bespoke code, it’s maintenance work. Things don’t remain miraculously bug free and vulnerability free just because you don’t use a framework.

                                                                                                                                                                        If you are arguing that maintaining bespoke code from a random developer (who might have moved on) is easier than updating a widely used framework, then I guess we simply disagree. I know I’ll pick any time a framework rather than reinventing my own.

                                                                                                                                                                        Also that “JS hype cycle” meme is getting a bit old, and I wonder where you pull that 18 months from. Svelte has been around for 4 years, React for 8 years, etc.

                                                                                                                                                                      2. 1

                                                                                                                                                                        wouldn’t it be more reasonable to defer to the browser/OS for selecting input characters? unless I’m misunderstanding the purpose of the emoji picker.

                                                                                                                                                                        1. 9

                                                                                                                                                                          The framework takes 1.4 kB out of 13.9 kB total, and as a result you get a self-contained component you can insert in any page.

                                                                                                                                                                          And this is precisely how your app ends up with 15 versions of Svelte built-in with say 9 of them having security bugs. Or weird bugs where version N+1 of the library destroys version N’s global state.

                                                                                                                                                                          1. 2

                                                                                                                                                                            I think you meant to reply to /u/lau

                                                                                                                                                                          2. 2

                                                                                                                                                                            For sure, I would love for browsers to standardize on some kind of emoji picker, or to just delegate to the OS’s built-in picker. I wrote down some thoughts here on that.

                                                                                                                                                                            1. 2

                                                                                                                                                                              In theory yes.

                                                                                                                                                                              But in practice a lot of sites these days want custom “emoji” which would be difficult to design into an OS/browser picker.

                                                                                                                                                                              For example look at Android where just about every keyboard has an emoji picker but many apps (especially messaging apps) have their own button to add emoji.

                                                                                                                                                                              1. 1

                                                                                                                                                                                custom emoji would not be present in a pre-packaged web component anyway

                                                                                                                                                                        1. 7

                                                                                                                                                                          So, we start with

                                                                                                                                                                          When I released emoji-picker-element, I made the decision to bundle its framework (Svelte) directly into the component.

                                                                                                                                                                          and then this:

                                                                                                                                                                          If you’re already using Svelte in your project, then you can import ‘emoji-picker-element/svelte’ and get a version that doesn’t bundle its own framework, ensuring de-duplication.

                                                                                                                                                                          Let me get this straight:

                                                                                                                                                                          1. If I import emoji-picker-element I get the Svelte-bundled version.
                                                                                                                                                                          2. If I import emoji-picker-element/svelte I do not get the Svelte part.

                                                                                                                                                                          Well, this is… horrible! The entire world out there works in exactly the opposite way. Everyone expects your library to have as few dependencies as possible, and any framework-related enhancements to be found in framework-specific modules like mylib/svelte. I’d rather just name the package svelte-mylib and spare myself and everyone else tons of grief.

                                                                                                                                                                          I could go on, but @soc summed it up quite nicely.

                                                                                                                                                                          1. 5

                                                                                                                                                                            True, this is the way most packages on npm work. But I tend to agree with posts like this one by Lea Verou arguing that web components are too newbie-unfriendly. A lot of developers are more comfortable with <script> tags than bundlers and build pipelines. They want something that “just works.”

                                                                                                                                                                            As-is, I can ship the same code for Node/npm and the browser using ES modules and sites like unpkg.com and jsdelivr.com. This JS file can be directly consumed in the browser, no bundler required. That wouldn’t be true if it had import 'svelte'.

                                                                                                                                                                            As mentioned in the post, though, Skypack is shaping up to be a good alternative: ship things in the npm/Node style, but then consume them on-the-fly in a pre-bundled format. I might switch to that.

                                                                                                                                                                            Again though, we’re talking about 1.4 kB of framework code out of 13.9 kB total, so it’s really tiddlywinks.

                                                                                                                                                                            1. 5

                                                                                                                                                                              The point is that with the standard import, the Svelte dependency is bundled and invisible. You do not need to even know Svelte exists as a framework to use it.

                                                                                                                                                                              The /svelte import is short for “I already have Svelte hanging around, so use that one instead”. It implies that the consumer is aware of and has set up Svelte.

                                                                                                                                                                              Makes perfect sense to me.

                                                                                                                                                                              1. 2

                                                                                                                                                                                The /svelte import is short for “I already have Svelte hanging around, so use that one instead”. It implies that the consumer is aware of and has set up Svelte.

                                                                                                                                                                                To me it implies the opposite. Why not say /without-svelte if you’re talking specifically about what you exclude?