Threads for gwenhael

  1. 29

    Timsort: A very fast , O(n log n), stable sorting algorithm built for the real world — not constructed in academia.

    I hope the article presents not the original version but the one that was fixed by academia.

    (See https://ercim-news.ercim.eu/en102/r-i/fixing-the-sorting-algorithm-for-android-java-and-python, https://link.springer.com/chapter/10.1007/978-3-319-21690-4_16)

    1. 10

      I hope so too, that’s a weird anti-intellectual statement to make to open a post about an idea.

      1. 10

        This guy used to post a lot of his lessons on lobsters and most of them had pretty glaring mistakes, like saying an O(2^n) algorithm was faster than a linear one, so I’m not surprised.

        1. 1

          Aye ya yaye… Sounds like they’re trying to reinforce by recall but at the same time feel good about themselves maybe…

        2. 3

          Being pro-pragmatism isn’t the same as anti-intellectual. “Academia” means different things to different people also, of course

          1. 1

            I was a bit surprised by the tone of this sentence, to say the least. Glad I’m not the only one!

        1. 7

          Post the excellent web apps that you know.

          I’ll start with https://www.photopea.com/

          1. 2

            https://www.soundslice.com/ (music learning/practice/editing via interactive sheet music)

            1. 1

              Note taking app: https://reflect.app/

              1. 1

                In the same spirit: https://mecabricks.com/

              1. 9

                I don’t get the fascination about computer-generated content. I haven’t delved much in it but I’ve seen glimpses of the DALL-E pictures, before that GPT-3 texts, all the various this-whatever-doesn’t-exist.

                I don’t get the interest. I only see empty attempts at mimicking conscience that ultimately fails to produce anything meaningful. I have the same feeling that I have when I listen to someone very good at talking without a purpose. Some people (especially politicians but not only) are very good at talking for a long time, catching the ear without ever really telling anything. It’s quite fascinating when you realize that the person has in fact only been using glue words, ideas and sentences but there’s no substance at all when you take out these fillers.

                It’s what I see in all of this. We got to a point where we make computers to churn out filler content for our mind, but there’s no nutritional value in it. We’re taking out the human producer of memes and honestly I’m a bit terrified we’ll end up brain-deadly consuming content produced by things. What happens when the art/ideas/entertainment/… is made without “soul”? What is the point of all this?

                I’m not very good at ordering and communicating my thoughts myself but I’m very scared already when my 12 years old son gets stuck scrolling short videos, the idea of taking the human “soul” out crushes my hope for the future of our species.

                1. 13

                  I’m excited about this for two principle reasons:

                  1. It’s fun. SO much fun. Getting DALL-E to generate a heavy metal album cover made of pelicans made of lightning? I found that whole process deeply entertaining. See also the “fantasy breakfast taco” game I describe at the end of my post.
                  2. I see these tools fitting in the category of “bicycles for the mind”. They help me think about problems, and they do things like break me out of writer’s block and help me get started writing something. In DALL-E’s case it’s an imagination enhancer: I can visualize my ideas with a 20s delay.

                  Aside from those, here’s a use-case you may not have considered which I tried recently on GPT-3. If you’re on disability benefits and a government office cancels them and you need to write a letter - but you don’t have much experience writing formal letters. I tried the prompt “Write a letter to the benefits office asking why my disability claim was denied”. Here’s the result: https://gist.github.com/simonw/6e6080a2f51c834c13b475743ef50148

                  I find this a pretty convincing attempt. I could do a better job, but I’ve written a lot of letters in my time. Give it a prompt with specific details of your situation and you’ll get something even more useful.

                  1. 4

                    In DALL-E’s case it’s an imagination enhancer: I can visualize my ideas with a 20s delay.

                    I’ve been using the free VQGAN-CLIP[1] to generate things like “Moomins as Dark Souls in the style of linocut / watercolour / screenprint etc.” to give me interesting things to practice linocutting, watercolours, acrylic painting, etc.

                    [1] https://github.com/nerdyrodent/VQGAN-CLIP

                  2. 8

                    Why do you expect it to have a vague notion of a “soul”? It’s a tool. It’s not an artist, it’s an automated Photoshop.

                    How do you feel about recorded music? Music used to exist only as a live human performance, and now we have soulless machines playing it. New music can be made with button presses, without fine motor skills of playing an instrument. Now we can create paintings without motor skills of using a brush.

                    To me DALL-E is a step as big as photography. Before cameras if you wanted to capture what you see, you had to draw it manually. With cameras it’s as trivial as pressing a button. Now we have the same power for illustrations and imaginary scenes.

                    Selfies have disrupted portrait painters, and this without a doubt will be disruptive for artists and photographers. Short term such commoditization sucks for creators, and the way ML is done is exploitative. Long term it means abundance of what used to be scarce, and that’s not necessarily a bad thing.

                    1. 3

                      Basically - I think at most, things like this will serve two purposes:

                      1. Inspiration stuff for artists - look at some ideas, see new possible connections.
                      2. Pornography (not in the sense you’re used to, but “meets my specific thing”)/pot-boilers; it follows a situation of like “I want X character in Y setting”, and generates something mostly coherent, but in a bland way. But maybe that’s all someone wants…
                      3. Shitposting.

                      I think these right now are devoid of much “soul” for lack of a better term - it’s impressive they can follow the prompt, but it’s impassionate and at times they feel like they’ve been painted by someone with dementia.

                      1. 1

                        Why is it seen as filler content? A lot of what the author ends up creating are descriptions that came from a person, and the AI is just doing its best to visualize it. There are thousands of instances of a description - why not generate them all?

                        I tell this to people from time to time that life is just state traversal… and AI helps traversing it

                      1. 2

                        Anecdata: I once decided to move my personal Nextcloud instance from MariaDB to PostgreSQL for getting acquainted to that other DB everyone seem to praise. It stopped working after I upgraded PostgreSQL to the new at the time major version. I promptly migrated back to MariaDB when I looked up why and saw what I had to do to perform the upgrade.

                        1. 7

                          MySQL/MariaDB is certainly much easier to upgrade, but the way you do that gives me the willies - basically, just replace the binaries with a newer version, then start the server with the old files. This server will happily run with the old tables and maybe even produce correct results, but they are considered “broken” until you run mysql_upgrade which then “repairs” them for you when needed.

                          1. 4

                            Postgres pg_restore does the same thing and works just fine. For the past few years the breaking on-disk changes have been to indexes and the database comes with both the old and new code for working with indexes in the server. You only get the new data structures on disk if you did one of the time-consuming updates like REINDEX, pg_dump/pg_restore or replication. Those do full rewrites of everything to perform the update which is equivalent to the mysql_upgrade rewrite.

                            1. 2

                              pg_upgrade is also very fast because depending on configuration it doesn’t do too much.

                              From the official docs:

                              Major PostgreSQL releases regularly add new features that often change the layout of the system tables, but the internal data storage format rarely changes. pg_upgrade uses this fact to perform rapid upgrades by creating new system tables and simply reusing the old user data files. If a future major release ever changes the data storage format in a way that makes the old data format unreadable, pg_upgrade will not be usable for such upgrades. (The community will attempt to avoid such situations.)

                            2. 1

                              thanks for warning..

                            1. 1

                              Genuinely sad that we’ll be losing the K9 name and logo at some point. 😅 A real part of that app’s appeal (alongside being an excellent mail client of course).

                              1. 1

                                Funnily I have the opposite reaction :P Finally that infuriating dog reference will disappear from my phone!

                                1. 1

                                  Sacrilege! More Doctor Who references on my home screen please.

                              1. 3

                                I’m a bit on the edge about this. I did learn a lot even if honestly I don’t write much HTML bu hand these days.

                                It does looks clean and challenges habits yet it relies a lot on knowing “obscure” details of HTML. It’s all implicit.

                                Nonetheless why not after all, it shows elegant HTML written with good knowledge of it.

                                1. 4

                                  Thank you for your kind words! Maybe we can get the word out and make these details less obscure ;)

                                1. 8

                                  My daughter is 5 - I don’t want her dialing 911.

                                  It’s weird to think of this as a problem since the entire world had this potential concern for a while and it was… fine?

                                  Super cool though. I’m jealous of the payphone.

                                  1. 10

                                    In Britain the number is 999, which lends itself very well to being dialled by any toddler who sees fit to mash a single button. Ask me how I know!

                                    1. 6

                                      Kids today will never know the joy of crank calling a random phone number.

                                      1. 3

                                        Maybe not, but the “It’s Lenny” crowd sure have fun with pranking the scammers who call them instead. https://old.reddit.com/r/itslenny/

                                        1. 3

                                          Hopefully not, but there are many folks that end up stuck doing that for a living!

                                          1. 1

                                            I wonder if there’s a YouTube video of someone playing The Jerky Boys for some kids and seeing what they make of it.

                                          2. 5

                                            I find it very weird. Why wouldn’t they want their child to be able to call for help?

                                            1. 4

                                              You know I’ve never thought about the positive case. What do people do nowadays? Instruct their kid on how to take the iPhone out of their pocket and make an emergency call from it?

                                              1. 4

                                                We bought our children their own phones, partly for this reason. Especially as we encourage them to wander the neighbourhood and catch trains from age 7.

                                            2. 4

                                              I actually did dial 911 on a dare on a payphone as a kid, and surprise, emergency services showed up (and I was being an idiot but what can I say). According to a friend who works in emergency dispatch, this used to be quite common (though because we’re both of around the same age I don’t know if any of us have stats on how 911 dialing has changed in the mobile phone era.)

                                              So yeah this is definitely a concern, but there’s also the positive case. My mother was in ill-health when I was a child and I did have to dial 911 for her a couple times. I was a technical kid, but it was nice that I could just take a few specific actions (namely dialing 9-1-1 on a phone) and get help for my mother. (You would think, having both dialed out to 911 as a prank and in legitimate need, that I would understand what a silly thing it was to have dialed out on a dare, but it took me a few years for that self-reflection ability to happen in my child brain.) I’m curious about what kids do these days especially since most folks keep their phones locked. Do folks keep a home phone around specifically to dial out to emergency services?

                                              1. 5

                                                Locked phones typically still have an “emergency call” button that allows dialing the local emergency number. On Android it also allows you to dial any of the contacts stored as emergency contacts in the phone. The question is still a valid one if a child was home without someone’s cell phone, but in general a locked phone shouldn’t stop an emergency call.

                                                EDIT: And, yes, I also have a landline for just such a situation.

                                                1. 4

                                                  Even phones without a SIM are supposed to be able to call the local emergency number, so that slightly narrows the use-case for a landline.

                                              2. 4

                                                My parents had a lot of older phone and computer equipment laying around that we liked to play with growing up. One item was an old rotary phone that belonged to one of my grandparents and I recall my father one day explaining the differences in dialing between it and DTMF. To let us hear the differences, he plugged it into one of the phone jacks and, after booping a few buttons on the touch-tone phone, he then showed us the difference in the rotary.

                                                He first dialed a “9” to let us hear all the clicks, then to show the difference he dialed a “1”. Because that was so short, he repeated the number to make sure we heard it. Two minutes later, the emergency operator called back and he had to explain himself…

                                                1. 1

                                                  I was almost in big trouble when a random kid I was playing with at the Burger King nearest my childhood home wouldn’t quit calling 911 on the pay phone. I don’t remember details. I’m not sure if I was just assumed to be the mastermind, or if the other kid tried to pin it on me.

                                                1. 1

                                                  I wonder if a similar setup is possible with QEMU?

                                                  1. 2

                                                    It’s definitely possible to run qemu/kvm VMs with storage on raw block devices, so I’d certainly expect so.

                                                    1. 2

                                                      I’ve used this trick in the past testing lilo tweaks to make sure I’d still have a bootable machine.

                                                  1. 2

                                                    So as usual, consider not writing Ruby, but if you do, use type signatures, and maybe 100% branch coverage too, as annoying as it seems.

                                                    Could someone more into Ruby ecosystem explain why the OP could have written this paragraph about “not writing Ruby”? Is Ruby a dying language?

                                                    1. 5

                                                      strong opinion syndrome

                                                      1. 4

                                                        People thinking only strictly typed languages are worthy are prone to not recommend not strictly typed languages like Ruby. I like Ruby and I have made extensive refactors without Sorbet&co by having confidence in my tests. My colleague who swears by C# and Typescript will talk your ears off “proving” how such a thing is impossible.

                                                        To each their own and let’s stop trying to find the one size fits all magical tool.

                                                        1. 4

                                                          OP was a big ruby community member with many contributions to his name, who fell out with ruby for, first, not solving the language and runtime issues he thought were the most important, then, for solving them in a way he likes. He does go now.

                                                          1. 3

                                                            If you want what this post wants (good types and safety rails) Ruby is a bad choice for those things. Sure there is sorbet and writing a billion tests, but even Crystal would be a better choice if those things are your priority.

                                                            If you want is to play with a highly malleable environment and don’t want smalltalk then Ruby is where it’s at, though.

                                                            1. 2

                                                              I can’t speak about the OP, but Ruby definitely isn’t dying. It might not be growing, but it is keeping a good level and there are new devs coming to Ruby all the time.

                                                              1. 1

                                                                They are into Go, based on the other posts on the blog.

                                                              1. 19

                                                                Stop using your politics against systemd’s politics makes for a weak rant. I don’t have a strong technical opinion on systemd but the TINA (There is no alternative) “politics” of systemd has been annoying for… ever, hence the push against it and the work to keep a systemd-free alternative alive and viable.

                                                                1. 2

                                                                  It’s interesting how the discussion is focused mainly on the number of lines. That seems to be the most benign of the factors at work here. I’d venture that the same untrained, unaware contributors to that nightmare would have done even worse if somehow someone had the idea at some point to split in multiple files.

                                                                  1. 1

                                                                    I made a similar switch a while ago, awesome to sway, because one game (Cyberpunk 2077 with wine) was crashing my whole computer randomly under awesome.

                                                                    I managed to make a usable configuration but I still miss awesome a lot because of:

                                                                    • the lack of workspace aware alt-tab: this is the most frustrating thing, I just can’t understand how it is considered acceptable to rotate through all windows regardless of workspace, I never want that;
                                                                    • my feeling that my configuration is made of a dozen loosely coupled pieces.

                                                                    I really just want awesome-on-wayland.

                                                                    1. 3

                                                                      An alternative could be Qtile. I have never tried it myself, but it looks like Awesome with Python and it works on both X11 and Wayland. I didn’t know about it before starting on i3, but I think I would have given a try.

                                                                      1. 1

                                                                        Interesting, I’ll take a look. That’d be the occasion to learn Python.

                                                                        1. 1

                                                                          Very interesting indeed, Qtile has a strong feeling of awesome-but-with-python. Sadly the lack of systray when in wayland and more importantly of XWayland support make it a non-starter for me. I’ll keep it on my radar nonetheless.

                                                                      2. 1

                                                                        Sometimes it’s worthwhile to run a dedicated xserver for a game only.

                                                                        1. 9

                                                                          Not for me. Having said that I really dislike medium. A post that talks about the dangers of monopolisation of crowd work could do a lot better than a for profit walled garden blogging platform.

                                                                          1. 4

                                                                            FWIW, reading through the “cached” link underneath the submission’s title is I believe the workaround for a number of paywalls.

                                                                            1. 1

                                                                              Does not work on iOS unfortunately, the archive.md website displays a captcha page which is impossible to solve (the page doesn’t have a mobile version, and the modal window to solve the captcha goes outside the browser’s viewport).

                                                                            2. 3

                                                                              I think opening the link in a private window does the trick as a workaround. It’s actually quite an illustration for the article point.

                                                                              1. 2

                                                                                Sure there are workarounds, but the sooner people who use Medium for publishing realize they’re cutting into their prospective audience, the better.

                                                                            1. 3

                                                                              An old laptop falling apart physically but perfectly fine computing-wise, given to me by my brother. Intel Core i3-3217U with 4GB of RAM running Slackware-current. An internal SSD for the OS and 3 external HDD for storage. Running mainly Nextcloud and gitea, plus a bunch of small services. No desire nor need to “containerize” any of this.

                                                                              1. 3

                                                                                Nextcloud News

                                                                                1. 4

                                                                                  All generalizations suck, but it needs to be stated that this isn’t a problem of untyped text streams, as much as the author (and others) would like to make it so.

                                                                                  The problem can generically be stated as organic accretion, but there’s probably a bit of human laziness underneath it all. I’d summarize it as “Compound piping among several commands is tough. Adding new options is easy!” While not true in each and every instance, and for many of these cases probably adding the option was tougher than the composition, it’s “true enough” where that over a period of decades the cruft begins to add up.

                                                                                  I wonder if further standards around the Unix Philosophy might help here, specifically in regards to code budgeting (tm) and other purposeful artificial (and perhaps functionally orthogonal) constraints on design.

                                                                                  1. 5

                                                                                    Organic accretion is a property of software. If something exists that works, it’s easier to add to it than remove from it. This ends up being desired by users too - you’re absolutely free to run Slackware 3.1 or at least the command line tools from it, but people don’t do so voluntarily.

                                                                                    But, imagine if command line options didn’t exist - we’d end up with a lot more binaries, and distributions where the number of binaries grows over time. Is that better or worse? It seems like it would be slower (piping means context switches), reading scripts seems of similar complexity since the number of things to know is essentially the same, and it may or may not improve flexibility, depending on the supported combinations of the command line options vs. the composability of separate commands.

                                                                                    It’s hard to read the article without sensing that the real complaint is that software is becoming more capable over time, as opposed to the form of that capability.

                                                                                    (That said, I would like a mini-man that tells me the most common options that’s faster to read, and be able to fall back to the “manual” when something obscure comes up.)

                                                                                    1. 4

                                                                                      (That said, I would like a mini-man that tells me the most common options that’s faster to read, and be able to fall back to the “manual” when something obscure comes up.)

                                                                                      https://tldr.sh/ ?

                                                                                      1. 1
                                                                                      2. 2

                                                                                        Organic accretion is a property of software, and it happens, oddly enough, organically.

                                                                                        I don’t think we’re so far apart.

                                                                                        Let’s take ls. Assume that ls has one and only one output: a series of columns with everything you’d like to know about the filesystem you’re pointing it at. Now there very well may be some command-line options, but they all would be around what the target is: whether to recurse, whether to follow symbolic links, and so on. No command-line options around display.

                                                                                        Would that work? Sure it would. You’d just pipe whatever came out to grep, awk, or whatnot to do the rest of the work that command-line options currently do. But wait! Your point is valid: that’s a lot of piping and context-switching. Heck, you’re tanking performance and you could easily end up with tons of binaries for any simple job. Think of the versioning problem alone. Much easier to add to something that already works (and you know) than use three or four tools you don’t know for the job, increasing coupling and dependencies and making whatever you’re doing much harder to understand later.

                                                                                        But what we’re both complaining about is an implementation detail: how various command-line functions work together and under-the-hood. Could we just smash up long command-line pipes into one executable with one binary, optimizing caching and piping under the hood? Sure we could. We don’t, but such optimizations take place all of the time in programming languages. There’s no reason that they couldn’t take place on the command line, except one: it’s easier to accrete off something you already understand and works than it is to join up several things you might not know so well into something that might be harder for others to grok. If we could smash up and break apart complex command-lines at will, we’d be doing the same thing as adding options without having to modify code or making any one command so bloody complex.

                                                                                    1. 2

                                                                                      Slackware (current), Emacs, AwesomeWM, xterm, Firefox (Developer), Nextcloud, Ruby, Typescript

                                                                                      1. 2

                                                                                        If you can find it, Scientific Forth by JV Noble (same author as the article) is a great book on doing practical work in Forth.

                                                                                          1. 1

                                                                                            Looks like it. I have a nearly new physical copy that I bought from a friend when he retired.

                                                                                        1. 3

                                                                                          The Children of the Sky, ending my re-reading of the Zones of Thoughts trilogy. Highly enjoyable!

                                                                                          1. 37

                                                                                            The “downsides” list is missing a bunch. I mean, I use Makefiles too, probably too much, but they do have some serious downsides, e.g.

                                                                                            • The commands are interpreted first by make, then by $(SHELL), giving some awful escaping at times
                                                                                            • If you need to do things differently on different platforms, or package things for distros, you pretty quickly have to learn autoconf or even automake, which adds greatly to the complexity (or reinvent the wheel and hope you didn’t forget some edge-case with DESTDIR installs or whatever that endless generated configure script is for)
                                                                                            • The only way to safely (e.g. parallelizable) do multiple outputs is by using the GNU pattern match extension, which is extremely limited (rules with multiple inputs to multiple outputs is hard to write without lots of redundancy)
                                                                                            • GNU make 4 has different features from macos (pre-GPL3) make 3.8 has different features from the various BSD makes
                                                                                            • You really have to understand how make works to avoid doing things like possibly_failing_command | sed s/i/n/g > $@ (which will create $@ and trick make into thinking the rule succeeded because sed exited with 0 even though the first command failed). And do all your devs know how to have multiple goals that each depend on a temp dir existing, without breaking -j?

                                                                                            and there’s probably lots more. OTOH, make been very useful to me over the years, I know its quirks, and it’s available on all kinds of systems, so it’s typically the first thing I reach for even though I’d love to have something that solves the above problems as well.

                                                                                            1. 14

                                                                                              Your additional downsides makes it sound like maybe the world needs a modern make. Not a smarter build tool, but one with less 40-year-old-Unix design sensibilities: a nicer, more robust language; a (small!) handful of missing features; and possibly a library of common functionality to limit misimplementations and cut down on the degree to which every nontrivial build is a custom piece of software itself.

                                                                                              1. 7

                                                                                                mk?

                                                                                                1. 3

                                                                                                  i’ve also thought of that! for reference: https://9fans.github.io/plan9port/man/man1/mk.html

                                                                                                2. 11

                                                                                                  I think the same approach as Oil vs. bash is necessary: writing something highly compatible with Make, separating the good parts and bad parts, and fixing the bad parts.

                                                                                                  Most of the “make replacements” I’ve seen make the same mistake: they are better than Make with respect to the author’s “pet peeve”, but worse in all other dimensions. So “real” projects that use GNU Make like the Linux kernel and Debian, Android, etc. can’t migrate to them.

                                                                                                  To really rid ourselves of Make, you have to implement the whole thing and completely subsume it. [1]

                                                                                                  I wrote about Make’s overlap with shell here [2] and some general observations here [3], echoing the grandparent comment – in particular how badly Make’s syntax collides with shell.

                                                                                                  I would like for an expert in GNU Make to help me tackle that problem in Oil. Probably the first thing to do would be to test if real Makefiles like the ones in the Linux kernel can be statically parsed. The answer for shell is YES – real programs can be statically parsed, even though shell does dynamic parsing. But Make does more dynamic parsing than shell.

                                                                                                  If there is a reasonable subset of Make that can be statically parsed, then it can be converted to a nicer language. In particular, you already have the well-tested sh parser in OSH, and parsing Make’s syntax 10x easier that. It’s basically the target line, indentation, and $() substitution. And then some top level constructs like define, if, include, etc.

                                                                                                  One way to start would be with the “parser” in pymake [4]. I hacked on this project a little. There are some good things about it and some bad, but it could be a good place to start. I solved the problem of the Python dependency by bundling the Python interpreter. Although I haven’t solved the problem of speed, there is a plan for that. The idea of writing it in a high-level language is to actually figure out what the language is!

                                                                                                  The equivalent of “spec tests” for Make would be a great help.

                                                                                                  [1] https://lobste.rs/s/ofu5yh/dawn_new_command_line_interface#c_d0wjtb

                                                                                                  [2] http://www.oilshell.org/blog/2016/11/14.html

                                                                                                  [3] http://www.oilshell.org/blog/2017/05/31.html

                                                                                                  [4] https://github.com/mozilla/pymake

                                                                                                  1. 6

                                                                                                    Several more modern make style tools exists - e.g. ninja, tu and redo.

                                                                                                    1. 2

                                                                                                      We need a modern make, not make-style tools. It needs to be mostly compatible so that someone familiar with make can use “modern make” without learning another tool.

                                                                                                      1. 8

                                                                                                        I think anything compatible enough with make to not require learning the new tool would find it very hard to avoid recreating the same problems.

                                                                                                    2. 2

                                                                                                      The world does, but

                                                                                                      s/standards/modern make replacements/g

                                                                                                    3. 5

                                                                                                      Do most of these downsides also apply to the alternatives?

                                                                                                      The cross platform support of grunt and gulp can be quite variable. Grunt and gulp and whatnot have different features. The make world is kinda fragmented, but the “not make” world is pretty fragmented, too.

                                                                                                      My personal experience with javascript ecosystem is nil, but during my foray into ruby I found tons of rakefiles that managed to be linux specific, or Mac specific, or whatever, but definitely not universal.

                                                                                                      1. 5

                                                                                                        I recommend looking at BSD make as its own tool, rather than ‘like gmake but missing this one feature I really wanted’. It does a lot of things people want without an extra layer of confusion (automake).

                                                                                                        Typical bmake-only makefiles rarely include shell script fragments piping output around, instead they will use ${VAR:S/old/new} or match contents with ${VAR:Mmything*}. you can use ‘empty’ (string) or (file) ‘exists’.

                                                                                                        Deduplication is good and good mk fragments exist. here’s an example done with bsd.prog.mk. this one’s from pkgsrc, which is a package manager written primarily in bmake.

                                                                                                        1. 2

                                                                                                          Hey! Original author here :). Thanks a bunch for this feedback. I’m pretty much a Make noob still, so getting this type of feedback from folks with more experience is awesome to have!

                                                                                                          1. 2

                                                                                                            You really have to understand how make works to avoid doing things like possibly_failing_command | sed s/i/n/g > $@ (which will create $@ and trick make into thinking the rule succeeded because sed exited with 0 even though the first command failed).

                                                                                                            Two things you need to add to your Makefile to remedy this situation:

                                                                                                            1. SHELL := bash -o pipefail. Otherwise, the exit status of a shell pipeline is the exit status of the last element of the pipeline, not the exit status of the first element that failed. ksh would work here too, but the default shell for make, /bin/sh, won’t cut it – it lacks pipefail.
                                                                                                            2. .DELETE_ON_ERROR:. This is a GNU Make extension that causes failed targets to be deleted. I agree with @andyc that this behavior should be the default. It’s surprising that it isn’t.

                                                                                                            Finally, for total safety you’d want make to write to .$@.$randomness.tmp and use an atomic rename if the rule succeeded, but afaik there’s no support in make for that.

                                                                                                            So yes, “you really have to understand how make works [to avoid very problematic behavior]” is an accurate assessment of the state of the things.

                                                                                                            1. 1

                                                                                                              Your temp directories dependency problem makes me think a GUI to create and drag drop your rules around could be useful. It could have “branching” and “merging” steps that indicate parallelism and joining too.