1. 1

    Wow, this (simple) data pipeline architecture is almost exactly what I was looking for 5 years ago for something at work. I ended up writing something very similar to this so we could wire up disparate APIs to talk to each other and synchronize data across different providers, platforms & environments.

    I’d love to see some in-depth, real-world (read: gnarly) setups that use this; I’m curious how it handles integrating all of these features into a functioning system.

    1. 2

      It’s a little old now but there’s a blog from Meltwater loosely outlining how they use it for enrichments: https://underthehood.meltwater.com/blog/2019/08/26/enriching-450m-docs-daily-with-a-boring-stream-processor/, the input/outputs are fairly basic as it’s consuming from Kafka to Kafka, but in between it has to perform a network of HTTP enrichments each with a unique API and format for batching messages (JSONL, JSON array, individual parallel requests, etc).

      The more modern way of doing it is outlined in a simpler example here: https://www.benthos.dev/cookbooks/enrichments/.

    1. 1

      This Clipboard / Drag&Drop inspector mentioned in the article looks really helpful as well: https://evercoder.github.io/clipboard-inspector/

      1. 2

        Great article! I was reminded of the “1-Bit Symphony” by Tristan Perich (2009). The physical version of the piece is a circuit built into a standard, clear plastic CD jewel case. Summary of the works with photos here: https://marijebaalman.eu/dafx/pages/1bitMusic.html

        1. 4

          Yes! I was given “1-Bit Symphony” as a gift about 10 years ago and blown away. I honestly was so mesmerized by the idea that polyphony could be achieved on a single pin. I’ve gone down a bit of a rabbit hole the last week after reading this article and have found that there is quite a bit more resources on this topic since the last time I attempted to look into it.

          Dr. Blake Troise, (aka PROTODOME), the author of the the article, has an album on a chip as well. You can actually view the source code for the album here

          Anyway, I’m glad you found some joy in the article as well :).

          1. 1

            Awesome walk-through of iterating towards a really neat ray-traced shader animation!

            And here’s Part 2: https://wallisc.github.io/rendering/2020/05/02/Volumetric-Rendering-Part-2.html

            1. 1

              Ha… ha… 😛

              Should be tagged satire

              1. 3

                Let’s use a small device like PC Engines APU2 (4 core AMD GX-412TC SOC 1Ghz, 4GB of RAM). On this device, how many jails executing a real process can we start?



                1. 1

                  With 4 GB RAM 1000 FreeBSD Jails is possible:


                  1. 1

                    Yes, from the article:

                    The current system limit of this test is due to 4GB of RAM consumed by all the bird processes.

                    Where bird is the application which is run inside the jails for the test. See https://bird.network.cz/

                1. 1

                  Heading to Strange Loop with my 10yo son!

                  We’re both super excited to do City Museum and see some awesome presentations at the conference.

                  1. 2

                    At least one of the techniques described allows expanding the amount of time the compiled binary takes to perform tasks, up to some defined usability threshold, so as to make branch exploration time-intensive. Which smells like Proof-of-Work in so far as “Let’s solve this problem by burning more CPU cycles and putting more carbon in the atmosphere”

                    I’d be scared to learn what kind of carbon footprint a technique like this would have if implemented in widely-used software.

                    1. 1

                      Two really important parts from this:

                      The software product industry (including mobile phone makers) has reaped excess profits for decades by selling risky products and offloading the risk onto their clients and society. My analogy is that they constructed financial products that yield a certain amount of excess return but blow up disastrously under certain geopolitical events, then sold some of the excess return and all of the risk to a third party that is not informed of the risk.

                      What is the right solution to such a conundrum? Lobbying, and a concerted PR effort to deflect the blame. Security researchers, 0-day vendors, and people that happen to sell tools that could be useful to 0-day vendors are much more convenient targets than admitting: All this risk that is surfaced by security research and 0-day vendors is originally created for excess profit by the tech industry.

                      1. 4

                        Ahhhh, I love this guy! He wrote and has hosted one of my favorite web-based “games” for the last … 10 years? The eyeballing game 🥰

                        1. 17

                          Database schema migration management system 🙃

                          1. 2

                            Hm, I actually built one at my current employer’s, and am quite happy with the result… umm, but then, I suppose, this one is thus the very definition of IH - and I should probably not be so sure my collegues are 100% happy with it, and wouldn’t want to reinvent it… uh, oh :/

                          1. 28

                            This is banal for PL nerds, but: sum types and pattern matching. Maybe it’s not niche, but if you look at the top 5 most popular programming languages—perhaps even top 10—then I don’t think any of them support these features properly. (No, template hacks in C++ don’t count.) They are easily one of the most important features that impact how I write code in languages with and without them. I find that with them, there’s a lot less friction between communicating the proper intent I had when writing code than without them.

                            Here’s an interesting question: how useful are sum types without generics? Without generics, you couldn’t for example have a generic Maybe/Option type, but you could still use sums for more specific circumstances. Are there any languages with sum types that don’t support generics? (Do tagged unions in C count?)

                            1. 3

                              I think one way to have useful sum types without generics would be to use dynamic typing. then you could have variants with associated data and even pattern match on them but the type would just be that of the variant. (pyret and pure have sum types in a dynamic language, though I’ve not really used either. it’s a rare combination of features.)

                              1. 13

                                I think the killer feature of sum types is statically checked exhaustive pattern matching. Without that, …

                                1. 2

                                  Checked exhaustive pattern matching is one of the things I’ve really been enjoying about Swift, warm fuzzies for sure.

                                2. 6

                                  If you’re not doing static typing, then you might as well fake it with tuples and atoms. Like Erlang does it.

                                  1. 14

                                    Tuples and atoms are also things more languages should have. :D

                                  2. 3

                                    Interesting. I guess I would have thought that sum types wouldn’t really be a thing in a dynamic language, at least not explicitly. Although, I’ve used them with Flow in Javascript, so there’s certainly some kind of middle ground there.

                                    Do pyret and pure have exhaustiveness checks? I believe Flow does.

                                    1. 2

                                      pyret does run-time exhaustiveness checking.

                                      for pure, i would guess not since (a) pattern matching is pervasive in the language and sum types aren’t really emphasised, and (b) pure seems to think of types in a very different way from most languages (“In Pure the definition of a type takes a somewhat unusual form, since it is not a static declaration of the structure of the type’s members, but rather an arbitrary predicate which determines through a runtime check which terms belong to the type.”)

                                    2. 3

                                      Yeah I noticed that Pyret has sum types, but it’s dynamic. (Never heard of Pure, thanks for the reference.)

                                      I have been using Zephyr ASDL in Oil, which is meant to give you sum types in languages like C++ and Java. You can fake it pretty well with the typing rules for inheritance (that was sort of surprising to me.).


                                      I’m using it in Python, which means there is no static type checking. But I still found it very useful, so I was thinking of putting that in the Oil language, which is dynamic.

                                      Although the latest development is that I used MyPy to statically type the OSH front end (9K lines of hand-written code and 7K lines of generated code), and I really liked it. I am now a fan of gradual typing. Between TypeScript, MyPy, and Hack, it seems like gradual typing is “here”.

                                      There is also a (dormant) shell-like language with tagwords:


                                      That is basically the dynamic version of sum types … although I now wonder if gradual typing makes sense in this context.

                                      1. 2

                                        if you are interested in gradual typing, it’s worth taking a look at stanza, which has it designed-in from the beginning.

                                      2. 2

                                        There exists a slightly-acceptable embedding of sum types into TypeScript. This is kinda neat for me because TypeScript is interoperable with a widely used dynamically typed language. You put different types into the branches of a union type. You can check switches on it for exhaustiveness by assigning the switched-upon variable to a variable with type never, which is the uninhabitable type.

                                        I’m 95% sure it’s possible to write this embedding in a way that has nicer ergonomics than what I’ve just written here. This is cribbed from memory of writing Redux reducers in TypeScript.

                                        Example of an imperfect embedding of Either<A, B> into TS

                                        enum LR {
                                            L = 'L',
                                            R = 'R',
                                        type Left<L> = { direction: LR.L, left: L };
                                        type Right<R> = { direction: LR.R, right: R }; 
                                        type Either<L, R> = Left<L> | Right<R>;
                                        // note: `LR.L as LR.L` sucks. I assume a better way exists but IDK.
                                        // I think you might be able to do it without needing the `direction` enum at all but I'm not sure.
                                        const aFailure = { direction: LR.L as LR.L, left: -1 };
                                        const aSuccess = { direction: LR.R as LR.R, right: "Yay it worked" };
                                        const aResult: Either<number, string> = Math.random() >= 0.5 ? aFailure : aSuccess;
                                        function interpretResult(result: Either<number, string>) {
                                            switch (result.direction) {
                                                case LR.L: {
                                                    // (This explicit variable isn't actually necessary, I just have it here to demonstrate that the compiler really does know for dead certain that, on this side of this branch, `result` is a `Left<number>`
                                                    // like how a nice compiler will know what all the types are when it gets to the right hand side of a `match` statement
                                                    const itIsKnownToBeLeft: Left<number> = result;
                                                    console.log(`Oh no! Error code ${String(itIsKnownToBeLeft.left)}`);
                                                case LR.R: {
                                                    // (Same here)
                                                    const itIsKnownToBeRight: Right<string> = result;
                                                default: {
                                                    // Note: if you add something like e.g. `| string` on the end of the definition of Either, you get a type error here because string is not compatible with the `never` type.
                                                    const shouldNotGetHere: never = result;
                                                    console.log("Someone else broke type safety. Oh no.", shouldNotGetHere);
                                    1. 3

                                      I love these guidelines; good stuff :)

                                      What opinions do people have about flags/conventions to control the level/quantity/nature of output and/or messaging?

                                      I haven’t really come to a personal standard, but many of my command-line interfaces implement one or more of the following:

                                      -v | --verbose ...... May be specified multiple times.Increase the
                                                            verbosity of output or messaging?
                                      -d | --debug ........ What kind of diagnostic messaging should be
                                                            displayed? Should this be handled instead by
                                                            just some repetition of "-v"?
                                      -j | --json ......... Make all output machine-readable via JSON
                                      -q | --quiet ........ Should this restrict output or messaging to
                                                            only warning or error-related information?
                                      -s | --silent ....... Same as "-q" ?
                                        1. 3

                                          At work: Continuing development of my file-syncing tool with Rust 🙌

                                          At home: Last week I had a sudden inspiration for a simple game, which would be a fun iPhone/Android app… and since then I’ve got a working prototype that my kids really enjoy playing and have been helping me to refine. Sooo… maybe launching my first phone app game soon? I’m 99.9999% sure if I do I’ll just make it $0.99 and have no ads whatsoever, maybe with a free version that just has a smaller number of levels. I haaaaaate advertising.

                                          And my favorite part about the game is that as a puzzle game, it’s basically teaching the user reverse-engineering.

                                          1. 3

                                            If you release a free version with a small number of levels, maybe think of way to skip those levels or track the progress of the user once he pay for the real version. It might be boring to re-do puzzles just to get back at where you were at.

                                            Edit: It’s really nice that have a nice prototype that your kids seems to enjoy :)! Was it hard to learn how to do the app? Where you familiar with making games before?

                                            1. 2

                                              That’s a good idea about skipping levels!

                                              I’ve never professionally made a game before, and never made a mobile app… so it’ll be interesting for sure. I can easily see falling down the rabbit hole of features like: achievements, preferences, tracking user metrics & engagement, etc.

                                              One of the features I think would be killer is auto-generated levels, which really raises the importance of testing & reliability so that for example users don’t get stuck.

                                          1. 1

                                            Dang it, I just saw the author’s comment that he’s not necessarily ready to share this w/the public yet. Can we delete this please?

                                            1. 2

                                              It’s still a nice summary of the starting point with helpful links. People can build on this. It might also make them think or inspire action. So, it still has value here.

                                              1. 1

                                                It’s fine, I’ve just reacted to that one comment, and added Double buffering notes this week.

                                              1. 2

                                                At work: Continuing infrastructure improvements & automation.

                                                At home: I started a side-project last week that I’m super excited about. It’s a file-synchronization thing written in Rust based on the notify crate. The idea is to watch a local directory in real-time and sync file updates to a remote server, which can then fan those updates out to other servers near it. I feel like I’ve gotten 90% of the fiddly bits working in my prototype. Hopefully this week I’ll have the skeleton working and then I’ll upload it somewhere public.

                                                1. 1

                                                  Given the length of the article I was surprised to see Ember.js missing, even though golden-oldies like Prototype and Backbone.js were.

                                                    1. 2

                                                      I just picked this up as well while on vacation.

                                                      I’ve been enjoying working through Project Euler doing the solutions in Rust.

                                                      Creating actual working code with Rust has been a lot of fun!

                                                    1. 4

                                                      Does anyone know if there’s been any more recent progress on getting FreeBSD support (back) into Nix?

                                                      I’d loooooove to try out Nix especially for stuff at work but not supporting FreeBSD is currently a deal-breaker.

                                                      1. 3

                                                        On a related note, is there any support planned for OpenBSD?