1. 9

    I’ve taken to describing this class of things as “Stuff so important that nobody ever talks about it.” As in, the assumptions that are deeply embedded but seem totally basic to an insider.

    But yeah, I get it. There are so many arenas where I could talk for days, or we could just sit down and work together for a few hours.

    1. 6
    1. 2

      Another reason the scaling costs are so cheap is that we basically have a monoculture for phones. Two operating systems on 2B devices makes the economics pretty favorable for the black hats.

      1. 1

        Maybe we need to look beyond HTTP as the protocol-for-everything. There are messaging protocols that are suited to spanning different organizations.

        1. 3

          Good read. I agree with most of it, but found the proposed solution a bit wierd. Maybe I misundersted, but isn’t long pooling a problem for setups like Django for example, where you have a limited number of worker processes?

          1. 5

            They’re hard to maintain in general. A lot of web technology assumes you have short-lived requests. Not to say it isn’t possible or even increasingly common, but it’s a moderate investment.

            I agree with the thrust of the article (webhooks are insufficient) but disagree with the conclusion that only polling or only listening to webhooks is the way. Poll, but also trigger a sync when you receive a webhook. It’s easy to reason about and doesn’t require much in the way of special infrastructure.

            1. 4

              it’s easy to reason about and doesn’t require much in the way of special infrastructure.

              Unless you’re a firewalled client. Then it requires special infrastructure, DNS, etc.

              I think the cost of long-polling is overestimated by a lot of people. I have systems with over a million open connections each: each handle costs only about 2kb. Maybe long-polling is hard in some frameworks, so perhaps it’s worth adding some middleware?

              1. 1

                Long polling, SSE, and web sockets all have one problem in common: the endpoint a client polls needs to know about new events. With plain polling the endpoints simply read from the database when they get a request.

                In your long polling setup, how do your client facing endpoints get new events? Do they themselves poll, or have events pushed to them, or something else?

                1. 2

                  the endpoints simply read from the database when they get a request.

                  This just kicks the can: How did the database know about the change? Someone did an INSERT statement or an UPDATE statement. If you’re using Postgres they could have additionally done a NOTIFY (or a TRIGGER could have been created to do this as well automatically).

                  A decade ago, when I used MySQL, I had a process read the replication log and distribute interesting events to fifos that a PHP client would be consuming.

                  In your long polling setup, how do your client facing endpoints get new events? Do they themselves poll, or have events pushed to them, or something else?

                  They have events pushed to them: On client-connection, a subscription is made to receive updates, and on disconnection the endpoint unsubscribes. If you’re using Postgres this is just LISTEN. If you’re using erlang, you just use rpc+disk_log (or whatever). If you’re using q, you -11! the client log and hopen the realtime. In PHP, I’ve had a bank of fifos in /clients that we just read the events from. And so on.

            2. 2

              isn’t long pooling a problem for setups like Django for example

              You have X customers, so you simply need to handle X connections. If you allow less than that you have queueing which may be fine for a while, but really you need to be able to handle that anyway: otherwise your customers get errors and one customer can (trivially) deny another.

              If X is larger than fits on one machine, you might want multiple machines anyway. Your load-balancer can route long-polls by customer and disconnect old pollers. Regular polling is a bit harder to capacity-plan.

              1. 1

                Long-polling also shifts cost from the client to the provider. The provider has to have sockets open, RAM allocated, and potentially a thread per request.

                There are ways to implement that don’t consume a thread per connection. These usually involve async I/O which is tricky in vanilla Java, C#, and the like. Actor based frameworks and languages make it easier, but the RAM and socket costs are still there.

                1. 1

                  Long-polling also shifts cost from the client to the provider. The provider has to have sockets open, RAM allocated, and potentially a thread per request.

                  There are ways to implement that don’t consume a thread per connection. These usually involve async I/O which is tricky in vanilla Java, C#, and the like. Actor based frameworks and languages make it easier, but the RAM and socket costs are still there.

                1. 3

                  I want a version of jq that’s… worse. I have been using it for what feels like a decade and the query language is just as mystifying now as it was on day one. I want something with a far simpler and less expressive DSL, which can maybe only do a single transformation per execution, so that I pipe using the shell rather than in an opaque string. And which only does JSON stuff, not anything I can do already with grep. Does that exist?

                  1. 2

                    I haven’t used it yet, but I bookmarked dasel for that purpose.

                    1. 2

                      maybe grom, especially now that it’s written in Go, instead of PHP? https://github.com/tomnomnom/gron

                      1. 1

                        Wow, this is more or less exactly what I was hoping to find. Thank you!

                        1. 1

                          I’ve been using it recently as well, it works quite well for what I want

                    1. 5

                      This is a good post, but it’s somewhat missing in clarity of thought – I suspect because it tries to cast failure analysis in a framework of counterfactuals and causality.

                      I prefer to instead view failure as a system that for some reason went outside the constraints placed on it. It can do this for primarily three reasons: a sub-system went out of constraints (disk full), a control mechanism was insufficient (disk space freed too slowly) or a control mechanism was missing in the first place (disk space not automatically freed.) Of course, the reasons for control mechanism failure is often failure of control mechanisms higher up in the system hierarchy.

                      I’m not yet very good at describing this view, which is known as the System theoretic accident model. It also emphasises how we shouldn’t stop at a fixable cause, but rather explore the causal links as far as economy allows us, because the more leveraged fixes tend to be deeper. (As an example in my company, a simple hardware failure led us to realise we were missing an entire department in our organisation, that we would need to prevent failures like this one – and many, many others – in the future.)

                      I strongly recommend the CAST handbook, which is available as a free PDF and was my intro to this way of looking at things. It presents many of the same things as this article, except in my opinion with a much more clear conceptual framework. It also goes much deeper.

                      1. 3

                        Hey thanks for that. Next time I’ll be sure to write a full-length monograph so I can hit every aspect of a topic.

                        (OK, I’ll set aside the sarcasm. For now.)

                        Look, when writing a blog, one always has to pick one thing to discuss in one post. Safety and system failure is a vast topic. Most readers, and from what I’ve seen most people in our industry, don’t know about any of it. Instead, they are shown examples of behavior in their companies. The typical company today still believes in human error (e.g. HBO Max throwing an intern under the bus last week), root cause analysis, and situational awareness. Absent external influences, people duplicate the practices they see.

                        My post was written to address a single, very specific pattern of behavior I see as unhelpful.

                      1. 4

                        As a very general take on the relationship between counterfactual reasoning and causality (the linked article is specifically about diagnosing causes of tech failures), I found the 1999 psychology paper “When Possibility Informs Reality: Counterfactual Thinking as a Cue to Causality” interesting.

                        Link is a PDF, but here’s the abstract:

                        People often engage in counterfactual thinking, that is, imagining alternatives to the real world and mentally playing out the consequences. Yet the counterfactuals people tend to imagine are a small subset of those that could possibly be imagined. There is some debate as to the relation between counterfactual thinking and causal beliefs. Some researchers argue that counterfactual thinking is the key to causal judgments; current research suggests, however, that the relation is rather complex. When people think about counterfactuals, they focus on ways to prevent bad or uncommon outcomes; when people think about causes, they focus on things that covary with outcomes. Counterfactual thinking may affect causality judgments by changing beliefs about the probabilities of possible alternatives to what actually happened, thereby changing beliefs as to whether a cause and effect actually covary. The way in which counterfactual thinking affects causal attributions may have practical consequences for mental health and the legal system.

                        1. 3

                          Looks interesting, thank you!

                          My thinking was influenced heavily by Douglas Hofstadter, who had some similar arguments in one of his books.

                        1. 2

                          If I read this correctly, it means that doc comments become actual values rather than just being discarded by the lexer/parser. Cool!

                          One of the tricks I like in Clojure is that docstrings are values in the metadata map for a var. That means you can use ordinary Clojure code to attach docs to a var, even outside the syntactic form that declares the var. When is this helpful? For a few libraries, it lets one provide very long docstrings (e.g., including examples) without unduly breaking up the source code.

                          That ability to manipulate the docs as values opens up some interesting abilities.

                          1. 1

                            The idea of identifying code by structure globally is fascinating but how does this accommodate abstraction? The hallmark of abstraction is an interface that can be implemented in different ways (and trade-offs). This does not seem possible (or encouraged) here.

                            1. 3

                              I think Unison has four ways to support polymorphism. (Though this is my novice-level interpretation so I might not get all the nuances.)

                              First and most visible, Unison is an FP language with functions as first class values. So you can always pass in a function (as long as the function’s type is acceptable) that contains a specific variation of behavior.

                              Second, functions can be polymorphic in their types.

                              Third, Unison has an effects system (called “abilities”) where the effect definition is an abstract interface and specific implements can vary. Abilities occupy a special position in the syntax and semantics though, so not everything can be abstracted this way.

                              Fourth, there is work going on now about how to incorporate typeclasses.

                              So it might be that there is a surplus of abstraction techniques!

                              1. 1

                                As I recall, it’s approach to abstraction is similar to Haskell’s; type-wise, if function a has the same types as function b, then the functions are treated the same to the rest of the code using them. Haskell seems to do ok with abstraction, but maybe I’m misinterpreting you.

                                1. 1

                                  Indeed, I see definite similarity.

                                  there are no “open source projects” - only “the open source Key-Value database of all functions”

                                  If you look at the github URL + function hash as a composite key, then we almost have that. (Though such a composite key still allows redundancy where the same function appears in multiple github projects.)

                                1. 6

                                  Note: I first submitted Tails a month ago when it was just a little gist. I’ve kept working on it, and it now has its own repo. I’ve added a parser, a stack-effect checker, more data types (strings, arrays, quotations), type-checking, and a dumb but effective garbage collector. There’s even a REPL so you can try it out interactively.

                                  1. 1

                                    Is there a particular compiler I should use? GCC is throwing a lot of errors.

                                    1. 1

                                      I’ve pushed a fix for the GCC compatibility issues, which were pretty trivial. Thanks for pointing them out.

                                      1. 1

                                        I build with Clang. I should install GCC and try it. The most likely cause of build errors is missing standard library includes…

                                    1. 5

                                      Looks really nice, clicks the right buttons… But sadly I need a split keyboard for health reasons ;-(

                                      1. 11

                                        May be someone could make a fork :D

                                        1. 3

                                          I highly recommend gboards.ca for split keyboards, but I’ve also heard people say good things about the Moonlander (from the ErgoDox EZ folks).

                                          1. 3

                                            I’m one of those Moonlander boosters. Using it right now!

                                            But I do have a caveat: I had to relearn how to type on the ortholinear layout. My typing speed plummeted at first. I had never learned touch typing and was apparently “crossing over” for a bunch of keys like “y” and “6”. Now I’m faster than I was before and back into to realm of “think about the words” rather than “think about the fingers”.

                                            1. 1

                                              The moonlander is easily my favorite keyboard. I just bought a second one for my work desk.

                                              1. 1

                                                Would love to see a direct comparison between the moonlander and Ergodox EZ. I have a (self built) Ergodox and I’m a big fan. But I wish I could have one that is just a tad smaller with keys being a bit closer to each other.

                                                1. 2

                                                  I saw this comparison video on YouTube (15 minutes, but indexed so you can skip to parts you are interested in) a few days ago and found it helpful. A friend of mine got the Moonlander and I have an Ergodox, so I hope to compare them for myself in the near future. I have loved my Ergodox and actually don’t find that the thumb clusters cause any trouble for me, so I doubt I will get a Moonlander anytime soon. I do envy the foldable wrist rests though. The ones for the Ergodox could pull double-duty as wheel chocks for a passenger plane and tend to wander away through a day of use.

                                            1. 4

                                              Edited: I judged too harshly. Retracted my previous criticism.

                                              1. 6

                                                It does look like an ad for teleport, but I would not reduce it to “just”.

                                                1. 4

                                                  how so? i found it a very good general discussion of the ux issues around exposing a terminal in a web browser.

                                                  1. 3

                                                    On the contrary, I was just about to comment that this post is a really good example showing courage talking about the internal decisions made in a company on a product.

                                                  1. 20

                                                    Man… I’ve reached the point where techniques I promoted are now historical questions.

                                                    1. 1

                                                      Man, I felt old because I remember doing this 20 years ago, and the answer was obvious to me :)

                                                    1. 2

                                                      That was a memorable game. Had it as a kid. There was something deeply disturbing about the packaging… the idea of the serene white mask covering over a terrified face really got into my head.

                                                      The other idea that got into my head was “different people see the same thing different ways.” Not a bad lesson for a 10 year old.

                                                      1. 1

                                                        The project homepage link on the author’s page is no longer current. The new project home is https://fsprojects.github.io/FSharp.Data/

                                                        1. 2

                                                          Seems like the author cherry picked a few examples that didn’t pan out, but I don’t think the generalization works. IME, “developer tools” have improved tremendously in the last twenty years that I’ve been working as a developer. Even the languages themselves have been moving in directions that make dev lives easier.

                                                          The biggest problem, IMO, is that by the time a language and its ecosystem has been around long enough to develop mature tooling, the trend followers have moved on to the next new thing. C++, Java, Lisp, and even Python have a ton of dev tools. The newest, most bleeding edge Javascript dialect probably not so much.

                                                          1. 5

                                                            Hmm, I’m not sure I agree.

                                                            The author picked tools from one lineage (Java and the community of researchers who attempted to improve it).

                                                            You could look at tools from another lineage (e.g. Smalltalk, Lisp) and note that big, useful ideas (e.g. the Smalltalk browser, Gemstone version control system, Common Lisp condition/restart exception handling, Interface Builder) tend to go by the wayside or emerge into popular practice hamstrung in crucial ways (Interface Builder’s interactivity subjugated to the edit-compile-run cycle).

                                                            I’m more familiar with the Lisp stuff than the Smalltalk stuff so I’ve provided links for reference for those.

                                                            1. 4

                                                              What is the best reference to read about the Gemstone system you mention?

                                                              1. 2

                                                                I don’t have a reference handy, I’m sorry :(

                                                                I learned about it from someone I met at bangbangcon a few years ago - they worked (many years ago) in finance on an all-Smalltalk system and we talked about it at length over lunch.

                                                                1. 2

                                                                  I used ENVY for version control, but only knew of Gemstone as an object database. I think they later morphed into a distributed object system with persistence ability.

                                                                  1. 2

                                                                    NP. I will dig around. Smalltalk inspired quite a few neat projects.

                                                                    1. 1

                                                                      I likely conflated Gemstone with ENVY, see mtnygard’s sibling comment!

                                                                      I also found this collection of descriptions of ENVY and how people used it - from this description it doesn’t seem obvious to me what specific technical feature was awesome about it.

                                                                      Several of those interviewed talk about method-at-a-time versioning reducing conflicts and the subjective experience of the “liveness” of the whole codebase.

                                                                      https://paulhammant.com/2017/09/01/smalltalk-envy/

                                                              2. 1

                                                                C++, Java, Lisp, and even Python have a ton of dev tools. The newest, most bleeding edge Javascript dialect probably not so much.

                                                                Let’s not ignore the advances that have been made relative to those languages, with regard to community, debugging, legibility, safety, ergonomics… etc. I admit, many people’s function for what to use is basically use?(thing, threshold) = (thing.hotness > threshold). But there are some legitimate reasons to use newer languages.

                                                                If using your newfangled tool required using a language whose inception took place at least 25 years ago, wouldn’t you consider that a problem for adoption? Would the kind of person to use such a well-established tool really be interested in trying out your new, shiny invention?

                                                              1. 5

                                                                Everything on Twitter is performative, no?

                                                                Speaking personally, I don’t have time to try every language and every tech stack. I do learn from the experiences of others. I’ve seen some good experience reports here on Lobste.rs over the years. That means I do care what languages people use, at least in two ways. First is the “huh, never heard of that before” moment. Second is the comparison between “thing I know” and “thing I don’t know”. This recent article by Kevin Lynagh hit both of those sweet spots for me.

                                                                There’s a world of difference between an experience report and cheerleading, of course. Twitter is best suited to cheerleading or mudslinging.

                                                                1. 18

                                                                  Fun article! I think a lot of people missed the very last paragraph though, where the author says:

                                                                  Collectively, the software industry simply has no idea how to hire software developers. Factorio is probably the best technical interview we have right now, and that’s embarassing.

                                                                  1. 6

                                                                    We certainly can’t switch to using Factorio as an interviewing method - you might as well just give a candidate a take-home assignment.

                                                                    … and everyone is discussing how they’d react to Factorio used as an interviewing method. 😔

                                                                    1. 3

                                                                      I can’t blame people for noping out before the author got to the point, though.

                                                                    2. 2

                                                                      Collectively, the software industry simply has no idea how to hire software developers.

                                                                      I’ve been exposed to bad interviews and good ones. I don’t think the author sufficiently justifies this point.

                                                                      Factorio is probably the best technical interview we have right now

                                                                      Again, insufficient justification is given for this extraordinary claim. The whole article hand waves about how there’s similarities between code and playing Factorio, like for example you have to fix bugs in Factorio (“woa bugs, we have those in code too! :o”), but the article’s conclusion does not follow from its body.

                                                                      1. 1

                                                                        If Factorio were multiplayer, and the other players just randomly rewrote the rules of the game for no well-explained reason, it would be more appropriate in evaluating how good a person was at software development.

                                                                        1. 2

                                                                          It is still too narrow of an environment to properly model the complexity that you can enconter while developing software. For example, the bugs that arise in Factorio arise from, essentially, a mismatch in the supply and capacity of the flow of materials. Its “complexity” arises from simply adding more of these pipelines. Software is usually not so simple.

                                                                    1. 9

                                                                      I use a ZSA Moonlander, which I absolutely love, and I doubt I’ll ever want to use another different keyboard again. I just filled out your survey!

                                                                      1. 4

                                                                        I also recently got a Moonlander. I’m not back at my full speed yet but I’ve only had it for a full week. So far I really enjoy it.

                                                                        1. 3

                                                                          I got a Moonlander last year just before Christmas. That way I could use the break to relearn how to type. I learned proper touch typing for the first time in my life. Found out that I was crossing over for the ‘y’ and ‘6’ keys all the time. It’s been about two and a half months now and I feel like I’m back up to speed.

                                                                          Recovery was partly spending 30 minutes a day on typing sites, and partly a matter of changing the layout to avoid my most common errors. (When trying to use my left index finger to hit the phantom ‘y’ key on the left side, I was hitting the “layer 2” modifier. Remapped those centerline keys to ‘(’, ‘{’, and ‘[’, which helped a lot.)

                                                                          1. 2

                                                                            I’m having exactly the same issue right now. Thanks for the suggestion.

                                                                        2. 3

                                                                          The Moonlander looks really good to type on! when I’m back in work I think I’ll invest in one!

                                                                          Thanks for doing the survey.

                                                                          1. 3

                                                                            I can’t recommend it enough. You’re clearly experienced with split/ergonomic keyboards, and I can’t speak to others as this is the only one I’ve used, but there isn’t a single thing about this keyboard I wish I could change, and I’ve been using it for a few months now.

                                                                            1. 2

                                                                              Thanks for the recommend, typing on the ErgoDox that I built has been a pleasure, and the Moonlander looks like a great extension of that.