Threads for friendlysock


    Have a codebase at work where somebody did this prior to the current team joining. So, of course, we now have both a massive sql ball and a bunch of migrations.

    If you want to make future maintainers’ lives really annoying for code discovery and refactors, by all means do this.

    I argue that this is an Ecto antipattern.


      At work we have a CI pipeline that checks out the committed SQL file from the “base branch” of a merge request and applies the migrations from the current branch, then dumps the schema to another SQL file. It then diffs the dumped file to ensure the schema matches the previous schema plus migrations.

      Another approach could be to simply generate the SQL file in a commit hook.

      Having an SQL file makes dumping and loading much faster, and if you have a large test suite, resetting the db to the schema’s version is faster too. Although nowadays we no longer do that, but work with a template db that gets restored at test start, that’s even faster.

      An SQL file is also a lot easier to search through when discovering the schema or quickly grepping for existing indexes, triggers and such. Sure, you can also connect to the db and inspect it that way, but navigating an a schema file in a good editor can be more efficient than using DB tools depending on what you’re doing.

      In some cases like the ORM-based migrations (let’s say Django), you might have to squash your migrations, if at some point you used a library that provided custom column types but that are no longer used. If you didn’t squash your migrations, you’d need to keep that library around as a dependency “forever”.

      In the past I’ve also seen Rails projects where people loaded the current version of the models inside Ruby migration scripts in order to make complex data migrations easier. Of course, this breaks as soon as the model is changed in any way that the migrations rely on. In those cases, using and committing a DB dump is also probably the only way to go. From that experience I’ve really come to appreciate how Django instantiates a pseudo-model that only has the in-memory representation of the model at the current point in the migration history, so you can still write idiomatic ORM code to do complex data migrations.

      In your specific case, if you really dislike the situation you could probably just drop the SQL file and always apply the migrations from scratch. Or mark it as an ignored file for your editor so it doesn’t show up when grepping.

    1. 31

      Friends and I have been playing with ChatGPT for the last couple of days, and a few things come to mind:

      • It has a decently good “understanding” of Elixir, perhaps on par with an intern or junior who can paraphrase all the blog posts they’ve recalled.
      • It will generate code with the right “shape” for some things, for example nix flakes. The details though it gets wrong.
      • If you start playing games with it to try and circumvent blocks, you can usually do so pretty easily.
      • It is capable of turning at least high-level English descriptions of “I want an app with thus-and-such data model, give me the boilerplate codegen to run” into usable steps. It isn’t capable of spotting the “smart” or “experienced” way of doing things _unless those are common knowledge on Stack Overflow or some equivalent in its training corpus.
      • It does okay at spotting and explaining some classes of code bugs, or parroting reasonable design considerations.
      • It is extremely easy to “taint” the results. I had it pretending to give a lecture on a 40-opcode VM machine for running GPT, and because it was a long session and required multiple “and then you continue” prompts from me, I had the chance to see it react in its replies to subtle suggestions in the “continue” prompts. One prompt, for example, said something like “and then you get excited for the next part talking about things that improve performance for the VM implementation” and like 3/5 of the next opcodes were “CACHE” or something like it.
      • It will give bogus information (say, asking it to generate certs or perform certain ciphers).

      My friends and my takeaways about it were that it could probably be an okay rubber ducking tool and a neat way of seeding a “I’m curious about X, but I don’t know who to start asking questions to” session. It is also extremely easy to gaslight yourself with it, because the human brain will pattern-match and anthropomorphize all the things. It almost caught me up once or twice when trying to get it to divulge information about its host–you get caught up in the story it’s weaving, and unless you keep a strong amount of skepticism you will fool yourself into thinking it’s doing something it isn’t.

      The singularity isn’t here yet, our jobs are safe for another decade, but we sure do now have a neat interactive assistant for searching Stack Overflow–and that’s really the other thing I noticed, is that I think most of the impressive results this thing gives are more a function of the sheer quantity of shit we as programmers have memoized via sharing than any sort of actual reasoning.

      Have fun with it, but be careful.

      1. 17

        I think one of the important take-home lessons from ChatGPT is that a lot of programming tasks require far more repetitive boiler plate than they should. The system can easily solve them because it has large numbers of examples with tiny differences to look at.

        1. 2

          Lisp Master Race rises again. Who cares about macros, all I care about is minimizing the stupid boilerplate.

          How well does ChatGPT do with a lisp?

        2. 4

          Thanks for mentioning that it’s only quite good at spotting some bugs. Based on what I have been seeing on Twitter and Discord, I have been assuming that it is very good at spotting bugs. But if it only spots some bugs, it’s not going to be nearly as good as a proofreading (or, I guess, static analysis?) tool, right?

          1. 8

            That’s the thing that’s annoying about it. It does such a good job for a lot of things that it lulls you into a false sense of security when you still need to be double-checking its work. IIRC it missed a format injection attack but caught the general bad usage of C for strings.

            I think it could be a force-multiplier to replace junior-level boilerplate generation in tasks where writing the code is slower than verifying/eyeballing it; my caution is that for sensitive stuff you still have to do the work yourself.

          2. 3

            Hm, my thoughts exactly. I had it generate docker files and k8s manifests. It’s good as a generic scaffolding tool but that’s about it.

            1. 2

              I asked it to generate valid boards for Sokoban, and then solve them. It did a decent job of generating boards, with valid goals. It gave a list of valid moves, but did not solve, unfortunately. :)

              It was able to get pretty close to writing a FizzBuzz implementation for a bytecode VM it wrote, even after I told it to use indirect threading instead of a switch statement. Was too late to ask about a garbage collector for the system.

            1. 15

              I find Pleroma much easier to admin, personally. Lots of things get less complicated on a platform designed for concurrency (here, the Erlang VM + Elixir).

              1. 2

                It’s also got a lot fewer moving parts. To get it running as a single user-instance was mostly a matter of a new Postgres DB, running the core service and some nginx copy/paste, especially once you look at the output of the setup script and start managing the config file yourself instead. No job queues to setup and worry about!

                1. 3

                  If it had an SQLite option, it would be perfect - I really don’t need the full Postgres shebango for a 5 user instance.

                  1. 2

                    It’s a bit surprising that something that runs on the Erlang VM doesn’t support mnesia. It’s been a long time since I did any serious Erlang programming, but I removed it being a much cleaner fit that an external SQL database.

                    1. 2

                      It’s just a different thing: mnesia isn’t a relational database.

                    2. 2

                      Pleroma’s access patterns on Postgres are rather weird too; the write load is so heavy that it will absolutely destroy SD cards if you try to run it on a single-board computer.

                      I’ve switched over to GoToSocial and it makes Pleroma look complicated–no need for nginx or postgres; just a set up your domain name and run a single binary in a tmux session to get started with SQLite; it even handles setting up the TLS certs for you and everything. It’s still missing a few features, but from a sysadmin perspective it’s a dream.

                      Plus it doesn’t have the whole “dev teams are good pals with abusive bigots” factor.

                      1. 3

                        Plus it doesn’t have the whole “dev teams are good pals with abusive bigots” factor.

                        To be fair, he seems to have been locked out of the Pleroma repo a few months back (assuming we’re talking about the same one.) But yeah, it’s one reason I’m considering a migration to Akkoma once I’ve got a few hours to do a dry-run (unless I can figure out how to migrate the Pleroma DB to GotoSocial or something similar - this is probably going to need more than a few hours though…)

                        1. 1

                          Plus it doesn’t have the whole “dev teams are good pals with abusive bigots” factor.

                          I can think of no more fitting protest than getting people to use the platform that those people hate.

                  1. 4

                    I always found the “Let it crash” very convincing. One of the most convincing talks I know is not even about Erlang. But I’ve never actually coded anything substantial in either Erlang or Elixir myself. I went the other way instead with Rust and static typing.

                    But I’m curious, for those of you who do work on such systems, does it deliver on its promise? Is it simpler? More robust? And is modern Elixir done in the same vein of “Let it crash” and very few tests or verification?

                    1. 5

                      Rust follows the “let it crash”-philosophy, its panic system is Erlang-inspired. I used to be even stronger baked into the language, when it still had language-level tasking with a runtime. The nomicon chapter on unwinding still calls it out

                      You can see that in the tasking/threading APIs where a panic crashes that component and another part of the system is responsible for handling.

                      1. 4

                        I’ve had to deal with more than one Rust service that take this philosophy to heart and so will fully crash the entire program in the presence of, say, a network connection timeout to a non-business-critical API endpoint. Maybe this isn’t the intended effect of the panic approach to error management, but it does seem to be a common outcome in my experience.

                        The problem here is a mismatch of expectations. It’s nominally OK to crash an Erlang actor in response to many/most runtime faults, because Erlang actors always operate in a constellation of redundant peers, and failure is a first order concern of their supervisor. That crash impacts a single request.

                        But e.g. systemd is not the OTP, and OS processes don’t operate in a cluster. A service running as an OS process is expected to be resilient to basically all runtime errors, even if those errors mean the service can’t fulfill its user-facing requirements. If an OS process crashes, it doesn’t impact a single request, it impacts every request served by the process, every other process with active connections to that process for any other reason, assumptions made by systemd about the soundness of that binary, probably some assumptions about off-host e.g. load balancers shuttling traffic to that instance, everything downstream from them, etc. etc.

                        If “crash-only” means “terminate the request” and not “terminate the process” then all good! But then “crash” isn’t the right verb, I don’t think, as crashing is pretty widely understood to mean the OS level process of the program. Alas.

                        1. 7

                          Yeah, I think this is an acute case of catchy, but wildly misleading terminology. What is really (as in Elang or Midori) understood as proper “let it crash” is two dual properties:

                          • abandoning the current “thing”
                          • containing abandonment to some well-defined boundary, such that:
                            • abandonment don’t propagate outside of this boundary
                            • tearing things down at this boundary doesn’t compromise the state
                            • restarting at the boundary is a well-defined operation which can fix transient errors
                            • the actual blast radius from abandonment is small

                          Everyone gets the first point, buts it’s the second one which matters, which is hard, and which leads to simplicity and reliability.

                          1. 3

                            To expand on this, Rust does only marginally, if at all, better here than you average $LANG:

                            • the build-in boundary is OS thread, which is often too coarse-grained, there’s catch_unwind for do-it-yourself boundaries. There’s nothing to protect from thread monopolizing the CPU due to an infinite loop bug. Some errors (stack overflow, OOM) abort the process bypassing the recovery mechanism.
                            • UnwindSafe machinery in theory helps somewhat with the tainted state problem. In practice, it’s too cumbersome to use and people often silence it. I had one spectacular bug where UnwindSafe would’ve saved couple of days of debugging, if it wasn’t silenced due to it tripping a compiler bug.
                            • nothing to make restart workable, do-it-yourself again.
                          2. 2

                            But e.g. systemd is not the OTP, and OS processes don’t operate in a cluster. A service running as an OS process is expected to be resilient to basically all runtime errors, even if those errors mean the service can’t fulfill its user-facing requirements.

                            I think this might be oversimplifying. Whether it’s reasonable to let a service continue to fulfill tasks despite encountering a serious fault is not clear cut. Example: A service that has a lot of shared state, say thread bound caches of various sensitive user data, a crash might lead to failed cleanups and subsequent data leaks.

                            1. 1

                              Let me rephrase my claim to be more precise.

                              A service running as an OS process is generally expected to be resilient to runtime errors. If a runtime error puts the service in a state where it can no longer fulfill user requirements, and that state is transient and/or recoverable, it is usually preferable for the service to continue to respond to requests with errors, rather than crashing.

                        2. 4

                          In my experience across several companies and codebases in Elixir, I’d say the following things.

                          “let it crash” can lead to clean code. It also originated out of a design space that I believe doesn’t map as directly onto modern webshit as folks want to believe. This is neither good nor bad, it’s just an occasional impedance mismatch between system design philosophies.

                          “let it crash” encourages some people, drunk on the power of OTP and the actor model, to grossly overcomplicate their code. They decide deep supervision trees and worker pools and things are needed when a simple function will do. This is the curse of the beginner Elixir or Erlang developer, and if properly mentored this goes away quickly. If not properly mentored it progresses to a case of conference talks and awkward libraries.

                          Testing and verification in the BEAM ecosystem is weird, and until recently was both best and worst in class depending on what languages you were up against. Dialyzer for example is a marvelous typechecker, but there is growing suspicion that it is severely stunted in the sorts of verification it is categorically capable of. On the other side, property-based testing is strictly old-hat over in at least the Erlang ecosystem and has been for quite some time iirc. Other folks are catching up.

                          (Testing is also–in my opinion–most often valuable to solve team coordination problems and guard against entropy caused by other humans. This is orthogonal to language concerns, but comes out when you have larger webshit-style teams using BEAM stuff when compared with the origin of Erlang.)

                          Robustness is quite possible. I’ve alluded elsewhere to how a running BEAM instance is more of a living thing (gasp, a pet!) than most contemporary app models (pour one out for Smalltalk)…this unlocks some very flexible things you can do in production that I haven’t really seen anywhere else and which make it possible to do things without downtime during an incident that most folks would just look at and go “wat.”. On the other hand, you have to design your systems to actually enable robustness–writing your standard webshit without structuring the application logic to have affordances for interactivity or process isolation or whatever means you’re basically using the BEAM like you would other more conventional systems.

                          (You can also, as I’ve done with Python on at least one occasion, build a BEAM-like actor model with fault tolerance. The idioms are baked into Erlang and Elixir, but you can with sufficient effort reproduce them elsewhere.)

                          (Also, “let it crash” doesn’t mean you won’t sometimes have to wrap your whole VM in a systemd unit to restart things when, say, an intern pushes to prod and blows all the way up the supervision tree.)

                          1. 2

                            In a sense, yes—an example I’ve run into several times is that a service you depend on becomes intermittently unresponsive. In a “regular” software service, unless you clutter your code up with retry and “fail soft” logic (basically your own ad-hoc OTP) this usually means a hard error, eg a end-user received an error message or a service needed to be restarted by the OS process manager.

                            In Erlang the system can usually deal with these kinds of errors by automatically retrying; if the operation keeps failing, the error will propagate up to the next level of the “supervision tree”. Unless it makes it all the way up to the root the application will keep running; sometimes the only indication that something went wrong is some log output.

                            The nice thing about “Let it crash” is that you don’t have to consider every possible failure scenario (eg what happens if this service call returns malformed data? What if it times out?). Instead of trying to preempt every possible error, which is messy and basically intractable, you can focus on the happy path and tell OTP what to do in case of a crash.

                            That said, “Let it crash” is not a silver bullet that will solve all errors in your distributed system; you still have to be acutely aware about which parts of the system can be safely restarted and how. The nice thing is that the base assumption of OTP is that the system will fail at some point, and it gives you a very powerful set of tools to deal with it.

                            Another thing that makes Erlang more robust is the process scheduling model: Processes are lightweight and use preemptive multitasking with fair scheduling, which means you’re less susceptible to “brownouts” from runaway processes.

                            1. 1

                              But I’m curious, for those of you who do work on such systems, does it deliver on its promise? Is it simpler? More robust? And is modern Elixir done in the same vein of “Let it crash” and very few tests or verification?

                              I can only speak to the first half, and it is completely dependent on the team culture. If the team is very OTP/Erlang native, it can work out incredibly well. These teams tend to be extremely pragmatic and focused on boring, obvious, naive ways to solve problems, using global state as needed even!

                              However, when OTP/Erlang collide with a team trying to treat it like anything else things can go badly…. quickly.

                            1. 39

                              This is a curious misunderstanding of what “let it crash” means.

                              “Let it crash” is the observation that if something unexpected, maybe spurious happens in a distributed, ever-changing system, it’s a good approach to crash the unit of work in a structured way and have the supervisor of that unit figure apply a strategy. e.g. if you are connecting to a socket, doing some work and then close - and the connection doesn’t work, the task should crash. The supervisor could then retry 3 times and then propagate the error. It’s a philosophy for error handling and structuring errors in a concurrent system, at far more high level then the example in the code.

                              That approach is very inspired from how init systems are doing things. Say, you run a cleanup task using your init system - it fails once because of a spurious connection error. The init system tries it 5 more times. If it fails 5 times, it then marks the task faulty. Same philosophy: your init system is a supervision system.

                              “Let it crash” does not mean “your code can be faulty”, it encourages crashing as a strategy for handling of spurious, hard to predict errors.

                              1. 26

                                This is a curious misunderstanding of what “let it crash” means.

                                Their understanding of “let it crash” is not your understanding of “let it crash”, and–regardless of the degree to which I prefer your explanation–classifying it as a misunderstanding when I believe they have more expertise in the Erlang ecosystem (7 years at least, judging solely from this blog) and idioms than you do is somewhat presumptuous.

                                More pithily: please don’t rustsplain to the erlangutans. :P

                                There are things I disagree with in the article–for example, I think that tests are helpful for refactoring and for things like numerical routines…here the author and I part ways. That said, perhaps the misunderstanding here is that you and author view bugs differently: the author says that “let it crash” covers all defects and failure conditions (including ones introduced by programmer negligence or bad luck or foresight) while you say it covers only spurious errors.

                                The additional cultural context that’s helpful here is that BEAM systems are (much to the annoyance of devops folks trained in the way of the times) handled as somewhat living critters. It is not uncommon (hell, did this last month during peak traffic with my team) to shell into a running system, handling production traffic, and debug and tweak and patch running code. There is a different mindset about this sort of things than I believe we have when working with bake-and-deploy-and-scrap languages and runtimes.

                                I agree with your interpretation of “let it crash”, but having also suffered through a lot of oddly-engineered Elixir (which is similar enough to Erlang in the relevant ways I feel comfortable saying so) that has forgotten or misunderstood “let it crash” I do have sympathy for and empathy with the author.

                                1. 6


                                  This article, and these comments, seem to be conflating multiple notions of crashing. I guess crashing is usually understood as something that happens at or above the level of an OS process.. That is, a function call that doesn’t return successfully isn’t usually described as having crashed. Or, an autonomous task within a process that hits a terminal error isn’t typically said to have crashed. A process crashes. A system crashes.

                                  Erlang defines a system model in which a supervisor (BEAM/OTP) manages a constellation of redundant entities (actors) which each serve isolated workloads (typically requests). When Erlang says “crash” it means something quite different, and far less impactful, than when Rust or Linux says “crash”. And Erlang’s definition is niche.

                                  Maybe this is a distinction that everyone here fully understands and I’m harping on a pointless point, could be. But it’s definitely misunderstood as a general concept, and IMO shouldn’t be advocated-for without serious qualifications.

                                1. 28

                                  I’d hold off a little bit…several of those are kinda navel-gazey, and the technical ones are covered decently by other tags.

                                  We’re currently seeing a whole bunch of noise due to the transient Twitter shakeout, but give it another month or two and I’d bet it’ll settle down. Once you add a tag, you’re explicitly giving it a home here, and as you’ve noted the Fediverse stories shade into culture/ideology stuff. Let’s not give that a beachhead simple due to a temporary event.

                                  1. 15

                                    the technical ones are covered decently by other tags

                                    I’m not sure I agree here. Several of the technical articles listed above have a tag only tangentially related to the article. For example, Self-hosting GoToSocial on NixOS is only tagged with “Nix”. Most other articles with this tag are either focused on the package manager (or related operating system), or otherwise have a tag more related to the central point of said article (e.g., performance).

                                    A similar case exists for Scaling Mastodon… with the sole tag of scaling. The vast majority of links submitted with this tag also have another tag that answers the question “scaling what?” The article Installing Mastodon inside a FreeBSD jail has only a freebsd tag, which again, is not typically used by itself for software that simply runs on FreeBSD.

                                    Do you know of another tag that currently exists that would bring these articles’ tags more in line with those published on other topics?

                                    Once you add a tag, you’re explicitly giving it a home here

                                    I do share some concern here. I think a number of users (myself included) would like to not see this site become overwhelmed with complaints about twitter ad nauseam (which some recent posts have unfortunately been). Perhaps some restrictions should be added to the tag, such as not permitting new users to post using that tag (similar to merkle-trees).

                                    give it another month or two and I’d bet it’ll settle down

                                    Though I am also concerned about waiting to add the tag. If the fediverse is something that is going to be posted about with any kind of frequency, we should start tagging that now. Uninterested users should be able to filter it out, and interested users should be able to specifically search for it. If we wait a few months, we’ll have that many months of articles that are harder to search for because they don’t have the tag, and uninterested users won’t have a way to filter it out during that upcoming timeframe.

                                    Even if/when this transient dies down, I don’t think it would necessarily be a bad candidate for a tag. We already have several tags about which stories are posted less than once per month (e.g., illumos and d).

                                    1. 2

                                      We already have several tags about which stories are posted less than once per month (e.g., illumos and d).

                                      Those tags tend not to cover a domain with a large amount of drama over politics, ideology, and parasocial relationships.

                                      Do you know of another tag that currently exists that would bring these articles’ tags more in line with those published on other topics?

                                      I think those are fine enough. As I understand your main complaint, it’s that you want to search for Mastodon/fediverse-flavored stuff specifically, instead of nix+plus+mastodon, bsd+plus+mastodon, etc.–e.g., you want to center the fediverse aspect of things with multiple tags.

                                      If we wanted to have a purely technical federation tag, to differentiate between the technical problem space of the fediverse and the flaming dumpsterfire of cliqueish highschool behavior and culturewar that it often actually represents, that might work–but in such a case, we already have networking and distributed.


                                      You’ve been here for four months. You and I have talked about an adjacent topic before. Perhaps take a bit more time to acclimate before insisting on further infiltration of social media topics.

                                      1. 19

                                        I’ve been here for three years, and I would prefer that you not use someone’s length of time on the site as an excuse to silence them when you disagree with their opinions.

                                        1. 1

                                          I haven’t “silenced” them; their posts are still here!

                                          Length of time in a community is a valid data point when evaluating meta discussions around how a community functions.

                                          If you mainly listen to long-standing members, you get one set of pathologies–say, not fixing broken steps. If you mainly listen to brand new members, you get a different set–say, no long-term shared identity or benefits of stare decisis.

                                          I think that one of the great successes of Lobsters is the community-driven meta process; one of the great ways to derail that is to implement each and every brand new request from new members who don’t find it quite to their liking due to inexperience.

                                          I don’t think that these are even particularly bad ideas but that a bit of time should be taken to suss out if the root issue is something genuinely missing in the Lobsters ecosystem or merely something they expect to be present that isn’t.

                                          1. 2

                                            Perhaps take a bit more time to acclimate before insisting on further infiltration of social media topics.

                                            That sure sounds like your intent was to have the other person stop posting/advocating for their opinion, based on the length of time they’ve been here. And I’m telling you I would prefer you not do that.

                                            1. -1

                                              And I’m telling you I would prefer you not do that.

                                              Please don’t silence me. :)

                                        2. 2

                                          a purely technical federation tag

                                          I think that would be an equally good solution. I’m still not sure why “fediverse” came into common use, and I suspect that name was only chosen for the tag proposal because it is currently popular.

                                          we already have […] distributed

                                          Adding a description to the distributed tag that explicitly includes federations of servers could be a decent alternative to making a separate federation tag. (Indeed, many posts recently tagged with distributed are about the fediverse.)

                                          Perhaps take a bit more time to acclimate before insisting on further infiltration of social media topics.

                                          @friendlysock, I think that is a rather uncharitable interpretation of my comments. I posted the adjacent topic you mentioned as an open-ended question specifically to acclimate myself to the community and determine the consensus on recent posts about Mastodon and related software. I’m not insisting that Lobsters allow social media posts. My point was that the community should take some kind of action based on this influx of new posts, and either (1) if social media talk is banned, then the posts should be deleted, (2) if social media talk is permitted, a tag should be made for it. I think you’ll find this true if you reread my original post.

                                          From the comments in my previous post, I concluded that the community would appreciate technical discussions of social media, but not other discussions about it (which you yourself agreed with). My comments above were based on this conclusion, combined with several technical links about the fediverse listed in the original post.

                                          1. 2

                                            My point was that the community should take some kind of action based on this influx of new posts,

                                            No action is an action, and one that’s fine to take until we have a few more months of data between us and the current Twitter drama.

                                            either (1) if social media talk is banned, then the posts should be deleted, (2) if social media talk is permitted, a tag should be made for it.

                                            There is a range of other, community-driven actions that fall between “pushcx manually scrubs all of these from the site” and “we explicitly endorse these articles due to their tag existing; please post more!”. There are also:

                                            • Observe post, upvote.
                                            • Observe post, do nothing.
                                            • Observe post, flag.
                                            • Observer post, flag and explain.

                                            Each of these has its own use in the site meta–the issue with your question in that thread (and similar to one here) is that it looks a great deal like forcing a black-or-white binary decision. The answer I gave there, and have given here, and I think others have given is “Look, we’re totally enthusiastic about the technical side of this, but are categorically unwilling to open the door for fediverse drama.”

                                            I’m sure you’ve asked both of these questions in good faith, but if you do so a third time it goes from being “okay, this new person is asking about site stuff instead of lurking to get clarification, that’s fine” to “okay, this person seems to be really forcing this issue every time they don’t get an answer they seem to like.” This is a real social attack that happens to sites like this everywhere, so please understand why I and others might be sensitive to it.

                                            Adding a description to the distributed tag that explicitly includes federations of servers could be a decent alternative to making a separate federation tag.

                                            I wanted to highlight this because I think you’ve nailed a really good incremental step that might get us 80% of the way with 20% of the effort–and honestly slightly expanding the tag definition for distributed to onramp new folks hadn’t even occurred to me. Good thinking! :)

                                            1. 2

                                              You make a number of good points. Regarding the comment that I’ve only been here for four months: I don’t think you’re wrong to point that out. However, it would have been great if you had contributed constructively by drawing upon your eight years’ time here to share previous tag proposals that you’ve seen succeed or fail, rather than discouraging new users from posting. Regardless, I’ve looked through some past posts on tag proposals (many of which you commented on), and I feel I somewhat better understand your perspective, and why you are sensitive to these discussions.

                                              I still have a few sticking points, though:

                                              I don’t think “flag” or “flag and explain” are really options. The about page makes it pretty clear that flags should only be used for requesting moderator action and should not be discussed in comments (emphasis mine):

                                              Users can flag stories and comments when there’s a serious problem that needs moderator attention.
                                              Please don’t use flagging to try to push down topics or people you don’t like; click “hide” on the story and move on rather than clutter up the mod dashboard with false alarms.
                                              To guide usage and head off distracting meta conversations (“Why was this flagged!?”, etc), flagging requires selecting from a preset list of reasons.
                                              [Flag comments with] “off-topic” for drifting into meta or topics that aren’t related to the story.

                                              If these statements on the about page are wrong, please fix them. That page is one of the few resources new users have to learn how this site works.


                                              Finally, I’d ask you to take a step back and look at my profile as a whole. You’ve stated your concern that I may be “really forcing this issue every time they don’t get an answer they seem to like” with the clear implication that the answer I want is to enable “further infiltration of social media topics.” I’ve already gotten the answer you seem to think I want: all social media is fine! (Posted by a user of this site for three years, no less.) But I haven’t commented on it nor referenced it.

                                              Your characterization of me is just… not accurate? I don’t want more social media drivel on this site. I’ve never submitted a story or posted a comment about any non-meta discussion of social media or the fediverse. I am only active in meta discussions about them because I personally have had trouble discerning fediverse-related stories from others when picking ones from the front page to read (sometimes, I would not like to read about [whatever new fediverse platform]).

                                              My primary complaint is that stories about the fediverse are tagged inconsistently. I’m happy to have them all move under distributed. But to have one tagged with only scaling, one with only freebsd, and one with only nix, etc., and have these posted multiple times a day really grinds my gears (and I suspect a number of other users’, based on this tag proposal having >100 votes).

                                      2. 3

                                        I agree with you, but it also makes me kind of sad because there’s a LOT of super interesting tech that goes into ActivityPub and surrounding protocols, server stacks, etc. It’s also a brand new niche with tons of room for innovation as more people pile in.

                                        1. 4

                                          Then that would be a great place for the networking or distributed tags.

                                      1. 68

                                        I read this and felt weird about it initially but couldn’t quite put my finger on it. From my experience, using Rust has lead to faster developer velocity (even for CRUD apps) simply because the strong type system allows you to encode invariants to be checked at compile time, and the tooling and libraries are amazing. This article seems to be about a project that was being developed right around async/await release, which was a rough time for using libraries. Most of the sharp edges are gone now, but I don’t doubt that the state of the ecosystem at that point affected this person’s experience.

                                        However, I do think there is a situation where this advice holds true (even for someone as heavily biased as I am), which is for a very specific kind of startup: a hyper-growth startup with 100%+ YoY engineer hiring. The issue with Rust is not that its slower to develop in, I don’t think that is true, its that in order to develop quickly in Rust you have to program in Rust. And frankly, most new developers to Rust have no idea how to program in Rust because so many languages do not feature strong type systems. And the problem is that if your influx of new developers who need to learn Rust is too large, you won’t be able to properly onboard them. Trying to write Java using Rust is horrible (I’ve worked with a number of colleagues who I’ve had to gently steer away from OO design patterns that they were used to, simply because they make for really difficult Rust code, and are largely obsoleted by the type system).

                                        It isn’t even lifetimes or borrowing that are necessarily tricky, in my experience issues with lifetimes are fairly rare for people and they almost always immediately seek out an experienced Rust dev for guidance (you only need a handful to deal with all questions on Lifetimes; my current team only has me and its been a non-issue). The bigger problems are around how to structure code. Type-driven development is not something most people have experience with, so they tend to stick to really simple Structs and Enums, to their detriment.

                                        For instance, I commonly see new Rust developers doing something like this:

                                        fn double_or_multiply(x: i32, y: Option<i32>, double: bool) -> Result<i32> {
                                            if double {
                                                if y.is_some() {
                                                    return Err("Y should not be set");
                                                x * 2
                                            } else {
                                                if y.is_none() {
                                                     return Err("Y should be set");
                                                x * y.unwrap()

                                        Yes, I know its a completely contrived example, but I’m sure you’re familiar with that kind of pattern in code. The issue is that this is using the shallow aspects of Rust’s type system – you end up paying for all of Rust but only reaping the benefits of 10% of it. Compare that to what you could do by leveraging the type system fully:

                                        enum OpKind {
                                           Multiply(x, y),
                                        fn double_or_multiply(input: OpKind) -> i32 {
                                            match input {
                                                 Double(x) => x * 2,
                                                 Multiply(x, y) => x * y,

                                        Note how the error has disappeared as there is no way to call this function improperly. That means fewer tests to write, less code to maintain, and APIs that can’t be used improperly. One of the most interesting questions I get commonly when I promote this style of code is “how do I test that it fails then?”; its always fun to explain that it can’t fail[1] and there is no need to write a test for the failure. The developer efficiency benefits from this style of thinking everywhere is massive and more than pays for the cost of using Rust.

                                        But developers from other languages take time to get to this point, and it does take time and effort from experienced Rust developers to get everyone on the same page. Too many new people and I can see more and more code leaking in as the first example, which means you get minimal benefits of Rust with all the cost.

                                        I can’t argue with this person’s experience, as much as I love Rust and think it has features that make it an incredibly productive language, but I think the core issue is that the majority of developers do not have experience with strong type-system thinking. As more languages start adding types I’m hopeful this problem becomes less prevalent, because the productivity differences between developers who understand type-driven development vs. those who don’t is large (in languages with strong type-systems).

                                        [1] Technically it can panic, which I do explain, but for the majority of cases that is a non-issue. Generally if there is a panic-situation that you think you might have to handle you use a different method/function which returns a Result and bubble that up. Panics are largely unhandled (and for good reason; they aren’t exceptions and should be considered major bugs).

                                        1. 11

                                          FWIW Matt Welsh is a very experienced programmer and former computer science professor at Harvard:


                                          (I know of him from his research on SEDA, an influential web server architecture, as well as having a good blog.)

                                          So this comments strikes me as a bit out in left field … I don’t doubt it’s your experience, but it doesn’t seem relevant to the article

                                          1. 9

                                            I’m not familiar with who Matt Welsh is, but I found his post to be well written and accurate to his experience. My comment was simply a reflection of my own experience, which I think differs.

                                            I don’t see how my comment isn’t relevant to the article, but I am open to feedback if there is something specific you felt made my comment out of left field!

                                            1. 7

                                              Not GP, and I hope this doesn’t come off as too negative, but your comment is pretty dismissive of the idea that Matt Welsh could have substantive issues with the design of Rust. You seem to imply that the problems he ran into stem from a lack of experience:

                                              I think the core issue is that the majority of developers do not have experience with strong type-system thinking.

                                              Your example about double_or_multiply is great, but IMO it’s a pretty elementary example for most readers here, as well as Matt Welsh.

                                              The general vibe is like this: someone complains that a particular proof in graduate-level mathematics is hard to read, and then you respond with a tutorial on pre-calculus. I like your comment, and it is relevant, but it doesn’t feel like it responds to the article, or takes the author seriously.

                                              1. 4

                                                Thanks for the explanation, I do appreciate it. I hoped to not make my comment be dismissive of the article, and more as a refinement of which aspect it was talking about. I think the fundamental issue is that I completely disagree with Matt Welsh that programming in Rust lowers developer efficiency, and so my comment was exploring a reason for why he may feel that way.

                                                The example was simple, and more just for having something to ground my argument in (ie. Rust makes you faster because you write less code overall), as well as having something for developers who are unfamiliar with type-driven development to see. I didn’t mean to imply it as Matt Welsh doesn’t know that, but more that type-driven development is a completely different style of programming, and onboarding devs to a new language is easy but to a new style is hard.

                                                Clearly my point wasn’t made as clear as I had hoped, and thank you for pointing out where it felt like I was off-base. I do think it’s important to disagree, and I’ve never been one to defer to another just because of their accomplishments or prestige.

                                                I’m thinking it might make sense for me to write my own post on my thoughts on developing quickly in Rust, and hopefully I can take these ideas and your feedback and make something that pushes the conversation forward on what makes a productive language without coming across as dismissive of others experience :)

                                                1. 1


                                                  …a hyper-growth startup with 100%+ YoY engineer hiring… in order to develop quickly in Rust you have to program in Rust. And frankly, most new developers to Rust have no idea how to program in Rust because so many languages do not feature strong type systems. And the problem is that if your influx of new developers who need to learn Rust is too large, you won’t be able to properly onboard them.


                                                  We hired a ton of people during my time at this company, but only about two or three of the 60+ people that joined the engineering team had previous experience with Rust.

                                                  This was in two years, which he claims was 10x growth in headcount, so from ~6 people to 60 in two years, with only 3 people who knew Rust. Basically, well above your 100% YoY hiring threshold for being able to onboard Rust engineers.


                                                  I completely disagree with Matt Welsh that programming in Rust lowers developer efficiency

                                                  I don’t think you do disagree with him :)

                                                  My interpretation of his claim is that in a rapidly growing organisation that needs to ship product rapidly, and where you don’t have a lot of Rust expertise already, Rust may not be a good fit. What you are claiming is that given enough experience with Rust, it can accelerate your day to day development. But that to do so you need to give developers sufficient time to onboard to Rust such that they can become familiar and efficient with it (which of course is true for any language, but Rust onboarding likely takes a lot longer than, say, Go, or Python, or C#).

                                                  Both of these claims can be true because they are discussing different aspects of software engineering at different scales.

                                                  Related, I’d be interested in hearing what your experience has been on how long it takes to onboard a complete Rust novice, to the point they are at close to 100% productivity.

                                                  1. 2

                                                    My interpretation of his claim is that in a rapidly growing organisation that needs to ship product rapidly, and where you don’t have a lot of Rust expertise already, Rust may not be a good fit.

                                                    You’re right, I do agree with this claim. My understanding of Matt’s article is that he argues this is a function of Rust as a language, whereas I argue this is a function of a general lack of understanding of type-driven programming.

                                                    It is effectively semantics, but important semantics, because I don’t think the solution is to avoid using Rust/Haskell/OCaml/etc except in niche situations, it should be to educate developers in type-driven development. But of course that is a much bigger problem, and not easy, so I can see where one might think I’m just arguing a point entirely separate from what the article is talking about.

                                                    I think you’re right, the point I am making is mostly tangential, and I could have made that more clear. Thanks for explaining your view!

                                                    Related, I’d be interested in hearing what your experience has been on how long it takes to onboard a complete Rust novice, to the point they are at close to 100% productivity.

                                                    I’ll take the easy way out and say it depends :P but it is hard to put an exact number on it simply because I don’t have a ton of data to really say. Rust is still fairly niche, even at companies using it (ie. Rust for core services, but most devs work on services in other langs talking to the Rust core), and usually devs wanting to work on the Rust stuff already know Rust to an extent. My current team is getting ~6 or 7 devs without Rust experience next quarter (who have no idea they’ll be working in Rust) so hopefully I’ll have much better data soon! (Absurd bureaucratic policies maybe have good side effects)

                                                    From my limited experience onboarding people with no Rust experience, I’ve seen devs without type-driven design experience pick up a productive level in two to three months (including new-grads), but take a lot longer to get to being comfortable with using the type system to its full power (maybe a year to get mostly comfortable, but still have gaps). However, they’ve largely been passionate and qualified devs I’ve worked with (even the ones that were new-grads), so I think my opinion is biased here.

                                                    I’d guess that a developer with strong type system experience can pick up a near 100% productivity in a few weeks, but I have yet to onboard a dev with this background who doesn’t already know Rust, so this is largely from my own experience learning Rust and talking to others.

                                                    I really wish I had experience with a hyper growth startup using Rust, and seeing first-hand the failure modes onboarding that many people to Rust. But I agree with Matt’s assessment that it’s the wrong language for that situation given the average software developer’s experience, and I have my doubts about the efficacy of hyper growth startups in the first place.

                                          2. 8

                                            On a related note, I’m very curious to see what happens with the recent Twitter situation. If Twitter manages to thrive, I think many companies are going to take notice and cut back on developers. The easy pay-day for software engineers could be at an end, and fewer developers will have to handle larger and larger systems. In that case, I’d imagine building things which are robust will outweigh building things quickly. If you have 10% the number of engineers you want to minimize the amount of incident response you are doing (1 out of 100 devs dealing with oncall every week is very different from 1 out of 10 devs dealing with oncall every week in terms of productivity; now the buggy systems have a 10% hit on productivity rather than 1%).

                                            I’m both worried (large scale cutbacks to mimic Twitter would not be fun) but also somewhat optimistic that it would lead to more reliable systems overall. Counter-intuitively, I think Rust/Haskell/Ocaml and friends would thrive in that world, as choosing a language for quick onboarding of hundreds of devs is no longer a constraint.

                                            1. 19

                                              I draw the exact opposite conclusion:

                                              Tighter pursestrings mean less resources allocated to engineers screwing around playing with new languages and overengineering solutions to stave off boredom.

                                              There will probably be cases where a business truly uses Rust or something else to differentiate itself technically in performance or reliability, but the majority of folks will go with cheaper tooling with easier-to-replace developers.

                                              1. 11

                                                I agree. People will go where the libraries are. If you have 1/10 the number of people you aren’t going to spend your time reimplementing the AWS SDK. You are going to stick to the beaten path.

                                                1. 2

                                                  I’m sure you meant that as more of a general example than a specific one, but:

                                                  1. 1

                                                    Yeah, I meant it generically. More niche languages are missing a lot of libraries. If you have fewer people you probably want to spend less time reinventing the wheel.

                                                    I know for any one language people will probably come out of the wood work and say “I don’t run into any issues.” but it’s more about perception in the large.

                                                2. 2

                                                  You make a really good point, and I’ve been mulling on it. My logic was based on the idea that if the software industry suddenly shrank to 10% of its size, the engineers maintaining buggy systems would burn out, while those maintaining robust systems would not. Sort of a forced evolution-by-burnout.

                                                  But I think you’re right, tighter purse strings means less experimentation, so the tried-and-true would benefit. So who knows! Hopefully it’s not something we will ever learn the answer to :)

                                                  1. 2

                                                    The department I run, after over 50% casualty rate this year, has made it a major focus to consolidate, simplify, and emphasize better-documented and smaller systems specifically to handle this. :)

                                                    I hope it works out, but these are going to be interesting times whatever happens. I just personally wish engineers in tech as a culture hadn’t overplayed their hand.

                                                3. 5

                                                  I can probably set your mind at ease about Twitter (but not the other tech companies having layoffs, nor the new management there who is utterly clueless). Since at least 2009, Twitter’s implicit/unspoken policy was that engineers are cheaper than machines. In other words, it’s more cost-effective to hire a team to optimize the bejeezus out of some service or another, if they can end up cutting server load by 10%. If their policy was based on any real financial data (I have no idea), good dev and devops people will continue to be in high demand, if only to reduce overall machine costs.

                                                4. 3

                                                  Any recommend way to learn about that from your experience (except than being lucky enough to have an experienced Rust programmer to help you out)?

                                                  Maybe something like

                                                  1. 4

                                                    I’m a fan of trial-by-fire, and if you really want to understand type-driven development then learning and using Haskell is what I’d recommend. Rust is a weird language because it seems really novel, but it really only has the ownership model as unique (and even then, Ada + spark had it first). Everything else is just the best bits borrowed from other languages.

                                                    Haskell’s type system is more powerful, and the docs for libraries heavily lean into the types-as-documentation. I’m not good enough at Haskell to write production software in it, but getting to that “aha!” moment with the language has paid dividends in using Rust effectively.

                                                    1. 3

                                                      Rust is a weird language because it seems really novel, but it really only has the ownership model as unique (and even then, Ada + spark had it first

                                                      Ada/SPARK did not have Rust’s affine types ownership model.

                                                    2. 4

                                                      Just wanted to mention that it is actually for others’ sake. Didn’t want to let that go unnoticed as it is a wonderful platform for learning!

                                                      1. 2

                                                        Elm! If you want a beginner-friendly way to epiphany, work through the Elm tutorial. That was my first, visceral, experience of the joy of sum types / zero runtime errors / a compiler that always has my back.

                                                        Why via Elm? Because it’s a small and simple language that is user-friendly in every fibre of its being. This leaves you with lots of brain cycles to play with sum types.

                                                        • Friendly error messages that always give you hints on what to do next. (Elms error messages were so spectacularly good, and that goodness was so novel, that for a while there was a whole buzz in all sorts of language communities saying “our error messages should be more like Elm’s”. Rust may be the most prominent success.)
                                                        • You’re building a web page, something you can see and interact with.
                                                        • Reliable refactoring, if your refactor is incomplete the compiler will tell you.
                                                      2. 2

                                                        fn double_or_multiply(x: i32, y: Option, double: bool)

                                                        my 2c: I know it’s a contrived example, but even outside of rust it’s generally (not always) a bad idea (e.g. it’s a code smell) to have a function that does different thing based on a boolean.

                                                        Also, a good linter/code review should help with the kind of issue you’re pointing to.

                                                        1. 2

                                                          In hopes it’s instructive, your code samples are an instance of parse don’t validate where you push all your error checking logic to one place in the code.

                                                          1. 3

                                                            Yes it is :) I’m a huge fan of that article, though I’ve found it can sometimes be difficult for someone who isn’t familiar with strong types already. Thank you for sharing the link, I think it’s a great resource for anyone interested in reading more!

                                                          2. 1

                                                            I’m new to Rust, could you provide an example of calling your second function? I’ve only just passed the enum chapter of the book and that is the exact chapter that made me excited about working with Rust.

                                                            1. 7

                                                              Of course! You would call it like so:

                                                              double_or_multiply(OpKind::Multiply(33, 22));

                                                              Its good to hear your excitement from enums in Rust, as I think they are an under-appreciated aspect of the language. Combining structs + enums is super powerful for removing invalid inputs, especially nesting them into each other. The way I think about designing any API is: how can I structure the input the user supplies such that they can’t pass in something incorrect?

                                                              I wish I could find a source for how to design APIs, as there is some place out there which lists the different levels of quality of an API:

                                                              • low: the obvious way to use the API is incorrect, hard to use correctly
                                                              • medium: can be used incorrectly or correctly
                                                              • high: the obvious way to use the API is correct, hard to use incorrectly
                                                              • best: no way to incorrectly use the API, easy to use correctly
                                                              1. 6

                                                                You may be thinking of Rusty Russell’s API design levels.

                                                                1. 1

                                                                  Yes! That was exactly what I was looking for, thank you!

                                                              2. 3
                                                                double_or_multiply(Double(2)) // = 2*x = 4
                                                                // Or
                                                                double_or_multiply(Multiply(3,7)) // = 3 * 7 = 21
                                                              3. 1

                                                                Can you explain how it could panic?

                                                                1. 2

                                                                  Multiplication overflow, which actually would only happen in debug mode (or release with overflow checks enabled). So in practice it likely couldn’t panic (usually nobody turns on overflow checks) (see below)

                                                                  1. 6

                                                                    It’s not that uncommon. Overflow checks are generally off because of perceived bad performance, but companies interested in correctness favor a crash over wrapping. Example: Google Android…


                                                                    Overflow checking is on by default in Android for Rust, which requires overflow operations to be explicit.

                                                                    1. 2

                                                                      I stand corrected! I’m curious what the performance impact is, especially in hot loops. Though I imagine LLVM trickery eliminates a lot of overflow checks even with them enabled

                                                                      1. 2

                                                                        I remember numbers flying around on Twitter, most of what I hear is that it is in neglectible ranges. Particularly that if it becomes a problem, there‘s API for actually doing wrapping ops.

                                                                        Sadly, as often, I can‘t find a structured document that outlines this, even after a bit of searching. Sorry, I‘d love if I had more.

                                                                    2. 1

                                                                      So, it’s specific for this example, if the enum was over structs with different types and the function did something else, it wouldn’t necessarily panic, right?

                                                                      Is there a way to make this design panic-proof?

                                                                      1. 5

                                                                        Yes, the panicking is specific to the example. And you can make it panic-proof if none of the function calls within can panic. IIRC its still an open design problem of how to mark functions as “no panic” in rust so the compiler checks it [1][2]. There are some libraries to do some amount of panic-proofing at compile-time[3] but I haven’t used them. I thought there was a larger RFC ticket for the no-panic attribute but I can’t find it right now.

                                                                        [1] [2] [3]

                                                                1. 9

                                                                  I kind of wish they wouldn’t do that. I have a currently dead code branch because of this that disables ads when you visit with a lobsters referfer.

                                                                  1. 3

                                                                    I’m not sure about the “more ethical aggregator”, but I appreciate the idea. :)

                                                                    I’m somewhat a fan of Grandpa Zawinski’s approach myself.

                                                                  1. 5

                                                                    I’m like 99% sure that’s to prevent abuse by people seeking to use us as marketing channels or SEO stuff.

                                                                    1. 28

                                                                      This is an important lesson. Rust is aimed at being a safer systems language. There are a lot of safe applications languages with more mature ecosystems, better tooling, and so on. If you are in a situation where you don’t need fine-grained control over memory layout, where you can tolerate the memory and jitter of a global garbage collector, then Rust is as much the wrong choice as C would be. Rust is a good choice for situations where, if Rust didn’t exist, C would be a good choice. If you wouldn’t consider using C for a project, don’t use Rust. Use Erlang/Elixir, C#/F#, JavaScript/TypeScript, Swift, Pony, or anything else that targets application programming problems.

                                                                      1. 20

                                                                        There are a lot of safe applications languages with more mature ecosystems, better tooling, and so on

                                                                        I think this is sadly not entirely true. On balance, I would say Rust’s “quality of implementation” story is ahead of mature languages, and that’s exactly what creates this “let’s write CRUD in Rust” pressure.

                                                                        Let’s say I write something where I don’t need Rust, what are my choices?

                                                                        • Erlang/Elexir – dynamic types, the same, if not bigger, level of weirdness in Rust.
                                                                        • F# last time I’ve checked, the build system was pretty horrible (specifying files in a specific order in an XML file)
                                                                        • C# – until recently, it had nullability problems, and was pretty windows-specific. maybe it is a viable option now. My two questions would be: a) how horrible is the build system? b) how hard is it to produce a static binary which I can copy from one linux PC to another without installing any runtime libraries?
                                                                        • JavaScript/TypeScript – npm is a huge time sink. Once deno matures (or Rome conquers the world) it might become a reasonable alternative
                                                                        • Swift – doesn’t really exist outside of Mac? A lot of core things were in mac-specific libraries last time I’ve checkd
                                                                        • Pony – super niche at this point?
                                                                        • Java – nullability, problems with static binaries, horrible build system
                                                                        • Kotlin – Java sans nullability
                                                                        • Go – if you value utmost simplicity more than sum types, that is a splendid choice. I think Go is the single language which is in the same class as Rust when it comes to “quality of implementation”, and, arguably, it’s even better than Rust in this respect. If only it had enums and null-checking…
                                                                        • OCaml – this one is pretty ideal when it comes to the language machinery, but two+ build systems, two standard libraries, etc, make it not a reasonable choice in comparison.

                                                                        Really, it seems that there’s “Go with enums / OCaml with channels and Cargo”-shaped hole in our language-space these days, which is pretty imperfectly covered by existing options. I really wish that such language existed, this would relieve a lot of design pressure from Rust and allow it to focus on systems use-case more.

                                                                        1. 9

                                                                          JavaScript/TypeScript – npm is a huge time sink. Once deno matures (or Rome conquers the world) it might become a reasonable alternative

                                                                          I’d have a hard time finding a productive lead or senior engineer for Node.js complaining that npm is a timesink and the number one thing that makes the language not productive. The teams I work with are incredibly productive with Node (and would have a very hard time onboarding with rust), and most of the time we only have to touch package.json once every few weeks.

                                                                          1. 6

                                                                            Ocaml 5 has channels along with the new multi core support. And algebraic effects.

                                                                            1. 4

                                                                              [… ] this would relieve a lot of design pressure from Rust and allow it to focus on systems use-case more.

                                                                              Considering that a lot of more serious “system use-cases” (the Linux kernel, Firefox) tend to either fight or ditch Cargo, could it be that its current design is optimized for the wrong use-cases?

                                                                              1. 3

                                                                                That’s a great question.

                                                                                The primary answer is indeed that’s not the use-case Cargo is optimized for, and that was the right thing to do. With the exception of the kernel, all folks who ditch Cargo still make use of ecosystem. And producing re-usable libraries is the use-case Cargo is very heavily skewed towards. The main shtick of Cargo is the (informal, implementation-defined) specification of Cargo.toml file and .crate archive which provide a very specific interface for re-usable units of code. Because every library on is build in exactly same inflexible way, re-using the libraries is easy.

                                                                                The secondary answer is that, while Cargo could be much better (defaults steer you towards poor compile times) for the case of developing a leaf artifact, rather than a library, it’s pretty decent. Stuff like ripgrep, cargo, rust-analyzer, wasmtime are pretty happy with Cargo. There’s a lot of “systems-enough” software which doesn’t need a lot of flexibility in the build process.

                                                                                But yeah, ultimately, if you build something BIG, you probably have a generic build-system building the thing, and the way to plug Rust there is by plugging rustc, not cargo. Though, yeah, Cargo could’ve been somewhat more transparent to make this easier.

                                                                                1. 1

                                                                                  The main shtick of Cargo is the (informal, implementation-defined) specification of Cargo.toml file […]

                                                                                  An even bigger shtick is, a rudimentary build system built with environment variables. For example, in build2 we could map all the concepts of Cargo.toml (even features), but dealing with is a bleak prospect.

                                                                                  1. 2

                                                                                    Yeah, is super-problematic. It’s a necessary evil escape hatch to be able to use C code, if you have too.

                                                                                    I think the most promising solution in this space is the metabuild proposal: rather than compiling C “by hand” in, you specify declarative dependencies on C libraries in Cargo.toml, and then a generic (eg, the same binary shared between all crates) reads that meta and builds C code. This would provide a generic hook for generic build systems to supply Cargo with C deps.

                                                                                    Sadly, that didn’t go anywhere:

                                                                                    • the design we arrived at required first-class support in Cargo, which didn’t materialize. I feel that an xtask-shaped polyfil would’ve worked better.
                                                                                    • we lacked a brave person to actually submit tonnes of PRs to actually move ecosystem towards declarative native deps.
                                                                                    1. 1

                                                                                      Yeah, in a parallel universe there is a standard build system and package manager for C/C++ with Rust (and other languages) that are built “on top” simply reusing that and getting all the C/C++ libraries for free.

                                                                              2. 3

                                                                                the same, if not bigger, level of weirdness in Rust.

                                                                                In the case of Elixir, I’d argue that it actually writes a great deal like Python or Ruby, plus some minor practical guardrails (immutability) and an idiom around message-passing.

                                                                                The really weird stuff you don’t hit until Advanced/Advanced+ BEAM wizard status.

                                                                                1. 2

                                                                                  I think this overstates the importance of the output being a single static binary. Sometimes that’s a desirable goal, no question. But if you are writing code that will be deployed to a containerized environment, which is the case for a lot of CRUD web apps, installing runtime dependencies (libraries, interpreter, etc.) is a no-op from an operational point of view because the libraries are bundled in the container image just like they’d be bundled in a static binary.

                                                                                  1. 3

                                                                                    But then you’re paying the cost of building container images - which is a huge part of deployment time costs if I understand people’s experience reports correctly.

                                                                                    1. 5

                                                                                      Even with static binaries you’re probably still looking at containerisation for a lot of intermediate things, or runtime requirements, if only because of the state of infrastructure right now. There isn’t really “k8s but without containers”.

                                                                                      1. 2

                                                                                        Not really. Building Docker images with proper caching is pretty quick, within about a couple of minutes in my experience. Far more time will be spent running tests (or in bad CI systems, waiting for an agent to become ready).

                                                                                        1. 1

                                                                                          👍 It only adds seconds on top of iterated builds if done well. Could take an extra minute or two on a fresh machine

                                                                                        2. 1

                                                                                          It’s pretty quick if you choose not to include a whole operating system

                                                                                        3. 2

                                                                                          Yeah, for specific “let’s build a CRUD app” use-case static linking is not that important.

                                                                                          But it often happens that you need to write many different smaller things for different use-cases. If you pick the best tool for the job for every small use-case, you’ll end up with a bloated toolbox. Picking a single language for everything has huge systemic savings.

                                                                                          This is the angle which makes me think that “small artifacts” is an important part of QoI.

                                                                                        4. 1

                                                                                          This is your regularly scheduled reminder that D exists. It recently added sumtypes to the standard library!

                                                                                          1. 6
                                                                                            $ cat main.d
                                                                                            import std.stdio;
                                                                                            void main(){
                                                                                                auto r = f();
                                                                                                writeln("hello, ub");
                                                                                            auto f() {
                                                                                                int x = 92;
                                                                                                return g(&x);
                                                                                            auto g(int* x) {
                                                                                                return x;
                                                                                            $ dmd main.d && ./main
                                                                                            hello, ub

                                                                                            If, by default, I can trivially trigger UB, the language loses a lot of points on QoI. I think there’s some safe pragma, or compilation flag, or something to fix this, but it really should be default if we think about language being applicable to CRUD domain.

                                                                                            1. 2

                                                                                              This is fixed by enabling the experimental DIP1000 feature via -dip1000:

                                                                                               11: Error: returning `g(& x)` escapes a reference to local variable `x`

                                                                                              DIP1000 is in testing and will hopefully become standard soon.

                                                                                              That said, just don’t use pointers. You practically never need them. We have large backend services in production use that never use pointers at all.

                                                                                          2. 1

                                                                                            Scala. The horsepower and ecosystem of the JVM, plus the type system inspired by much more academic languages. In reality though, all of the above mainstream languages are plenty good enough for almost all line of business work. And where they have shortcomings, certainly Rust does have its own shortcomings as outlined in the OP. So it’s all about tradeoffs, as always.

                                                                                            1. 1

                                                                                              tbh, I think the only reason to use Scala over Kotlin is having an already existing large Scala code-base which went all-in on type-level programming (which I think is surprisingly the case for a lot of bank’s internal departments).

                                                                                              Scala deserves a lot of credit for normalizing functional programming in the industry, it was the harbinger of programming language renaissance, but, as a practical tool, I don’t think it is there.

                                                                                          3. 6

                                                                                            Use Erlang/Elixir, C#/F#, JavaScript/TypeScript, Swift, Pony, or anything else that targets application programming problems.

                                                                                            Didn’t Wallaroo move from Pony to Rust around this time last year?

                                                                                            I remember the corecursive interview with Sean Allen in 2020 talking about his developer experience with Pony “All of us who worked on Wallaroo in the early days have a bit of scar tissue where even though none of us had hit a compiler bug in forever, we were still like, ‘Is that a bug in my code or is that a bug in the compiler?’”

                                                                                            Having worked at a startup where we used Haskell for an elaborate CRUD app, this sounds about as painful. They sacked the engineering director after I left and I believe they’ve moved to Typescript for everything now.

                                                                                            I wouldn’t put Pony on that list just like I wouldn’t put Haskell on that list.

                                                                                            I can also say that my buddy who’s a manager at a big-data company has had everyone under him switch over to Java from Scala.

                                                                                            So I’d probably put Java and Golang at the top of the list for a CRUD app.

                                                                                            1. 2

                                                                                              Great interview, enjoyed reading it.

                                                                                              Tbh I think the trade off here is how “business logicky” your app is, rather than crud or whatever. At a certain point something like scala really helps you find cascading implications of changes. If you have that kind of app, all else being equal, a rich static type system is going to improve productivity. The more you’re mostly just doing io with code that’s fairly self contained, the easier it is to crank out code with a golang or mypy like type system.

                                                                                              Similarly the more you need to spend time wrangling for throughput the better your life will be with systems that focus on safety and control of execution.

                                                                                              Twitter started out on rails, and that makes sense when you don’t need your application to actually be bug free or fast but you do need to just get stuff out there. We continue to have so many different programming systems because there are so many different points in the space that it’s worth optimizing for.

                                                                                              (For anyone who thinks I’m being dismissive- I spend my days writing golang with only api tests mostly to run sql, and it’s a solid choice for that kind of thing).

                                                                                          1. 8

                                                                                            Couple of things I want to highlight from this article that bear repeating (indeed, perhaps, stamping onto the arms of developers tempted to forget):

                                                                                            The only reason we were using Rust was because the original authors of the system were Rust experts, not because it was an especially good fit for building this kind of service.

                                                                                            There can be tremendous advantages your team is missing out on due to what is effectively an accident of history. In the case of a CRUD app with a couple of routes making a database call, Express (JS) or Sinatra (Ruby) or Flask (Python) can just bury the competition in productivity. It took longer to write this comment than a simple index API route in those frameworks/languages. Just because your founding team likes X doesn’t mean that decision shouldn’t be reviewed.

                                                                                            We started having weekly “learn Rust” sessions for the team to help share knowledge and expertise. This was all a significant drain on the team’s productivity and morale as everyone felt the slow rate of development.

                                                                                            Developers need to learn the domain, learn the process, and learn the tooling during the course of doing their work. Choosing non-standard or difficult tooling means that their efficiency will suffer. It sounds like the author’s team navigated this as best they could with professional development, but the real solution of using an easier and better-fit tool would have been preferable.

                                                                                            I’m often just farting out code trying to get some basic idea working and checking whether my assumptions about how things should work are more-or-less correct.

                                                                                            My favorite point that the author makes. In my experience, especially on high-velocity teams trying out lots of stuff, the exact things that make languages like Rust, Haskell, and others so good at their domains make them liabilities. If your language requires a strong and carefully-expressed model of the business rules and business logic in order to do anything, until you have that model you will do nothing.


                                                                                            Overall, a neat article, and I’m glad to hear some skepticism.

                                                                                            1. 29

                                                                                              Why Twitter didn’t go down … yet

                                                                                              I was hoping for some insights into the failure modes and timelines to expect from losing so many staff.

                                                                                              This thread has some interesting peeks into some of the infrastructure underneath Mesos / Aurora.

                                                                                              1. 12

                                                                                                I also liked this thread a lot:

                                                                                                And yesterday it was possible to post entire movies (in few-minute snippets) in Twitter, because the copyright enforcement systems were broken.

                                                                                                1. 5

                                                                                                  That tweet got deleted. At this point it’s probably better to archive them and post links of that.

                                                                                                  1. 11

                                                                                                    It wasn’t deleted - there’s an ongoing problem over the last few days where the first tweet of a thread doesn’t load on the thread view page. The original text of the linked tweet is this:

                                                                                                    I’ve seen a lot of people asking “why does everyone think Twitter is doomed?”

                                                                                                    As an SRE and sysadmin with 10+ years of industry experience, I wanted to write up a few scenarios that are real threats to the integrity of the bird site over the coming weeks.

                                                                                                    1. 12

                                                                                                      It wasn’t deleted - there’s an ongoing problem over the last few days where the first tweet of a thread doesn’t load on the thread view page.

                                                                                                      It’s been a problem over the last few weeks at least. Just refresh the page a few times and you should eventually see the tweet. Rather than the whole site going down at once, I expect these kinds of weird problems will start to appear and degrade Twitter slowly over time. Major props to their former infrastructure engineers/SREs for making the site resilient to the layoffs/firings though!

                                                                                                      1. 2

                                                                                                        Not only to the infra/SREs but also to the backend engineers. Much of the built-in fault-tolerance of the stack was created by them.

                                                                                                    2. 2


                                                                                                      I have this URL archived too, but it seems to still be working.

                                                                                                      1. 1

                                                                                                        hm, most likely someone would have a mastodon bridge following these accounts RT-ing :-)

                                                                                                      2. 2

                                                                                                        FWIW, I just tried to get my Twitter archive downloaded and I never received an SMS from the SMS verifier. I switched to verify by email and it went instantly. I also still haven’t received the archive itself. God knows how long that queue is…

                                                                                                        1. 2

                                                                                                          I think it took about 2 or 3 days for my archive to arrive last week.

                                                                                                      3. 2

                                                                                                        oh, so they still run mesos? thought everyone had by now switched to k8s…

                                                                                                        1. 13

                                                                                                          I used to help run a fairly decent sized Mesos cluster – I think at our pre-AWS peak we were around 90-130 physical nodes.

                                                                                                          It was great! It was the definition of infrastructure that “just ticked along”. So it got neglected, and people forgot about how to properly manage it. It just kept on keeping on with minimal to almost no oversight for many months while we got distracted with “business priorities”, and we all kinda forgot it was a thing.

                                                                                                          Then one day one of our aggregator switches flaked out and all of a sudden our nice cluster ended up partitioned … two, or three ways? It’s been years, so the details are fuzzy, but I do remember

                                                                                                          • some stuff that was running still ran – but if you had dependencies on the other end of the partition there was lots of systems failing health checks & trying to get replacements to spin up
                                                                                                          • Zookeeper couldn’t establish a quorum and refused to elect a new leader so Mesos master went unavailable, meaning you didn’t get to schedule new jobs
                                                                                                          • a whole bunch of business critical batch processes wouldn’t start
                                                                                                          • we all ran around like madmen trying to figure out who knew enough about this cluster to fix it

                                                                                                          It was a very painful lesson. As someone on one of these twitter threads posted, “asking ‘why hasn’t Twitter gone down yet?’ is like shooting the pilot and then saying they weren’t needed because the plane hasn’t crashed yet”.

                                                                                                          1. 8

                                                                                                            Twitter is well beyond the scale where k8s is a plausible option.

                                                                                                            1. 2

                                                                                                              I wonder what is the largest company that primarily runs on k8s. The biggest I can think of is Target.

                                                                                                              1. 3

                                                                                                                There’s no limit to the size of company that can run on kube if you can run things across multiple clusters. The problem comes if you routinely have clusters get big rather than staying small.

                                                                                                                1. 1

                                                                                                                  Alibaba, probably.

                                                                                                                  1. 1

                                                                                                                    Oh, I didn’t realize that was their main platform.

                                                                                                                  2. 1
                                                                                                                    1. 2

                                                                                                                      I was thinking about that too, but I’m guessing that CFA has a fraction of the traffic of Target (especially this time of year). Love those sandwiches though…

                                                                                                                2. 2

                                                                                                                  Had they done so, I bet they’d already be down :D

                                                                                                                  1. 1

                                                                                                                    I work at a shop with about 1k containers being managed by mesos and it is a breath of fresh air after having been forced to use k8s. There is so much less cognitive overhead to diagnosing operational issues. That said, I think any mesos ecosystem will be only as good as the tooling written around it. Setting up load balancing, for instance . . . just as easy to get wrong as right.

                                                                                                                1. 3

                                                                                                                  Having been a user of VRChat going back some years, I’m really bummed that the development story is so tightly-bound to Unity. I get it, right, but it’s still a disappointment.

                                                                                                                  1. 2

                                                                                                                    A fairly old version too at this point (2019.4.31f1). I’m not familiar with the process of upgrading Unity versions for games but at this point it’s likely they will never be able to upgrade and that’s really going to hurt.

                                                                                                                  1. 2

                                                                                                                    Flagged, please don’t use Lobsters as your marketing channel.

                                                                                                                    1. 18

                                                                                                                      Todd said, “But that’s all in their heads!” and I agreed. Overwhelmed is a feeling, not a physical problem.

                                                                                                                      This. One thing I’ve learned from dealing with neurodivergent people, including myself, is that just because something only exists in your head doesn’t mean it’s not real. This applies to neurotypical people too! It’s just less noticeable because people are used to dealing with it a certain way. Once you accept this as a fact, no matter how silly or trivial something might look from the outside of someone’s head, then you can start actually addressing the problem.

                                                                                                                      1. 7

                                                                                                                        It’s just less noticeable

                                                                                                                        Is this even true? “it’s all in your head” is massively dismissive, no matter who you are. In fact, everything is “in your head” since we can’t even prove the physical world exists! What is in a person’s head is arguably the most real and the most noticeable to them

                                                                                                                        1. 3

                                                                                                                          is that just because something only exists in your head doesn’t mean it’s not real.

                                                                                                                          This line of thinking as written I believe to be wrong and to be a large part of the issues we have as a civilization (arguably, species).

                                                                                                                          An amended version I’m 100% fine with would be:

                                                                                                                          is that just because something only exists in your head doesn’t mean it’s not real to you.

                                                                                                                          It’s incredibly important to have empathy and a theory of mind that allows for interacting with irrational and insane people. It’s incredibly problematic to let others essentially dictate reality on their own whims, especially when there are more than one persons involved.

                                                                                                                          Sometimes, there is objectively a problem. Sometimes, there is a misinterpretation of an objective reality. Sometimes, the problem is rooted in an interpretation of nonreality. The techniques for addressing problems in these different categories are severely limited if we must take everything as actual reality–if you’ve ever dealt with a narcissist, schizophrenic, or borderline person, this should be obvious. For example, many of the successful techniques for dealing with anxiety require that you do not assume the brain/self has a correct view of reality.

                                                                                                                          If you’re ever in a management position, there are many cases where you’ll have to deal (as the author’s friend did) with folks who “feel” a certain way that is incongruous with facts on the ground. Burnout is one such circumstance, imposter syndrome another, plain old grudges and paranoia a third. You cannot effectively manage people if you automatically assume that what they’re telling you is the unvarnished truth–you can only assume that they believe those things to be true (and in pathological cases, you can’t even reliably assume that).

                                                                                                                        1. 5

                                                                                                                          Please slow down on the sourcegraph spam. Lobsters is not your marketing channel.

                                                                                                                          1. 5

                                                                                                                            I see what you mean, but in that instance, the article was actually useful to me. (I don’t use or work for Sourcegraph.)

                                                                                                                            1. 1

                                                                                                                              For sure my bad.

                                                                                                                              1. 4

                                                                                                                                To be clear, the problem is not the number of submissions pushing your own stuff, it is the ratio between those and your other contributions. You have submitted more stories than comments and all except one of your stories has been sourcegraph (the other one is currently sitting at -2). No one cares if active contributors to the community plug their own stuff, people care if people just treat this place as a marketing channel. Join the discussion on other topics, submit interesting things you read elsewhere and no one will complain when you submit things like this.

                                                                                                                                The rule of thumb I was told was that no more than 10% of your contributions should be self-promotion. I’d qualify that slightly and suggest that one-line comments don’t count towards the other 90%. Spend some time thinking about how your unique perspective can enrich other discussions.

                                                                                                                                1. 2

                                                                                                                                  Starting yesterday, I’m going to be a better community member and not treat Lobsters like an open mic night.

                                                                                                                                  Spend some time thinking about how your unique perspective can enrich other discussions.

                                                                                                                                  Will do. I want to apologize to you, @friendlysock, and the rest of the community for my behavior, it was unacceptable and I’ll do better going forward.

                                                                                                                            1. 35

                                                                                                                              I flag those all with extreme prejudice.

                                                                                                                              Once normalized, they attract the worst sort of circlejerking and navelgazing, because everybody has an opinion about them and few people will learn something “new” technically. Social media stuff, especially the monetized kind, tends to function off of network effects and pulling people in to add to the collective. I don’t like feeding Moloch here in such a way.

                                                                                                                              Posts about the implementation of social media, of course, are something I totally dig. Things like dealing with fan-in/fan-out, subscriptions, realtime analytics, how X solved production problem Y at Z scale…all that stuff is great. I just can’t abide by the non-technical stuff that tends to accumulate like plaque from sugary foods.

                                                                                                                              Just because something is interesting to you or you have an opinion about it doesn’t automatically make for a good Lobsters submission.

                                                                                                                              1. 10

                                                                                                                                This. I come to lobsters to learn cool things, not ponder the social issues of the day.

                                                                                                                                1. 4

                                                                                                                                  Maybe social issues are part of technical issues?

                                                                                                                                  1. 2

                                                                                                                                    If there are interesting technical solutions to social issues, I’m all for it. But I have less than zero interest in Twitter drama, Mastodon evangelizing, or any other non-technical discussion of social issues when I come here.

                                                                                                                                    I care deeply about many social issues but I don’t come to lobsters to discuss them, and I don’t want to.

                                                                                                                                2. 6


                                                                                                                                  From Cambridge Dictionary

                                                                                                                                  the activity of spending too much time considering your own thoughts, feelings, or problems

                                                                                                                                  At least I learned a new word :)

                                                                                                                                1. 6

                                                                                                                                  This is the tweet that I believe prompted the blog post.

                                                                                                                                  I was one of the voices saying that I wish people had some other go to datastructure than linked lists for learning a new language.

                                                                                                                                  Mainly because I love Rust (especially the borrow checker), and I hate the idea people get turned off the language due to ownership rules messing with their first attempt coding something.

                                                                                                                                  1. 8

                                                                                                                                    Now I’m wondering how many C books and tutorials include buggy linked lists with behavior worse than O(n) append.

                                                                                                                                    There are some pretty bad C books, and even good ones do have errors.

                                                                                                                                    1. 3

                                                                                                                                      Thanks for sharing that link. I fear I have permanently lost some coding iq from stepping through the (mentally corrected) version of the code sample from that page.

                                                                                                                                    2. 7

                                                                                                                                      messing with their first attempt coding something.

                                                                                                                                      It’s totally fine if Rust is not a good choice for somebody’s first language. C++ is a horrible first language, as are several others.

                                                                                                                                      1. 2

                                                                                                                                        Yeah. I meant “coding something in Rust”.

                                                                                                                                        Wheter Rust is a good first language, I don’t know. I think it could be, because there’s plenty of stuff you can do without being even close to “fight the borrow checker”.

                                                                                                                                        Maybe it comes down to whether teaching pass by reference vs value is something that’s OK to learn early or not.

                                                                                                                                      2. 3

                                                                                                                                        The borrow checker vs linked lists conflict is so unfortunate. I wonder if borrow checking hasn’t been done before, because every time a language researcher considered such design they’ve thought “it won’t even work on linked lists, obviously a dead-end idea”.

                                                                                                                                        1. 6

                                                                                                                                          I wonder if borrow checking hasn’t been done before, because every time a language researcher considered such design they’ve thought “it won’t even work on linked lists, obviously a dead-end idea”.

                                                                                                                                          I think the historical record (read: breadcrumbs through the literature) suggests otherwise; there’s been continual progress on this kind of thing stretching back 30 years (some original type theory work in the early 90’s, some work on applying it to systems programming in the context of Cyclone in the early 00’s, various research-grade projects, and Rust first showed up on the scene in 2010 but it takes a while for a language to become mature). I think the reason it hasn’t been done before is because it wasn’t actually trivial figuring out how, and it took a while to get there.

                                                                                                                                          1. 4

                                                                                                                                            I wonder if borrow checking hasn’t been done before, because every time a language researcher considered such design they’ve thought “it won’t even work on linked lists, obviously a dead-end idea”

                                                                                                                                            This makes me wonder if a prerequisite for Rust’s creation was a critical mass of people who all hold the opinion “eh, linked lists suck anyway, no big deal”.

                                                                                                                                            1. 10

                                                                                                                                              prerequisite for Rust’s creation was a critical mass of people who all hold the opinion “eh, linked lists suck anyway, no big deal”.

                                                                                                                                              I don’t know why this is how people think about rust. For me, as a low-level dev that does tons of crazy stuff that isn’t necessarily borrowck-friendly, I just figured “free safety checks when/where possible” then back to unsafe when I can’t. It’s not like you lose anything compared to before, you just don’t get to take advantage of the safety guarantees the language has to offer as much as you would like to at all times.

                                                                                                                                              (I also don’t end up reaching for unsafe as often as I thought I would.)

                                                                                                                                              1. 13

                                                                                                                                                The Rust community is pretty big now, and opinions on unsafe vary a lot. Some people write really twisted or inefficient code just to avoid unsafe {}. I guess it depends whether you see Rust as safer C, or faster Python.

                                                                                                                                                1. 1

                                                                                                                                                  I’ll bite my tongue here and refrain from saying anything other than I agree with you.

                                                                                                                                                2. 3

                                                                                                                                                  It’s not like you lose anything compared to before,

                                                                                                                                                  Ah, this is a very good way to look at it

                                                                                                                                            2. 2

                                                                                                                                              Anyone know what Rust blog post antirez was reading?