1. 2

    But, perhaps your problem is CPU bound. In that case — you still shouldn’t use threads! Consider using multiple processes instead. A good model is an overseer program, which organizes the work to be done and aggregates the results, spawning worker programs to run the actual computations.

    From extensive experience trying to wrangle services written this way (Ruby/Unicorn, etc.) I can state that it is emphatically not a good model. Balkanizing your application’s logic and state across process boundaries to meet performance requirements is like a single-host version of the microservices fallacy: the knock-on costs to overall system coherence, as well as the impact to secondary concerns like observability, dominate whatever benefit you think the approach gives you.

    Bluntly: a single process should always be able to saturate whatever the physical bottleneck is on your system for your use case. If your programming language or runtime or design model doesn’t allow for this, fix it or pick a different one, don’t work around it.

    1. 2

      This account has never commented and only posted links to their own blog. This is the definition of a spammer. Can an admin please boot them?

      1. 12

        0x6. I like monospaced fonts.

        Unfortunately, I learned about kerning and kerning is impossible to do even decently with monospace fonts.

        1. 6

          Kerning is useless for monospaced fonts, almost by definition.

          Kerning is so that combinations like “AV” don’t have a wide space between them. AV will have that, because the horisontal space taken up by each character is the same.

          1. 3

            There are advantages to kerning, and you miss out on them with monospaced fonts. Obviously you gain other benefits while writing code with monospaced fonts, but fit prose? Not so clear.

          2. 5

            Ditto. Maybe it’s just me, but I find it very easy to lose my place when reading monospace text.

            1. 3

              So don’t do kerning? Not sure if there are any readability studies or something that you’re thinking about but as a programmer I am also happy to read articles in monospaced font.

              1. 17

                Monospaced fonts make prose objectively harder to read. They’re an inappropriate choice for body text, unless you’re trying to make a specific e.g. stylistic statement.

                1. 1

                  Do you have any links for some studies about it? I’m wondering since you’ve used the objectively term, which I find confusing, since I’m not impacted by monospaced formatting at all. Film scripts are being written in monospaced script, books were written in it (at least in the manual typewriter days), I think this wouldn’t be the case if monospaced fonts would be objectively harder to read?

                  1. 4

                    Do you have any links for some studies about it? I’m wondering since you’ve used the objectively term, which I find confusing, since I’m not impacted by monospaced formatting at all.

                    This is a subject that has been studied for a long time. A quick search turned up Typeface features and legibility research, but there is a lot more out there on this topic.

                    The late Bill Hill at Microsoft has a range of interesting videos on ClearType.

                    1. 2

                      Your first link was fascinating, thanks!

                    2. 1

                      Manuscripts and drafts are not the end product of a screenplay or a book. They’re specialized products intended for specialized audiences.

                      There are no works intended for a mainstream audience that are set in a monospaced typeface that I know of. If a significant proportion of the population found it easier to read monospaced, that market would be addressed - for example, in primary education.

                      1. 1

                        Market could prefer variable width fonts because monospaced are wider, thus impacting the space that is taken by the text, which in turn impacts production cost. This alone could have more weight for market preference than the actual ease of reading. Bigger text compression that is achieved by using variable width could improve speed of reading by healthy individuals, but that isn’t so obvious for people with vision disability.

                        Individuals with central loss might be expected to read fixed-pitch fonts more easily owing to the greater susceptibility of crowding effects of the eccentric retina with which they must read. On the other hand, their difficulty in making fixative eye movements in reading should favor the greater compression of variable pitch. Other low-vision patients, reading highly magnified text, might benefit from the increased positional certainty of characters of fixed pitch. Our preliminary results with individuals with macular disease show fixed pitch to be far more readable for most subjects at the character size at which they read most comfortably. (“Reading with fixed and variable character pitch”: Arditi, Knoblauch, Grunwald)

                        Since at least some research papers attribute superiority of variable width font to the horizontal compression of the text – which positively influences the reading speed and doesn’t require as many eye movements – I’m wondering if the ‘readability’ of monospaced typefaces can be improved with clever kerning instead of changing the actual width of the letters.

                        The reading time (Task 1) with the variable-matrix character design was 69.1 s on the average, and the mean reading time with the fixed-matrix character set was 73.3 s, t (8) = 2.76, p < 0.02. The difference is 4.2 s or 6.1% (related to fixed-matrix characters). (“RESEARCH NOTE Fixed versus Variable Letter Width for Televised Text”: Beldie, Pastoor, Schwarz)

                        The excerpt from the paper above suggests that the superiority of variable width vs monospaced isn’t as crushing as one could think when reading that human preference for variable width is an “objective truth”.

                        Also, the question was if monospaced fonts are really harder to read than variable fonts, not if monospaced fonts are easier to read. I think there are no meaningful differences between both styles.

                        1. 1

                          Market could prefer variable width fonts because monospaced are wider, thus impacting the space that is taken by the text, which in turn impacts production cost.

                          So it’s more readable and costs less? No wonder monospaced fonts lose out.

                          I’d love to read the paper you’ve referenced, but cannot find a link in your comment.

                          1. 1

                            So it’s more readable and costs less? No wonder monospaced fonts lose out.

                            Low quality trolling.

                            I’d love to read the paper you’ve referenced, but cannot find a link in your comment.

                            They could be paywalled. I’ve provided the name of papers plus authors, everyone should be able to find them on the internet.

                            1. 2

                              Low quality trolling.

                              What?! I put a lot of effort into my trolling!

                              (To be honest: you’re right and I apologize. It was a cheap shot).

                              I found the first paper (https://www.ncbi.nlm.nih.gov/pubmed/2231111), and while I didn’t read it all I found a link to a font that’s designed to be easier to read for people who suffer from macular degeneration (like my wife). The font (Maxular), shares some design cues from monospaces fonts, but critically, wide characters (like m and w) are wider than narrow ones, like i.

                              That’s what I think is a big problem with monospaced fonts, at small sizes characters like m and w get very compressed and are hard to distinguish.

                  2. 5

                    I also tried to code with a variable width font. It works ok with Lisp code but not the others. The tricky part is aligning stuff. You need elastic tabstops.

                    1. 1

                      Oh, wow. That’s a cool idea. Yeah, that might be enough.

                      1. 1

                        Very cool idea, but that means using actual tabs in a file, and I know a lot of programmers who hate tabs in files.

                        1. 1

                          Good point. I think the cases when different sized tabs would cause problems should also cause problems with a system like this.

                  1. 4

                    To me it seems the most logical solution is to have a convention where all libraries that make backwards incompatible changes just change name.

                    sqlite3 is a perfect example of this, nobody worries or cares about sqlite1 or sqlite2. The system worked perfectly with no special support from the language.

                    The version suffix stuff being built into the tool seemed like a pointless complication to me.

                    1. 5

                      The version suffix stuff being built into the tool seemed like a pointless complication to me.

                      Yes.

                      Semantic Import Versioning is not only a complication, but actively user-hostile, in many circumstances.

                      1. 1

                        Also strong agree with this from me. I had a great time reading all design docs about modules when they we’re first happening and was and still am largely excited about them and their design approach. My main disappointment is SIV and some of the tooling changes and I think they are partially related. I thought maybe i’d get used to SIV but i’m less convinced then ever. I suspect SIV is responsible for the majority of significant hiccups and bugs people run into when adopting and understanding modules.

                        Adopting SIV seems more painful then the convention of just changing the import path. Additionally, I feel that the convention of changing your import path on major upgrades would result in more semver compliance from module authors then the world of SIV.

                        Honestly, I’d just stay on v0/v1 forever with maybe minor breakages depending on the scope of the project and userbase. Then just make a new import path if I need to make significant breaking changes.

                      1. 7

                        That’s not what unit tests are. Once you’re testing something else, you need a different specific test for that, that tests that other thing properly. If you’re testing for B or ‘other stuff’ while you’re testing for A you decrease accuracy of testing A and you’re not explicitly and thoroughly testing B and definitely not ‘other stuff’.

                        1. 2

                          Different philosophies of unit testing I’m afraid! I don’t subscribe to that view at all. My problem with the approach you describe for testing is that a) it’s incredibly laborious to write tests function-by-function in this way and b) the benefit of tests in this style as an aid-to-correctness is much reduced because testing a function without reference to what it interacts with massively reduces the value of the test. As a result I’m happy to do database work in a unit test, I’m happy to involve multiple system components, etc, etc - pretty much touch anything except third party services.

                          I do try to address this in the first paragraph of the article - mostly because I want to quickly move on from debates on what constitutes and “unit” test vs an “integration” test as I don’t think that ends up being very valuable, but since we’re on it anyway: my opinion is that taking a reductive view of the word “unit” in unit test is not helpful.

                          1. 7

                            Different philosophies of unit testing I’m afraid

                            When people talk about the differences between unit and integration testing being partly philosophical, usually they are referring to the fact that an aggregation of several functions or classes can still be considered a “unit” from a testing point of view, so there is no clear cut-off at which a complex unit is no longer a unit.

                            Unit testing is not the only type of testing, and of course there is no agreed definition of what exactly unit testing is. However, what you are doing (deliberately integrating complex systems and also introducing unknown state) is contrary to the idea of what many people consider to be a unit test. Common ideas about what a unit test is include:

                            • “small” - doesn’t try to test everything, just a single “unit”
                            • “isolated” - we try to remove the unit from any external influences besides the test input.

                            This is not to say that your testing methods are necessarily wrong: I think ultimately, this is a language issue. You are free to define words however you like, but if your definition flies in the face of conventional usage, you’re just going to make things difficult for yourself. If you talk about what you are doing as an integration test or some sort of ad hoc fuzzing, I think you might have an easier job discussing the ideas themselves, rather than having people get hung up on whether they are really unit tests.

                            1. 3

                              Including dirty data is definitely well outside of most people’s definition of a unit test and I’m fine with that and perhaps should have been more explicit about it. However I think including first-party collaborators is not - though since unit testing gained popularity there are now more microservices about and I think what was previously a collaborator class/object is now often a collaborator microservice.

                              I disagree that the (IMO) over-reductive view that a unit test should only cover a single function or class (analogous to the “Mockist” approach described in Martin Fowler’s article) is in complete common parlance and I think that’s why “that’s not a (unit|integration|system) test, that’s a (unit|integration|system) test!” is such a common complaint. I think a large chunk of people would accept that a unit test should include at least some collaborators - and most of those people would include the database in that, including the creator of Ruby on Rails.

                              1. 5

                                There is no authoritative definition of a unit test, but I’m confident that very few definitions have a scope large enough to include actual database access.

                                The definition I’m most fond of is: cloning your repository and running the unit tests should always work, and shouldn’t require anything other than the language tool chain. No internet access, no implicit dependencies on local services, no Docker commands, etc.

                                1. 3

                                  This.

                                  Additionally, the “who has claim to the term ‘unit test’” argument is beside the point. Whatever terms you use, there is an important conceptual distinction between the kinds of tests that have no outside dependencies (unless they’re mocked) and the kinds of tests that do. Just stamping your foot and calling the latter “unit tests” doesn’t magically give them the same design and behavioral properties as the former, and those properties are the reason we write those kinds of tests.

                                  Of course, you are free to make the case against the value of those kinds of tests, but that has not been done persuasively in this article.

                                  1. 2

                                    While that’s it’s rare for unit tests to include the database, it’s not universal. Rails devs often consider tests that involve models, which save and read from the database, as unit tests.

                          1. 3

                            Generally very good set of rules. A couple of exceptions, at least in part.

                            Avoid defining variables used only once

                            If a function takes a few parameters, especially when they’re repeated primitive types, it’s often extremely useful to use single-use variables to disambiguate. I’d generally prefer to see

                            var (
                              cfg   = x.GetConfig()
                              addr  = config.Address(id)
                              level = y.GetLevel()
                              retry = config.ShouldRetry(z)
                            )
                            return other.NewThing(addr, level, retry)
                            

                            than

                            cfg := x.GetConfig()
                            return other.NewThing(cfg.Address(id), y.GetLevel(), cfg.ShouldRetry(z))
                            

                            especially as the set of intermediate values grows.

                            Prefer github.com/pkg/errors#Wrap to stdlib errors wrapping

                            IMO, the rationale isn’t convincing enough to justify the dependency.

                            1. 3

                              I have a mixed feelings about pkg/errors vs stdlib; I switched my app to stdlib a few weeks ago, and I just woke up to have 24 of these errors in my mailbox:

                              Session.GetOrCreate: context deadline exceeded
                              

                              There’s only one place this can logically occur in the function, but I kinda miss the stack trace to confirm that this is really the right location, and also because right now I’m not sure if this happens from the a HTTP call or a cron for example. I know this is fixable with better error wrapping context, but it’s easy to forget and sometimes also tricky to get right, as you don’t want to add too much context as that will just lead to duplicate info in a long error string.

                              I also miss Wrap() returning nil on nil errors; it removes the need for a lot of if err != nil checks at the end of functions.

                              I already wrap the errors package to add Wrap() and Wrapf(), and I think I’ll add stack traces too. Either way, I think pkg/errors still has some value.

                              1. 1

                                you don’t want to add too much context as that will just lead to duplicate info in a long error string

                                IME, never been an issue, and strictly preferable to automated (file:line) stack traces.

                                1. 1

                                  I’ve seen it quite a few times with stuff like:

                                  _, err := os.Open(file)
                                  if err != nil {
                                      return fmt.Errorf("somefun %q: %w", file, err)
                                  }
                                  

                                  Which, in Go 1.14, produces:

                                  somefun "/file": open /file: no such file or directory
                                  

                                  There are some other functions as well. The standard library is better about this now compared to a few years ago, but in the past it didn’t always add useful context like the filename or whatnot. With 3rd party libraries you never know what they do (return err is still a common pattern, in spite of many people’s best efforts to convince Gophers that blindly returning errors is not a good idea), so you actually need to check.

                                  It’s all just a certain amount of cognitive overhead that I’d rather spend on actually solving the issue at hand. It kind of reminds me of C and the like where you need to worry about finicky memory management details. It’s not unworkable or anything, but I think it could be improved (we’ll see what happens with the “Go 2” proposals, although they don’t really solve this specific issue IIRC, and arguably even makes it worse).

                              2. 1

                                IMO, the rationale isn’t convincing enough to justify the dependency.

                                What makes a dependency justified to you? To me it’s simple deep interfaces. SQL, unix, etc. pkg/errors while not deep, is very very simple. Wrapf is an intern project at best.

                                1. 1

                                  What makes a dependency justified to you?

                                  In your terms, the dependency must be good (i.e. simple, or in John Ousterhout terms, narrow) and substantial (i.e. deep). If it’s substantial but not good, it’s a non-starter. If it’s good but not substantial, I’ll copy the functionality into my project.

                                  1. 1

                                    I’ll copy the functionality into my project.

                                    That sounds like vendoring with extra steps? If you copy/rewrite while staring at the code, you’re already taking on the IP burden, why not just copy the dependency, and add patches where necessary?

                                    1. 1

                                      Because a little copying is better than a little dependency.

                              1. 8

                                Debian. The OS/distro is not the correct layer for experimentation.

                                1. 5

                                  Fellow Debian user here. I think there is room for experimentation at the distro and even OS levels, but … not at my day job, and not on my personal machines until I retire.

                                1. 4

                                  I was half expecting a 40-character line length limit.

                                  1. 2

                                    Why?

                                    1. 6

                                      50% off

                                  1. 29

                                    The 6-week release cycle is a red herring. If Rust didn’t have 6-week cycles, it would have bigger annual releases instead, but that has no influence on the average progress of the language.

                                    It’s like slicing the same pizza in either 4 or 42 slices, but complaining “oh no, I can’t eat 42 slices of pizza!”

                                    Rust could have had just 4 releases in its history: 2015 (1.0), 2016 (? for errors), 2018 (new modules) and 2020 (async/await), and you would call them reasonably sized, each with 1 major feature, and a bunch of minor standard library additions.

                                    Async/await is one major idiom-changing feature since 2015 that actually caused churn (IMHO totally worth it). Apart from that there have been only a couple of syntax changes, and you can apply them automatically with cargo fix or rerast.

                                    1. 17

                                      It’s like slicing the same pizza in either 4 or 42 slices, but complaining “oh no, I can’t eat 42 slices of pizza!”

                                      It’s like getting one slice of pizza every 15 minutes, while you’re trying to focus. I like pizza, but I don’t want to be interrupted with pizza 4 times. Being interrupted 42 times is worse.

                                      Timing matters. Treadmills aren’t fun as a user.

                                      1. 13

                                        Go releases more frequently than Rust, and I don’t see anyone complaining about that. Go had 121 releases, while Rust less than half of that.

                                        The difference is that Go calls some releases minor, so people don’t count them. Rust could do the same, because most Rust releases are very minor. If it had Go’s versioning scheme it’d be on something like v1.6.

                                        1. 20

                                          People aren’t complaining about the frequency of Go releases because Go doesn’t change major aspects of the language, well, ever. The most you have to reckon with is an addition to the standard library. And this is a virtue.

                                          1. 8

                                            So, what major aspects of the language changed since Rust 1.0, besides async and perhaps the introduction of the ? operator?

                                            1. 10

                                              The stability issues are more with the Rust ecosystem than the Rust language itself. People get pulled into fads and then burned when they pay the refactoring costs to move to the next one. Many of those fad crates are frameworks that impose severe workflow constraints.

                                              Go is generally far more coherent as an overall ecosystem. This was always the intent. Rust is not so opinionated and structured. This leads to benefits and issues. Lots of weird power plays where people write frameworks to run other people’s code that would just be blatantly unnecessary in Go. It’s unnecessary in Rust, too, but people are in a bit of daze due to the complexity flying around them and it’s sometimes not so clear that they can just rely on the standard library for a lot of things without pulling in a stack of 700 dependencies to write an echo server.

                                              1. 2

                                                Maybe in the server/web part of the ecosystem. I am mostly using Rust for NLP/ML/data massaging and the ecosystem has been very stable.

                                                I have also use Go for several years, but I didn’t notice much difference in the volatility.

                                                But I can imagine that it is different for networking/services, because the Go standard library has set strong standards there.

                                              2. 6

                                                Modules have changed a bit, but it was optional change and only required running cargo fix once.

                                                Way less disruptive than GOPATH -> go modules migration.

                                            2. 5

                                              That is kind of the point. I love both Go and Rust (if anything, I’d say I’d like Rust more than Go if working out borrow checker issues wasn’t such a painstaking, slow process), but with Go I can go and update the compiler knowing code I wrote two years ago will compile and no major libraries will start complaining. With Rust, not so much. Even in the very short time I was using it for a small project, I had to change half of my code to use async (and find a runtime for that, etc.) because a single library I wanted to use was ‘async or the highway’.

                                              Not a very friendly experience, which is a shame because the language itself rocks.

                                              1. 9

                                                In Rust you can upgrade the compiler and nothing will break. Rust team literally compiles all known Rust libraries before making a new release to ensure they don’t break stuff.

                                                The ecosystem is serious about adherence to semver, and the compiler can seamlessly mix new and old Rust code, so you can be selective of what you upgrade. My projects that were written for Rust 1.0.0 work fine with the latest compiler.

                                                The async addition was the only change which caused churn in the ecosystem, and Rust isn’t planning anything that big in the future.

                                                And Go isn’t flawless either. I can’t upgrade deps in my last Go project, because migration to Go Modules is causing me headaches.

                                                1. 3

                                                  Ah, yeah, the migration to modules was a shit show. It took me about six months to be able to move a project to modules because a bunch of the dependencies took a while to upgrade.

                                                  Don’t get me wrong, my post wasn’t a criticism of my Rust. As I said, I really enjoy the language. But any kind of big changes like async and so on introduce big paradigm shifts that make the experience extra hard for newcomers. To be fair to Rust, Python took 3 iterations or so until they figured out a proper interface for async, while rust figured the interface and left the implementation to the reader… Which has created another rift for some libraries.

                                          2. 4

                                            I can definitely agree with the author, since I do not write Rust in my day job it is pretty hard for me to keep up with all the minor changes in the language. Also, as already stated in the article, the 6 week release cycle exacerbates the problem.

                                            I’m not famliar with Rust’s situation, but from my own corporate experience, frequent releases can be awful because features are iterated on continuously. It would be really nice to just learn the final copy of something rather than all the intermediate steps to get there.

                                            1. 3

                                              Releasing “final copy” creates design-by-commitee. Features have to get real-world use to prove they’re useful.

                                              There’s a chicken-egg problem here. Even though Rust has nightlies and betas, features are adopted and used in production only after they’re declared stable. But without using a feature for real, you can’t be sure it’s right.

                                              Besides, lots of changes in 6-week releases are tiny, like a new command-line flag, or allowing few more functions to be used as initializers of global variables.

                                              1. 6

                                                Releasing “final copy” creates design-by-commitee. Features have to get real-world use to prove they’re useful.

                                                Design-by-committee can be a lot more thoughtful than design-by-novice. I think this is one of the greatest misonceptions of agile.

                                                Many of the great things we take for granted are done by committee including our internet protocols and core infrastructure. There’s a lot of real good engineering in there. Of course there’s research projects and prototyping which are super useful but it’s a full time job to keep up with developments in research. Most people don’t have to care to learn it until it’s stable and published.

                                                1. 2

                                                  Sorry, I shouldn’t have mentioned an emotionally-charged “commitee” name. It was not the point.

                                                  The point is that language features need iteration to be good, but for a language with strong stability guarantee the first iteration must be the final one.

                                                  So the way around such impossible iteration is release only obvious core parts, so that libraries can iterate on the rest. And the rest is going to be blessed as official only after it proves useful.

                                                  Rust has experience here: the first API of Futures turned out to have flaws. Some interfaces caused unfixable inefficiencies. Built-in faillibility turned out to be more annoying than helpful. These things came out to light only after the design was “done” and people used it for real and built large projects around them. If Rust held that back and waited for the full async/await to be feature-complete, it’d be a worse design, and it wouldn’t have been released yet.

                                                2. 3

                                                  Releasing “final copy” creates design-by-commitee.

                                                  I’m not convinced that design-by-crowd is substantively different from design-by-committee.

                                                  1. 1

                                                    Releasing “final copy” creates design-by-commitee. Features have to get real-world use to prove they’re useful.

                                                    There’s a chicken-egg problem here. Even though Rust has nightlies and betas, features are adopted and used in production only after they’re declared stable. But without using a feature for real, you can’t be sure it’s right.

                                                    I deny the notion that features must be stabilized early so that they get wide spread or “production use.” It may well be the case that some features don’t receive enough testing on nightly/beta and in order to get more users it must hit stable, but limited testing on nightly or beta is not a reason to stabilize a feature. Either A) wait longer until it’s been more thoroughly tested on nightly/beta or B) find a manner to get more testers of features on nightly/beta.

                                                    I’m not necessarily saying that’s what happened with Rust, per se, but it’s close as I’ve seen the sentiment expressed several times over my time with Rust (since 0.9 days).

                                                3. 10

                                                  It’s not a red herring. There might be bigger annual releases if there weren’t 6-week releases, but you’re ignoring the main point: Rust changes frequently enough to make the 6-week release cycle meaningful. The author isn’t suggesting the same frequency of changes less often, but a lower frequency of changes - low enough, perhaps, that releasing every 6 weeks would see a few “releases” go by with no changes at all.

                                                  No one is trying to make fewer slices out of the pizza. They’re asking for a smaller pizza.

                                                  1. 7

                                                    How is adding map_or() as a shorthand for map().unwrap_or() a meaningful language change? That’s the scale of changes for the majority of the 6-week releases. For all but handful of releases the changes are details that you can safely ignore.

                                                    Rust is very diligent with documenting every tiny detail in release notes, so if you don’t pay attention and just gloss over them only counting the number of headings, you’re likely to get a wrong impression of what is actually happening.

                                                    1. 3

                                                      How is adding map_or() as a shorthand for map().unwrap_or() a meaningful language change?

                                                      I think that’s @ddevault’s point: the pizza just got bigger but it didn’t really get better. It’s a minor thing that doesn’t really matter, but it happens often and it’s something you may need to keep track of when you’re working with other people.

                                                      1. 9

                                                        Rust also gets criticised for having too small standard library that needs dependencies for most basic things. And when it finally adds these basic things, that’s bad too…

                                                        But the thing is — and it’s hard to explain to non-users of the language — that additions of things like map_or() is not burdensome at all. From inside, it’s usually received as “finally! What took you so long!?”.

                                                        • First, it follows a naming pattern already used elsewhere. It’s something you’d expect to exist already, not really a new thing. It’s more like a bugfix for “wtf? why is this missing?”.

                                                          Back-filling of outrageously missing features is still a common thing in Rust. 1.0 was an MVP rather than a finished language. For example, Rust waited 32 releases before add big-endian/little-endian swapping.

                                                        • There’s cargo clippy that will flag too unidiomatic code, so you don’t really need to keep track of it.

                                                        • It’s OK to totally ignore this. If your code worked without some new stdlib function, it’ll doesn’t have to care. And these changes are minor, so it’s not like you’ll need to read a book on a new method you notice. You’ll know what it does from it’s name, because Rust is still at the stage of adding baby things.

                                                        1. 7

                                                          In the Haskell world, there’s a piece of folklore called the Fairbairn Threshold, though we have very clean syntax for composing small combinators:

                                                          The Fairbairn threshold is the point at which the effort of looking up or keeping track of the definition is outweighed by the effort of rederiving it or inlining it.

                                                          The term was in much more common use several years ago.

                                                          Adding every variant on every operation to the Prelude is certainly possible given infinite time, but this of course imposes a sort of indexing overhead mentally.

                                                          The primary use of the Fairbairn threshold is as a litmus test to avoid giving names to trivial compositions, as there are a potentially explosive number of them. In particular any method whose definition isn’t much longer than its name (e.g. fooBar = foo . bar) falls below the threshold.

                                                          There are reasonable exceptions for especially common idioms, but it does provide a good rule of thumb.

                                                          The effect is to encourage simple combinators that can be used in multiple situations, while avoiding naming the explosive number of combinations of those combinators.

                                                          Given n combinators I can probably combine two of them in something like O(n^2) ways, so without the threshold as a rule of thumb you wind up with a much larger library, but no real greater utility and much higher cognitive overhead to track all the combinations.

                                                          Further, the existence of some combinations tends to drive you to look for other ever larger combinations rather than learn how to compose combinators or spot the more general usage patterns yourself, so from a POSIWID perspective, the threshold encourages better use of the functional programming style as well.

                                                      2. 1

                                                        Agreed. It has substantially reduced my happiness all around:

                                                        • It’s tiring to deal with people who (sincerely) think adding features improves a language.
                                                        • It’s disappointing that some people act like having no deprecation policy is something that makes a language “stable”/“reliable”/good for business use.
                                                        • It’s mind-boggling to me that the potential cost of removing a feature is never factored into the cost of adding it in the first place.

                                                        Mainstream language design is basically living with a flatmate that is slowly succumbing to his hoarding tendencies and simply doesn’t realize it.

                                                        What I have done to keep my sanity is to …

                                                        • freeze the version of Rust I’m targeting to Rust 1.13 (I’m not using ?, but some dependencies need support for it), and
                                                        • playing with a different approach to language design that makes me happier than just watching the constant mess of more-features-are-better.
                                                        1. 2

                                                          Mainstream language design is basically living with a flatmate that is slowly succumbing to his hoarding tendencies and simply doesn’t realize it.

                                                          I like that analogy, but it omits something crucial: it equates “change” with “additional features/complexity” – but many of the changes to Rust are about removing special cases and reducing complexity.

                                                          For example, it used to be the case that, when implementing a method on an item, you could refer to the item with Self – but only if the item was a struct, not it it was an enum. Rust 1.37 eliminated that restriction, removing one thing for me to remember.

                                                          Other changes have made standard library APIs more consistent, again reducing complexity. For example the Option type has long had a map_or method that calls a function on the Some type or, if the Option contains None, uses a default value. However, until Rust 1.41, you had to remember that Results didn’t have a map_or method (even though they have nearly all the other Option methods). Now, Results have that method too, making the standard library more consistent and simpler.

                                                          I’m not claiming that every change has been a simplification; certainly some have not. (For example, did we really need todo!() as a shorter way to write unimplemented!() when they have exactly the same effect?).

                                                          But some changes have been simplifications. If Rust is a flatmate that is slowly buying more stuff, it’s also a flatmate that’s throwing things out in an effort to maintain a tidy space. Which effect dominates? As a pretty heavy Rust user, my personal feeling is that the language is getting simpler over time, but I don’t have any hard evidence to back that up.

                                                          1. 3

                                                            But some changes have been simplifications.

                                                            I think what you are describing is a language that keeps filling some gaps and oversights, they are probably not the worst kind of additions, but they are additions.

                                                            If Rust is a flatmate that is slowly buying more stuff, it’s also a flatmate that’s throwing things out in an effort to maintain a tidy space.

                                                            What has Rust thrown out? I have trouble coming up with even a single example.

                                                            As a pretty heavy Rust user, my personal feeling is that the language is getting simpler over time, but I don’t have any hard evidence to back that up.

                                                            How would you distinguish between the language getting simpler and you becoming more familiar with the language?

                                                            I think this is the reason why many additions are “small, simple, obvious fixes” to expert users, but for new/occasional users they present a mountain of hundreds of additional things that have to be learned.

                                                            1. 1

                                                              How would you distinguish between the language getting simpler and you becoming more familiar with the language?

                                                              That’s a fair question, and is part of the reason I added the qualification that I can only provide my personal impression – without data, it’s entirely possible that I’m mistaking my own familiarity for language simplification. But I don’t believe that’s the case, for a few reasons.

                                                              I think this is the reason why many additions are “small, simple, obvious fixes” to expert users, but for new/occasional users they present a mountain of hundreds of additional things that have to be learned.

                                                              I’d like to focus on the “additional things” part of what you said, because I think it’s key: if a feature is revised so that it’s consistent with several other features, then that’s one fewer thing for a new user to learn, not one more. For example, match used to treat & a bit differently and require as_ref() method calls to get the same effect, which frequently confused people learning Rust. Now, & works the same with match as it does with the rest of the language. Similarly, the 2015 Edition module system required users to format their paths differently in use statements than elsewhere. Again, that confused new users (and annoyed pretty much everyone) and, again, it’s been replaced with a simpler, more consistent, and easier-to-learn system.

                                                              On the other hand, you might have a point about occasional Rust users – if a user understood the old module system, then switching to the 2018 Edition involves learning something new. For the occasional user, it doesn’t matter that the new system is simpler – it’s still one more thing for them to learn.

                                                              But for a new user, those simplifications really do make the language simpler to pick up. I firmly believe that the current edition of the Rust Book describes a language that is simpler and more approachable – and that has fewer special cases you have to “just remember” – than the version of the language described in the first edition.

                                                              1. 1

                                                                A lot of effort is spent “simplifying” things that “simply” shouldn’t have been added in the first place:

                                                                • do we really need two different kind of use paths (relative and absolute)?
                                                                • do we really need both if expressions and pattern matching?
                                                                • do we really need ? for control flow?
                                                                • do we really need to have two different ways of “invoking” things, (...) for methods (no support for named parameters) and {...} for structs (support for named parameters)?
                                                                • do we really need the ability to write foo for foo: foo in struct initializers?

                                                                Most often the answer is “no”, but we have it anyway because people keep conflating familiarity with simplicity.

                                                                1. 2

                                                                  You’re describing redundancy as if it was some fault, but languages without any redundancy are a turing tarpit. Not only we don’t need two kinds of paths, the whole use statement is unnecessary. We don’t even need if. Smalltalk could live without it. We don’t really need anything more than a lambda and a Y combinator or one instruction.

                                                                  I’ve used Rust v0.5 before it had if let, before there was try!(). It required a full match on every single Option. It was a pure minimal design, and I can tell you it was awful.

                                                                  So yes, we need these things, because convenience is also important.

                                                                  1. 2

                                                                    You’re describing redundancy as if it was some fault, but languages without any redundancy are a turing tarpit.

                                                                    I’m very aware of the turing tarpit, and it simply doesn’t apply here. A lack of redundancy is not the problem – it’s the lack of structure.

                                                                    Not only we don’t need two kinds of paths, the whole use statement is unnecessary. We don’t even need if. Smalltalk could live without it. We don’t really need anything more than a lambda and a Y combinator or one instruction.

                                                                    Reductio ad absurdum? If you think it’s silly to question why we have both if-then-else and match, why not add ternary operators, too?

                                                                    It required a full match on every single Option. It was a pure minimal design, and I can tell you it was awful.

                                                                    Pattern matching on options is pretty much always wrong, regardless of the minimalism of design. I think the only reasons Rust users use it is because it makes the borrow checker happy more easily.

                                                                    I’ve used Rust v0.5 before it had if let, before there was try!(). It required a full match on every single Option. It was a pure minimal design, and I can tell you it was awful.

                                                                    In my experience, the difference in convenience between Rust 5 years ago (which I use for my own projects) and Rust nightly (which is used by some projects I contribute to) just isn’t there.

                                                                    There is no real point in upgrading to a newer version – the only thing I get is a bigger language and I’m not really interested in that.

                                                      3. 1

                                                        This discussion suffers from “Monday morning quarter backing” to an extent. We now (post fact) know which releases of Rust contained more churn than others. “churn” being defined as a change that either introduced a different (usually better IMO) way of doing something already possible in Rust, or a fundamental change that permeated the ecosystem either to due to being the new idiomatic way, or being the Next Big Thing and thus many crates in the ecosystem jumped in early. Either way, my code needs to change due to new warnings (and the ecosystem doesn’t care for warnings) or since many crates are open source I’ll inevitably get a PR to switch to the new hotness.

                                                        With that stated, my actual point is that Rust releases every 6 weeks. I don’t know if the next release (1.43 at the time of this writing) will contain something that produces churn or not without closely following upcoming releases. I don’t know if the release after that will contain big changes. So I’m left with either having to follow all releases (every 6 weeks), or closely follow upcoming releases. Either way I’m forced to stay in tune with Rust development. For many this is fine. However in my industry (Government) where dependencies must go through audit, etc, etc. It’s really hard to keep up with. If Rust had “major” (read churn inducing releases) every year, or say every 3 years (at new editions) that would be far, far easier to keep up with. Because then I don’t need to check every 6 weeks, I can check every year, or three years whatever it may be. Minor changes (stdlib additions, etc.) can still happen every 6 weeks, almost as Z releases (in semver X.Y.Z speak), but churn inducing changes (Y changes) happen on a set much slower schedule.

                                                        1. 2

                                                          When your deps updated to ?, you didn’t need to change anything. When your deps started using SIMD, you didn’t need to change anything. When your deps switched to Edition 2018, you didn’t need to change anything because of that.

                                                          Warnings from libraries are not displayed (cap-lints), so even if you use deprecated stuff, nobody will notice. You could sleep through years of Rust changes and not adopt any of them.

                                                          AFAIK async/await was the first and only language change after Rust 1.0 that massively changed interfaces between crates, causing a necessary ecosystem-wide churn. It was one change in 5 years.

                                                          Releases are backwards compatible, so you really don’t need to pay attention to them. You need to update the compiler to update dependencies, but this doesn’t mean you need to adopt any language changes yourself.

                                                          The pain of going through dependency churn is real. But apart from async, it’s not caused by compiler release cycle. Dependencies won’t stop changing just because the language doesn’t change. Look at JS for example: Node has slow releases with long LTS, the language settled down after ES2016, IE and Safari put breaks on language evolution speed. And yet, everything churns all the time! People invent new frameworks weekly on the same language version.

                                                      1. 3

                                                        Personally I name my machines a 3-letter onomatopoeia. I’m particularly fond of “ugh” for my work laptop.

                                                        Servers get a logical site code, typically the city the datacenter is in, plus an optional integer.

                                                        1. 6

                                                          This totally falls apart when you need to collaborate with, well, anyone.

                                                          1. 2

                                                            Actually it doesn’t. I’ve used this style in multiple organisations starting around mid 90s starting with CVS, later subversion, and now git.

                                                            It was called “Concurrent Version System”, because multiple people could productively work on the same codebase at the same time. We created “branches” (tags you could check out later) only for releases in case we ever needed to apply an emergency bugfix to something we shipped.

                                                          1. 9

                                                            Another thing.

                                                            x, err := strconv.ParseInt("not a number", 10, 32)
                                                            // Forget to check err, no warning
                                                            doSomething(x)
                                                            

                                                            This comes up all the time in critiques of the language. Sure, it’s possible. But, like — I’ve never had to catch this in a code review. In practice it just isn’t something that happens. I dunno. I wish Go had a Result type, too. But this class of error is, in my experience, almost purely theoretical.

                                                            1. 3

                                                              I’ve definitely seen this in the wild, both in FOSS projects and at my company. The upside is that popular go linters do catch this, and depending on your editor, this type of check may be enabled by default.

                                                              That said, I much prefer it when a language disallows cases like this than depending on linters.

                                                            1. 10

                                                              It seems like the author is misconstruing things that are operating-as-intended as “quirks”, because it didn’t fit their initial expectations. Much of these points are non-issues for seasoned Go developers

                                                              Edit: For example, the gripe about time.Duration wouldn’t be a problem if the author checked the documentation, which explicitly states:

                                                              A Duration represents the elapsed time between two instants as an int64 nanosecond count […]

                                                              You can’t really blame the language for something if you didn’t read the documentation…

                                                              1. 22

                                                                It seems far too easy to just handwave things away with “it’s documented”; people make tiny mistakes all the time; it’s normal. Minimize the surface area for these kind of tiny mistakes and facilitating easy learning are explicitly among Go’s goals.

                                                                context.WithTimeout() accepts a time.Duration; it’s not unreasonable to expect that it will reject a simple int value. It’s an (arguably silly) mistake, but also he kind that people make all the time, since you can’t remember everything and it’s an easy one to make. All other things being equal, Go would be better and more robust of the typechecker was more strict about this.

                                                                We could, by the way, use your same “read the documentation”-argument against using any type checking at all: “why did you pass an int to this function? It’s clearly documented it only accepts string!”

                                                                1. 10

                                                                  Much of these points are non-issues for seasoned Go developers

                                                                  What about beginners or people new to the language? To them it may just be, like the author points out, surprising.

                                                                  1. 2

                                                                    Just because something is surprising doesn’t make it a quirk

                                                                    1. 7

                                                                      Doesn’t it? I thought that was pretty much the definition.

                                                                      Quirks aren’t a killer and I don’t think the author is arguing against using Go. But if its behaviour is surprising or unexpected to newcomers then that’s something to be aware of and hopefully improve upon.

                                                                  2. 1

                                                                    Much of these points are non-issues for seasoned Go developers.

                                                                    Go’s goal is to enable inexperienced developers to churn out code fast. So I think what you are arguing is missing the point on Go’s target demographic.

                                                                    Especially when it comes to time, I think Go does not have a good track record, as described in I want off Mr. Golang’s Wild Ride.

                                                                    1. 2

                                                                      Go’s target demographic, experienced or not, still has to read the documentation even if the features are intended to be easy to learn/use.

                                                                      Quirks are peculiarities or minor departures from the intended effect of a feature; it’s fair to argue some of these even constitute bugs, but I don’t think that’s the case in several of the cases addressed in the article.

                                                                      1. 13

                                                                        You seem to be misinterpreting the word “quirks”. The author is talking about design quirks, meaning parts of Go’s design which, while working as designed, is still surprising or awkward in some way. The most extreme case I can think of right now is probably Javascript’s == operator; it does exactly what it was designed to do, but the consequence is that == isn’t a transitive operator, which is really surprising (a “quirk”).

                                                                      2. 1

                                                                        I don’t think that article supports the conclusion you draw from it.

                                                                        1. 0

                                                                          No, the goal of Go was to fix the issues Google had with large C++ project.

                                                                          1. 2

                                                                            The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

                                                                            – Rob Pike

                                                                            1. 4

                                                                              Also Rob Pike:

                                                                              The Go programming language was conceived in late 2007 as an answer to some of the problems we were seeing developing software infrastructure at Google. The computing landscape today is almost unrelated to the environment in which the languages being used, mostly C++, Java, and Python, had been created. The problems introduced by multicore processors, networked systems, massive computation clusters, and the web programming model were being worked around rather than addressed head-on. Moreover, the scale has changed: today’s server programs comprise tens of millions of lines of code, are worked on by hundreds or even thousands of programmers, and are updated literally every day. To make matters worse, build times, even on large compilation clusters, have stretched to many minutes, even hours.

                                                                              Go was designed and developed to make working in this environment more productive. Besides its better-known aspects such as built-in concurrency and garbage collection, Go’s design considerations include rigorous dependency management, the adaptability of software architecture as systems grow, and robustness across the boundaries between components.

                                                                              talk

                                                                              1. 2

                                                                                Where does this quote come from?

                                                                                1. 2

                                                                                  That quote is from the talk From Parallel to Concurrent. He says it at 20:40.

                                                                        1. 17

                                                                          In the docs for http.Transport, . . . you can see that a zero value means that the timeout is infinite, so the connections are never closed. Over time the sockets accumulate and you end up running out of file descriptors.

                                                                          This is definitely not true. You can only bump against this condition if you don’t drain and close http.Reponse.Body you get from http.Clients, but even then, you’ll hit the default MaxIdleConnsPerHost (2) and connections will cycle.

                                                                          Similarly,

                                                                          The solution to [nil maps] in not elegant. It’s defensive programming.

                                                                          No, it’s providing a constructor for the type. The author acknowledges this, and then states

                                                                          nothing prevents the user from initializing their struct with utils.Collections{} and causing a heap of issues down the line

                                                                          but in Go it’s normal and expected that the zero value of a type might not be usable.

                                                                          I don’t know. They’re not bright spots, but spend more time with the language and these things become clear.

                                                                          1. 4

                                                                            If you really want to prevent users of your library from using the {} sntax to create new objects instead of using your constructor, you can choose to not export the struct type & instead export an interface that is used as the return value type of the constructor’s function signature.

                                                                            1. 10

                                                                              You should basically never return interfac values, or export interfaces that will only have one implementation. There are many reasons, but my favourite one is that it needlessly breaks go-to-definition.

                                                                              Instead, try to make the zero value meaningful, and if that’s not possible provide a New constructor and document it. That’s common in the standard library so all Go developers are exposed to the pattern early enough.

                                                                              1. 2

                                                                                Breaking go-to-definition like that is the most annoying thing about Kubernetes libraries.

                                                                              2. 4

                                                                                That would be pretty nonidiomatic.

                                                                                1. 1

                                                                                  Yea this is a good approach sometimes but the indirection can be confusing.

                                                                              1. 5

                                                                                Not sure how much to trust an author who gets the name of the language wrong…

                                                                                1. 1

                                                                                  Unkind flag was me. Honestly, come on. By a couple sentences in, I have a sense that the author’s first language is not english, and by the referenced link at the end (with a sketchy looking URL, but, eh), it’s clear that this was adapted from an article written in Chinese. Also, the author, pxlet, is a member here. Maybe at least point out the correct way the language should be named?

                                                                                  (I honestly don’t even know if you’re correcting the casing to “Golang” or the spelling to “Go”)

                                                                                  I’ve been messing around with Go, and hadn’t come across unsafe yet. From a very quick scan, it doesn’t look like spam or marketing copy to me… Is it incorrect? Is it bad? I honestly would like to know, because I’m curious to give it a read. I’m curious if Go’s unsafe is conceptually similar or different from Rust’s, which I’m more familiar with.

                                                                                  1. 4

                                                                                    The author’s contributions to lobste.rs have been primarily links to their own blog.

                                                                                    1. 2

                                                                                      Worth pointing out, thank you. If I’d clicked through, I would have seen that they have made zero comments in their 11 months on this site, which does change the framing a little.

                                                                                    2. 2

                                                                                      As far as incorrectness goes:

                                                                                      • The section on accessing unexported fields is subtly wrong: it doesn’t account for padding between struct fields. In the example, there happens to be no padding, so it’s fine, but if you change the age field to a boolean, it would not be right (https://play.golang.org/p/enZ9FuEfK1t).

                                                                                      • Both of the usages of reflect.SliceHeader and reflect.StringHeader are incorrect. Because the reflect package defines the pointer fields as type uintptr, they are not recognized as containing pointers to the garbage collector. Since the usage of the bytes/string is dead after creating the bh or sh variables, they could be collected at that moment. (https://play.golang.org/p/_lUYicIFUzW)

                                                                                      There’s a strict set of rules and valid things to do with unsafe documented at https://golang.org/pkg/unsafe/#Pointer and it is very very easy to get them wrong.

                                                                                      1. 1

                                                                                        Thank you!

                                                                                      2. 1

                                                                                        Go is very popular in China, which I attribute to some mix of the strong unicode support the fact that Baidu’s heavy usage of it has spurred many Chinese universities to teach it to undergrads.

                                                                                        Unlike Rust, there’s no ‘unsafe’ keyword in Go; there’s just a package with that name.

                                                                                        1. 3

                                                                                          I guess my point about language was that your comment boiled down to a cheap shot at how they wrote, and one that I think reflects a real, harmful cultural bias against folks whose first language isn’t English. I couldn’t tell if your comment was based on more than the title, and I really would value your opinion of the content as trustworthy or not.

                                                                                          unsafe— thanks yeah. I have gotten that far :) As far the comparison, what I meant was more, given that they both let you be dangerous with pointers, what are the allowances and limits of those abilities, whether designed as part of the language or the standard library. But that’s just my own motivating curiosity about this post.

                                                                                          (ps. I undid my downvote on your comment. I was grumpy this morning, and that action wasn’t called for since I was making a reply to your comment anyway. Sorry.)

                                                                                          1. 1

                                                                                            There’s also no safe in Go. The Go type system does nothing to prevent two goroutines from concurrently modifying the same object. That doesn’t have to be unsafe, but Go also has large value types, and so assigning from one goroutine and reading from the other can result in tearing. That isn’t intrinsically unsafe, except that this also applies to some built-in types, such as slice (unless this has been fixed recently). You can read the base of one slice and the bounds of another, spuriously pass the bounds check, and then access memory out of bounds. If you’re careful, you can avoid this kind of subtle bug in Go, but the language doesn’t help you at all.

                                                                                            1. 1

                                                                                              The language helps by providing channels. But you have to use them.

                                                                                              1. 1

                                                                                                Channels don’t help. If you pass a pointer over a channel, now you and the receiver have pointers to the object and you get into exactly this problem. Design patterns built over channels to guarantee unique ownership may work, but the language doesn’t help.

                                                                                      1. 3

                                                                                        None of the examples in this article appear to be examples of premature optimization. All of them appear to be examples of inappropriate design, or picking the wrong tool (approach) for the job.

                                                                                        Premature optimization means assuming that a reasonable, idiomatic, default way of solving some problem is going to be too slow, and using instead a complex, obscure, nonidiomatic, or otherwise difficult-to-maintain version of that solution without having justified its costs.

                                                                                        (I find that developers who tend toward premature optimization have a high tolerance for [local] complexity, and don’t tend to appreciate, or leverage, consistent high-order system design. This seems to be a personality thing more than anything else. I’m not sure they’re better or worse developers for it, just something I’ve noticed.)

                                                                                        1. 7

                                                                                          Total agree. All of my nontrivial programs now look like this:

                                                                                          func main() {
                                                                                              if err := exec(os.Args[1:], os.Stdin, os.Stdout, os.Stderr); err != nil {
                                                                                                  fmt.Fprintln(os.Stderr, err)
                                                                                                  os.Exit(1)
                                                                                              }
                                                                                          }
                                                                                          
                                                                                          func exec(args []string, stdin io.Reader, stdout, stderr io.Writer) error {
                                                                                              // ...
                                                                                          }
                                                                                          
                                                                                          1. 1

                                                                                            What do you do about flags?

                                                                                            1. 1
                                                                                              1. 1
                                                                                                import (
                                                                                                    "github.com/peterbourgon/ff/v3"
                                                                                                )
                                                                                                
                                                                                                func exec(args []string, ...) error {
                                                                                                    fs := flag.NewFlagSet("myprogram", flag.ContinueOnError)
                                                                                                    var (
                                                                                                        addr = fs.String("addr", "localhost:1234", "listen address")
                                                                                                        // ....
                                                                                                    )
                                                                                                    if err := ff.Parse(fs, args); err != nil {
                                                                                                        return fmt.Errorf("error parsing flags: %w", err)
                                                                                                    }
                                                                                                    
                                                                                                    // ...
                                                                                                
                                                                                            1. 7

                                                                                              It’s nice that more people are leaning into deterministic simulation while building correctness-critical distributed systems.

                                                                                              It’s not clear to me what they mean by strong consistency for a system that replicates multiple concurrent uncoordinated modifications on different devices, it would be nice if they went into that claim a bit more.

                                                                                              1. 7

                                                                                                yeah, the deterministic simulation is my favorite tech in the whole project. it’s caught all types of bugs, from simple logic errors to complicated race conditions that we would have never thought to test. I think there’s some interesting work out there to bring more of this “test case generation” style of testing to a larger audience…

                                                                                                It’s not clear to me what they mean by strong consistency for a system that replicates multiple concurrent uncoordinated modifications on different devices, it would be nice if they went into that claim a bit more.

                                                                                                ah, sorry this wasn’t worded too clearly. we broke the sync protocol down into two subproblems: 1) syncing a view of the remote filesystem to the clients and 2) allowing clients to propose new changes to the remote filesystem. then, the idea is that we’d solve these two problems with strong consistency guarantees, and then we’d use these protocols for building a more operational transform flavored protocol on top.

                                                                                                we took this approach since protocol-level inconsistencies were very common with sync engine classic’s protocol. we spent a ton of time debugging how a client’s view of the remote filesystem got into a bizarre state or why they sent up a nonsensical filesystem modification. so, it’d be possible to build a serializable system on our core protocol, even though we don’t, and that strength at the lowest layer is still really useful.

                                                                                                1. 2

                                                                                                  deterministic simulation is my favorite tech in the whole project

                                                                                                  Any tips on where to get started on this?

                                                                                                  1. 2

                                                                                                    Any tips on where to get started on this?

                                                                                                    the threads on this post are a good place to start: https://lobste.rs/s/ob6a8z/rewriting_heart_our_sync_engine#c_8zixa2 and https://lobste.rs/s/ob6a8z/rewriting_heart_our_sync_engine#c_ab2ysi. we also have a next blog post on testing currently in review :)

                                                                                                    1. 1

                                                                                                      Thank you! I’m looking forward to the next blog post too :)

                                                                                                  2. 2

                                                                                                    yeah, the deterministic simulation is my favorite tech in the whole project. it’s caught all types of bugs, from simple logic errors to complicated race conditions that we would have never thought to test. I think there’s some interesting work out there to bring more of this “test case generation” style of testing to a larger audience…

                                                                                                    I’ve been digging into a whole bunch of approaches as to how people do deterministic simulation. I’m really curious—how does your approach work? Can you provide some sort of gist/code example as to how those components are structured?

                                                                                                    1. 7

                                                                                                      ah, I don’t have a good code sample handy (but we’ll prepare one for our testing blog post). but here’s the main idea –

                                                                                                      1. we write almost all of our logic on a single thread, using futures to multiplex concurrent operations on a single thread. then, we make sure all of the code on that thread is deterministic with fixed inputs. there’s lots of ways code can sneak in a dependency on a global random number generator or time.
                                                                                                      2. have traits for the interfaces between the control thread and other threads. we also mock out external time behind a trait too.
                                                                                                      3. then, wrap each real component in a mock component that pauses all requests and puts them into a wait queue.

                                                                                                      now, instead of just calling .wait on the control thread future, poll it until it blocks (i.e. returns Async::NotReady). this means that the control thread can’t make any progress until some future it’s depending on completes. then, we can look at the wait queues and psuedorandomly unpause some subset of them and then poll the control thread again. we repeat this process until the test completes.

                                                                                                      all of these scheduling decisions are made psuedorandomly from a fixed RNG seed that’s determined at the beginning of the test run. we can also use this seed for injecting errors, generating initial conditions, and “agitating” the system by simulating other concurrent events. the best part is that once we find a failure, we’re guaranteed that we can reproduce it given its original seed.

                                                                                                      in fact, we actually don’t even log in CI at all. we run millions of seeds every day and then if CI finds a failure, it just prints the seed and we then run it locally to debug.

                                                                                                      1. 4

                                                                                                        There are so many beautiful properties of a system that is amenable to discrete event simulation.

                                                                                                        • You can use the seed to generate a schedule of events that happen in the system. When invariants are violated, you can shrink this generated history to the minimal set that reproduces the violation, like how quickcheck does its shrinking (I usually just use quickcheck for generating histories though). This produces minimized histories that are usually a lot simpler to debug, as causality is less blurred by having hundreds of irrelevant concurrent events in-flight. Importantly, separating the RNG from the generated schedule allows you to improve your schedule generators while keeping the actual histories around that previously found bugs and reusing them for regression tests. Otherwise every time you improve your generator you destroy all of your regression tests because the seeds no longer generate the same things.
                                                                                                        • Instead of approaching it from a brute force exploration of the interleaving+fault space, it’s often much more bug:instruction efficient to start with what has to go right for a desired invariant-respecting workload, and then perturbing this history to a desired tree depth (fault tolerance degree). Lineage Driven Fault Injection can be trivially applied to systems that are simulator friendly, allowing bugs to be sussed out several orders of magnitude more cheaply than via brute force exploration.
                                                                                                        • This approach can be millions of times faster than black-box approaches like jepsen, allowing engineers to run the tests on their laptops in a second or two that would have taken jepsen days or weeks, usually with far less thorough coverage.

                                                                                                        Simulation is the only way to build distributed systems that work. I wrote another possible simulation recipe here but there are many possible ways of doing it, and different systems will benefit from more or less complexity in this layer.

                                                                                                        1. 3

                                                                                                          thanks for the pointer for the molly paper, looks really interesting.

                                                                                                          here’s another idea I was playing with a few months ago: instead of viewing the input to the test as random seed, think of it as an infinite tape of random bits. then, the path taken through the program is pretty closely related to different bits in the input. for example, sampling whether to inject an error for a request is directly controlled by a bit somewhere in the input tape.

                                                                                                          this is then amenable to symbolic execution based fuzzing, where the fuzzer watches the program execution and tries to synthesize random tapes that lead to interesting program states. we actually got this up and working, and it found some interesting bugs really quickly. for example, when populating our initial condition, we’d sample two random u64s and insert them into a hashmap, asserting that there wasn’t a collision. the symbolic executor was able to reverse the hash function and generate a tape with two duplicate integers in the right places within minutes!

                                                                                                          but, we didn’t actually find any real bugs with that approach during my limited experiments. I think the heuristics involved are just too expensive, and running more black box random search in the same time is just as effective. however, we do spend time tuning our distributions to get good program coverage, and perhaps with a more white box approach that wouldn’t be necessary.

                                                                                                          1. 3

                                                                                                            I’ve also had disappointing results when combining fuzzing techniques with discrete event simulation of complex systems. My approach has been to have libfuzzer (via cargo fuzz) generate a byte sequence, and have every 2-4 bytes serve as a seed for a RNG that generates a scheduled event in a history of external inputs. This approach actually works extremely well for relatively small codebases, as a burst of Rust projects experienced a lot of nice bug discovery when this crate was later released, but the approach really never took off for my use in sled where it dropped the throughput so much that the coverage wasn’t able to stumble on introduced bugs anywhere close to as fast as just running quickcheck uninstrumented.

                                                                                                            I’ve been meaning to dive into Andreas Zeller’s Fuzzing Book to gain some insights into how I might be able to better apply this technique, because I share your belief that it feels like it SHOULD be an amazing approach.

                                                                                                            1. 4

                                                                                                              here’s another pointer for interesting papers if you’re continuing down this path: https://people.eecs.berkeley.edu/~ksen/cs29415fall16.html?rnd=1584484104661#!ks-body=home.md

                                                                                                              I’ve kind of put it on the backburner for now, but it’s good to hear that you’ve reached similar conclusions :)

                                                                                                      2. 4

                                                                                                        I don’t know anything about this project, but I do work on a system that has this property, I guess. There are lots of approaches but for me it’s just an exercise in designing the components carefully.

                                                                                                        First, you want to draw strict borders between the core protocol or domain or business logic, and the way the world interacts with that core. This is a tenet of the Clean Architecture or the Hexagonal Architecture or whatever, the core stuff should be pure and only know about its own domain objects, it shouldn’t know anything about HTTP or databases or even the concept of physical time. Modeling time as a dependency in this way takes practice, it’s as much art as it is science, and it depends a lot on your language.

                                                                                                        Second, you want to make it so that if the core is just sitting there with no input, it doesn’t change state. That means no timers or autonomous action. Everything should be driven by external input. This can be synchronous function calls — IMO this is the best model — but it can also work with an actor-style message passing paradigm. There are tricks to this. For example, if your protocol needs X to happen every Y, then you can model that as a function X that you require your callers to call every Y.

                                                                                                        Once you have the first step, you can implement in-memory versions of all of your dependencies, and therefore build totally in-memory topologies however you like. Once you have the second step, you have determinism that you can verify, and, if you’re able to model time abstractly rather than concretely, you can run your domain components as fast as your computer can go. With the two combined you can simulate whatever condition you want.

                                                                                                        I hope this makes sense. I’m sure /u/spacejam has a slightly? majorly? different approach to the challenge.

                                                                                                        1. 3

                                                                                                          this is spot on! we still have periodic timers in our system, but we hook them into our simulation of external time. there’s some special casing to avoiding scheduling a timer wakeup when there’s other queued futures, but it more-or-less just works.

                                                                                                          1. 2

                                                                                                            this is spot on!

                                                                                                            Nice to hear I’m not totally off base :)

                                                                                                          2. 2

                                                                                                            I totally agree that if you can make your core business logic a pure function, it dramatically improves your ability to understand your system. What you said about time is also possible for most things that flip into nondeterminism in production:

                                                                                                            • random number generators can be deterministic when run in testing
                                                                                                            • threads/goroutines can be executed in deterministic interleavings under test with the use of linux realtime priorities / instrumented scheduling / ptrace / gdb scripts etc…

                                                                                                            Determinism can be imposed on things in test that need a bit of nondeterminism in production to better take advantage of hardware parallelism. You lose some generality - code that you compile with instrumented scheduler yields that runs in a deterministic schedule for finding bugs will trigger different cache coherency traffic and possibly mask some bugs if you’re relying on interesting atomic barriers for correctness, as the scheduler yield will basically shove sequentially consistent barriers at every yield and cause some reordering bugs to disappear, but that’s sort of out-of-scope.

                                                                                                            There are some fun techniques that allow you to test things with single-threaded discrete event simulation but get performance via parallelism without introducing behavior that differs from the single threaded version:

                                                                                                            • software transactional memory
                                                                                                            • embedded databases that support serializable transactions
                                                                                                            • commutative data structures
                                                                                                    1. 3

                                                                                                      I don’t understand the obsession with videoconferencing in all these remote work discussions. It seems like people want to replicate normal work environments remotely, which just seems to defeat the whole purpose. It’s a bit like when the dominant UI style on phones was skeuomorphic: the notes app looked like a little notebook, etc. It’s good for familiarity, I guess, but it’s not really taking advantage of the benefits of the different situation/platform/etc while still getting the downsides.

                                                                                                      To me the biggest advantage of remote work is surely that you don’t have to sit in synchronous face-to-face meetings. With a ‘real’ office you get the advantage of face-to-face meetings being easy, whether they’re informal or formal. So you take advantage of that by synchronising everyone’s work hours (or mostly synchronised with a bit of flexibility, perhaps).

                                                                                                      When working remotely, formal face-to-face meetings are hard and informal face-to-face meetings don’t exist. So why bother? Abandon the whole notion of synchronised work hours and face-to-face meetings and take advantage of the advantages of remote work, like being able to work entirely offline and distraction-free at your own pace in your own time. As long as you get the assigned work done, why should your boss care if you do it at 3am in your underwear while watching Netflix?

                                                                                                      If I were working remotely I would want to work my own hours, at my own pace, and communicate via email with the rest of the people I was working with. When I need their help, I send them an email. They’ll respond within 24 hours (and probably much sooner) unless it’s the weekend, and in the meantime I’d get on with some other work.

                                                                                                      Obviously this is a bit different if you’re doing something like a system administration role where you’re expected to monitor the status of a system during your assigned hours, but I’m talking about development roles.

                                                                                                      1. 5

                                                                                                        This is unfortunately constrained. If you are doing assigned tasks, then sure you can work down your inbox. If you are in one of the more meta business roles (what are we doing, where are we going, what are our concerns) then you need a face to face. One in person meeting can set the stage for an entire year of tasks as you describe them, but it could take ages to manage what been only be achieved in person.

                                                                                                        1. 5

                                                                                                          Having a discussion to make a tricky but important decision - fairly common during software development in my experience - takes far longer over email (asynch) than via a meeting (synchronous).

                                                                                                          1. 1

                                                                                                            The Linux kernel development process seems to work a lot better than most commercial development processes. Perhaps it’s actually better for people to have a good long time to think between their messages rather than trying to think things through in depth ‘live’.

                                                                                                            1. 4

                                                                                                              The Linux kernel development process seems to work a lot better than most commercial development processes. Perhaps it’s actually better for people to have a good long time to think between their messages rather than trying to think things through in depth ‘live’.

                                                                                                              The Linux kernel is not bound by the kind of market pressure that dictates velocity and decision-making in most commercial organizations.

                                                                                                              1. 1

                                                                                                                A huge part of the Linux kernel’s development is done by commercial organisations that experience market pressure. The Linux kernel changes at a rate almost unprecedented for software projects, the sheer volume of commits in each release is huge and it just keeps growing. So I don’t know about that really.

                                                                                                                How many decisions are made on a daily basis that really couldn’t wait a day? There’s not really any less throughput on decisions, it’s just decision latency that’s affected. If you have enough other work to be doing it doesn’t affect your throughput for one task to be put on the back burner for a day. Latency only affects throughput when you have low concurrency, or if task switching is high overhead.

                                                                                                                1. 2

                                                                                                                  How many decisions are made on a daily basis that really couldn’t wait a day?

                                                                                                                  I mean, have you worked in a high-efficacy, market-driven organization? Tons. It’s never the single decision, but rather that every decision exists in an extremely long chain of decisions, each dependent on the previous. The communication praxis of these organizations is making those decisions with imperfect information, getting agreement among stakeholders, and moving to the next synchronization point. Sometimes you do like dozens of these in a single meeting. If every step cost a day of latency things would break totally.

                                                                                                                  The total transparency and async processes of GitLab provide a great real-life example of what I mean. Almost every decision-making thread you see on the GitLab internal boards (or whatever they’re called) tend to span multiple days, weeks, sometimes months, with tons of input from tons of people who aren’t direct stakeholders. In a high-efficacy on-site organization these threads would probably be single half-hour meetings with a handful of people and be done. Is the GitLab process better, by some metrics? Probably, surely, yes. Is GitLab slower in the market because of the constraints imposed by that process? Without a doubt.

                                                                                                                  1. 1

                                                                                                                    Sure, you make the decisions faster, but do you actually make them better? Almost certainly not. You make them much worse because you don’t have any time to really sit down and THINK about the consequences.

                                                                                                                    1. 1

                                                                                                                      And very often, in these environments, faster is better than better.

                                                                                                              2. 3

                                                                                                                Better by what metric?

                                                                                                                1. 2

                                                                                                                  Quality of code. The Linux kernel is a well-maintained codebase with little legacy code. There’s a strong culture of removing and replacing outdated code, and of replacing internal uses of legacy code with new code so that that legacy code can be removed. It’s also very well-documented.

                                                                                                                  The ability for many different organisations to all contribute in different ways that fit with their workflows. Because contributions are made by mailing patches to a mailing list, you don’t have to use specific proprietary tools or web-based interfaces to contribute. Any workflow that can result in someone or something running git format-patch and git send-email (which is any workflow that uses git at all) can result in sending patches to the LKML. You can use GitHub pull requests internally, you can use your own mailing list, or an internal GitLab instance, or sourcehut, or just plain git. You can have individual contributors within your organisation contribute patches individually, or you can have a few maintainers within your organisation bundle up all your work and send it, or a single person that does that. You don’t even have to use git, diff -uN works if all you want to do is write a single small patch.

                                                                                                                  Openness to contribution from drive-by contributors. The Linux project doesn’t require you to sign CLAs or provide evidence your employer has signed off on you contributing changes. All you need to do is agree to the contribution agreement, which basically just says ‘I have the right to contribute these changes’, and you indicate this by including a Signed-off-by: First Last <first.last@example.com> line in your commit messages, which you can do automatically with a flag to git commit. No complicated legalese, no contracts, no giving up your copyright to some company that could close it down at any moment. Just a flag to git commit.

                                                                                                                  Speed of changes. When a new security fix is required, it’s usually fixed and published as a new stable release within a couple of hours. What commercial products get that kind of maintenance? I’ve played online competitive games where there have been bugs that allow any player to crash the game server and all connected clients at any time that have taken weeks to be fixed. Losing your game? Just crash the server.

                                                                                                                  Access to developers. One of the great things about free software is that the people working on it are just people. Have an issue with something? Documentation unclear? Send an email. The email addresses of the maintainers of each part of the kernel are all listed in the MAINTAINERS file, and they’re pretty good at replying to email, even though they get an enormous amount of it. About a decade ago when I was basically a child I emailed Linus Torvalds a random question about the kernel and he emailed me back in quite some detail. I don’t even know who the lead developer of the NT kernel is, and even if I did I doubt he would respond to my emails. A company that I worked at loved to use PostgreSQL over commercial databases for the same reason: it’s a product where you can email the lead developer of the project and expect to get a detailed technical email back within 24 hours. Good luck doing that with Oracle unless you’re one of Oracle’s top 5 biggest customers.

                                                                                                                  Permanent record of design thought process. The Linux kernel mailing list archives are available on the web. If you’re wondering about the thought process that went into the development of a feature or a choice made during its development you can just go look at the mailing list archives. The responses to the patch series when posted, the various different versions of the patch series, etc. are all permanently archived. All the feedback it got, all the discussion around it. All archived in the same format on many different mirrors. This is invaluable. And it’s not just theoretically useful. It’s actually used in practice all the time. LWN articles often include quotes from design discussions that were had 10 or 15 years ago that provide valuable insight that would be really hard to discern just from the code or the documentation that exists today. Good luck doing this if you’ve used half a dozen different proprietary chat systems over the last decade and 95% of your design discussions are had in non-minuted face-to-face meetings.

                                                                                                                  1. 2

                                                                                                                    I appreciate you taking the time to lay that out so clearly. You’ve convinced me that asynch discussions are superior for OSS code or even commercial code that’s not being developed in a competitive enviroment. I’m not sure if that applies to commercial code being developed in a competitive environment though.

                                                                                                          1. 1

                                                                                                            ISTM that using macros to do something as normal as defining routes and handlers in an HTTP library is a sign that something is… off in the design. Like, I always understood macros as a feature-of-last-resort in most languages (c.f. e.g. Lisps) because of the way they subvert typical language constructs and our ability to mentally model programs. Is that not the case, or not accepted wisdom, in Rust?

                                                                                                            1. 5

                                                                                                              Is that not the case, or not accepted wisdom, in Rust?

                                                                                                              I think the macros being used in examples like this are mostly a gimmick. They make for nice hello worlds and can make for pretty concise code for simple web servers. They are also familiar to anyone who has built simple web servers in Python (using, say, bottle or Flask). Except in Python, you do it with “decorators.”

                                                                                                              Like any language feature, folks will abuse it. Macros do have some very compelling use cases and they are used well there. The one that comes to mind is automatic serialization. Does something as “normal” as serialization using macros mean there is a design flaw somewhere? What about something as “normal” as serialization needing runtime reflection in languages like Go? In both cases, serialization without macros (or runtime reflection) is super annoying.

                                                                                                              I’d say macro abuse in the Rust ecosystem is no better or worse than, say, reflection abuse in the Go ecosystem, if that makes sense.

                                                                                                              1. 1

                                                                                                                That makes things clearer, thank you. I guess that makes this a little like http.HandleFunc(”/route”, handler), all the way down to the implicit global http.DefaultServiceMux?

                                                                                                                1. 2

                                                                                                                  That’s my understanding, yes.