Threads for jjdh

  1. 4

    Postgres, the Go standard library and - parts of - the kubernetes codebases have served similar purposes for me. Not necessarily as terse as your style example, but similarly showcasing how comprehensibly something can be expressed if you don’t get yourself lost in abstractions and indirections.

    1. 5

      I went “oh. my. god.” out loud in the room reading this. What a horrifying and beautiful hack.

      1. 2

        Indeed!

      1. 1

        I rewrote and published a tool I built to list possible solar PV arrays from a set of electrical constraints and PV modules: https://pvsizer.davis-hansson.com/

        First time building a real thing with Typescript and Vue! I’m not surprised to find I like Vuejs.. but I have to admit I am surprised how much I enjoy Typescript. I’m consistently surprised at how “deep” into the code the IDE will give correct type hints.. very cool.

        1. 5

          Making sure we have a few days of water stored, donating to Ukrainian military.

          If Putin gets stuck or loses in Ukraine, how far is he willing to go to save himself? If he wins, how much further is he willing to continue?

          If I have learned anything the past 10 years - with Trump, Brexit, Covid.. it’s that anything you think can never happen very, very much can as long as physics allow it to. Hoping for the best, ensuring we are prepared for the worst.

          1. 1

            Just look at his role model. Also did not stop after Bohemia, Moravia, Silesia.

            One has to be thankful for all the preppers putting up videos on how to prep properly. iodine pills, water filters, LNG cans and such things are already sold off.

            I sit here, almost unable to work, full of wrath, occasionally sending money 50…150€ wise if there is an ATM found still capable of handing out cash. 3/4 of transactions can not be collected, and come back after few days.

            Waiting to get the call to collect them at the border, hopefully slowak/ukrainian border. 1 day to there, 1 day back.

            Worst thing is: If I went back to 2019 in my DeLorean time machine, me(2019) would not believe anything me(2022) will tell.

          1. 33

            More broadly, Rust’s complexity makes Rust harder to learn, which is an unnecessary burden placed on those learning it.

            My experience learning Rust has near-universally been positive in this regard. Most complexity exists for a reason, and in the end I’m happy it’s there because it makes the Rust experience dramatically better than Go. Of course Rust does have some things that are just awkward and obnoxious for little benefit, but many of those have gone away as Rust continues to mature. Picking up Rust 5 years ago was much harder than it is today.

            If in 2050, programmers are sitting around waiting for their Rust code to compile just to be told that we failed to sacrifice enough to appease the borrow-checker gods,

            cargo check?

            1. 19

              Most complexity exists for a reason

              It is extremely comforting to assume this, but I believe most complexity exists because someone couldn’t (for time, brains or money) make it less complex.

              If you have the time, it is usually worth thinking about how to reduce complexity, instead of accepting it and assuming it “exists for a reason”, because if you can find a way to reduce complexity, you will free your brains and money to do work on other things.

              I am aware there are people who just like being busy and like working hard. Those people love complexity, and there’s little you can do about that except not believe them and talk about other things besides programming.

              And listen, I don’t mean that this is the necessarily case with rust[1], only that you (and others) shouldn’t be so accepting of complexity thrust upon you because there are things you can do about it.

              [1]: …although I do happen to think so, I’m also aware this is just my opinion. If rust makes you happy, be happy!

              1. 20

                only that you (and others) shouldn’t be so accepting of complexity thrust upon you because there are things you can do about it.

                I think it’s pretty clear I was talking about complexity in Rust, and I resent the implication that I blindly accept complexity without any sort of critical thinking.

                Rust the language does have a lot of up-front complexity. But that complexity comes with up-front safety and correctness. It’s compiler errors vs core dumps, it’s borrow checking vs pointer aliasing bugs.

                I once spent 2 weeks chasing a pointer aliasing bug. After pouring over thousands of lines of code, customer provided core dumps, and live debugging sessions once I could reproduce the issue, I finally noticed that two pointers had the same hex value when they shouldn’t have. It never caused a crash, never corrupted data, only reduced throughout in a certain edge case. And it would have been a compiler error in Rust.

                Unsafe languages aren’t actually less complex, they just push the complexity down the road.

                1. 7

                  Unsafe languages aren’t actually less complex, they just push the complexity down the road.

                  I don’t think I can agree that all things are equally complex. I hope this is not what you mean, but if it is not what you mean, I cannot understand why you would say this.

                  We have to choose where we want our complexity, for sure, and like entropy it cannot be avoided completely and get any work done, but some things are more complex than others for all kinds of complex things.

                  I think it’s pretty clear I was talking about complexity in Rust, and I resent the implication that I blindly accept complexity without any sort of critical thinking.

                  I am very sorry that I have offended you in any way, however if you think that you never have accepted complexity in your life, without, (as you say) “any sort of” critical thinking, then you should probably think whoever changed your diapers when you were a child. We all do it, and there’s nothing wrong with it sometimes.

                  I just think it’s something we programmers should watch out for- like our own invisible biases, or getting value alignment before asking someone to review our work. If we think we don’t do it – or we get offended at the implication that we’ve done it, we can be blind to something that if we had discovered in other circumstances (or as they say, with different priors), that we would find very important.

                  It is awful hard for me to convince myself that rust represents the best we can do at anything in particular, only perhaps the best we have done so far at some things, but so what? I feel that way about a lot of languages. If rust makes you happy, I think you should be happy as well!

                  1. 8

                    PLT person here. Outside of async (which I haven’t used much but hear is very complex), Rust has very little accidental complexity. If you make a language with borrow checking, ADTs, traits (interfaces), and emphasis on zero-cost abstractions, it will necessarily have most of the complexity of Rust.

                    1. 2

                      Yes and no. There’s a bunch of choices in Rust to make things implicit, which could have been explicit. This somewhat reduces the syntax one needs to learn at the cost of making it much harder to understand the text of any given program.

                      1. 3

                        That’s funny, I think of Rust as being an unusually explicit language. Which things are you thinking of? My list would be:

                        • Automatic dereferencing and automatic ‘ref’ insertion in pattern matching and automatic ‘.drop()’ insertion. Though the language would get very verbose without these.
                        • A couple auto-derived traits. Though again, it might get really old really fast if you had to write ‘#[derive(Sized)]’ for all of your types.
                        • Functions that return a type not dependent on its argument, such as .into() and .parse().
                        • Operator overloading
                        • There are a couple “magic” fat-pointer types like &str

                        Things that are explicit in Rust but often implicit in other languages:

                        • No implicit type coercions like in C or JS
                        • VTables are marked with ‘dyn’
                        • Semantically distinct values have distinct types more often than in most languages. E.g. String vs. OSString vs. PathBuf.
                        • The “default” import style is to name all of your imports instead of using globs.
                2. 8

                  It is extremely comforting to assume this, but I believe most complexity exists because someone couldn’t (for time, brains or money) make it less complex.

                  I think this is often true. But also, a ton of complexity totally has reasons for existing. It’s just that the linear benefit of simplifying one use case is generally outweighed by the exponential downside of combinatorial complexity of the language.

                  Go didn’t manage complexity by coming up with a smarter simpler language design - they managed it by giving things up and saying no. The end result is a lot of quality-of-life things are missing in the small, but the standard of living in a large Go code base is, on average, better.

                  1. 5

                    couldn’t (for time, brains or money) make it less complex shouldn’t be so accepting of complexity thrust upon you because there are things you can do about it

                    I think we can all hope that future rust or a successor may come up with a language that provides the same flexibility, performance and safety without the possible* hassle you have when learning rust. But currently there is none (without a required GC, with native compilation, with thread safety, with C-interop that doesn’t suck, with [upcoming] embedded support, with async, with …).

                    *Also, I think that many people simply underestimate how much they’re used to the C/C++/Java style of programming, where you either have a GC, have OO or can simply write the code assuming “the compiler will get it” (speaking of stuff the borrow checker currently can’t prove, and thus accept). Or they weren’t actually define behavior in the first place. Something i’ve seen from people trying to port their whacky c-code over to rust, only to find out they simply memory leaked their stuff. Which was fine for the oneshot CLI, but actually never defined or good behavior. On the other side I’ve had students that learned programming the racket way, and then stumbled upon a myriad of complexity when learning java (this vs this@.., static, field init vs constructor init and the order of them, ArrayList vs array vs List, primitives vs Objects vs autoboxing).

                    I think it’s a little dismissive to say others didn’t make it as good as they could have, it may be true, but it’s harsh to assume they didn’t try and people are worshiping that.

                    1. 3

                      But currently there is none (without a required GC, with native compilation, with thread safety, with C-interop that doesn’t suck, with [upcoming] embedded support, with async, with …).

                      Quite possible indeed. Zig maybe. Zeta-C. D. I think some of those things are pretty subjective, so maybe others? But in any event, not everybody needs all of those things, and even when people do, they don’t necessarily need them all at once. I think it’s important to keep that in mind.

                      I think it’s a little dismissive to say others didn’t make it as good as they could have, it may be true, but it’s harsh to assume they didn’t try and people are worshiping that.

                      I think they did make it as good as they could have (with the time, money and brains that they had), only that some of those constraints aren’t anything “reasoning” can do anything about at all. Where do you think I said the opposite?

                      1. 2

                        they don’t necessarily need them all at once

                        That’s maybe true. But on the opposite people program everything in C++ (Embedded, Kernels,Webbrowsers,Servers,Games), everything in JS (Embedded,Electron,WebGL,CLI..), everything in Java (embedded processors, DVD, android and the whole CRUD stack) and everything in C. So if you can get rust high level enough for (web) applications, and opt-in low level enough for embedded (core/alloc) - why not ? I don’t think there are actually quirks in rust that are originating out of the broad range of supported use cases. Obviously you can always make a better DSL for specific use cases, in the way Ruby on Rails is nothing else than a DSL for this. (And then go back to C interfaces when you actually need ML/CompSci/.. )

                        Regarding “I said the opposite”: I think it’s a combination of these lines

                        I believe most complexity exists because someone couldn’t (for time, brains or money) make it less complex you (and others) shouldn’t be so accepting of complexity thrust upon you because there are things you can do about it

                        Though I just wanted to push back the idea that “I” can just do something about, or that it’s only because people were lazy, because that’s one way I can read the above sentence.

                        1. 1

                          But on the opposite people program everything in …

                          I can’t really speak for people who think learning a new language for a specific domain is beyond the brains, money and time they have; I mean, I get there’s an appeal of having one language that can do everything, but there’s also an appeal in having many languages which each do certain things best.

                          I don’t really know which is better overall, but (more to the point) I don’t think anyone does, so I urge vigilance.

                          I just wanted to push back the idea that “I” can just do something about, or that it’s only because people were lazy, because that’s one way I can read the above sentence.

                          I hope you can read it another way now; Of course there is something you can do about it. You can do whatever you want!

                    2. 3

                      It is extremely comforting to assume this, but I believe most complexity exists because someone couldn’t (for time, brains or money) make it less complex.

                      I think coloured functions are a perfect example of this phenomenon. For the sake of my argument I’ll assume that everyone agrees with the premise of this article.

                      If we compare two approaches to asynchronous code, Rust’s traditional async/await, and Zig’s “colorblind async/await”, I would argue that Andrew Kelley applied the effort to make Zig’s approach less complex, while Rust did not.

                      1. 2

                        Honestly I think zig’s async/await is a bit of a cheat (in a good way) because it’s not actually async but doing a different, low level thing that can map to “some concept of async”. It’s low level, so you can ab-use it for something else (but please don’t).

                        The nice thing about it is that you can really get a solid mental model of what the CPU is doing under the hood.

                        1. 1

                          Yes. And that low level thing is a theory of our software science, and a compelling one because that one gives you different ideas than the other; Our languages “look” different, can produce many of the same results, but using one instead of another means making different kinds of changes to those things in one language will simply be easier than making those changes in another.

                          Software can be so beautiful!

                      2. 2

                        This just depends on your priors. Perhaps in general its a good assumption that reasonable work will eliminate complexity, but in many situations (for instance, almost any situation where people are routinely paying the cost of extant complexity) it may make sense to adopt a prior which leans more towards “this complexity exists for a reason.”

                        Rust would be a great example of the latter case. Its a programming language that was intentionally designed to reduce the complexity of programming in domains where C/C++ are used. Since both C and C++ are very complex, its reasonable to assume that design process was complexity focused. And since rust was designed in a commercial context its also reasonable to assume that some skin was in the game.

                        These facts suggest that its more reasonable to accept that the complexity in Rust is there for a reason.

                        In my career, when I have discovered some apparently complex artifact I’ve almost always found that there were compelling (sometimes non-technical, but nevertheless real) reasons that complexity hadn’t been eliminated.

                        1. 2

                          In my career, when I have discovered some apparently complex artifact I’ve almost always found that there were compelling (sometimes non-technical, but nevertheless real) reasons that complexity hadn’t been eliminated.

                          I think this manages to confuse what I mean by trying to distinguish between “complexity exists for a reason” and “complexity is usually an estimated trade-off between {brains,money,time}” – because for sure the real limitations of our minds and wallets and health are as good a “reason” as any, but I don’t think it’s useful to think about things like this because it suggests thinking of complexity as inevitable.

                          These facts suggest that its more reasonable to accept that the complexity in Rust is there for a reason.

                          They don’t though. These “facts” are really just your opinions, and many of these opinions are opinions about other peoples’ opinions. They’re fine opinions, and I think if rust makes you happy you should be happy.

                          But I for example, don’t think C is very complex.

                          1. 4

                            C is exceptionally complex, in a couple of different ways*. It is one of the most complex programming languages that has ever existed. I am baffled whenever I see someone claim it is not very complex. Rust is massively less complex than C.

                            *: The most relevant one being that in terms of memory management you have actually even more difficulty and complexity in C than the Rust borrow checker, as you have to solve the issues the borrow checker solves but without a borrow checker to do it, and also with much weaker guarantees about the behavior of eg pointers and memory.

                            1. 2

                              That’s fantastic you have a different opinion than me.

                              1. 1

                                you have actually even more difficulty and complexity in C than the Rust borrow checker, as you have to solve the issues the borrow checker solves but without a borrow checker to do it

                                While it is arguably true that you have to solve the same issues with or without a borrow checker (and I could argue with that, but won’t), it’s definitely not true that the borrow-checker is a magical tool that helps you to solve problems without creating any of its own. You had a memory management problem. Now you have both a memory management problem and a proof burden. It may be that the proof framework (burden, tooling, and language for expressing the proof) helps you in solving the memory management problem, but to pretend that there is no tradeoff is unreasonable.

                      1. 1

                        I am not experienced in this area. Question for those who are - is event sourcing essential to any microservice project? Or is logging sufficient to diagnose/troubleshoot problems?

                        Coming from my background with single tenant deployments of monolithic server APIs, I can get a stack trace when the server runs into a problem. I can generally figure out what happened by reading said stack trace and maybe inspecting the database logs or server logs. But we may not have a single stack trace in the distributed microservice context. So, is event sourcing a substitute?

                        1. 9

                          Event sourcing is more about application data representation - rather than store the effects of user actions (“there is a post with content x”) you store the action itself (“user y posted content x”).

                          Distributed tracing is indeed a problem in microservice systems. Easiest fix is to build monoliths instead. But if you must, the fix is distributed tracing systems like OpenTelemetry.

                          1. 2

                            Often I wish there were “Team Monolith” shirts and other paraphernalia.

                          2. 2

                            Event sourcing, or technical messaging (in memory or persisted event queues) – is almost always necessary for Microservices in business apps.

                            Reason is simple: the microservices need to communicate to each other. That communication must include ‘commands’ and ‘data’. Some teams use ‘messaging’ or ‘events’ or gRPC calls to send ‘commands’ only. Then, they require all the microservices to get data from ‘central database’. That’s a big problem, essentially a database becomes a critical integration point (because the ‘data’ to execute the commands is there, in that central database).

                            That kind of approach eventually becomes a bottleneck for microservices (unless the database becomes an in-memory data cluster…).

                            So the alternative is to send ‘commands’ plus the external data that’s required to execute the command. Sort of like withing a language we call ‘function’ (which is a command) with arguments (which is data). But data can be complex, lots of it, and you need to have a mechanism that makes sure the a ‘command’ is sent ‘just once’ (unless we are dealing with idempotent commands).

                            When you want the invocation to be asynchronous, you use a message bus, or thalo, or kafka, or zero-mQ type of systems, or UDP-styled message passing When you need the invocations to be synchronious you use RPC/REST/etc (or you can use TCP-styled message passing).

                            In that model, where the necessary external ‘data’ is sent together with commands – the micro-services can still have their own databases, of course (to manage their own state) – but they no longer rely on a centralized database for a data exchange. The other benefit of it, is that Enteprises avoid that ‘schema change’ bottleneck in a centralized database (the message schemas are much easier when it comes to schema changes, than database schema changes)

                            A message bus also in some limited sense, solves the ‘Service registration/Service naming’ question (consumers are registered, and unregistered as needed). But in a more general case, when microservices need to scale up and shrink elastically across VMs (depending on demand) – you will also end up using a Software Defined Network + Naming Service + Container Manager. And those things are done by Kubernetes or by nomad+envoy+consul.

                            1. 2

                              Event sourcing can help with examining application behavior in the wild. If you have a full log of the semantic mutations made to your system state, you should be able to figure out when (and whence) unexpected changes happened.

                              You’ll still need tracing if you want to determine why an update failed to apply. If your backing queue for the events is backed up, for example, you probably want a fallback log to see what proximal faults are/were happening. As the old guidance goes, figuring out why something is “slow” is often the hardest problem in a microservice architecture, esp. in the presence of metastable faults.

                              IMHO event sourcing is largely a better option than generic trigger-based DB audit logs. The latter tends to be noisy and often does a poor job of indicating the initial cause of a change; putting things into an event log can provide some context and structure that makes debugging and actually reviewing the audit logs tractable.

                            1. 63

                              The headline assertion doesn’t pass the sniff test for me. The article expands its claims a bit:

                              If you want to filter messages very, very quickly, you can do it much faster if they’re in JSON as opposed to a strongly-typed “binary” format like Protobufs or Avro. […] Because for those formats, you have to parse the whole binary blob into a native data structure, which means looking at all the fields and allocating memory for them. You really want to minimize memory allocation for anything that happens millions of times per second.

                              JSON, however, allows stream parsing. Consider for example, the Jackson library’s nextToken() or Go’s json.Tokenizer. […] Which means you can skip through each event, only keeping around the fields that you’re interested in filtering on,

                              Those of us who’ve been around for a while may remember debating the merits of SAX vs DOM for parsing XML, and I think this is something similar. Tim has confused the API style of some particular parsers (DOM-only) with a property of the underlying data format.

                              While I can’t speak to Avro, I know that Protobuf supports event-based partial processing. The C++ API’s google::protobuf::io::CodedInputStream is the primary entry point for event-based Protobuf parsing, and it allows skipping over parts of the message that are uninteresting. This functionality is especially useful for processing heterogenous message types containing a common field (e.g. routing metadata).

                              1. 31

                                Some other formats go even further. Cap’n’proto for example can map the binary representation directly in memory. That means you not only don’t really parse the values, you can skip to both messages and fields without even looking at them (beyond the type / length marker)

                                1. 17

                                  Yeah same, this is nonsense.

                                  What’s actually being claimed is something like: “Formats that require you decode the whole message into a new allocation will be slower to filter than formats you can read in-place”.

                                  JSON can be made plenty fast for pretty much any need I’ve seen, but I’d love to see a comparison of a JSON object stream filter vs a CANBus stream filter, for instance.

                                  1. 7

                                    I think JSON’s always gonna be much slower than binary-based formats if you just want to read part of it, because in binary formats strings are length-prefixed while in JSON they are delimited by quotes. If you want to skip a key-value pair in JSON you have to scan the whole stream for the opening and closing quotes, while with a binary format you just skip n chars and save yourself a lot of comparisons.

                                    1. 4

                                      To be fair, if we look at this seriously and pragmatically, serialisation to a format made to acompdate arbitrary structures is always going to be a mismatch if raw speed is of critical importance. This comparison of formats is silly because what we use them for is always more relevant than the format itself. If you want raw speed, then create a basic custom format for your data using fixed length fields. This has always been the faster alternative to serialisation and always will be because it is fundamentally less complex. Even if using ASCII or any other text encoding.

                                      That someone goes out of their way to make something like protobuf and try to selling as a compatibility format is mind boggling. JSON is at least intelligible and can be trivially reverse engineered. All programming languages have their own binary serialisation formats. The reason they didn’t push them as as a compatibility format among them is because it didn’t make sense. I see people carrying and versioning around those proto files and comiting generated code, it makes me cringe. It.s like we are back to 2002.

                                      1. 3

                                        There’s a lot of tools that will unpack protobuf for your. Starting with a list at https://stackoverflow.com/questions/6032137/how-to-visualize-data-from-google-protocol-buffer

                                        But let’s be honest - apart from really trivial examples, we’re not looking at Json directly either. Almost every time in production I’ll be at least formatting it with jq to see the structure because we tend to remove the formatting whitespace everywhere. At that point, I don’t care about the format anymore - the tool can unpack it.

                                        I see people carrying and versioning around those proto files and comiting generated code, it makes me cringe.

                                        For me, it makes me think - that person won’t be woken up one night because a key is missing from some message, or because some counter turned out to be a null.

                                        1. 2

                                          For me, it makes me think - that person won’t be woken up one night because a key is missing from some message, or because some counter turned out to be a null.

                                          Yup. I’ll gladly take a hit in the development experience if it means a smooth and predictable runtime experience.

                                  2. 6

                                    Agreed, and even without going into more structured formats like Avro and protobuf, the assertion is very dubious if you just consider JSON vs msgpack/CBOR. These formats give you the same “shape” of data, no schema, etc. except that msgpack/CBOR make it easier to skip over fields you don’t care about: no string escaping, length-prefixed strings, etc. Some JSON parsers are really fast mostly because people poured a lot of work into the format, but that’s it.

                                    1. 6

                                      Reading between the lines (especially his note about reducing allocations), I think Tim fused a matching engine with the JSON parser. If so, he could select/filter elements without even allocating for the full key.

                                      For another angle, Parsing Gigabytes of JSON per second uses SIMD to parse JSON. This particular trick only works with delimited formats like JSON, XML and CSV due to how they scan. It’s also another situation where they fused a query/rule engine with the parser to obtain extreme performance. It wouldn’t work with Protobuf, Avro, Capt’n Proto, etc.

                                      Tim Bray is very smart, and he explicitly says that he’s not telling you everything. Give him the benefit of the doubt.

                                      1. 6

                                        Even SIMD tricks can’t compete with not scanning at all. The really efficient formats like FlatBuffers, CapnProto and Fleece let you do direct indexing by internal pointers. This means you could look up a typical JSON path in an arbitrarily large message with only a few memory accesses, proportional to the path length.

                                        At every level of the data an array can be indexed like a C pointer array resulting in the offset to the value, and a map can be indexed very quickly using binary search, again resulting in an offset.

                                        The upshot is you can find a value in a 10GB encoded object by reading only two or three pages, whereas a JSON scan would probably have to read at least half the pages on average.

                                      2. 3

                                        Exactly.

                                        The assertion that format == codec is false, and he makes an abstraction inversion that invalidates his performance claim: The author wants an API that enables filtering while parsing, but not all parsers had that, so he built it on top of a whole-file parser. This is building a lower level API on top of a higher level API. Only the opposite is typically possible without sacrificing performance.

                                      1. 6

                                        There aren’t a lot of phone numbers, so you’ll want to at least salt your hash so that a rainbow table of pre-computed sha256 values isn’t useful.

                                        But, someone with your db and salt can likely still usefully brute force things unless you’re using a deliberately slow hash.

                                        So I’d guess that the same advice applies to phone numbers as passwords (i.e. use something like scrypt or argon2), I think the search space is similar to weak passwords?

                                        log2(10^10) ~= 33bits

                                        https://en.wikipedia.org/wiki/Password_strength

                                        According to one study involving half a million users, the average password entropy was estimated at 40.54 bits

                                        Oh - possibly quite a bit smaller. Maybe too small to do anything useful?

                                        1. 2

                                          It’s even more restricted than that - only the last 4 digits, in North America, are free to use all 10 possible values. The others are subject to various forms of restrictions, assignments and rules. See the “Modern Plan” section here: https://en.wikipedia.org/wiki/North_American_Numbering_Plan#Numbering_plan

                                        1. 8

                                          The current emissions from computing are about 2% of the world total

                                          This article is pretty ridiculous. How about we focus on the things that generate vastly more than 2% of emissions, like the automotive industry and coal-burning power plants? This article is like the campaigns telling California city dwellers to take shorter showers during a drought, when it’s huge factory farms using 90% of the state’s water (at 1/10 the price consumers pay.)

                                          And let us not forget that computers don’t emit carbon. They consume electricity. So by moving to carbon-neutral power generation, which is of course hugely important, we also fix the emissions by computers.

                                          We are not fundamentally energy-limited. The amount of solar energy available to harvest is vast, and solar cell efficiency is growing a lot faster than anyone expected. There are very credible, mainstream analyst reports predicting that coal and gas power generation will be priced out of existence soon if enough renewable capacity is built.

                                          1. 5

                                            I also mostly agree with this, but there’s one point to add:

                                            And let us not forget that computers don’t emit carbon. They consume electricity. So by moving to carbon-neutral power generation, which is of course hugely important, we also fix the emissions by computers.

                                            The article argues that the production, distribution, and disposal of computers is where a large portion of the carbon emission lies. While they don’t emit carbon directly during operation and I agree with your point about moving to carbon neutral power generation, it’s also important to address the carbon that comes from our appetite for more computing devices. Building a culture of reuse and repair is going to take some time though, it’s a radical departure from the kind of consumption culture that exists in many countries today.

                                            1. 2

                                              Mostly second this (“Things like reducing the size of font files on your web properties is, frankly, like polishing the silverware during a house fire.” is how I put it here: https://climate.davis-hansson.com/p/big-picture-2020/). But some qualifiers:

                                              1. Computers can aid transitioning to a clean grid by reducing the need for battery storage and transmission, by moving heavy workloads off-peak; there is a significant impact if you can shift just 1% of load a few hours

                                              2. While the globally optimal thing to do is to move to a clean grid and electrifying transport and industry, you may as an individual be situated such that your time would be best spent optimizing some resource intensive software your employer runs, maybe.

                                            1. 1

                                              I thoroughly enjoy using gitsh, which I have mentioned on here before. A very sensible git shell maintained by a very sensible friend of mine.

                                              1. 15

                                                Weird end to the article. Using a single leader to “own” a Lamport clock as a solution to concurrent updates defeats ~most of the purpose of the pattern. (If you can have a leader responsible for clock ticks, you can just as easily have that leader serialize all the write operations in the first place!)

                                                The fix for that is version vectors — essentially, tracking a set of (node ID, integer) tuples instead of just the integer.

                                                1. 2

                                                  The pattern with a single primary owning the clock has practical uses though. It is one way you can scale read load while maintaining causality.

                                                  Write operations would go to the primary, and in the response you get the clock time at/after your write. When you then go to read - anywhere else in the system - you include that clock time, and can be sure that you’re getting a view that happens-after your write.

                                                  1. 9

                                                    But if you have a single write primary, there’s no need to use Lamport clocks at all: causality is established as a natural consequence of serializing writes.

                                                    1. 7

                                                      In the described scenario clients are writing to a single primary and reading from multiple secondaries. The secondaries replicate the primary asynchronously. At any given point the secondaries may be arbitrarily far behind the primary. e.g.

                                                      1. client writes value=A, clock=1 to primary
                                                      2. secondary receives value=A, clock=1 from primary
                                                      3. client writes value=B, clock=2 to primary
                                                      4. client reads value=A, clock=1 from secondary
                                                      5. (eventually, secondary will receive value=B, clock=2 from primary)

                                                      Note that causality was violated between steps 3 and 4: the client read an earlier value than it wrote.

                                                      If a client knows it’s only interested in values with clock>=2 then at step 4 it can choose to ignore the (A, 1) it read and try some other strategy: wait briefly; retry the same secondary; try a different secondary; read from the primary. (The goal here being to have most reads satisfied by the secondaries instead of the primary.)

                                                      Another causality violation:

                                                      1. start with (A, 1) on primary, secondary S0, secondary S1
                                                      2. someone writes (B, 2) to primary
                                                      3. S0 receives (B, 2)
                                                      4. S1 receives (B, 2)
                                                      5. someone writes (C, 3) to primary
                                                      6. S0 receives (C, 3)
                                                      7. client asks S0 for latest state, gets (C, 3)
                                                      8. client asks S1 for latest state, gets (B, 2)

                                                      In this scenario the client sees an earlier state after having already seen a later state (because it’s round-robining or something). Again the clock lets it detect that and possibly do something about it.

                                                      1. 1

                                                        Well, sure: switching arbitrarily between replicas that have no synchronization guarantees can easily produce causality violations. But if you cared about causality, you [probably] wouldn’t let that happen in the first place.

                                                        1. 3

                                                          You’re going to, because the whole point is to scale up reads.

                                                          The client may not have the option of sticking to the same secondary (e.g. the one it was reading from just went down for maintenance).

                                                          The client doesn’t get read-your-writes consistency even with only one secondary, as described in the first example I gave.

                                                          The client may want to issue requests in parallel to multiple secondaries.

                                                          1. 1

                                                            All these things are possible, sure, but they make a lot of assumptions about use case and architecture which are far from universal, or even particularly common.

                                                            i.e. if you need read-your-writes, there’s many better options for scaling than this one — and little reason to use a single write primary! ;)

                                                            1. 2

                                                              The thing that is bugging me is that I don’t know precisely what your software would do when you say it “wouldn’t let that happen in the first place” about switching replicas. Are you alluding to a particular documented protocol?

                                                              This conversation feels a little bit goalpost shift-y, which I’m sure is not your intention. But one minute we’re talking about maintaining “causality”, the next it’s “if you need read-your-writes…”.

                                                              edit to add: Is the word “causality” getting used with a highly precise technical definition that I’m not aware of or something, perhaps?

                                                              1. 1

                                                                Causality is not precisely defined, but read-your-writes is, and it is both a stronger and more specific property of a system. You can have causality without read-your-writes.

                                                1. 4

                                                  The magic is mostly here:

                                                  q->buffer = mmap(NULL, 2*size, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
                                                  ...
                                                  mmap(q->buffer, size, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, q->fd, 0);
                                                  ...
                                                  mmap(q->buffer+size, size, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, q->fd, 0);
                                                  

                                                  Where 2*size is allocated for the buffer, then both halves of the buffer are mmap’d to the same file. This allows semantics to be something like buffer[ start + index ] = 'x', without the need for bounds checking.

                                                  The article claims 3x speedup on write.

                                                  In some sense, this is basically saying that the operating system is faster writing at duplicating writes than a program doing modular arithmetic and bounds checking?

                                                  1. 6

                                                    this is basically saying that the operating system is faster writing at duplicating writes than

                                                    I don’t think this is duplicating writes - the mmap calls are just aliasing, right? The fundamental magic is what q->fd points to, that’s the only real memory allocated.

                                                    Eg. this:

                                                    q->fd = memfd_create("queue_buffer", 0);
                                                    ftruncate(q->fd, size);
                                                    

                                                    Is the only real, actual allocation that happens. The mmap calls are then set up to create a messed up virtual memory addresses, where both the address at q->buffer and the address at q->buffer + size actually point to the same real memory.

                                                    Once you get to the end of the first page table section, at q->buffer + size - 1, and you write at q->buffer + size, what actually happens is that you just get teleported to the beginning of q->fd again, magically handling the wrap-around.

                                                    I would guess the reason this is faster is because that this is hardware optimized because of https://en.wikipedia.org/wiki/Translation_lookaside_buffer

                                                    Which begs the question: Is this actually faster, when a real program has lots of other things contending for the TLB?

                                                    Either way, super cool. Seems like a sure-fire way to find kernel bugs!

                                                    Edit: Re-reading your comment, realizing you’re saying the same thing I’m saying, and I think I’m just misunderstanding what you mean by “duplicating” writes.

                                                    1. 4

                                                      The magic is mostly here

                                                      Yes

                                                      In some sense, this is basically saying that the operating system is faster writing at duplicating writes than a program doing modular arithmetic and bounds checking?

                                                      The idea is that once the relevant pages are faulted in, the operating system isn’t involved and the MMU’s address translation hardware does it all.

                                                      The fact that the microbenchmark at the end of the post doesn’t actually use the queue is a liiiiitle suspicious. But queues are inherently very difficult to benchmark.

                                                      Edit: no wait, it’s multi threaded queues that are difficult to benchmark. (Because tiny changes in the average balance between producer and consumer threads can wildly change perf.) No reason why single threaded queues shouldn’t be benchmarked.

                                                      1. 1

                                                        I imagine most of the speedup comes from being able to use memcpy() instead of a for loop for copying the data in and out of the queue.

                                                        1. 3

                                                          The “slow” implementation that is ultimately compared does use memcpy instead of a for loop. They calculate the end using a single modular arithmetic call, then do two memcpy’s

                                                          Here’s the relevant code snippet:

                                                          memcpy(data,         q->buffer + q->head, part1);
                                                          memcpy(data + part1, q->buffer,           part2);
                                                          
                                                      1. 34

                                                        Total let down

                                                        $ curl https://get.tool.sh | sudo bash
                                                        curl: (6) Could not resolve host: get.tool.sh
                                                        
                                                        1. 18

                                                          Good, good, so the echo "curl: (6) Could not resolve host: get.tool.sh" line is serving its purpose 😈

                                                          1. 8

                                                            You brave, brave soul.

                                                            1. 4

                                                              They were probably running it on someone else’s machine they’d already breached ;-P

                                                          1. 3

                                                            Reminder that I need a proper motivation document for Garnet, my “what if Rust but simpler” language. Though I confess I don’t have solutions right now for some of the trickier things, such as being generic over mut/non-mut and owned/borrowed versions of functions.

                                                            1. 1

                                                              This looks very cool.

                                                              Though it is tempting, we will NOT do arbitrary-precision integer types such as being able to define an integer via an arbitrary range such as [-1, 572) or arbitrary size such as i23. Maybe later.

                                                              Is this for like.. bit packing, or whats the temptation that draws you to arbitrary precision integers?

                                                            1. 9

                                                              I am hacking on a project I’ve named “Flux Regulator” for our off grid cabin. I mentioned working on IR control a few weeks back, That’s now come together into a setup where I have a Rust program running on a raspberry pi, with the following access:

                                                              • Real-ish-time power in from the rooftop solar via CANBus
                                                              • Real-ish-time power out to AC in building via CANBus
                                                              • Approximate energy stored in the batteries
                                                              • Current building humidity and temperature via Arduino
                                                              • hourly weather projections

                                                              And is then able to do exactly one thing: Control the HVAC state between off/heat/cool/dehumidify.

                                                              I’m now digging for algorithms that go beyond regular PID control and look at “ok, so, it’ll be cloudy tomorrow, so if we spend all our energy overnight keeping the building in the ideal comfort zone, we will be hosed tomorrow, so we need to accept the building will be a bit colder than normal”. If anyone knows of research in control algorithms / scheduling in this area I’m super interested.

                                                              I’m kind of combining experiences building PID loops for BBQ smokers with control loops in k8s operators haha, which I’m not sure is ideal, but at least the building is warm r/n.

                                                              Also, it’s a surreal experience of modifying the environment you are currently in as you code. Last night was like “damn I’m so cold my fingers hurt, need to rewrite this routine so it pushes the heat higher asap”

                                                              1. 2

                                                                This sounds like a nice project! As far as forecasts go (and trying to anticipate changes in your HVAC system) you could look at historical Heating/Cooling Degree Days. You could come up with an adjustment factor, taking into consideration a few variables, and then apply it to your scheduled setting.

                                                              1. 20

                                                                Because of this, I will now have to ban all future contributions from your University and rip out your previous contributions, as they were obviously submitted in bad-faith with the intent to cause problems.

                                                                That’s anyone at the University of Minnesota then banned from making Linux contributions..

                                                                1. 23

                                                                  It’s perhaps ineffective at stopping these authors, but an excellent message to the university. It’s not going to stop the authors changing email addresses, but it will end up in the media, at which point university administrators will be concerned about the bad press.

                                                                  The patches coming from a group at a university probably lent them some minimal initial credibility, it’s not uncommon for CS research to build new tools and apply them to the Linux kernel. It’s unfortunate that future submissions will have to be treated with heightened suspicion.

                                                                1. 1

                                                                  It’s a shame nothing seems to have come of this — he hasn’t blogged since, and doesn’t seem to have anything related on GitHub.

                                                                  My main uncertainty about a protocol like this, where you can run sandboxed code remotely, is how to deal with DoS vulnerabilities by way of infinite loops, expensive queries, etc. Do you just kill a sandboxed task after it runs for X ms or allocates Y amount of memory?

                                                                  1. 2

                                                                    Seems like a losing battle, doesn’t it? If you somewhat trust the people sending you code, you could probably get away with Lua-style “max N instructions / max M ram” or something. But if you don’t trust the caller.. at least intuitively it feels like you end up in the realm of browser vendors, mounting an endless arms race as people learn to exploit new features you add.

                                                                    1. 1

                                                                      I’ve recently written about half of a spellserver (entrypoint, support library) in Monte, a dialect of E which supports auditors like DeepFrozen. Indeed, Monte modules are DeepFrozen and could plausibly be used as spells. The main difficulty is that we really did need both the cryptographic libraries as builtin routines and also E-style auditors, and we’ll need to develop orthogonal persistence as well.

                                                                      Along similar lines, I imagine that the author would argue that their work with Agoric is towards a platform which can host spellservers; it includes the necessary cryptographic primitives, code auditing, and orthogonal persistence. It’s also blockchain-oriented, which is explicitly something I don’t want to to require, but I am happy to see diversity in our ecosystem.

                                                                      I agree that even the most trivial of expensive queries can be debilitating. I have an old Monte issue about integer exponentiation, for example. The only solution I know of which reliably works is metainterpretation, where the runtime hosting the code is able to delimit itself somehow and then execute the entirety of the hosted code within that delimiter.

                                                                    1. 5

                                                                      Many objects with getters/setters are just structs with extra steps. As much as Meyer’s gets cited, most people seem to have skipped over the Uniform Access Principle when bringing OOP to languages.

                                                                      Getters and setters are a violation of UAP and a major failing of most languages which allow OOP, with the notable exceptions of Ruby, Objective-C and C#.

                                                                      1. 2

                                                                        Can you expand on what you mean here? I thought the whole argument for getters and setters was to make field access by clients be method calls, to follow UAP?

                                                                        The idea being that by making clients do ‘x.getFoo()’ instead of ‘x.foo’ you leave space to refactor without breaking clients later on.

                                                                        ie. in what way are getter/setters a violation of UAP?

                                                                        In my mind the thing I disagree with is not getters and setters, its UAP thats the problem.

                                                                        1. 3

                                                                          If I understand correctly “planning ahead” and using getter and setter methods are a workaround for the lack of UAP, it’s being treated as a property of a language not a program. Ruby and Objective-C don’t allow you to access members without going through a method (they force UAP), C# lets you replace member access with methods since it supports member access syntax for methods.

                                                                          1. 1

                                                                            C# lets you replace member access with methods since it supports member access syntax for methods.

                                                                            Python has the same feature (in fact, I’m pretty sure had it first). You can start with a normal member, then replace it with a method via the property decorator and if you want to, implement the full set of operations (get, set, delete) for it without breaking any previous consumers of the class.

                                                                            Of course, Python also doesn’t actually have the concept of public versus private members, aside from socially-enforced conventions, but your concern seems to be less whether someone can find the private-by-convention attributes backing the method, and more with whether x.foo continues to work before and after refactoring into a method (which it does, if the new foo() method is decorated with property).

                                                                            1. 1

                                                                              Of course, Python also doesn’t actually have the concept of public versus private members, aside from socially-enforced conventions

                                                                              I’m genuinely curious, as at first glance you seem to be an advocate of this, what’s the benefit of socially-enforced conventions over compiler-enforced privacy? Also, another thing I’ve been curious about not having programmed much in a language with these semantics, how does a language like Python handle something like the following:

                                                                              There’s a class I want to extend, so I inherit it. I implement a “private member” __name. The base class also implemented __name, does my definition override it?

                                                                              I’ve been wondering about that because if that’s the case, it seems like that would require people to know a lot of implementation details about the code their using. But for all I know, that’s not the case at all, so I’d be happy to hear someone’s perspective on that.

                                                                              1. 3

                                                                                There’s a class I want to extend, so I inherit it. I implement a “private member” __name. The base class also implemented __name, does my definition override it?

                                                                                It doesn’t. Double underscores mangle the attribute name in this case by prepending _<class name> to the original attribute name, which obfuscates external access. The attribute is only accessible by its original name in the class they are declared.

                                                                                Python docs: https://docs.python.org/3/tutorial/classes.html#private-variables

                                                                                1. 2

                                                                                  I’m genuinely curious, as at first glance you seem to be an advocate of this, what’s the benefit of socially-enforced conventions over compiler-enforced privacy?

                                                                                  It was my first day at the new job. I’d been shown around the office, and now I was at my new desk, laptop turned on and corporate email set up. I’d been told my first ticket, to help me get to know the process, would be changing the user-account registration flow slightly to set a particular flag on certain accounts. Easy enough, so I grabbed a checkout of the codebase and started looking around. And… immediately asked my “onboarding buddy”, Jim, what was going on.

                                                                                  “Well, that’s the source code”, he said. “Yeah, but is it supposed to look like that? “Of course it is, it’s encrypted, silly. I though you were supposed to be coming in with years of experience in software development!” Well, I said I’d seen some products that shipped obfuscated or encrypted code to customers, but never one that stored its own source that way. “But this way you have proper access control! When you’re authorized to work on a particular component, you reach out to Kevin, who’s the senior engineer on our team, and he’ll decrypt the appropriate sections for you, then re-encrypt when you’re done. That way you never see or use any code you’re not supposed to know about. It’s called Data Hiding, and it’s a fundamental part of object-oriented programming. Are you sure you’ve done this before?”

                                                                                  I sighed. And then recognition dawned. “Hey, wait”, I said, “this isn’t really encrypted at all! It’s just ROT13! Look, here this qrs ertvfgre_nppbhag is actually just def register_account…”

                                                                                  THWACK!

                                                                                  I’d been pointing at the code on my screen excitedly, and didn’t notice someone sneaking up behind me. Until he whacked me across the fingers, hard, with a steel ruler. “YOU ARE NOT AUTHORIZED TO READ THAT CODE!” he yelled, and then walked away.

                                                                                  “Who was that?”

                                                                                  “That was Kevin, the senior engineer I told you about. You really should be more careful, you’ll get written up to HR for an access violation. And if you accumulate three violations you get fired. Maybe they’ll let this one slide since you’re new and obviously inexperienced.”

                                                                                  “But how does anyone get anything done here?”, I asked.

                                                                                  “I told you – you ask Kevin to decrypt the code you’re supposed to work on.”

                                                                                  “But what if I need to use code from some other part of the codebase?”

                                                                                  “Then Kevin will liaise with senior engineers on other teams to determine whether you’re allowed to see their code. It’s all very correct according to object-oriented design principles!”

                                                                                  I goggled a bit. Jim finally said, “Look, it’s obvious to me now that you’ve never worked somewhere that followed good practices. I’m not going to tell on you to HR for whatever lies you must have put on your résumé to get hired here, but maybe you could tell me what you used to do so I can help you get up to speed on the way professionals work.”

                                                                                  So I explained that at previous jobs, you could actually see all the code when you checked it out, and there was documentation explaining what it all did, how to perform common tasks, what APIs each component provided, and so on, and you’d look things up and write the code you needed to write for your tasks and file a pull request that eventually got checked in after review.

                                                                                  Now Jim was goggling at me. “But… what if someone used the code in a way the original team didn’t want it to be used? How would you protect against that?”

                                                                                  “Well, there were conventions for indicating and documenting which APIs you were committing to support and maintain, and the policy was anyone could use those APIs any time. But if you needed something that wasn’t provided by any supported API, you’d talk to the team that wrote the component and work something out. Maybe they would say it was OK to use a non-supported API as long as you took responsibility to watch for changes, maybe they’d work with you to develop a new supported API for it, or come up with a better solution.”

                                                                                  Jim couldn’t believe what I was telling him. “But… just knowing which team wrote some other code is a violation of the Principle of Least Knowledge! That’s a very important object-oriented principle! That’s why everything that crosses boundaries has to go through Kevin. Why, if you could just go talk to other teams like that you might end up deciding to write bad code that doesn’t follow proper object-oriented principles!”

                                                                                  I tried my best to explain that at my previous jobs people trusted and respected each other enough that there wasn’t a need for fanatically-enforced controls on knowledge of the code. That we did just fine with a social-convention-based system where everybody knew which APIs were supported and which ones were “use at your own risk”. That there certainly weren’t senior engineers wandering among the desks with steel rulers – that senior engineers had seen it as their job to make their colleagues more productive, by providing tools to help people write better code more quickly, rather than being informational bottlenecks who blocked all tasks.

                                                                                  After I finished explaining, Jim shook his head. “Wow, that sounds awful and I bet the code they produced was pretty bad too. I bet you’re glad to be out of those old jobs and finally working somewhere that does things right!”

                                                                                  1. 1

                                                                                    So just to make sure I’m following, your argument is that if you need to use something that’s not included in the public API, compiler-enforced privacy requires you to talk to the team that developed the code if you need an extension to the API, while convention-enforced privacy requires that in order to make sure you don’t break anything you… talk to the team that developed the code so that you can work out an extension to the API?

                                                                                    1. 1

                                                                                      My argument is that in languages with enforced member-access/data-hiding, I can’t even think about using a bit of API that hasn’t been explicitly marked as available to me. If I try it, the compiler will thwack me across the hand with a steel ruler and tell me that code is off-limits. My only options are to implement the same thing myself, with appropriate access modifiers to let me use it, or somehow convince the maintainer to provide public API for my use case, but even that won’t happen until their next release.

                                                                                      In Python, the maintainers can say “go ahead and use that, just do it at your own risk because we haven’t finalized/committed to an API we’re willing to support for that”. Which really is what they’re saying when they underscore-prefix something in their modules. And Python will let me use it, and trust that I know what I’m doing and that I take on the responsibility. No steel rulers in sight.

                                                                                      A lot of this really comes down to Python being a language where the philosophy is “you can do that, but it’s on you to deal with the consequences”. And that’s a philosophy I’m OK with.

                                                                              2. 1

                                                                                I have mixed feelings about UAP, because I want to know when accessing a field is a trivial operation, and when it can run some expensive code.

                                                                                If a field is just a field, then I know for sure. If it could be a setter/getter, then I have to trust that author of the class isn’t doing something surprising, and will not make it do something surprising in the future.

                                                                                1. 1

                                                                                  You can’t know that in languages like Java which default to get_X() and set_Y() for even the simplest field access, if only to not break their interface when they need to add a simple range guard.

                                                                                  Languages without UAP will go to such lengths to emulate it using methods that you will seldom see a bare field exposed, at which point you can no longer know if a simple getter will return a local field or fire nuclear missiles.

                                                                                  1. 1

                                                                                    Yeah, IMO it’s a terrible idea for multiple reasons.

                                                                                    One, like you’re saying, it gives up a super powerful tool for improving readability in client code. If you’re using a language that has “property accessor” nonsense, every access to fields provided by a library may - for all you know - throw IO exceptions or have any other arbitrary behavior. With method calls being explicitly separate, you add a huge aid in optimizing for readers by reducing the number of possible things that can happen.

                                                                                    Two, it makes library authors think they can swap client field access for computation without breaking backwards compatibility, which is some sort of post-modernist academic fiction.

                                                                            1. 25

                                                                              Note a couple things:

                                                                              • With the 2021 edition, Rust plans to change closure capturing to only capture the necessary fields of a struct when possible, which will make closures easier to work with.
                                                                              • With the currently in-the-works existential type declarations feature, you’ll be able to create named existential types which implement a particular trait, permitting the naming of closure types indirectly.

                                                                              My general view is that some of the ergonomics here can currently be challenging, but there exist patterns for writing this code successfully, and the ergonomics are improving and will continue to improve.

                                                                              1. 12

                                                                                I have very mixed feelings about stuff like this. On the one hand, these are really cool (and innovative – not many other languages try and do async this way) solutions to the problems async Rust faces, and they’ll definitely improve the ergonomics (like async/await did).

                                                                                On the other hand, adding all of this complexity to the language makes me slightly uneasy – it kind of reminds me of C++, where they just keep tacking things on. One of the things I liked about Rust 1.0 was that it wasn’t incredibly complicated, and that simplicity somewhat forced you into doing things a particular way.

                                                                                Maybe it’s for the best – but I really do question how necessary all of this async stuff really is in the first place (as in, aren’t threads a simpler solution?). My hypothesis is that 90% of Rust code doesn’t actually need the extreme performance optimizations of asynchronous code, and will do just fine with threads (and for the remaining 10%, you can use mio or similar manually) – which makes all of the complexity hard to justify.

                                                                                I may well be wrong, though (and, to a certain extent, I just have nostalgia from when everything was simpler) :)

                                                                                1. 9

                                                                                  I don’t view either of these changes as much of a complexity add. The first, improving closure capturing, to me works akin to partial moves or support for disjoint borrows in the language already, making it more logically consistent, not less. For the second, Rust already has existential types (impl Trait). This is enabling them to be used in more places. They work the same in all places though.

                                                                                  1. 11

                                                                                    I’m excited about the extra power being added to existential types, but I would definitely throw it in the “more complexity” bin. AIUI, existential types will be usable in more places, but it isn’t like you’ll be able to treat it like any other type. It’s this special separate thing that you have to learn about for its own sake, but also in how it will be expressed in the language proper.

                                                                                    This doesn’t mean it’s incidental complexity or that the problem isn’t worth solving or that it would be best solved some other way. But it’s absolutely extra stuff you have to learn.

                                                                                    1. 2

                                                                                      Yeah, I guess my view is that the challenge of “learn existential types” is already present with impl Trait, but you’re right that making the feature usable in more places increases the pressure to learn it. Coincidentally, the next post for Possible Rust (“How to Read Rust Functions, Part 2”) includes a guide to impl Trait / existential types intended to be a more accessible alternative to Varkor’s “Existential Types in Rust.”

                                                                                      1. 6

                                                                                        but you’re right that making the feature usable in more places increases the pressure to learn it

                                                                                        And in particular, by making existential types more useful, folks will start using them more. Right now, for example, I would never use impl Trait in a public API of a library unless it was my only option, due to the constraints surrounding it. I suspect others share my reasons too. So it winds up not getting as much visible use as maybe it will get in the future. But time will tell.

                                                                                    2. 3

                                                                                      eh, fair enough! I’m more concerned about how complex these are to implement in rustc (slash alternative compilers like mrustc), but what do I know :P

                                                                                      1. 7

                                                                                        We already use this kind of analysis for splitting borrows, so I don’t expect this will be hard. I think rustc already has a reusable visitor for this.

                                                                                        (mrustc does not intend to compile newer versions of rust)

                                                                                        1. 1

                                                                                          I do think it is the case that implementation complexity is ranked unusually low in Rust’s design decisions, but if I think about it, I really can’t say it’s the wrong choice.

                                                                                      2. 4

                                                                                        Definitely second your view here. The added complexity and the trajectory means I dont feel comfortable using Rust in a professional setting anymore. You need significant self control to write maintainable Rust, not a good fit for large teams.

                                                                                        What I want is Go-style focus on readability, pragmatism and maintainability, with a borrow checker. Not ticking off ever-growing feature lists.

                                                                                        1. 9

                                                                                          The problem with a Go-style focus here is: what do you remove from Rust? A lot of the complexity in Rust is, IMO, necessary complexity given its design constraints. If you relax some of its design constraints, then it is very likely that the language could be simplified greatly. But if you keep the “no UB outside of unsafe and zero cost abstractions” goals, then I would be really curious to hear some alternative designs. Go more or less has the “no UB outside of unsafe” (sans data races), but doesn’t have any affinity for zero cost abstractions. Because of that, many things can be greatly simplified.

                                                                                          Not ticking off ever-growing feature lists.

                                                                                          Do you really think that’s what we’re doing? Just adding shit for its own sake?

                                                                                          1. 9

                                                                                            Do you really think that’s what we’re doing?

                                                                                            No, and that last bit of my comment is unfairly attributed venting, I’m sorry. Rust seemed like the holy grail to me, I don’t want to write database code in GCd languages ever again; I’m frustrated I no longer feel confident I could corral a team to write large codebases with Rust, because I really, really want to.

                                                                                            I don’t know that my input other than as a frustrated user is helpful. But I’ll give you two data points.

                                                                                            One; I’ve argued inside my org - a Java shop - to start doing work in Rust. The learning curve of Rust is a damper. To me and my use case, the killer Rust feature would be reducing that learning curve. So, what I mean by “Go-style pragmatism” is things like append.

                                                                                            Rather than say “Go does not have generics, we must solve that to have append”, they said “lets just hack in append”. It’s not pretty, but it means a massive language feature was not built to hammer that one itch. If “all abstractions must have zero cost” is forcing introduction of language features that in turn make the language increasingly hard to understand, perhaps the right thing to do is, sometimes, to break the rule.

                                                                                            I guess this is exactly what you’re saying, and I guess - from the outside at least - it certainly doesn’t look like this is where things are headed.

                                                                                            Two, I have personal subjective opinions about async in general, mostly as it pertains to memory management. That would be fine and well, I could just not use it. But the last few times I’ve started new projects, libraries I wanted to use had abandoned their previously synchronous implementations and gone all-in on async. In other words, introducing async forked the crate community, leaving - at least from my vantage point - fewer maintained crates on each side of the fork than there were before the fork.

                                                                                            Two being there as an anecdote from me as a user. ie. my experience of async was that it (1) makes the Rust hill even steeper to climb, regressing on the no. 1 problem I have as someone wanting to bring the language into my org; (2) it forked the crate community, such that the library ecosystem is now less valuable. And, I suppose, (3) it makes me worried Rust will continue adding very large core features, further growing the complexity and thus making it harder to keep a large team writing maintainable code.

                                                                                            1. 9

                                                                                              the right thing to do is, sometimes, to break the rule.

                                                                                              I would say that the right thing to do is to just implement a high-level language without obvious mistakes. There shouldn’t be a reason for a Java shop to even consider Rust, it should be able to just pick a sane high-level language for application development. The problem is, this spot in the language landscape is currently conspicuously void, and Rust often manages to squeeze in there despite the zero-cost abstraction principle being antithetical to app dev.

                                                                                              That’s systems dynamics that worries me a lot about Rust: there’s a pressure to make it a better language for app development at the cost of making it a worse language for systems programming, for the lack of an actual reasonable app dev language one can use instead of Rust. I can’t say it is bad. Maybe the world would be a better place if we had just a “good enough” language today. Maybe the world would be better if we wait until “actually good” language is implemented.

                                                                                              So far, Rust resisted this pressure successfully, even exceptionally. It managed to absorb “programmers can have nice things” properties of high level languages, while moving downwards in the stack (it started a lot more similar to go). But now Rust is actually popular, so the pressure increases.

                                                                                              1. 4

                                                                                                I mean, we are a java shop that builds databases. We have a lot of pain from the intersection of distributed consensus and GC. Rust is a really good fit for a large bulk of our domain problem - virtual memory, concurrent B+-trees, low latency networking - in theory.

                                                                                              2. 5

                                                                                                I guess personally, I would say that learning how to write and maintain a production quality database has to be at least an order of magnitude more difficult than learning Rust. Obviously this is an opinion of mine, since everyone has their own learning experiences and styles.

                                                                                                As to your aesthetic preferences, I agree with them too! It’s why I’m a huge fan of Go. I love how simple they made the language. It’s why I’m also watching Zig development closely (I was a financial backer for some time). Zero cost abstractions (or Zig’s case, memory safety at compile time) isn’t a necessary constraint for all problems, so there’s no reason to pay the cost of that constraint in all cases. This is why I’m trying to ask how to make the design simpler. The problem with breaking the zero cost abstraction rule is that it will invariably become a line of division: “I would use Rust, but since Foo is not zero cost, it’s not appropriate to use in domain Quux, so I have to stick with C or C++.” It’s antithetical to Rust’s goals, so it’s really really hard to break that rule.

                                                                                                I’ve written about this before, but just take generics as one example. Without generics, Rust doesn’t exist. Without generics (and, specifically, monomorphized generics), you aren’t able to write reusable high performance data structures. This means that when folks need said data structures, they have to go implement them on their own. This in turn likely increases the use of unsafe in Rust code and thus significantly diminishes its value proposition.

                                                                                                Generics are a significant source of complexity. But there’s just no real way to get rid of them. I realize you didn’t suggest that, but you brought up the append example, so I figured I’d run with it.

                                                                                                1. 2

                                                                                                  I guess personally, I would say that learning how to write and maintain a production quality database has to be at least an order of magnitude more difficult than learning Rust.

                                                                                                  Agree, but precisely because it’s a hard problem, ensuring everything else reduces mental overhead becomes critical, I think.

                                                                                                  If writing a production db is 10x as hard as learning Rust, but reading Rust is 10x as hard as reading Go, then writing a production grade database in Go is 10x easier overall, hand wavingly (and discounting the GC corner you now have painted yourself into).

                                                                                                  One thing worth highlighting is why we’ve managed to stick to the JVM, and where it bites us: most of the database is boring, simpleish code. Hundreds of thousands of LOC dealing with stuff that isn’t on the hot-path. In some world, we’d have a language like Go for writing all that stuff - simple and super focused on maintainability - and then a way to enter “hard mode” for writing performance critical portions.

                                                                                                  Java kind of lets us do that; most of the code is vanilla Java; and then critical stuff can drop into unsafe Java, like in the virtual memory implementation. The problem with that in the JVM is that the inefficiency of vanilla Java causes GC stalls in the critical code.. and that unsafe Java is horrifying to work with.

                                                                                                  But, of course, then you need to understand both languages as you read the code.

                                                                                              3. 4

                                                                                                I think the argument is to “remove” async/await. Neither C or C++ have async/await and people write tons of event driven code with them; they’re probably the pre-eminent languages for that. My bias for servers is to have small “manual” event loops that dispatch to threads.

                                                                                                You could also write Rust in a completely “inverted” style like nginx (I personally dislike that, but some people have a taste for it; it’s more like “EE code” in my mind). The other option is code generation which I pointed out here:

                                                                                                https://lobste.rs/s/rzhxyk/plain_text_protocols#c_gnp4fm

                                                                                                Actually that seems like the best of both worlds to me. High level and event driven/single threaded at the same time. (BTW the video also indicated that the generated llparse code is 2x faster than the 10 year old battle-tested, hand-written code in node.js)

                                                                                                So basically it seems like you can have no UB and zero cost abstractions, without async/await.

                                                                                                1. 5

                                                                                                  After our last exchange, I don’t really want to get into a big discussion with you. But I will say that I’m quite skeptical. The async ecosystem did not look good prior to adding async/await. By that, I mean, that the code was much harder to read. So I suppose my argument is that adding some complexity to language reduces complexity in a lot of code. But I suppose folks can disagree here, particularly if you’re someone who thinks async is overused. (I do tend to fall into that camp.)

                                                                                                  1. 2

                                                                                                    Well it doesn’t have to be a long argument… I’m not arguing against async/await, just saying that you need more than 2 design constraints to get to “Rust should have async/await”. (The language would be a lot simpler without it, which was the original question AFAICT.)

                                                                                                    You also need:

                                                                                                    1. “ergonomics”, for some definition of it
                                                                                                    2. textual code generation is frowned upon

                                                                                                    Without constraint 3, Rust would be fine with people writing nginx-style code (which I don’t think is a great solution).

                                                                                                    Without constraint 4, you would use something like llparse.

                                                                                                    I happen to like code generation because it enables code that’s extremely high level and efficient at the same time (like llparse), but my experience with Oil suggests that most “casual” contributors are stymied by it. It causes a bunch of problems in the build system (build systems are broken and that’s a different story). It also causes some problems with tooling like debuggers and profilers.

                                                                                                    But I think those are fixable with systems design rather than language design (e.g. the C preprocessor and JS source maps do some limited things)

                                                                                                    On a related note, Go’s philosophy is to fix unergonomic code with code generation (“go generate” IIRC). There are lots of things that are unergonomic in Go, but code generation is sort of your “way out” without complicating the language.

                                                                                          2. 1

                                                                                            I didn’t know about the first change. That’s very exciting! This is definitely something that bit me when I was first learning the language.

                                                                                          1. 7

                                                                                            It’s your code. If it crashes, you fix it. We like this idea because it pushes developers to deliver higher-quality code. The dread of taking those middle-of-the-night phone calls provides a little extra incentive to take another pass through your work before you ship.

                                                                                            Is that… even legal?

                                                                                            1. 14

                                                                                              Found the European!

                                                                                              TL;DR: This is very likely illegal in the EU, you can’t do that to employees. Part of our SRE team is in Sweden, and we had to to a lot of threading the needle to meet Swedish laws about right to rest when we set up on-call rotas. I would assume “you are always on call” would be illegal in most non-US countries.

                                                                                              1. 2

                                                                                                Thanks, I thought so.

                                                                                              2. 5

                                                                                                Yes, there is no law to absolve workers of responsibility.

                                                                                                1. 3

                                                                                                  Why would it be illegal?

                                                                                                  1. 8

                                                                                                    you can’t have all your employees on call all the time

                                                                                                    1. 14

                                                                                                      I parsed this as I’ve seen it: each team has its own on-call rotation.

                                                                                                  2. 1

                                                                                                    I don’t think this literally means they have a pager system that runs git blame on stack traces and pages the poor soul that touched it last (or even worse, just pages an entire team simultaneously).

                                                                                                    I took this to be an allusion to some permutation of full service ownership — as in, the engineering team responsible for building a service also maintains on an on-call rotation to take responsibility for that service in production. This is in contrast to the older pattern of “engineers chuck code over the wall into production and go home, while the SRE/ops folks deal with whatever hell rains down at 3AM”. I don’t see why this would be illegal, unless pages were happening so often that folks couldn’t sleep / live a life.