1. 1

    It seems odd that they don’t compare Superpack to their other homegrown compression tool, Zstd.

    1. 3

      For Rust, why does this say that you need no_std to get portable binaries? I don’t feel like this is true. If you just want a portable binary within the same OS, you can use the MUSL libc target and statically compile everything. It’s crazy easy.

      1. 1

        Suppose you need to write something on a bare metal architecture, no operating system even, just an assembly bootstrap.

        1. 1

          Ok, sure, then you’d need no_std. But if we’re just talking about a binary being portable between different environments with the same kernel, then Rust + musl libc gets you there.

          The biggest hit to portability is dynamic linking. Once you start dynamically linking, you have to worry about what version of your dependencies are on your users’ machines. Or your users have to worry about it, at least.

          Portability between different operating systems is a less important problem, though seemingly still interesting given how much attention Cosmopolitan libc gets.

          Portability to the extent of “I don’t even need an OS” is probably not that valuable.

      1. 1

        These are all interesting optimizations (had no idea char was 2x slower to iterate over than the u8s in a &str, though that makes some sense in hindsight).

        But I feel like this post is missing a full comparison of all the various algorithms and strategies tested. What was the final result? How much faster did it get?

        1. 2

          This post is part of a series, and benchmark results were in a previous post (this is seemingly out of order, because author first implemented them, and now it’s distilling this knowledge).

        1. 12

          Isnt this… Really big?

          1. 15

            It does seem like it. This is, to my knowledge, the first hugely popular I/O library which now lets its users use io_uring in a way which just looks like normal async file I/O.

            Rust seems like it is in a special position in that it’s a language with good enough performance for io_uring to matter, but with powerful enough facilities to make clean abstractions on top of io_uring possible.

            1. 6

              Isn’t the problem that Rust bet the farm on readiness-based APIs and now it turns out (surprise) that completion-based APIs are generally “better” and finally coming to Linux (after Windows completely embarrassed Linux on that matter for like a decade).

              1. 1

                It’s not a problem in practice. Rust’s futures model handels io-uring just fine. There was some debate over how to handle “cancellations” of futures, e.g. when Rust code wants to just forget that it asked the OS for bytes from a TCP socket. But the “ringbahn” research prototype found a clean solution to that problem.

                Actually, that entire blog is a wealth of information about Rust futures.

                1. 1

                  found a clean solution

                  I’d call that a stretch, considering that the “solution” pretty much foregoes futures altogether (and with that async/await) and largely rolls its own independent types and infrastructure.

                  So I’m not seeing how this is evidence for:

                  futures model handels io-uring just fine

                  I’d say its evidence of the opposite.

                  Actually, that entire blog is a wealth of information about Rust futures.

                  Actually, that blog is the reason why I asked the question in the first place.

                  1. 1

                    I’m getting a little out of my depth here, but my understanding is that ringbahn (which inspired the tokio implementation) is meant to be used under the hood by a futures executor, just like epoll/kqueue are used under the hood now. It’s a very thin interface layer.

                    Basically, from application code you start up a TCP socket using an async library with io-uring support. Then whenever you read from it and await, the executor will do ringbahn-style buffer management and interface with io-uring.

              2. 1

                There’s also hugely popular https://github.com/libuv/libuv/pull/2322

                (Since libuv isn’t modular it hasn’t officially landed yet, but the way I understand it, both projects are at about the same level of completion)

              3. 4

                Everything about rust is big now =)

                1. 3

                  Rust is the epitome of Big FOSS.

              1. 9

                There’s a lot wrong with this rant.

                First and foremost, the post is largely framed like there are two groups of people: the producers and consumers of Xorg/Wayland. You can see that framing in calls to “supply and demand” and mostly talking about what is most valuable to users, and ignoring ease of development as something to care about. Essentially, the post treats FOSS display servers as a product to consume.

                But this is free software. There’s no fundamental difference between producers and consumers. You aren’t paying for the development of a display server. You aren’t owed anything. If the maintainers of Xorg no longer enjoy maintaining it, they should stop. And they did. If that doesn’t work for you, the onus is on you to pick up development where it left off.

                If that model doesn’t work for you, go pick up Windows or macOS, where you are literally paying money to not have these problems. That’s awesome. But people who use Linux as a desktop are opting into a different model where they are using software distributed “WITHOUT WARRANTY”. It’s unreasonable to then complain that the people who screw around with this software in their free time have decided to stop or switch gears to something they enjoy more.

                And then there are some small nitpicks.

                Words like DPI, display scaling, framebuffer and such are meaningless

                Plenty of users have issues with wanting different DPI and display scaling on different monitors. I’m one of them. This is a very real feature of Wayland that impacts my life, and I’m sure many would agree.

                Nvidia makes hardware. Their job is not to sabotage their product so it can work with arbitrary bits of code out there. It is the job of software vendors to make their software work with the hardware in the best fashion (if they want to)

                Open sourcing a driver (or even just making it source-available) is not “sabotaging their product”. AMD does it just fine with their GPUs. I won’t speculate on why Nvidia doesn’t cooperate with open source, but I suspect that the reason would frustrate me.

                1. 1

                  If that model doesn’t work for you, go pick up Windows or macOS, where you are literally paying money to not have these problems. That’s awesome. But people who use Linux as a desktop are opting into a different model

                  This claim is fundamentally incompatible with… Something that I don’t know if there’s a name for.

                  Basically, there are two camps in Linux: the “LibreApple” camp (who want to replace Windows/etc with a FOSS equivalent and improve computing for everyone) and the “PowerUserOS” camp (who want an OS by power users for power users, a lot of whom see Linux’s niche status as a benefit, and who like to say “Linux is like Lego”).

                  By definition, LibreApple can’t be exclusive to hackers (“everyone” means “everyone”), which means you either literally teach the entire world to code, or you build Linux with a fundamental difference between producers and consumers.

                  Point is, you come down firmly in the “PowerUserOS” camp, to the point where you’re not even acknowledging the existence of LibreApple.

                  You could claim that “LibreApple means FSF and PowerUserOS means the open source crowd”, but 1) I’m not so sure that’s always true (see below), 2) you used the term “Free Software” so me describing you as in the “open source crowd” would make things less clear, and 3) there’s enough semantic confusion around the phrases “free software” and “open source” that I wanted to exclude them from my comment if at all possible.

                  From point 1 above: For instance, a lot of “LibreApple” people see Steam as a good thing for Linux, despite being a proprietary platform for (mostly) proprietary games. I don’t really want to get into this semantic discussion about FOSS though, I’m just preemptively responding to an expected response.

                  1. 1

                    The problem is that “LibreApple” doesn’t work. Apple has funds to be Apple. The Linux ecosystem is developed by volunteers for fun. For the majority of FOSS, development will always be by volunteers for fun.

                    Maybe if there were a company out there whose business model was “buy our linux-compatible software and we promise it will just work”, then there could be a LibreApple. But I don’t know of any such company. System76 is the closest I can come up with. Buy their stuff and it’ll probably just work. But it’s not like they have tons of control over Wayland/X11, since the display servers are written by volunteers for fun. So unless they want to take on the whole stack, there will still be problems.

                    I don’t know if the Linux ecosystem can house a company that’s paid to make the whole software stack Just Work. I hope it can. I know I’d pay for it.

                    1. 2

                      The Linux ecosystem is developed by volunteers for fun. For the majority of FOSS, development will always be by volunteers for fun.

                      This hasn’t been true for a long time. Most of the big projects (for example the kernel, glibc, systemd, GNOME) are backed by big companies. RedHat is not part of IBM and employs a load of the core developers for key parts of the system. It’s difficult for a volunteer-run project to keep up with these.

                      1. 1

                        I guess that’s fair. But I think the foundation of what I was saying is still true. If you’re not paying for a product, and instead are essentially relying on the enthusiasm of a volunteer (or a company donating to FOSS, which is not unlike volunteering), then we can’t look at things from the perspective of the software being a product. It’s not a product because no one is buying it.

                        That said, I admit the situation becomes more complicated with companies involved. They have more funds to throw at developers to maintain the small details that individual volunteers often don’t want to. But the incentive structure to do things like “support Xorg forever” is still missing. If there were a giant user base paying for Xorg that might stop paying if all their programs stopped working, maybe Wayland wouldn’t exist, or would have better backwards compatibility.

                1. 3

                  It’s much worse than that in some languages. Swift, for example, defines grapheme clusters as the unit for the length of a string and so the length of a string can change between releases of your Swift compiler.

                  1. 4

                    Unfortunately, grapheme clusters are the closest thing there is to a “character”, so any other option (except relying on the ICU library in your OS instead) would yield even more nonsensical results.

                    1. 1

                      I haven’t yet seen a situation where I actually want to know how many grapheme clusters are in a string. When would this be useful? Maybe if you’re impmementint a text rendering engine?

                      1. 1

                        Not grapheme clusters; characters.

                        1. 1

                          Ok, but same question. Even if there were a reliable way to count the number of “characters” in a string of UTF8, I’ve never encountered the need to do that. Maybe it’s just the kind of code I work on that it doesn’t come up?

                          Even if you’re presenting a string to the user, don’t we 99.9% of the time just pass the UTF8 bytes to a rendering library and let it do the rest?

                          On the other hand, I very consistently need the length of the string in bytes for low-level, memory-management type work.

                          1. 2

                            If you write web apps, everything you receive from an HTTP request is effectively stringly-typed simply because of the nature of HTTP. And many of the data types you will convert to as you parse and validate the user’s input will involve rules about length, about what can occur in certain positions, etc., which will not be defined in terms of UTF-8 bytes. They will be defined in terms of “characters”. Which, sure is a bit ambiguous when you translate into Unicode terminology, but generally always means either code points or graphemes.

                            And you can protest until you’re blue in the face that the person who specified the requirements is just wrong and shouldn’t ask for this, but the fact is that the people specifying those requirements sign your paychecks and have the power to fire you, and a large part of your job as a web app developer is to translate their fuzzy human-language specifications into something the computer can do. Which, again, almost never involves replacing their terms with “UTF-8 bytes”.

                            Amusingly this has become a problem for browsers, because the web specs are still written by people who think at least partly like you do, and so have a significant disconnect between things like client-side validation logic setting max length on a text input (which the spec writers tend to specify in terms of either a byte limit or a UTF-16 code-unit limit), and the expectations of actual humans (who use languages that don’t cleanly map one “character” to one byte and/or one UTF-16 code unit).

                            IIRC Twitter actually went for the more human definition of “character” when deciding how to handle all the world’s scripts within its (originally 140, now 280) “character” limit, which means users of, say, Latin script are at a disadvantage in terms of the byte length of the tweets they can post compared to, say, someone posting in Chinese script.

                            1. 2

                              I think the twitter example is an interesting one that I hadn’t thought of. Limits on lengths are usually due to concerns about denial-of-service attacks, but not for Twitter. It’s actually supposed to be a limit on some kind of abstract human idea of “characters”. I don’t envy having to figure that one out.

                  1. 3

                    That is a really complicated setup.

                    At first, I was thinking that I can’t imagine what you’d want 25Gbit for. But then again, I moved recently from a 400Mbit cable to a 50-ish DSL and I really, really don’t like DSL. (Side note, they’ve just announced they’re laying fiber in my place, contract is signed and this time next year I could be on gigabit).

                    I assume jumping from < 1Gbit to 10+ Gbit is just a natural next step. I mean, yes I don’t need that speed all the time. But it’d still be nice to click “Download” and just a minute later, the entire 100+ GB Elder Scrolls online is here.

                    1. 3

                      That is a really complicated setup.

                      It seems that he’s not even using multiple subnets - I’d say his setup is a lot simpler than mine. :)

                      I could imagine having 25 Gbps at home, but I’ve just started to deploy 10 Gbps in my internal network (between a few hosts) so it might be slightly overkill for me as well. My current max is 1000/100 but I’ve only opted for 100/100 as I don’t need more, and since I can’t have 1000 Mbps in upload…

                      1. 2

                        might be useful to scan the entire ipv4 address space in a couple of minutes (or even faster)

                        1. 3

                          If your ISP doesn’t block you, that’s a great way to end up on threat intelligence feeds and labelled as a bot.

                          1. 1

                            fine take a few more minutes :)

                      1. 5

                        tl;dr; They are sum types just like in Swift or F#.

                        1. 5

                          I hope we are going to see a post about how Rust immutability is amazing soon. After that we can have articles about every single ML feature that existed for decades that Rust successfully implemented.

                          1. 8

                            You mean, it’s not a good thing to implement great features of a language that basically nobody is using into a new language that makes things better and more appealing?

                            1. 2

                              No, I mean presenting it as a novel idea when in fact it is around for 30 years is kind of funny.

                              that basically nobody is using into a new language

                              Factually incorrect. Please have a look at the languages in the ML family.

                            2. 4

                              It’s no secret that Rust was heavily inspired by ML, C, C++, etc. Rust itself has very few new language features, the borrow checker being one of them.

                              But Rust appeals to the low-level systems crowd, and brings with itself a ton of nice-to-haves from ML and the like. So people who were previously stuck with C-like enums suddenly have nice options and want to talk about it.

                              1. 4

                                What’s so bad about highlighting the strengths of a programming language?

                              2. 2

                                PEP 634 is also bringing sum types to Python.

                              1. 3

                                What is stopping the community from building a net new (fully compatible) web browser at this point?

                                I would love to hear from those who have the relevant experience (Chromium/Firefox developers, hobbyist browser developers).

                                I see the answer to this come up often enough, that the endeavour is simply too big to try and make a new one at this point. I think if it’s worth it, then the amount of effort shouldn’t stop people from at least attempting to build something better.

                                I’m intentionally being naive here with the hopes to spark some discussion.

                                1. 4

                                  Rendering basic HTML is easy enough. Ensuring complex modern webapps like Google Docs work performantly is multiple orders of magnitude harder. Even Microsoft with all its corporate backing struggled to get the old Edge engine to run competitively.

                                  1. 3

                                    I’m curious what makes it orders of magnitude harder? Is it the amount of moving pieces? Is it the complexity of a specific piece needed to make modern web apps work? Maybe existing browser code bases are difficult to understand as a point of reference for someone starting out?

                                    1. 5

                                      A good way to understand the complexity of modern browsers is to look at standards that Mozilla is evaluating for support.

                                      You’ve got 127 standards from “new but slightly different notification APIs” to “enabling NFC, MIDI I/O, and raw USB access from the browser”. Now, obviously lots of these standards will never get implemented - but these are the ones that were important enough for someone to look at and consider.

                                  2. 3

                                    Drew DeVault goes through some of the challenges here. Short version: enormous complexity.

                                  1. 1

                                    I have a hard time figuring out how to get matrix setup and working. Like what the backend and frontend are and how they work. Am I not understanding what it is?

                                    1. 10

                                      TL;DR: If you want to try it out, download the Element client and let it walk you through making an account.

                                      You’ll have to choose a Matrix homeserver (like an email provider). If you won’t use it that frequently, the free matrix.org homeserver is good but slow. For more serious use, consider a subscription to Element Matrix Services, where they host a homeserver for you. Or you can try to self-host synapse. I wouldn’t.

                                      Other homeservers are being developed right now (Conduit is pretty cool). But none are ready for production just yet. And unfortunately the choice of homeserver is still important because your account will be tied to it. In the future, the “multi-homed accounts” feature will make this initial choice less important (hopefully).


                                      There are two basic components to understand if you’re just getting into Matrix, and the two components are best understood as an analog to email, which is really the only popular federated protocol today.

                                      There’s the Matrix homeserver, which is like your email provider. It’s where the messages and account information are stored. It’s what actually takes care of sending and receiving your messages. It’s where you sign up. Multiple people can have an account on the same homeserver. Synapse is the most popular homeserver right now, it’s developed by the team that founded Matrix, and it’s considered to be slow (Python) and is slated to be replaced.

                                      Then there’s Matrix clients. Just like email, Matrix is standardized. You can use any email client to get/send your Gmail, and you can use any Matrix client to get/send Matrix messages. Element is the most popular Matrix client (again made by the team that created Matrix). It’s the most feature-complete by far. It’s written in Electron, so it’s bloated. But it works fairly well.

                                      1. 3

                                        I wouldn’t.

                                        Can you elaborate ? We use a synapse server at work and it works.

                                        1. 1

                                          I should have been clearer. I meant that I don’t advise trying to self-host the homeserver at all. Self-hosting anything is a ton of work if done properly. Timely security updates, migrations, frequent (and tested) backups, and reliability all come to mind as things I personally don’t want to have to worry about. Element Matrix Services seems like a good deal for just a few people.

                                          1. 3

                                            But these challanges aren’t at all Synapse-specific, are they? Updates. migrations and proper backups are something you have to do with any server that you self-host. And after running a homeserver for a few years, the only migration I ever had to do is from an older to a newer PostgreSQL version by simply dumping the whole database and reading it back in. All schema migrations are done automatically by Synapse and I never had any problems with that. Hosting a Matrix server with Synapse is so easy if you compare it e.g. to hosting your own email server. And Synapse really is battle-tested because it’s dogfooded at the huge matrix.org homeserver instance.

                                            1. 1

                                              No they’re definitely not specific to Synapse. That was pretty much my point.

                                              And I know Synapse has put a ton of work into being easy to deploy. But I still won’t ever recommend managing infrastructure to anyone. It’s awesome that Synapse makes it as easy as possible for people like us to self-host it, but $5/month is probably well worth the lack of headache for most people.

                                        2. 2

                                          As far as I can tell, none of the homeserver implementations are ready for self-hosting – unless you disable federation, and then what’s the point?

                                          1. 3

                                            I’m not sure where you’re getting that impression. I’m hosting two different Synapse instances myself. I just update them when new releases come out; it’s been relatively painless.

                                            1. 1

                                              Can you please give a reason why you don’t think Synapse is ready for self-hosting? I’ve been doing it for years with enabled federation and I never had any serious problems.

                                              1. 1

                                                Sure. I’ve heard again and again that if you enable federation on Synapse and someone joins a large room, the server bogs down and starts chewing through resources. Has that changed?

                                                Also note that I’d be running it on an old laptop or a raspberry pi, just like I would run any chat server – IRC, Jabber, etc.

                                          2. 1

                                            .. I mean, probably? What exactly are you struggling with?

                                            1. 2

                                              Uh oh, now I feel even dumber. The main website has information about something called Synapse and there is “element” which is a frontend I believe, but how do you install a matrix server and start using it?

                                              1. 13

                                                My attempt at clarification:

                                                • Matrix is a protocol for a federated database, currently primarily used for chat
                                                • Synapse is the reference home server (dendrite, conduit, construct etc. are other alternatives)
                                                • Element is the reference client (there are versions of element for the web (electron), android and ios)
                                                • A user account is (currently) local to a home server
                                                • A chat room is federated and not located on a specific home server. The state is shared across home servers of all users that have joined the room.
                                                • There are P2P tests where the client and home server are bundled together on e.g. a mobile phone
                                                • Spaces are a way to organize rooms. Spaces are just a special case of a room and can include other rooms and spaces.
                                                1. 4

                                                  Thank you! That clarifies a lot. I was stuck thinking Matrix is the server. So, Matrix is a protocol for a federated database, that’s very interesting and cool.

                                                  1. 1

                                                    Is it legitimate for me, as a user rather than someone who’s interested in the infrastructure, to just think of Matrix being like a finer-grained version of IRC, where instead of (mainly) a few big networks there are more smaller networks and instead of joining e.g. #linux on freenode, I’d join e.g. #linux:somewhere …

                                                    Would I now discover ‘rooms’ by starting from a project’s website, for example, rather than just joining some set of federated servers and looking for rooms with appropriate names?

                                                    I just searched for ‘linux room matrix’ and the top hit was an Arch Linux room #archlinux:archlinux.org

                                                    (I don’t really want to join a general Linux room - just using it as an example)

                                                    1. 3

                                                      Well, generally NO. Most all matrix home servers are all joined together via the federated protocol. So if you join #archlinux:archlinux.org on homeserver A, and your BFF uses homeserver B, you will still see each other and communicate with each other in that room like if you were both on homeserver A.

                                                      One COULD create a non-federated home server, but that’s not the typical use case, and the reasons to do so would be odd. If you are doing for example a chat server for internal chat @ $WORK, using Matrix is probably a terrible idea. Zulip, Mattermost, etc are all better solutions for that use-case.

                                                      1. 2

                                                        Discovering rooms is currently a bit problematic, as room directories are per server. But a client can query room directories from any server (that allows public queries). Spaces will probably help a lot with room discovery, as they can form deep hierarchies.

                                                    2. 8

                                                      I did a video to try to explain it last year (which i should update, but it’s still usable, even if it calls Element by its old name of Riot): https://www.youtube.com/watch?v=dDddKmdLEdg

                                                      1. 3

                                                        I recommend starting off by just creating an account at app.element.io and using the default homeserver so you don’t have to host anything youself

                                                        1. 2

                                                          Synapse is a server implementation and Element is one of the clients.

                                                          Installing Synapse: https://matrix.org/docs/guides/installing-synapse

                                                          1. 1

                                                            Uh oh, now I feel even dumber.

                                                            Don’t. The Matrix project is pretty terrible at both naming and the new user experience.

                                                            1. 2

                                                              Not trying to hate on them or anything. @ptman ‘s comment above really helped.

                                                              1. 1

                                                                Yeah, I wish them every success - but what I guess I’ll call the introductory surface of the system is fairly painful.

                                                      1. 30

                                                        Last month, I had to install a package manager to install a package manager. That’s when I closed my laptop and slowly backed away from it.

                                                        The whole article is good, but this was my favorite part.

                                                        1. 7

                                                          i came here to quote that line too :) i laughed out loud when i hit it.

                                                        1. 3

                                                          In 2014 I decided that I was willing to spend more than the usual $50-150 on a router/firewall, if I could be sure that:

                                                          • it would not be a bottleneck on a gigabit symmetrical fiber connection
                                                          • I would not find it de-supported in 1-3 years, or lacking in an upgrade to some vital subsystem to prevent RCEs and DDOS.

                                                          After a bit I concluded that building a mini-ITX x86 machine and running Debian stable on it would work well.

                                                          It’s now seven years later and I should probably replace the USB stick that I use for a daily rsync of the contents.

                                                          The machine is described here: https://blog.randomstring.org/2014/11/09/a-new-firewall/

                                                          Because it runs a nicely supported OS, I get mail from it when there are necessary package updates. It handles firewalling, IPv6, wireguard, DNS, DHCP, and some basic stats collection.

                                                          It connects to a dumb 24-port gigabit switch that handles the other machines in the home office, and then has cross-house runs of ethernet to two other switches and three wifi access points, which are configured in bridge mode and don’t supply NAT or DHCP. I tend to assign consistent IPs via DHCP to every non-guest device, and keep a pool of 30 addresses for guests.

                                                          1. 1

                                                            What’s the advantage of reusing IPs from the pool for guests instead of generating new ones?

                                                            1. 3

                                                              I’m not sure what you mean. When a machine I don’t recognize asks for a DHCP-assigned IP, it comes from a pool that I allow to access the Internet and a very few local services; machines owned by my family or a few close friends get a subnet that allows rather more access.

                                                              (And certain machines that I “own”, like my TV, get addresses that aren’t allowed to talk to the outside world at all.)

                                                          1. 7

                                                            This is something Rust gets right with the from_{be,le}_bytes function defined on integer types. The author’s point is valid, but we can have the best of both worlds by just writing the function well once in the standard library and having everyone else use that.

                                                            No, there isn’t a huge speed difference between the author’s version and just casting an array of bytes into an integer. But with Rust’s way, you get the ergonomics, safety, and the extra speed.

                                                            1. 10

                                                              This; the argument is basically one of API. Instead of saying

                                                              #ifdef LITTLE_ENDIAN
                                                              int x = swap_bytes(read_from_network());
                                                              #else
                                                              int x = read_from_network();
                                                              #endif
                                                              

                                                              it’s nicer if you can always write:

                                                              int x = bigendian_to_native(read_from_network());
                                                              
                                                            1. 10

                                                              What I want to know is if I start learning Rust now, how confident can I be that my knowledge is applicable in a few years, or if the best practices will be deprecated until then. The impression I always get from listening into Rust conversations is that there is a lot of change going on, old compilers (such as those usually distributed by system package managers) are easily outdated making me feel like it might be better to wait a bit for the language to stabilize. Is that mistaken or is it true that Rust isn’t a stable language to learn, like C or Lisp?

                                                              1. 21

                                                                Mistaken. I started learning Rust in 2013 (pre-1.0), and most things I learned about Rust then is still applicable now and I am confident it will continue to be applicable in 5 years. Rust is a stable language that has been stable since 2015. I actually don’t recall any best practices that has been deprecated.

                                                                Hope this helped.

                                                                1. 3

                                                                  I am probably not the most qualified person to answer this (i.e. i am not a rust expert), but, as i understand, rust has actually been a quite stable language since v 1.0, so i wouldn’t worry too much about stability.

                                                                  I’d add to that, that in general in the software industry technologies and best practices change all the time (within reason), so I’d keep an open mind and give it a shot. If you end up liking it, I am sure you’d find ways to stay up to date with any evolution of the ecosystem without even noticing.

                                                                  My 2 cents…

                                                                  1. 2

                                                                    old compilers are easily outdated

                                                                    Rust has a rare approach to version management: they guarantee backwards compatibility of ‘stable’ features across upgrades, but expect you to always use a very recent version of the compiler (and provide automatic updates).

                                                                    I’m not sold on it, personally:

                                                                    • I don’t trust anyone not to introduce regressions via automatic updates. In fairness to the rust team, I’m not aware of any regressions that made it out, and they have very good automatic testing to ensure they don’t introduce any.
                                                                    • If you use a library that requires ‘nightly’ (non-stable) features, your code will most likely be broken when those features change in an update.

                                                                    If you want stability, install a ‘stable’ release; your code will continue to compile without modification for the foreseeable future, thanks to the ‘editions’ approach to backwards compatibility.

                                                                    It’s true that there are many changes to the language in each (frequent) release, but the vast majority are “something that you would expect to work was broken, has now been fixed”.

                                                                    1. 8

                                                                      I used Rust in production for algorithmic trading for a year. (I have since moved on to other jobs and I no longer use Rust in production.) I was responsible for technical decisions, and I insta-updated Rust compiler as soon as releases were out. I had no problems. It does work as advertised.

                                                                      1. 1

                                                                        I insta-updated Rust compiler as soon as releases were out. I had no problems. It does work as advertised.

                                                                        But this is what annoys me. I run apt upgrade maybe once every week or two, and would like everything to be done, having to pay attention to compiler updates because a library depends on it not something I want to see.

                                                                        1. 6

                                                                          The apt-provided compiler is the compiler used by Debian/Ubuntu for building their system and their packages. It is not meant for general development - especially if it involves e.g. cross-compilation - unless you target Debian/Ubuntu. If you would like that policy to be different, I recommend giving feedback to the packager team, they are a great bunch :). FWIW: This notion is far more pronounced when talking about Rust then about C, but it is also present there - the C compiler in “build-essentials” is the one used by the system. C just happens to be way more settled.

                                                                          I also want to add that libraries switching to higher minimum versions is becoming rarer and rarer, I often find myself lagging the stable compiler by at least 2 versions (and that as a Rust project member!).

                                                                          1. 1

                                                                            If you would like that policy to be different, I recommend giving feedback to the packager team, they are a great bunch :).

                                                                            Can you tell me where I can do this?

                                                                            1. 1
                                                                      2. 7

                                                                        In maintaining ggez for about four years, I’ve had compiler updates break the build twice that I can recall. Both times the breakage was in a dependency. Once it was a small but for realsies regression, once it was tightening the rules around unsafe code a little. In both cases it was a bit of a nuisance but that’s all.

                                                                        IMO this is acceptable. It’s kinda hateful, but dependencies break sometimes; this is not a Rust-specific thing. If you don’t want to put up with it, don’t use them. A lot of the more stable crates outright say what the oldest version of rustc they support is, which is nice, but for something like ggez that has a huge transitive dependency tree you’re basically stuck using the most recent one.

                                                                        1. 3

                                                                          Using a library on nightly has been, in my experience, problematic with updating compiler versions. That’s exactly how it’s supposed to be, though.

                                                                          Libraries on stable, however, have never once broken for me on a compiler upgrade, and it’s something I’ve done fairly often over the last two years using Rust in production.

                                                                          1. 3

                                                                            The (valid!) problem most people highlight is that if e.g. the stdlib adds a function in 1.51 and a dependency starts using it, you have to update your compiler. Given that Rust just added const generics and a lot of common libraries like heapless and bitvec have been waiting for it for a while, 1.51 is also a release that will most likely be a release where that is more pronounced.

                                                                          2. 2

                                                                            Rust has a rare approach to version management: they guarantee backwards compatibility of ‘stable’ features across upgrades, but expect you to always use a very recent version of the compiler (and provide automatic updates).

                                                                            I’m not sure why you consider this rare. “Automated upgrades, declaring the version of the API that you want in a manifest file, and a fanatical approach to backwards compatibility” sounds like Windows 10, which might be one of the single most widely-used pieces of software in existence.

                                                                            1. 1

                                                                              Rust has a rare approach to version management: they guarantee backwards compatibility of ‘stable’ features across upgrades, but expect you to always use a very recent version of the compiler (and provide automatic updates).

                                                                              I want to push back here slightly: it’s true, but this is the status quo. Currently, we support the current compiler only and rarely backport fixes. We have a lot of change control going on and ensuring the backwards compatibility here is already a task we spend much time on. But that does not mean that this policy is set in stone, but supporting older compilers requires resources and we (still) are a young language. In general, we prefer to either do things proper or not at all and currently, the needle is still on “not at all”. For that reason, I wouldn’t call it “an approach”, only “the current approach”.

                                                                              We make the approach easy by releasing often and predictably (we never missed a release date), which ensures small deltas. Plus above mentioned change control mechanism (CI, Code Review and the big hammer: crater).

                                                                              https://github.com/rust-lang/crater

                                                                          1. 6

                                                                            Not to be confused with RON, Rusty Object Notation.

                                                                            1. 3

                                                                              So this sounds pretty good. Are there detailed code examples of using code intended for shared nothing, and “normal” code? Basically the blurb sounds great but I want to see some of the substance.

                                                                              1. 8

                                                                                Hi! I’m the author of Lunatic. I have built a demo telnet chat application with it in Rust and also written a blog post about the structure of the app. I hope this helps you to get a feeling for the code structure.

                                                                                There is no separation between shared nothing and “normal” code. When you spawn a new process, the closure you pass to the spawn function will run in its own process. Everything you do inside this closure can’t access any memory from the parent process. That’s why the spawn function takes an initial context that will be copied into the child’s memory space. From the developers perspective it should be just writing “regular” Rust code.

                                                                                This is a Rust specific example, but for different languages this interaction is going to look differently. I always try to make it feel native to the language and libraries. Lunatic should feel like “just a library” for the developer.

                                                                                1. 10

                                                                                  Can you really achieve “the massive concurrency of Erlang” using web assembly? The Erlang virtual machine and runtime are tuned for this use case (millions of communicating processes) and web assembly is not. Erlang processes have a very small memory footprint, with an initial default heap size of 233 words; the heap grows and shrinks dynamically as needed. A quick peek at Lunatic suggests a process memory is sized as a multiple of 64K blocks. There are other issues, like the speed of IPC. If you want to make this claim, it would be interesting to show comparative benchmarks of massive concurrency in Erlang vs Lunatic.

                                                                                  1. 8

                                                                                    The wasm spec itself requires that all memory used by a wasm process is a multiple of the wasm page size, which is 64Ki. So there’s probably nothing the author can do about that. At least nothing simple.

                                                                                    But you raise a valid point. If every wasm process uses its minimum amount of memory, then you can only spawn ~15k processes per GB of host memory (I’m sure I introduced error somewhere due to GB/GiB).

                                                                                    This is probably plenty for most applications, but I guess Erlang could fill use cases where you need 100x that.

                                                                                    1. 8

                                                                                      There is nothing inherently “heavy” about WebAssembly. In practice allocating 64Kb/process doesn’t mean much, as most operating systems utilise virtual memory and if you don’t use the memory you don’t pay for it. I’m not sure what you mean with IPC speed, the overhead of sending messages between processes in Lunatic should be as optimised as it gets. Lunatic also uses one of the fastest context switching implementations available. I think that it’s a bit unfair to dismiss a new project just because it can’t compete with one that had decades of optimisations put into it right from the get-go.

                                                                                      1. 5

                                                                                        I didn’t dismiss your project, I asked to see a justification of your claims, which I think is a reasonable request. And you provided a justification, which was helpful, thanks.

                                                                                    2. 3

                                                                                      My superficial reaction to this is that it sounds more like a goroutine model than an actor model. Which is not bad, but basically more low level (for better and worse).

                                                                                      Given async/await’s backlash you’re probably arriving at the right time with this.

                                                                                      1. 2

                                                                                        Actors aren’t hard to implement on top of (goroutines/ dispatch queues/ etc.) I’ve done it, in C++, as the concurrency substrate for my work project. Mostly you just need lambdas and a reasonably efficient producer/consumer queue thereof.

                                                                                        The lambdas here might be more like IPC messages since the actors would often be in separate processes. But again, that’s not rocket science, and I know of other Actor libraries that already support it because they support distributed actors.

                                                                                        (In my experience, Actors and async/await are not an either/or. One of the frustrations of working with my library is that actor messages are one-way, so you have to use callbacks (delivered asynchronously) to get responses, which leads to the familiar callback-hell, or else formalize the response as another actor message sent the opposite direction, which requires the initiator to keep intermediate state data around. I’ve really, really been wishing for C++20 coroutines with their async/await features!

                                                                                  1. 2

                                                                                    Very creative. I think if I were trying to automate a solution to the same problem, I would just power cycle the fridge for fifteen minutes every hour or so regardless of the temperature.

                                                                                    This guy’s solution is better tho.

                                                                                    1. 47

                                                                                      I’m so tired of rehashing this. Pointing out that SemVer is not 100% infallible guarantee, or that major versions don’t always cause major breakage adds nothing new.

                                                                                      Lots of projects have a Changelog file where they document major changes, but nobody argues that reading changelogs would hurt you, because it may not contain all tiniest changes, or mention changes that would discourage people from upgrading, staying on insecure versions forever, etc.

                                                                                      SemVer is just a machine-readable version of documentation of breaking changes.

                                                                                      1. 23

                                                                                        Yes, and the article tries to succinctly sum up what value can be derived from that and what fallacies await. I’d have to lie to have ever seen it summed up thought that lens in one place.

                                                                                        I’m sorry it’s too derivative to your taste, but when the cryptography fire was raging, I was wishing for that article to exist so I can just paste it instead of extensive elaborations in the comments section.

                                                                                        1. 11

                                                                                          I thought the same thing initially, but it could also be coming from the perspective of using Rust frequently, which is strongly and statically typed. (I don’t actually know how frequently you use it; just an assumption.)

                                                                                          A static/strong type system gives programmers a nice boundary for enforcing SemVer. You mostly just have to look at function signatures and make sure your project still builds. That’s the basic promise of the type system. If it builds, you’re likely using it as intended.

                                                                                          As the author said, with something like Python, the boundary is more fuzzy. Imagine you write a function in python intended to work on lists, and somebody passes in a numpy array. There’s a good chance it will work. Until one day you decide to add a little extra functionality that still works on lists, but unintentionally (and silently) breaks the function working with arrays.

                                                                                          That’s a super normal Python problem to have. And it would break SemVer. And it probably happens all the time (though I don’t know this).

                                                                                          So maybe for weakly/dynamically typed languages, SemVer could do more harm than good if it really is unintentionally broken frequently.

                                                                                          1. 8

                                                                                            That’s all very true!

                                                                                            Additionally what I’m trying to convey (not very successfully it seems) is that the reliance on that property is bad – even in Rust! Because any release can break your code even just by introducing a bug – no matter what the version number says. Thus you have to treat all versions as breaking. Given the discussions around pyca/cryptography this is clearly not common knowledge.

                                                                                            The fact that this is much more common in dynamic languages as you’ve outlined is just the topping.

                                                                                            I really don’t know what I’ve done wrong to warrant that OP comment + upvotes except probably hitting some sore point/over-satiation with these topics in the cryptography fallout. That’s a bummer but I guess nothing I can do about it. 🧘

                                                                                            1. 7

                                                                                              Car analogy time: You should treat cars as dangerous all the time. You can’t rely on seatbelts and airbags to save you. Should cars get rid of seatbelts?

                                                                                              The fact that SemVer isn’t 100% right all the time is not a reason for switching to YOLO versioning.

                                                                                              1. 3

                                                                                                Except that SemVer is not a seatbelt, but – as I try to explain in the post – a sign saying “drive carefully”. It’s a valuable thing to be told, but you still have to take further measures to ensure safety and plan for the case when there’s a sign saying “drive recklessly”. That’s all that post is saying and nothing more.

                                                                                                1. 2

                                                                                                  Seatbelts reduce the chance of death. Reading a changelog reduces the chance of a bad patch. Trusting semver does not reduce the chance of an incompatible break.

                                                                                                  1. 6

                                                                                                    I really don’t get why there’s so much resistance to documenting known-breaking changes.

                                                                                                    1. 3

                                                                                                      I really don’t get why there’s so much resistance to documenting known-breaking changes.

                                                                                                      I mean you could just…like…read the article instead of guessing what’s inside. Since the beginning you’ve been pretending the article’s saying what it absolutely isn’t. Killing one straw man after another, causing people to skip reading because they think it’s another screech of same-old.

                                                                                                      I’m trying really hard to not attribute any bad faith to it but it’s getting increasingly harder and harder so I’m giving up.

                                                                                                      Don’t bother responding, I’m done with you. Have a good life.

                                                                                                      1. -1

                                                                                                        mean you could just…like…read the article instead

                                                                                                        So where in that article do you say why people don’t want to document known breaking changes ?

                                                                                                        Offtopic: That was really hard to read. Too many fat prints and

                                                                                                        quotes

                                                                                                        with some links in between. It just destroyed my reading flow.

                                                                                                        I also think the title “will not save you” is obviously telling everything about why people are just not reading it. It’s already starting with a big “it doesn’t work”, so why should I expect it to be in favor of it ?

                                                                                                        1. 4

                                                                                                          So where in that article do you say why people don’t want to document known breaking changes ?

                                                                                                          Well, the pyca/cryptography team documented that they were rewriting in Rust far in advance of actually shipping it, and initially shipped it as optional. People who relied on the package, including distro package maintainers, just flat-out ignored it right up until it broke their builds because they weren’t set up to handle the Rust part.

                                                                                                          So there’s no need for anyone else to cover that with respect to the cryptography fight. The change was documented and communicated, and the people who later decided to throw a fit over it were just flat-out not paying attention.

                                                                                                          And nothing in SemVer would require incrementing major for the Rust rewrite, because it didn’t change public API of the module. Which the article does point out:

                                                                                                          Funny enough, a change in the build system that doesn’t affect the public interface wouldn’t warrant a major bump in SemVer – particularly if it breaks platforms that were never supported by the authors – but let’s leave that aside.

                                                                                                          Hopefully the above, which contains three paragraphs written by me, and only two short quotes, was not too awful on you to read.

                                                                                                          1. 1

                                                                                                            Thanks, your summary is making a good point, and yes the original blogpost was hard to read, I did not intend this to be a troll.

                                                                                                            And nothing in SemVer would require incrementing major for the Rust rewrite

                                                                                                            Technically yes, practically I know that many rust crates do not increment the minimum required rust compiler version until a major version. So fair enough, semver in its core isn’t enough.

                                                                                                2. 3

                                                                                                  AFAIU, I think the OP comment may be trying to say that they agree with and in fact embrace the following sentence from your article:

                                                                                                  Because that’s all SemVer is: a TL;DR of the changelog.

                                                                                                  In particular, as far as I can remember, trying to find and browse a changelog was basically the only sensible thing one could do when trying to upgrade a dependency before SemVer became popular (plus keeps fingers crossed and run the tests). With the main time waster being trying to even locate and make sense of the changelog, with basically every project showing it elsewhere, if at all. (Actually, I seem to remember that finding any kind of changelog was already a big baseline plus mark for a project’s impression of quality). As such, having a hugely popular semi-standard convention for a tl;dr of the changelog is something I believe many people do find super valuable. They know enough to never fully trust it, similarly as they’d know to never fully trust a changelog. Having enough experience with changelogs and/or SemVer, they however do now see substantial value in SemVer as a huge time saver, esp. compared to what they had to do before.

                                                                                                  Interestingly, there’s a bot called “dependabot” on GitHub. I’ve seen it used b a team, and what it does is track version changes in dependencies, and generate a summary changelog of commits since last version. Which seems to more or less support what I wrote above IMO.

                                                                                                  (Please note that personally I still found your article super interesting, and nicely naming some phenomena that I only vaguely felt before. Including the one I expressed in this post.)

                                                                                                  1. 2

                                                                                                    I think there is something a bit wrong about the blanket statement that others shouldn’t rely on semver. I suspect that for many projects, trying one’s best to use the API as envisioned by the author, and relying on semver, will in practice provide you with bugfixes and performance improvements for free, while never causing any major problems.

                                                                                                    I like the parts of this blog post that are pointing out the problems here, but I think it goes way too far in saying that I “need to” follow your prescribed steps. Some of my projects are done for my own enjoyment and offered for free, and it really rubs me the wrong way when anyone tells me how I “should” do them.

                                                                                                    [edited to add: I didn’t upvote the top level comment, but I did feel frustrated by reading your post]

                                                                                                    1. 1

                                                                                                      I’m not sure to respond to that. The premise of the article it that people are making demands, claiming it will have a certain effect. My clearly stated goal is to dissect those claims, so people stop making those demands. Your use case is obviously very different so I have no interest to tell you to do anything. Why am I frustrating you and how could I have avoided it?

                                                                                                      1. 3

                                                                                                        My negative reaction was mostly to the section “Taking Responsibility”, which felt to me like it veered a bit into moralizing (especially the sentence “In practice that means that you need to be pro-active, regardless of the version schemes of your dependencies:”). On rereading it more carefully/charitably, I don’t think you intended to say that everyone must do it this way regardless of the tradeoffs, but that is how I read it the first time through.

                                                                                                  2. 9

                                                                                                    Type systems simply don’t do this. Here’s a list of examples where Haskell’s type system fails and I’m sure that you can produce a similar list for Rust.

                                                                                                    By using words like “likely” and “mostly”, you are sketching a sort of pragmatic argument, where type systems work well enough to substitute for informal measures, like semantic versioning, that we might rely on the type system entirely. However, type systems are formal objects and cannot admit such fuzzy properties as “it mostly works” without clarification. Further, we usually expect type-checking algorithms to not be heuristics; we expect them to always work, and for any caveats to be enumerated as explicit preconditions.

                                                                                                    1. 2

                                                                                                      Also there were crate releases where a breaking change wasn’t catched because no tests verified that FooBar stayed Sync/Send.

                                                                                                      1. 1

                                                                                                        All I meant is that languages with strong type systems make it easier to correctly enforce semver than languages without them. It’s all a matter of degree. I’m not saying that languages like Rust and Haskell can guarantee semver correctness.

                                                                                                        But the type system does make it easier to stay compliant because the public API of a library falls under the consideration of semver, and a large part of a public API is the types it can accept and the type it returns.

                                                                                                        I’m definitely not claiming that type systems prevent all bugs and that we can “rely entirely on the type system”. I’m also not claiming that type systems can even guarantee that we’re using a public API as intended.

                                                                                                        But they can at least make sure we’re passing the right types, which is a major source of bugs in dynamically typed languages. And those bugs are a prominent example of why OP argues that SemVer doesn’t work—accidental changes in the public API due to accepting subtly different types.

                                                                                                  1. 4

                                                                                                    It’s much easier to convince people with a PoC than with a paper.

                                                                                                    1. 7

                                                                                                      Some would even say PoC||GTFO

                                                                                                      1. 1

                                                                                                        That’s really not how it works for most real-world vulnerabilities today.

                                                                                                      2. 2

                                                                                                        A method of computing a factorisation more efficiently than current state of the art methods doesn’t imply that actually computing such a factorisation is cheap.

                                                                                                        1. 2

                                                                                                          Could be dangerous to submit a PoC for this kind of thing

                                                                                                          1. 2

                                                                                                            Probably a couple of days less dangerous than submitting the paper, at best. There’s been revisions of this paper around for a while as far as I can tell. If it posed any real threat to RSA, we’d have seen something by now.

                                                                                                            That said, I hope I’m wrong. A world with broken RSA is a more interesting world to live in.

                                                                                                          2. 1

                                                                                                            There is some pseudocode in the paper. I don’t know if that counts.

                                                                                                            1. 2

                                                                                                              PoC

                                                                                                              Proof of concept isn’t some pseudocode for me. Proof would be an example with code you can execute to verify it along with some measured times in this case. Should be easy enough to publish some python code that can do this.

                                                                                                          1. 18

                                                                                                            TL;DR: “I don’t know”