1. 19
  1.  

  2. 6

    This is one of the warts in rust that contributed to me writing way less rust code

    It was very frustrating to have the compiler reject programs that were obviously correct, and then have to model how the compiler was approximating borrow-correctness, and come up with an alternative solution that satisfied it

    Makes me excited to get back into rust at some point in the future :)

    1. 3

      Rust has a very high bar for “breaking” user code. When they say some code will break, it doesn’t mean you’ll spend a week fixing your project. It means 1 in 1000 libraries may need a couple of lines changed, because they already had a bug that wasn’t detected before.

      1. 5

        It is mostly an change that empowers developers; but it also is a change that will cause some existing code to break.

        This has been my impression of Rust, and why I’ve avoided doing much with it. It seems like it’s just a constantly moving target. I remember in its earlier days whole features (like classes) just being nuked out of existence. It may have settled down some now; I haven’t been keeping as close an eye on it.

        1. 27

          Since 1.0, this is the first major breaking change that I can recall. It’s also worth pointing out that any code that does break under NLL was fundamentally unsound – it shouldn’t have been compiling in the first place, but was, due to the limitations of the then-current lifetime analyzer. It’s also going to downgrade these breakages to just issuing warnings that this unsound code is going to stop compiling sometime in the future.

          My personal take is that, in a language that’s trying to prioritize safety and correctness, privileging correctness over the sanctity of “existing code that compiles” is the right move.

          Sure, in pre-release, there was a lot of churn as all manner of different ideas were tried out (like Classes) and subsequently removed. That’s, uh, why it was in pre-release. You need to get feedback somehow.

          1. 2

            It’s also going to downgrade these breakages to just issuing warnings that this unsound code is going to stop compiling sometime in the future.

            What’s the practical difference between “unsound” code that compiles and runs versus “undefined” or “implementation defined” behavior in C and C++?

            I suppose it’s a moot issue because there’s only one Rust implementation?

            1. 4

              What’s the practical difference between “unsound” code that compiles and runs versus “undefined” or “implementation defined” behavior in C and C++?

              I don’t think there’s any relationship there at all. Undefined/implementation defined code in C is code where the standard says, essentially, “in these cases the compiler can do whatever: compile, reject, crash, launch nethack”.

              Unsound code that was compiling under the AST based borrow checker in Rust were simply compiler bugs — explicitly disallowed things that slipped past due to defects in the borrow checker. The analogous situation in C is again a compiler bug

              1. 1

                What’s the practical difference between “unsound” code that compiles and runs versus “undefined” or “implementation defined” behavior in C and C++?

                There might be a difference between “this is unsound, we don’t know if the code is valid”, and “there is a data race” or some other genuine undefined behaviour.

                If you overlap borrows, one of which is mutable, your program should be rejected. But if you’re single threaded, overlapping borrows should be just fine. Perhaps confusing and bug prone, but as long as you don’t have any actual concurrent access, it should work.

                That’s most likely why there’s an unsafe mode: it’s like telling the compiler “I know you can’t prove my program doesn’t have any data races, but trust me, there isn’t any”.

                1. 4

                  But if you’re single threaded, overlapping borrows should be just fine.

                  Overlapping borrows are unsound even on a single thread. The canonical example of the borrow checker people use in intro talks is single threaded:

                  let v: &mut Vec<T> = ...;
                  let x0 = &v[0]; // overlapping borrow
                  v.push(...); // potential resize invalidating x0
                  println!("{:?}", x0); // use-after-free
                  

                  Or this more blatant case:

                  let o: &mut Option<String> = ...;
                  if let Some(s) = o { // overlapping borrow
                      *o = None; // string no longer has owner, is dropped
                      println!("{}", s); // use-after-free
                  }
                  
                  1. 1

                    Crap, didn’t think of pointer invalidation. Good point.

              2. 1

                It’s also worth pointing out that any code that does break under NLL was fundamentally unsound

                I don’t think this is true:

                https://github.com/rust-lang/rust/issues/59159

                This issue doesn’t mention unsafety, just inconvenience for them.

                1. 1

                  That’s not my reading of it.

                  From reading the various issues and reasoning behind stacked borrows it seems that:

                  That particular lint identifies an unintended pattern that was allowed to compile as a two-phase borrow. The pattern is unintended, and undesirable, because it violates fundamental validity guarantees that all Rust code, safe or unsafe, must adhere to in order for there to be a coherent set of rules (“stacked borrows”) about pointer aliasing in Rust that unsafe code and compiler optimizations can be asked to adhere to.

                  The two phase borrow pattern in question, in violating those rules, creates situations in which unsafe code or compiler optimizations that follow the rules will nevertheless result in safety and/or validity issues. This was argued about for several months with a bunch of people proposing various ways to make this particular 2PB pattern “work” with aliasing rules — but none of them seems to have managed a working solution.

                  Hence, the lint and eventual removal.

              3. 12

                oh, maybe my standard is lower than yours but with the language itself, I was always impressed how remarkedly well managed stability. But yes, they fix compiler bugs eventually. I was once affected and got a pull request to my crate from the compiler team to fix it. The compiler release before issued a warning.

                So, this affects compiler bugs which might lead to unsafe code. The compiler issued warnings for some releases, now it will stop compiling that code.

                In my view, a remarkable balance between stability and upholding the fundamental guarantees of rust.

                (the library ecosystem is a different story)

                1. 4

                  Maybe not the best wording on the author’s part, but if you read the rest of the post you would have read that only unsound code is affected, and it will continue to compile for now; only a future-compatibility warning will be raised.

                  Rust has strong backwards compatibility guarantees, and has since the 1.0 release in 2015. It’s only a moving target if you want to use the latest and greatest features, which are admittedly being added at a considerable rate. But that’s only a drawback if the features are not worth the maintenance overhead, which so far has not been a problem for Rust.

                  1. -1

                    ‘Unsound’ does not mean incorrect, it just means they can’t prove it is sound.

                    only a future-compatibility warning will be raised.

                    It is not useful to be told working code will maybe fail to compile at some nebulous point in the future when you specifically chose a stable release and edition to work with.

                    1. 3

                      ‘Unsound’ does not mean incorrect, it just means they can’t prove it is sound.

                      That’s what the unsafe blocks are for. If you didn’t intend a particular piece of code to be unsound, it shouldn’t be, and you should fix it.

                      1. 0

                        This is a bad argument taken to the logical extreme - as a compiler gets more and more intelligent it can reject more and more code, to the point is just rejects all code ever written due to all code we write having bugs.

                        My argument is simple, a rust edition should keep compiling things it accepted in the past. If they want to fix soundness problems, they should emit very loud and serious warnings and make a new edition. They shouldn’t retroactively stop compiling things they used to accept without a new rust edition.

                        1. 4

                          My argument is simple, a rust edition should keep compiling things it accepted in the past.

                          That argument is wrong. The soundness rules are not just defined by the (only) reference implementation. The soundness rules stipulated that multiple borrows (one of which is mutable) is not allowed to overlap, and it used scope as an approximation. If the compiler allow such an overlap, this is a bug and it should be fixed.

                          Likewise, if you unwittingly took advantage of this bug, and wrote unsound code outside of an unsafe block, you have a bug. Perhaps not a genuine bug, but you at least did not abide the soundness rules you should have. Thus, you should fix your code. I don’t care it’s something you consider “done” and no longer want to maintain. At the very least, you should accept a patch. If you don’t, well… there’s always the possibility of forking.

                          If they want to fix soundness problems, they should emit very loud and serious warnings and make a new edition.

                          Well, this is almost what they did: there’s a legacy compatibility mode, and warnings about that not compiling any more. That’s better than what C compilers do right now: when a new compiler spot an undefined behaviour it didn’t spot before, it could introduce a bug, without any warning, it didn’t used to introduce. (The magic of optimisations.)

                          But this is not about undefined behaviour. This is about soundness. Which unlike undefined behaviour in general is perfectly checkable statically. This won’t get worse and worse as compilers get better. It’s just a matter of fixing compiler bugs.

                          1. 0

                            I think it might annoy and cause reputational damage when all they need to do is make edition 2019-nll to avoid breaking the ecosystem.

                            Any crate with unsafe in it already has as much risk as something that has been there for years and nobody noticed any problem. They just need a warning and some bold flashy lights instead of permanently breaking our ability to compile a portion of crates.io. To maintain soundness they could make calling 2018 crates from 2019-nll an unsafe operation.

                            I think the right action depends how many crates they have obsoleted, which I don’t know. They should probably check and make it public, but I feel like they would rather not know.

                            1. 3

                              I think it might annoy and cause reputational damage when all they need to do is make edition 2019-nll to avoid breaking the ecosystem.

                              I dispute the assumption that they broke anything. Some code will stop compiling by default, but the old ways are just an option away. The warnings and the bold flashy lights is exactly what they have done. They have not broken your code, let alone permanently. Your code still compiles.

                              Sure, some users will get the warnings. Of course those users will file a bug. But your code still compiles, and will do for quite some time.

                              Any crate with unsafe in it already has as much risk as something that has been there for years and nobody noticed any problem

                              That’s just unsafe doing its job: a promise from the programmer that there isn’t any undefined behaviour, even though the borrow checker can’t verify it.

                              I think the right action depends how many crates they have obsoleted,

                              “Obsoleted” is such a strong word. I bet the changes required to fix the soundness errors in those crates will be minimal. I strongly suggest you take a look at your own crates, see where the new borrow checker got displeased, and do whatever is needed to please it again. This should be a quick fix.

                              1. 0

                                They have not broken your code, let alone permanently. Your code still compiles.

                                The warning specifically says they plan to stop it from compiling. Fine if it isn’t broken yet, but they told us the plan is to break things. They seem to have proposed adding these changes to edition 2015 to prevent even more code from compiling in the future.

                                I bet the changes required to fix the soundness errors in those crates will be minimal.

                                You also need to back port the changes to every major version release of your package to keep builds working for people depending on older API’s of your crates. Then you need to spend time testing each release, and publish them all. “Minimal” can quickly add up to an hour per crate., I don’t think all authors will do it, nobody is paying them to do it. Some probably don’t program rust anymore. It is just a loss for everyone.

                                “Obsoleted” is such a strong word. I bet the changes required to fix the soundness errors in those crates will be minimal.

                                Those crates simply won’t compile with newer versions of rustc 2018 edition without changes. The old versions just won’t work anymore without change, that sounds like obsoleting to me.

                                Anyway, obviously there are positives, like a lower maintenance burden so it isn’t totally bad.

                                1. 2

                                  I don’t think all authors will do it,

                                  I agree, they won’t. But I think it’s reasonable to assume that every single noteworthy crate will be addressed. The other can fall into oblivion like they would have anyway.

                                  Those crates simply won’t compile with newer versions of rustc 2018 edition without changes.

                                  The original post says: the 2018 edition of Rust […] has had NLL enabled ever since its official release.

                                  It’s a new edition. If you don’t want to update your code, well, just put a note that you’re only Rust 2015 compatible. C++ broke compatibility in similar ways, by the way: remember how it hijacked the auto keyword ? Nobody complained, because everyone understood it was a major release of the language (and the auto keyword could easily be removed from old code).

                                  And then there’s the migration mode, that allows you to take advantage of Rust 2018 even if you fall into one of those soundness bugs. Yes, they will turn it off eventually. But really, if you stopped maintaining your package, you are still using Rust 2015, and you will be fine. If you do maintain your package, well… what’s an hour per crate, really?


                                  It is not possible, nor (I think) even desirable to add special cases to the NLL borrow checker so it is bug compatible with the old AST borrow checker. It doesn’t work the same way at all, and even if you could reach for bug compatibility, you’d still have a soundness bugs, and with it the possibility of data races or pointer invalidation in supposedly “safe” code. Is bug compatibility worth sacrificing correctness? Not in my book.

                                  Then there are the benefits of the NLL borrow checker to begin with. Now pleasing the borrow checker will be much easier. Old complaints about it will likely fade away, and learning Rust will likely be less painful. Would you seriously sacrifice that just so you can be bug-compatible?

                                  Make no mistake, this is bug compatibility we’re talking about. Breaking code that relied on a buggy version of the Rust compiler, and as a result were not as safe as rustc claimed it were. They did not remove any feature, they fixed a bug (several bugs, actually). Sucks that the bug lead rustc to accept unsound code, but it’s really really not the same as, say, removing support for some syntax sugar or whatever.

                  2. 3

                    Rust has been stable for a long time now, so the moving target you mention has long settled. And as /u/pkolloch mentioned, the breaking changes are just bug fixes essentially, and they are downgrading the errors to warnings for an indefinite time. You should definitely check the language again if you feel like it.

                    1. 3

                      I thought the same about Rust, but then I realized that it was only going to break things if I upgraded my toolchain. This wasn’t something where a “yum/apt/brew install” or even a rustup would nuke my existing binaries. Instead, it only applies to upgrading your toolchain.

                      If the wrong version of python (2.6 instead of 2.7) is installed in a container or on a host I’m trying to use, I’ll see bugs or failures to run. God help me if I want to run Python 3 code. I hit that snag all the time.

                      With Rust, that problem is more or less nonexistent. I use Python A LOT. However, I still see the merit of Rust on this front. I’ve seen and used @yonkletron’s work written in Rust in an incredibly risk-averse environment and I was quite impressed at how little I had to think about it compared to my own Python work.

                      Rust may be less stable for me as a developer (and even that I’d question!) but it sure as hell seems pretty stable for me as an end user of apps written in it.

                      1. 2

                        It seems like it’s just a constantly moving target.

                        It’s definitely pretty active. But IMO this particular case strikes me as fixing a compiler defect (permitting unsound code to compile). It was a defect not to emit an error for this case, IMO. Fixing defects that result in new errors is not instability from the toolchain, it’s a bugfix that Rust customers should welcome (even though fixing their bug is a prerequisite to accepting the new toolchain release).

                        Maybe I didn’t quite read this article right but the fact that their 2015 example yields a warning that explicitly states that it was downgraded for compatibility makes it sound like the designation “will cause some existing code to break” is too pessimistic.

                        1. 0

                          They added warnings saying working code of mine in already released crates was going to stop compiling. I complained about it and nobody replied.

                          To put it another way, rust already broke code in one of my ALREADY RELEASED crates WITHOUT AN EDITION CHANGE. It pissed me off so much.

                          Not everyone has resources to go back and rewrite and retest code at their whims. In my opinion this change should be a strongly worded warning about potential unsafety + an edition change if they were serious about language stability. Don’t tell end users of my crate “hey, we might break your code maybe sometime, we don’t know when.”

                          If by future versions, they meant “future editions” I would be much more okay with it.

                          1. 2

                            If by “broke code”, you mean “emit warnings on newer versions of the compiler”, then maybe. What’s the crate? Where did you notify the developers (“complain”)? Are you sure the code is sound?

                            1. 0

                              Here is the issue, A report from a user of a released package:

                              https://github.com/andrewchambers/orderly/issues/20

                              Here is the complaint in the official thread for complaints of this issue:

                              https://github.com/rust-lang/rust/issues/59159

                              Are you sure the code is sound?

                              My reading of that issue is that it is sound, but inconvenient for them, but I don’t really care, they should do an emergency edition change for unsound code if the change isn’t backwards compatible. If code can’t last 6 months, what use is it? I want to write code that keeps working as when I wrote it for 20 years, 200 years if possible. They invented the edition system, why don’t they use it.

                              I don’t like how it is deemed ok to disregard small projects who put faith in rust.

                              1. 5

                                they should do an emergency edition change for unsound code if the change isn’t backwards compatible

                                No… Compiler bugs should be warned about once it becomes feasible to do so, with ample time before making them hard errors. Which is exactly what they’re doing… This looks like a great example of that system working. The warning drew your attention to your dependence on the compiler bug, and you fixed it the same day. All dependencies continued to compile, and assuming you released a patch version and yanked the previous version they always will.

                                If fixing any compiler bug required an edition bump, then the compiler would be riddled with bugs. Don’t pretend that you would prefer that.

                                1. 0

                                  Now 0.1, 0.2,0.3,0.4 and 0.5 versions of my project on crates.io are useless junk. What is the point of them being there?

                                  If my crate were a library this would also turn all crates depending on them into useless junk too. In my complaint I asked if they checked how many crates they are no only deprecating, but outright obsoleting.

                                  If you also note, the issue I linked doesn’t say it is a soundness issue, just a mistake they made accepting things that makes some future changes more annoying for them.

                                  1. 1

                                    Traceability. Studying the history of the code. Reproducing the results if they download the needed version of Rust. Not too many practical reasons, though.