1. 2

    No, SemVer does not solve every problem with dependencies, but at least it tells people to please stop intentionally breaking compatability in “minor” releases. If you’re going to break compatibility in every release, then just have a single version number. Is that really too much to ask?

    1. 1

      Funny enough, a change in the build system that doesn’t affect the public interface wouldn’t warrant a major bump in SemVer – particularly if it breaks platforms that were never supported by the authors – but let’s leave that aside.

      1. 1

        Here’s how this plays out over and over and over. First, people like me suggest that universal adoption of SemVer would significantly simplify and improve dependency management. Second, people like you point to a specific case where SemVer does not solve your problem, as if that somehow disproves that universal adoption of SemVer would significantly simplify and improve dependency management.

        I’ll repeat what I said above, yet again: No, SemVer does not solve every problem with dependencies, but at least it tells people to please stop intentionally breaking compatability in “minor” releases.

    1. 2

      This looks interesting, but it’s a bit beyond me. Any chance we could get a slightly dumbed-down explanation, with a bit less category theory jargon?

      1. 5

        I’ll give it a try. The main idea is to treat a programming language not based on its advertised type system, but on its actual computational rules. Because all programming languages must have such rules, the derived type theory must be a valid logic for reasoning about the behavior of programs, even if the language is traditionally considered untyped.

        For example, let’s take a snippet of Python 3.

        x = 1; x + 2
        

        This snippet is closed; it doesn’t refer to any outer names. By the rules of Python, we could remove the first statement and put x into an outer scope with the primitive value 1 already assigned. So, assuming that x = 1 already, we have:

        x + 2
        

        Now, again by the rules of Python, we can substitute 1 for x in our snippet, rewriting the syntax:

        1 + 2
        

        This might seem like ordinary big-step semantics. But we also inherit type information from the term constructors of the language. For Python 3, we know that 1 is not just a value, but has a type int of integers. This applies to operators, too; we know that if x and y are of type int, then so is x + y. Or, in a phrasing closer to type theory:

        x :int, y :int |- x + y :int
        

        The paper’s main contribution is a seamless synthesis of these two ways of approaching the behaviors of programming languages.

      1. 2

        This looks quite brilliant. I love that state updates are explicitly grouped into transactions via the next keyword. I love that promises are automatically awaited. I love that there’s strong static typing, algebraic data types, and pattern matching. I love that styling is first-class and scoped (as it is in svelte). I love that routing is built-in.

        My only concerns are over 1) cancelation, which seems like it could be handled by the compiler, and 2) interop with npm packages, which could be difficult to give up. I understand and commend the break from npm, but it introduces significant risk for real-world use. I will definitely be giving this a go for any future web front-end side projects, though.

        1. 3

          Are there other examples of SQLite being used as a website backend database in production? What kind of scale could you reach with this approach? And what would be the limiting resource?

          1. 10

            Expensify was based exclusively on sqlite for a long time, then they created a whole distributed database thing on top of it.

            1. 7

              Clojars used SQLite for a good 10 years or so, only recently moving away to Postgres for ease of redeployment and disaster recovery. The asset serving was just static assets, but the website and deployments ran against SQLite pretty well.

              1. 3

                If I remember correctly, the trouble that Clojars ran into had more to do with the quality of the JVM-based bindings to SQLite than they did with SQLite itself, at least during the portion of time that I was involved with the project.

                1. 2

                  Yeah, looking back at the issues, “pretty well” is maybe a little bit generous. There were definitely settings available later on which would have helped the issues we were faxing around locking.

              2. 4

                I can’t remember whom but at least one of the well funded dynamoDB style distributed database-y products from the mid 10s used it as the storage layer.

                So all the novel stuff that was being done with data was the communication and synchronisation over the network, and then for persistence on individual nodes they used sqlite instead of reinventing the wheel.

                1. 6

                  That was FoundationDB, purchased by Apple in 2013, then gutted, and then returned as open-source in 2018. I’m a bit annoyed, because it was headed to be CockroachDB half a decade earlier, and was taken off the market with very little warning.

                  1. 1

                    Thanks!

                2. 3

                  You probably will get really fast performance for read-only operations. The overhead of client/server and network stack could be more than10x times of function calls from same address space. The only real limitation might be single server, since you cannot really efficiently scale sqlite beyond single system. But when you reach that scale, you usually needs much more than sqlite.

                  1. 3

                    The sqlite website claims to run entirely on sqlite.

                    They also have this page, though most of those aren’t websites: https://sqlite.com/mostdeployed.html

                  1. 20

                    Python package maintainers rarely use semantic versioning and often break backwards compatibility in minor releases. One of several reasons that dependency management is a nightmare in Python world.

                    1. 18

                      I generally consider semantic versioning to be a well-intentioned falsehood. I don’t think that package vendors can have effective insight into which of their changes break compatibility when they can’t have a full bottom-up consumer graph for everyone who uses it.

                      I don’t think that Python gets this any worse than any other language.

                      1. 20

                        I’ve heard this opinion expressed before… I find it to be either dangerously naive or outright dishonest. There’s a world of difference between a) the rare bug fix release or nominally-orthogonal-feature-add release that unintentionally breaks downstream code and b) intentionally changing and deprecating API’s in “minor” releases.

                        In my view, adopting SemVer is a statement of values and intention. It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.

                        1. 18

                          In my view, adopting SemVer is a statement of values and intention. It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.

                          A “statement of values and intention” carries no binding commitment. And the fact that you have to hedge with “as much as is reasonably possible” and “only knowingly break” kind of gives away what the real problem is: every change potentially alters the observable behavior of the software in a way that will break someone’s reliance on the previous behavior, and therefore the only way to truly follow SemVer is to increment major on every commit. Which is the same as declaring the version number to be meaningless, since if every change is a compatibility break, there’s no useful information to be gleaned from seeing the version number increment.

                          And that’s without getting into some of my own direct experience. For example, I’ve been on the Django security team for many years, and from time to time someone has found a security issue in Django that cannot be fixed in a backwards-compatible way. Thankfully fewer of those in recent years since many of them related to weird old functionality dating to Django’s days as a newspaper CMS, but they do happen. Anyway, SemVer’s answer to this is “then either don’t fix it, or do but no matter how you fix it you’ve broken SemVer and people on the internet will scream at you and tell you that you ought to be following SemVer”. Not being a fan of no-win situations, I am content that Django has never and likely never will commit to following SemVer.

                          1. 31

                            A “statement of values and intention” carries no binding commitment.

                            A label on a jar carries no binding commitment to the contents of the jar. I still appreciate that my salt and sugar are labelled differently.

                            1. 2

                              Selling the jar with that label on it in many countries is a binding commitment and puts you under the coverage of food safety laws, though.

                            2. 6

                              Anyway, SemVer’s answer to this is “then either don’t fix it, or do but no matter how you fix it you’ve broken SemVer and people on the internet will scream at you and tell you that you ought to be following SemVer”.

                              What do you mean? SemVer’s answer to “this bug can’t be fixed in a backwards-compatible way” is to increment the major version to indicate a breaking change. You probably also want to get the message across to your users by pushing a new release of the old major version which prints some noisy “this version of blah is deprecated and has security issues” messages to the logs.

                              It’s not perfect, I’m not saying SemVer is a silver bullet. I’m especially worried about the effects of basing automated tooling on the assumption that no package would ever push a minor or patch release with a breaking change; it seems to cause ecosystems like the NPM to be highly fragile. But when taken as a statement of intent rather than a guarantee, I think SemVer has value, and I don’t understand why you think your security issue anecdote requires breaking SemVer.

                              1. 7

                                What do you mean? SemVer’s answer to “this bug can’t be fixed in a backwards-compatible way” is to increment the major version to indicate a breaking change.

                                So, let’s consider Django, because I know that well (as mentioned above). Typically Django does a feature release (minor version bump) every 8 months or so, and every third one bumps the major version and completes a deprecation cycle. So right now Django 3.1 is the latest release; next will be 3.2 (every X.2 is an LTS), then 4.0.

                                And the support matrix consists of the most recent feature release (full bugfix and security support), the one before that (security support only), and usually one LTS (but there’s a period at the end of each where two of them overlap). The policy is that if you run on a given LTS with no deprecation warnings issued from your code, you’re good to upgrade to the next (which will be a major version bump; for example, if you’re on 2.2 LTS right now, your next LTS will be 3.2).

                                But… what happens when a bug is found in an LTS that can’t be fixed in a backwards-compatible way? Especially a security issue? “Support for that LTS is cut off effective immediately, everybody upgrade across a major version right now” is a non-starter, but is what you propose as the correct answer. The only option is to break SemVer and do the backwards-incompatible change as a bugfix release of the LTS. Which then leads to “why don’t you follow SemVer” complaints. Well, because following SemVer would actually be worse for users than this option is.

                                1. 3

                                  But… what happens when a bug is found in an LTS that can’t be fixed in a backwards-compatible way?

                                  Why do people run an LTS version, if not for being able to avoid worrying about it as a dependency? If you’re making incompatible changes: forget about semver, you’re breaking the LTS contract, and you may as well tell drop the LTS tag and people to run the latest.

                                  1. 1

                                    you may as well tell drop the LTS tag and people to run the latest

                                    I can think of only a couple instances in the history of Django where it happened that a security issue couldn’t be fixed in a completely backwards-compatible way. Minimizing the breakage for people – by shipping the fix into supported releases – was the best available option. It’s also completely incompatible with SemVer, and is a great example of why SemVer is at best a “nice in theory, fails in practice” idea.

                                    1. 3

                                      Why not just tell them to upgrade? After all, your argument is essentially that stable APIs are impossible, so why bother with LTS? Every argument against semver also applies against LTS releases.

                                      1. 3

                                        After all, your argument is essentially that stable APIs are impossible

                                        My argument is that absolute perfect 100% binding commitment to never causing a change to observable behavior ever under any circumstance, unless also incrementing the major version at the same time and immediately dropping support for all users of previous versions, is not practicable in the real world, but is what SemVer requires. Not committing to SemVer gives flexibility to do things like long-term support releases, and generally people have been quite happy with them and also accepting of the single-digit number of times something had to change to fix a security issue.

                                  2. 2

                                    “Support for that LTS is cut off effective immediately, everybody upgrade across a major version right now” is a non-starter

                                    If it’s a non-starter then nobody should be getting the critical security patch. You’re upgrading from 2.2 to 3.0 and calling it 2.2.1 instead. That doesn’t change the fact that a breaking change happened and you didn’t bump the major version number.

                                    You can’t issue promises like “2.2.X will have long term support” because that’s akin to knowing the future. Use a codename or something.

                                    1. 7

                                      It’s pretty clear you’re committed to perfect technical adherence to a rule, without really giving consideration to why the rule exists. Especially if you’re at the point of “don’t commit to supporting things, because supporting things leads to breaking SemVer”.

                                      1. 4

                                        They should probably use something like SemVer but with four parts, e.g. Feature.Major.Minor.Patch

                                        • Feature version changes -> We’ve made significant changes / a new release (considered breaking)
                                        • Major version change -> We’ve made breaking changes
                                        • Minor version change -> Non breaking new features
                                        • Patch version change -> Other non-breaking changes

                                        That way 2.*.*.* could be an LTS release, which would only get bug fixes, but if there was an unavoidable breaking change to fix a bug, you’d signal this in the version by e.g. going from 2.0.5.12 to 2.1.0.0. Users will have to deal with the breaking changes required to fix the bug, but they don’t have to deal with all the other major changes which have gone into the next ‘Feature’ release, 3.*.*.*. The promise that 2.*.*.*, as an LTS, will get bug fixes is honored. The promise that the major version must change on a breaking change is also honored.

                                        SemVer doesn’t work if you try to imbue the numbers with additional meanings that can contradict the SemVer meanings.

                                        1. 3

                                          This scheme is very similar to Haskell’s Package Versioning Policy (PVP).

                                        2. 1

                                          I’m saying supporting things and adhering to SemVer should be orthogonal.

                                  3. 5

                                    every change potentially alters the observable behavior of the software

                                    This is trivially false. Adding a new helper function to a module, for example, will never break backwards compatibility.

                                    In contrast, changing a function’s input or output type is always a breaking change.

                                    By failing to even attempt to distinguish between non-breaking and breaking changes, you’re offloading work onto the package’s users.

                                    Optimize for what should be the common case: non-breaking changes.

                                    Edit: to expand on this, examples abound in the Python ecosystem of unnecessary and intentional breaking changes in “minor” releases. Take a look at the numpy release notes for plenty of examples.

                                    1. 7

                                      Python’s dynamic nature makes “adding a helper function” a potentially breaking change. What if someone was querying, say, all definitions of a module and relying on the length somehow? I know this is a bit of a stretch, but it is possible that such a change would break code. I still value semver though.

                                      1. 3

                                        The number of definitions in a module is not a public API. SemVer only applies to public APIs.

                                        1. 4

                                          If you can access it at run-time, then someone will depend on it, and it’s a bit late to call it “not public”. Blame Python for exposing stuff like the call stack to introspection.

                                          1. 2

                                            Eh no? SemVer is very clear about this. Public API is whatever software declares it to be. Undeclared things can’t be public API, by definition.

                                            1. 7

                                              Python has no concept of public vs private. It’s all there all the time. As they say in python land, “We’re all consenting adults here”.

                                              I’m sure, by the way, when Hettinger coined that phrase he didn’t purposely leave out those under the age of 18. Language is hard. :P

                                      2. 1

                                        Adding a new helper function to a module, for example, will never break backwards compatibility.

                                        Does this comic describe a violation of SemVer?

                                        You seriously never know what kinds of things people might be relying on, and a mere definition of compatibility in terms of input and output types is woefully insufficient to capture the things people will expect in terms of backwards compatibility.

                                        1. 6

                                          No, it does not descripbe a violation of SemVer, because spacebar heating is not a public API. SemVer is very clear about this. You are right people will still complain about backward compatibility even if you are keeping 100% correct SemVer.

                                    2. 6

                                      I would agree if violations were rare. Every time I’ve tried to solve dependency issues on Python, about 75% of the packages I look into have broken semver on some level. Granted, I probably have a biased sampling technique, but I find it extremely hard to believe that it’s a rare issue.

                                      Backwards compatibility is hard to reason about, and the skill is by no means pervasive. Even having a lot of experience looking for compatibility breaks, I still let things slip, because it can be hard to detect. One of my gripes with semver is that it doesn’t scale. It assumes that tens of thousands of open source devs with no common training program or management structure all understand what a backwards breaking change is, and how to fix it.

                                      Testing for compatibility breaks is rare. I can’t think of any Python frameworks that help here. Nor can I think of any other languages that address this (Erlang might, but I haven’t worked with it first-hand). The most likely projects to test for compatibility between releases are those that manage data on disk or network packets. Even among those, many rely on code & design review to spot issues.

                                      It communicates that you value backwards compatibility and intend to maintain it as much as is reasonably possible, and that you will only knowingly break backwards compatibility on major release increments.

                                      It’s more likely that current package managers force you into semver regardless if you understand how it’s supposed to be used. The “statement of values” angle is appealing, but without much evidence. Semver is merely popular.

                                      1. 7

                                        I guess this depends on a specific ecosystem? Rust projects use a lot of dependencies, all those deps use semver, and, in practice, issues rarely arise. This I think is a combination of:

                                        • the fact that semver is the only option in Rust
                                        • the combination of guideline to not commit Cargo.lock for libraries + cargo picking maximal versions by default. This way, accidental incompatibilities are quickly discovered & packages are yanked.
                                        • the guideline to commit Cargo.lock for binaries and otherwise final artifacts: that way folks who use Rust and who have the most of deps are shielded from incompatible updates.
                                        • the fact that “library” is a first-class language construct (crate) and not merely a package manager convention + associated visibility rules makes it easier to distinguish between public & private API.
                                        • Built-in support for writing test from the outside, as-if you are consumer of the library, which also catches semver-incompatible changes.

                                        This is not to say that semver issues do not happen, just that they are rare enough. I’ve worked with Rust projects with 200-500 different deps, and didn’t pensive semver breakage being a problem.

                                        1. 5

                                          I would add that the Rust type system is expressive enough that many backwards incompatible changes require type signature changes which are much more obvious than violations of some implicit contract.

                                      2. 6

                                        I don’t think I have a naïve view of versioning; putting on my professional hat here, I have a decade of experience dealing with a dependency modeling system that handles the versions of hundreds of thousands of interrelated software artifacts that are versioned more or less independently of each other, across dozens of programming languages and runtimes. So… some experience here.

                                        In all of this time, I’ve seen every single kind of breaking change I could imagine beforehand, and many I could not. They occurred independent of how the vendor of the code thought of it; a vendor of a versioned library might think that their change is minor, or even just a non-impacting patch, but outside of pure README changes, it turns out that they can definitely be wrong. They certainly had good intentions to communicate the nature of the change, but that intention can run hard into reality. In the end, the only way to be sure is to pin your dependencies, all the way down, and to test assiduously. And then upgrade them frequently, intentionally, and on a cadence that you can manage.

                                        1. 1

                                          I don’t think I have a naïve view of versioning; putting on my professional hat here, I have a decade of experience dealing with …

                                          Here here. My experience isn’t exactly like @offby1’s but I can vouch for the rest.

                                        2. 4

                                          to be either dangerously naive or outright dishonest

                                          This phrase gets bandied around the internet so much I’m surprised its not a meme.

                                          SemVer is … okay, but you make it sound like lives depend on it. There’s a lot of software running mission critical systems without using SemVer and people aren’t dying everyday because of it. I think we can calm down.

                                      3. 3

                                        Thats the problem of the package management being so old. Back then semantic versioning wasnt that common and it never really caught on. In my opinion the PyPA should make a push to make more packages use semantic versioning. I‘m seeing this trend already, but its too slow…

                                      1. 5

                                        I don’t get this.

                                        It’s tagged [rust], but the project seems to be in Python?

                                        Someone (the maintainers?) added a Rust dependency, and now the software won’t build on some architectures?

                                        I’ll unflag this as off-topic if I can a good explanation to why this on-topic, and not some misguided attempt to get more upvotes on a random GH issue.

                                        1. 12

                                          As from the sheer amount of comments and reaction on the issue, this is a large deal. I’ve never seen a Github issue this active. It is a Python project, but the main problem is that LLVM/Rust is not available on some platforms. So I think both tags are justified.

                                          1. 12

                                            It’s a real-life case of a practical problem at an intersection of quite a few technical and social topics important for programmers (package management, backwards compatibility, security, cross-platform-ness, FFI), with no clear “perfect” solution, which makes various compromises in how to solve it worth pondering and discussing as to possible consequences/pros and cons.

                                            A high “value density” case-study which has potential to bear learnings that can later be used to educate smaller day-to-day cases, by giving insights as to potential consequences of specific decisions.

                                            1. 3

                                              Yes, the discussion here has been enlightening. The discussion on GH was way too in media res for someone who wasn’t already caught up on the issue to get much out of it.

                                              For example, I’m still unclear about exactly what “pyca” is. It’s not a standard part of Python? If it isn’t, why is this project specifically so interesting that people who build docker images use it?

                                              A thoughtful blog post (like the top comment by @ubernostrum) would have been a much better submission imho.

                                              But I’ve removed my flag now, anyway.

                                              1. 4

                                                pyca stands for Python Cryptographic Authority. It’s an organization that regroups python libraries and tools related to cryptography. They are regrouped to allow better interoperability between them. The term “Authority” is, in my opinion, a little bit excessive as nobody requires you to use them; but as they are used by building block tools, you still use them transitively, hence the shitload of comments in the issue.

                                                This is not the only “Authority” in the python world, there is also the Python Packaging Authority (pypa), the Python Code Quality Authority (PyCQA), and maybe other ones that I don’t know about. As they often involve python core developers, their “Authority” status looks genuine.

                                            2. 8

                                              I found it to be relevant and informative, but that’s probably just because my day job includes writing and maintaining Python and because I use Rust in several of my side projects.

                                            1. 3

                                              I’m not entirely convinced a new model is needed. We already have memory mapped files in all the major operating systems. And file pages can already be as small as 4KiB, which is tiny compared to common file sizes, these days. Perhaps it would make sense to have even smaller pages for something like Opteron, but do we really need to rethink everything? What would we gain?

                                              1. 4

                                                What we’d gain is eliminating 50+ years of technical debt.

                                                I recommend the Twizzler presentation mentioned a few comments down. It explains some of the concepts much better than I can. These people have really dug into the technical implications far deeper than me.

                                                The thing is this: persistent memory blows apart the computing model that has prevailed for some 60+ years now. This is not the Von Neumann model or anything like that; it’s much simpler.

                                                There are, in all computers for since about the late 1950s, a minimum of 2 types of storage:

                                                • primary storage, which the processor can access directly – it’s on the CPUs’ memory bus. Small, fast, volatile.
                                                • secondary storage, which is big, slow, and persistent. It is not on the memory bus and not in the memory map. It is held in blocks, and the processor must send a message to the disk controller, ask for a particular block, wait for it to be loaded from 2y store and place into 1y store.

                                                The processor can only work on data in 1y store, but everything must be fetched from it, worked on, and put back.

                                                This is profoundly limiting. It’s slow. It doesn’t matter how fast the storage is, it’s slow.

                                                PMEM changes that. You have RAM only RAM, but some of your RAM keeps its contents when the power is off.

                                                Files are legacy baggage. When all your data is in RAM all the time, you don’t need files. Files are what filesystems hold; filesystems are an abstraction method for indexing blocks of secondary storage. With no secondary storage, you don’t need filesystems any more.

                                                1. 7

                                                  I feel like there are a bunch of things conflated here:

                                                  Filesystems and file abstractions provide a global per-device namespace. That is not a great abstraction today, where you often want a truly global namespace (i.e. one shared between all of your devices) or something a lot more restrictive. I’d love to see more of the historical capability systems research resurrected here: for typical mobile-device UI abstractions, you really want a capability-based filesystem. Persistent memory doesn’t solve any of the problems of naming and access. It makes some of them more complicated: If you have a file on a server somewhere, it’s quite easy to expose remote read and write operations, it’s very hard to expose a remote mmap - trying to run a cache coherency protocol over the Internet does not lead to good programming models.

                                                  Persistence is an attribute of files but in a very complicated way. On *NIX, the canonical way of doing an atomic operation on a file is to copy the file, make your changes, and then move the old file over the top. This isn’t great and it would be really nice if you could have transactional updates over ranges of files (annoyingly, ZFS actually implements all of the machinery for this, it just doesn’t expose it at the ZPL). With persistent memory, atomicity is hard. On current implementations, atomic operations with respect to CPU cache coherency and atomic operations with respect to committing data to persistent storage are completely different things. Getting any kind of decent performance out of something that directly uses persistent memory and is resilient in the presence of failure is an open research problem.

                                                  Really using persistent memory in this way also requires memory safety. As one of the The Machine developers told me when we were discussing CHERI: with persistent memory, your memory-safety bugs last forever. You’ve now turned your filesystem abstractions into a concurrent GC problem.

                                                  1. 1

                                                    Excellent points; thank you.

                                                    May I ask, are you the same David Chisnall of “C is not a low-level language” paper? That is probably my single most-commonly cited paper. My compliments on it.

                                                    Your points are entirely valid, and that is why I have been emphasizing the “just for fun” angle of it. I do not have answers to some of these hard questions, but I think that at first, what is needed is some kind of proof of concept. Something that demonstrates the core point: that we can have a complex, rich, capable environment that is able to do real, interesting work, which in some ways exceeds the traditional *nix model for a programmer, which runs entirely in a hybrid DRAM/PMEM system, on existing hardware that can be built today.

                                                    Once this point has been made by demonstration, then perhaps it will be possible to tackle much more sophisticated systems, which provide reliability, redundancy, resiliency, and all that nice stuff that enterprises will pay lots of money for.

                                                    There is a common accusation, not entirely unjust, that the FOSS community is very good at imitating and incrementally improving existing implementations, but not so good at creating wholly new things. I am not here to fight that battle. What I was trying to come up with was a proposal to use some existing open technology – things that are already FOSS, already out there, and not new and untested and immature, but solid, time-proven tools that have survived despite decades in obscurity – and assemble them into something that can be used to explore new and largely uncharted territory.

                                                    ISTM, based on really very little evidence at all, that HPE got carried away with the potential of someting that came out of their labs. It takes decades to go from a new type of component to large-scale highly-integrated mass production. Techies know that; marketing people do not. We may not have competitive memristor storage until the 2030s at the earliest, and HPE wanted to start building enterprise solutions out of it. Too much, too young.

                                                    Linux didn’t spring fully-formed from Torvalds’ brow ready to defeat AIX, HP-UX and Solaris in battle. It needed decades to grow up.

                                                    The Machine didn’t get decades.

                                                    Smalltalk has already had decades.

                                                  2. 4

                                                    I think files are more than just an abstraction over block storage, they’re an abstraction over any storage. They’re crucial part of the UX as well. Consider directories… Directories are not necessary for file systems to operate (it could just all be flat files) but they exist, purely for usability and organisation. I think even in the era of PMEM users will demand some way to organise information and it’ll probably end up looking like files and directories.

                                                    1. 2

                                                      Most mobile operating systems don’t expose files and directories and they are extremely popular.

                                                      1. 3

                                                        True, but those operating systems still expose filesystems to developers. Users don’t necessarily need to be end users. iOS and Android also do expose files and directories to end users now, although I know iOS didn’t for a long time.

                                                        1. 3

                                                          iOS also provides Core Data, which would be a better interface in the PMEM world anyway.

                                                          1. 1

                                                            True, but those operating systems still expose filesystems to developers.

                                                            Not all of them don’t, no.

                                                            NewtonOS didn’t. PalmOS didn’t. The reason being that they didn’t have filesystems.

                                                            iOS is just UNIX. iOS and Android devices are tiny Unix machines in your pocket. They have all the complexity of a desktop workstation – millions of lines of code in a dozen languages, multiuser support, all that – it’s just hidden.

                                                            I’m proposing not just hiding it. I am proposing throwing the whole lot away and putting something genuinely simple in its place. Not hidden complexity: eliminating the complexity.

                                                          2. 2

                                                            They tried. Really hard. But in the end, even Apple had to give up and provide the Files app.

                                                            Files are an extremely useful abstraction, which is why they were invented in the first place. And why they get reinvented every time someone tries to get rid of them.

                                                            1. 4

                                                              Files (as a UX and data interchange abstraction) are not the same thing as a filesystem. You don’t need a filesystem to provide a document abstraction. Smalltalk-80 had none. (It didn’t have documents itself, but I was on a team that added documents and other applications to it.) And filesystems tend to lack stuff you want for documents, like metadata and smart links and robust support for updating them safely.

                                                              1. 1

                                                                I’m pretty sure the vast majority of iOS users don’t know Files exist.

                                                                I do, but I almost never use it.

                                                              2. 1

                                                                And extremely limiting.

                                                        1. 5

                                                          Defenders of dynamically typed languages sometimes counter that these pitfalls do not matter when runtime failures are mostly harmless. If you want to find errors in your program, just run the program!

                                                          … We do?

                                                          1. 2

                                                            Gabriel qualified the statement (“sometimes”) and gives an example (Nix). Is it not a reasonable accusation? What else can one do with a dynamically typed program, except running it?

                                                            1. 2

                                                              An non exhaustive list:

                                                              • unit tests against your code
                                                              • property tests against your code
                                                              • runtime type pattern matching
                                                              • runtime type conditionals
                                                              • contracts
                                                              • other compile-time checks besides a type system
                                                              • defensive coding (for instance, calling int(x) BEFORE opening a file handle, not after)

                                                              I love type systems, and even more so I love compile time guarantees, but any static typing advocate should be able to speak about the value of typing as a trade-off against other techniques and paradigms. On the other hand, no advocate of dynamic typing would (or should, I suppose) claim that, in the absence of the tangible and often productivity-enhancing benefits of strong static typing, “just run the program” is an acceptable alternative.

                                                              1. 2

                                                                An non exhaustive list:

                                                                unit tests against your code property tests against your code runtime type pattern matching runtime type conditionals contracts other compile-time checks besides a type system defensive coding (for instance, calling int(x) BEFORE opening a file handle, not after)

                                                                All but one of these is a variant on running the code.

                                                                ** EDIT ** Well, the formatting didn’t stay and I’m on mobile so I’m just going to leave this as is for now…

                                                                1. 4

                                                                  All but one of these is a variant on running the code.

                                                                  In that sense, one can say that even compilation is just a variant on running the code.

                                                                  1. 2

                                                                    I disagree. The compiler, a linter, or another static analyzer is another program operating on your program.

                                                                    Though, I suppose you could argue the same about unit tests and property tests. I think the difference is that they exercise your code whereas your code may not execute with a compiler. Things get a bit weird with compile time execution/macros/whatever.

                                                                    I don’t feel completely satisfied with my response.

                                                                    1. 6

                                                                      A typed program is two programs at once; the type system’s annotations themselves form a second program which is intertwined with the original program’s shape. This second program is the one which is run by the type-checker, and this is what gives us Rice’s Theorem for type systems, where e.g. GHC has some extensions that allow for type-checking to be undecideable.

                                                                      This also justifies why static type systems cannot possibly replace runtime testing of some sort; in order to complete in a reasonable time, the typical type system must be very simple, because its expressions are evaluated by the type-checker.

                                                                      I waffled over whether to post this at the top or the bottom of the thread, since I feel that this fact is at odds with how the discussion progressed. After all, what else can one do with a statically-typed program (that is, a pair of interleaved programs where one is not Turing-complete and is either sound or conservative over the second program). except running it or running just the type-checker portion? Similarly, it is not just obvious that compilation is a variant on running the code, in that is is an evaluation of one entire layer of the program’s code which produces a residual single-layer program, but also that compilation must run the code.

                                                                      1. 2

                                                                        A typed program is two programs at once; the type system’s annotations themselves form a second program which is intertwined with the original program’s shape. This second program is the one which is run by the type-checker…

                                                                        Okay, I am reasonably convinced by this. My major hang ups might have been implementation details: the distinction between your program and a program running your program can be blurry in some environments (e.g., Dr. Racket).

                                                                        Given that there are two interleaved programs that may be run, the typed program’s advantage is that it is side effect free by it’s nature. Unit tests, property tests, fuzzers, etc. are running arbitrary code and could trigger arbitrary effects. I think that is ultimately the distinction that matters: how protected are you from mistakes in your program?

                                                                        (Side note: running property tests on a function that sends email could be hilariously bad.)

                                                                        I waffled over whether to post this at the top or the bottom of the thread, since I feel that this fact is at odds with how the discussion progressed.

                                                                        Well, I appreciate being called out on my position.

                                                                        1. 2

                                                                          On this view, what makes the type annotations the second and final program? Could it be three+ programs, since the linter/documentation generator also operate on a language that is intertwined with your program? This is a serious question, btw. I feel like I can think of some answers, but I don’t know that I can reason through them precisely.

                                                                          1. 2

                                                                            You’re quite right. There’s the view that syntax and semantics are the only two components to consider, and it’s a compelling view, rooted in the typical framing of Turing’s and Rice’s Theorems. There’s also the view that dependently-typed theories are at the ceiling of expressive power, and that our separation of types and values is just a happenstance due to choosing weak type systems, which also is tantalizing because of the universal nature of cubical and opetopic type theories. And to answer your point directly, there’s nothing stopping one syntax from having multiple different kinds of annotations which are projected in different ways.

                                                                            I suppose I’m obligated to go with a category-oriented answer. Following Lawvere, syntax and semantics are adjoint and indeed form a Galois connection when viewed in a certain way. So, when we use Rice’s framing to split the properties of a program along such a connection, what we get is a pair of adjoint functors:

                                                                            • On the left, semantics is a functor from programs to families of computer states
                                                                            • On the right, syntax is a functor from families of computer states to programs
                                                                            • The left side of the adjunction: the semantics of a program can include some particular behaviors, if and only if…
                                                                            • The right side of the adjunction: those behaviors are encoded within the syntax of that program

                                                                            And this is a family of adjunctions, with many possibilies for both functors. Rice’s Theorem says that we can’t decide whether most of the left-hand functors are computable, but the right-hand functor is not so encumbered; we can compute some properties of syntax, like well-typedness.

                                                                            We can use the free-forgetful paradigm for analysis too: Semantics freely takes any of the possible execution paths, and syntax forgets which path was taken.

                                                                            There is another functor which is left adjoint to semantics, the structure functor (discussed further here).

                                                                            • On the left, structure is a functor from families of computer states to programs
                                                                            • On the right, the same semantics as before
                                                                            • The left side of the adjunction: the structure of some computations can all be found in a particular program, if and only if…
                                                                            • The right side of the adjunction: the semantics of that program can reach all of those computations

                                                                            In the free-forgetful paradigm, structure freely generates the program which it embodies, and semantics forgets structure.

                                                                            Rice didn’t forbid us from analyzing this structure functor either! As a result, we have two different ways to examine a program without running it and invoking the dreaded semantics functor:

                                                                            • We can study syntax backwards: Which machine states are possibly implied by our program? We can logically simplify the program text if it would help.
                                                                            • We can study structure backwards: Which machine states do our program require or assume? We can simplify the states if it would help.

                                                                            This means that analyzing structure is abstract interpretation! Or I’ve horribly misunderstood something.

                                                                            Now I’m quite sorry to have not posted this at the top of the thread! Oh well.

                                                                          2. 1

                                                                            How does this interpretation handle fully inferred types? Is that not truly an analysis pass on the program, I.e. running a program on the source code, not running the source code itself?

                                                                            I’m not sure if there’s any value in making the distinction… But it feels like there must be, because static types are qualitatively different from dynamic types.

                                                                            1. 1

                                                                              The inference is usually syntactic and doesn’t require examining the semantics of the program. This is the sense in which the second program is intertwined with the first program.

                                                                              For example, in many languages, 42 has a type like Int. This fact could be known by parsers. Similarly, (\x -> x + 1) has a type like Int -> Int. Parsers could know this too, since function types are built out of other types. In general, syntactic analysis can discover both the concrete parse tree and also the inferred type.

                                                                        2. 2

                                                                          Types categorise values, they don’t run code.

                                                                1. 0

                                                                  Apologies for the brief comment, but I find it fascinating that I knew almost immediately we were talking about category theory and the answer is “no”

                                                                  I do not believe it is possible to explain why, though! As one cmomenter began to realize, this is a meta/sytem-talking-about itself issue. Perhaps you could define your way into proving it, but you’d just be left with loose threads at a higher level of abstraction. A system cannot contain a symbol that completely describes the system itself. This is semiotics, not CS.

                                                                  Very cool. Thanks for the link.

                                                                  1. 3

                                                                    A system cannot contain a symbol that completely describes the system itself.

                                                                    Not my area of expertise, but isn’t Godel’s incompleteness theorem significantly weaker than this statement? If this were true, how could a compiler be self hosting?

                                                                    1. 1

                                                                      The question is “Is there a category of actors?”

                                                                      That’s a representational question. that’s why I said this was semiotics, not CS. Your reply was about calculations: can a thing calculate itself? Sure it can.

                                                                      Godel is something else entirely, a statement about the nature of categories and transforms in general, ie a computational statement.

                                                                      It category-ese, the question was about categories of stuff. Your response was about transforms. In most cases these two are mirrors: a category is indicated by transforms, and transforms indicate the presence of a category. But not always. (And admittedly I’m doing a lot of hand-waving with my terminology here for sake of brevity)

                                                                      Godel is something else entirely, a statement about the nature of both categories and transforms.

                                                                      1. 1

                                                                        Right, Gödel’s statement is entirely about objects on the inside of a category/universe, but the interpretation which creates the paradox is always sitting outside. Indeed, the generic statement from Lawvere is mundane and talks of surjective arrows and fixed points in any category. It is only when we interpret these statements as being self-referential that we obtain the various paradoxes.

                                                                    1. 1

                                                                      Good read. A few of take-aways, for me:

                                                                      1. If you’re debugging a reproducibility problem, sprinkle in debug tracing (at full precision) to a file, and diff the traces.
                                                                      2. Focus on isolating the problem to the smallest reproduction you can, as quickly as you can.
                                                                      3. If you’ve eliminated everything but the compiler, use Godbolt to compare different compiler outputs on your minimal reproduction.
                                                                      1. 2

                                                                        It wasn’t the compiler.

                                                                      1. 6

                                                                        This question was prompted by a discussion I had about how I felt like all the momentum in programming languages is pointing toward Rust these days, and I felt like there’s no point in keeping my Go current (it’s been languishing for a couple of years now anyway).

                                                                        So, I asked this question to see (among other things) if I’m right or wrong.

                                                                        1. 12

                                                                          What is driving that feeling? Genuinely curious, because I feel the opposite. I am seeing more and more enterprises adopt Go. In conversations I have with other engineers, Rust still feels a little underground.

                                                                          I also think that Go and Rust have slightly different use cases & target audiences.

                                                                          1. 9

                                                                            Well, lobste.rs, for one. I feel like everywhere I look on here people talk about how they’d re-do everything in Rust if they could. The number of Rust advocates I see here seems to dwarf the number of Go advocates. Maybe that’s perception because Rust is “newer” and its advocates louder, but who knows.

                                                                            The things that really stuck with me, though, were Linus indicating that he’d be open to allowing Rust in the kernel, Microsoft starting to switch to Rust for infrastructure, and Dropbox migrating their core technologies to Rust.

                                                                            I just don’t see stories like that for Go. I don’t know if I’m not looking in the right place, or what.

                                                                            1. 22

                                                                              Go and Rust more or less solve the same problem (although the overlap isn’t 100%), just in different ways, not too dissimilar to how Perl and Python more or less solve the same problems in very different ways.

                                                                              I have the impression that, on average, Go tends to attract people who are a little bit jaded by the Latest Hot New Thing™ churn for 20 years and just want to write their ifs and fors and not bother too much with everything else. This is probably one reason why the Go community has the (IMHO reasonably deserved) reputation for being a bunch of curmudgeonly malcontents. These are not the sort of people who go out and enthusiastically advocate for Go, or rewrite existing tools in Go for the sake of it: they’re happy with existing tools as long as they work.

                                                                              Another reason I don’t really like to get involved in Go discussions is because some people have a massive hate-on for it and don’t shy away from telling everyone that Go is stupid and so is anyone using it every chance they get. It gets very boring very fast and I have better things to do than to engage with that kind of stuff, so I don’t. There’s some people like that on Lobsters as well, although it’s less than on HN or Reddit. It’s a major reason why I just stopped checking /r/programming altogether, because if the top comment of damn near every post is “lol no generics” followed by people ranting about “Retards in Go team don’t think generics are useful” (which simply isn’t true) then … yeah… Let’s not.

                                                                              1. 14

                                                                                Go and Rust more or less solve the same problem

                                                                                Hard disagree, rust is way more useful for actual problems like c/c++ are. I can write kernel modules in it, there is almost no way i’d want to do that with go. Having a garbage collector, or even really a runtime in go means a different use case entirely to being able to run without an os and target minis. Yes I know go can be used for that too but just due to having a GC you’re limited on how far down the horsepower wagon you can go.

                                                                                I hold no views outside of that rust actually has solutions and community drive (aka people using it for that upstreaming things) for programming things like avr processors etc… I don’t hate go it just strikes me as redundant and not useful for my use cases. Kinda like if you learn python not much use for learning ruby too kind of a deal. And if I’m already using rust for low level stuff, why not high level too?

                                                                                I don’t however miss debugging goroutine and channel bugs though, go is way easier to shoot yourself in a concurrent foot without realizing it. It might be ‘simple’ but that doesn’t mean its without its own tradeoffs. I can read and write in it but prefer the rust compiler telling me i’m an idiot for trying to share data across threads to goroutine debugging where two goroutines read off one channel and one of ems never gonna complete cause the other already got it. I’m sure “I’m holding it wrong” but as I get older these strict and more formal/functional languages like rust/haskell/idris/blah strike my fancy more. I”m not talking about generics really but stuff like Monads (Option/Result essentially) read to me way better than the incessant if the thing i did isn’t nil constantly. Its closer to what I’ve done in the past in C with macros etc…

                                                                                Its not that I hate it though, just that the language seems a step back in helping me do things. Idris as an example though is the coolest thing I’ve used in years in that using it was like having a conversation with the compiler and relearning how to move my computation to the type system. It was impressive how concise you can make things in it.

                                                                                As a recovering kernel/c hacker, you’d think go would appeal but to be honest it just seems more of the same as c with less ways of stopping me from shooting myself in the foot needlessly.

                                                                                But to each their own, functional languages with types just strike me as actually doing a lot of things that OO languages from the mid 90’s always said could be done with open form polymorphism but never seemed to happen.

                                                                                In 10 years we’ll see where the ball landed in the outfield so whatever.

                                                                                1. 11

                                                                                  Yes, hence the “more or less”. Most people aren’t writing kernel modules; they’re writing some CLI app, network service, database app, and so forth. You can do that with both languages. TinyGo can be used for microcontrollers, although I don’t know how well it works in practice – it does still have a GC and a (small) runtime (but so has e.g. C).

                                                                                  I don’t however miss debugging goroutine and channel bugs though, go is way easier to shoot yourself in a concurrent foot without realizing it.

                                                                                  Yeah, a lot of decisions are trade-offs. I’ve been intending to write a “why Go is not simple”-post or some such, which argues that while the syntax is very simple, using those simple constructs to build useful programs is a lot less simple. In another thread yesterday people were saying ‘you can learn Go in two days”, but I don’t think that’s really the case (you can only learn the syntax). On the other hand, I’ve tried to debug Rust programs and pretty much failed as I couldn’t make sense of the syntax. I never programmed much in Rust so the failure is entirely my own, but it’s a different set of trade-offs.

                                                                                  In the end, I think a lot just comes down to style (not everything, obviously, like your kernel modules). I used to program Ruby in a previous life, which is fairly close to Rust in design philosophy, and I like Ruby, but it’s approach is not without its problems either. I wrote something about that on HN a few weeks ago (the context being “why isn’t Ruby used more for scripting?”)

                                                                                  1. 4

                                                                                    Most people aren’t writing kernel modules; they’re writing some CLI app, network service, database app, and so forth. You can do that with both languages.

                                                                                    CLI yes, database probably, but I don’t think Rust’s async or concurrency or whatever story is mature enough to say it’s comparable with Go for network services.

                                                                                    1. 1

                                                                                      Cooperative concurrency is just more complicated (as a developer, not as a language designer) than preemptive concurrency. The trade-off is that it’s more performant. Someone could build a Rust-like language, i.e. compiler-enforced data race freedom, with green threads and relocatable stacks. And someday someone might. For now, the choice is between performance and compiler-enforced correctness on the Rust side, or “simpler” concurrency on the Go side.

                                                                                2. 3

                                                                                  There’s just a lot of toxicity about programming languages out there, and subjectively it feels particularly bad here. Rust has a lot to like and enough to dislike (abandoned libraries, inconsistent async story, library soup, many ways to do the same thing), but something about its culture just brings out the hawkers. I still heartily recommend giving Rust a try, though you won’t be super impressed if you’ve used Haskell or Ocaml in the past.

                                                                                  1. 6

                                                                                    I came to Rust after having used Haskell and the main thing about it that impressed me was precisely that it brought ML-style types to language with no GC that you could write an OS in.

                                                                                    1. 5

                                                                                      with no GC

                                                                                      I guess I find this often to be a solution with very few problems to solve. It’s understandable if you’re writing an OS or if you’re working on something real-time sensitive, but as long as the system you’re making can tolerate > 1ms p99 response times, and doesn’t require real-time behavior, Go, JVM languages, and .NET languages should be good enough. One could argue that there exists systems in the 1-10ms range where it’s easier to design in non-GC languages rather than fight the GC, and I can really see Rust succeeding in these areas, but this remains a narrow area of work. For most systems, I think working with a GC keeps logic light and easily understandable. When it comes to expressive power and compiler-driven development, I think both Haskell and Ocaml have better development stories.

                                                                                      1. 1

                                                                                        Rust also has a much cleaner package management story and (ironically) faster compile times than Haskell. And first-class support for mutability. And you don’t have to deal with monad transformers.

                                                                                        Haskell is still a much higher level language, though. I experimented last night with translating a programming language core from Haskell to Rust, and I quickly got lost in the weeds of Iterator vs Visitor pattern vs Vec. Haskell is pretty incredible in it’s ability to abstract out from those details.

                                                                                    2. 0

                                                                                      Your descriptions of both advocates and Go haters match my experience exactly.

                                                                                    3. 14

                                                                                      I’ve been using Go since around the 1.0 release and Rust for the last year or so. I don’t think either of them is going away any time soon. Go has less visible advocates, but it’s definitely still used all over the place. Datadog has a huge Go repo, GitHub has been looking for Go engineers, etc.

                                                                                      Rust is more visible because it’s newer and fancier, but it still loses for me in multiple aspects:

                                                                                      • Development speed - It’s a much pickier language and is much slower to develop in, though it forces you to get things right (or at least handle every case). Rust-Analyzer is great, but still fairly slow when compared to a simpler language.
                                                                                      • Compilation speed - Go compiles way faster because it’s a much simpler language.
                                                                                      • Library support/ecosystem - Because Go has been around for quite a while, there are a wealth of libraries to use. Because Rust hasn’t been around as long, many of the libraries are not as mature and sometimes not as well maintained.

                                                                                      However, Rust makes a number of improvements on Go.

                                                                                      • Error handling - Rust’s error handling is miles above Go. if err != nil will haunt me to the end of my days.
                                                                                      • Pattern matching - extremely powerful. This is an advantage Rust has, but I’m not sure how/if it would fit in to Go.
                                                                                      • Generics - In theory coming to Go soon… though they will be very different feature-set wise

                                                                                      They’re both great languages, and they have different strengths. For rock-solid systems software, I’d probably look at Rust. For web-apps and services, I’d probably look to Go first.

                                                                                      1. 4

                                                                                        Rust also loses massively in concurrency model. Tokio streams is so subpar to channels.

                                                                                        1. 2

                                                                                          Tokio also has channels - MPSC is the most common variant I’ve seen.

                                                                                          When Stream is added back to tokio, these will also impl Stream

                                                                                          I do agree that having goroutines as a part of the language and special syntax for channels makes it much easier to get into though.

                                                                                          1. 1

                                                                                            Rust’s async/await is definitely more complicated than Go’s green threads, and is almost certainly not worth it if Go has everything you need for your project. However, Rust’s standard threads are extremely powerful and quite safe, thanks to the Sync and Send marker traits and associated mechanics, and libraries like rayon make trivial parallelism trivial. Just yesterday I had a trivially parallel task that was going to take 3 days to run. I refreshed myself on how to use the latest rayon, and within about 10 minutes had it running on all 8 hyperthreads.

                                                                                          2. 2

                                                                                            Spot on. They both have their annoyances and use cases where they shine.

                                                                                          3. 6

                                                                                            A further difference is that Go code is hard to link with other languages due to its idiosyncratic ABI, threading and heaps. Rust fits in better. Not just in OS kernels but in mobile apps and libraries. (I’m still not a fan of Rust, though; Nim fits in well too and just feels a lot easier to use.)

                                                                                            1. 1

                                                                                              I would say that is entirely compiler dependant, and not a property of the language Go. https://github.com/tinygo-org/tinygo

                                                                                              1. 2

                                                                                                As long as the language has highly scalable goroutines, it won’t be using native stacks. As long as the language has [non ref-counted] garbage-collection, it won’t be using native heaps.

                                                                                                1. 1

                                                                                                  Well, tinygo != go, and e.g. gccgo still reuses the official standard library from the main implementation (which is where lots of the frustrating stuff is located, e.g. the usage of raw syscalls even on platforms like FreeBSD where it’s “technically possible but very explicitly not defined as public API”).

                                                                                              2. 5
                                                                                            2. 9

                                                                                              Golang is awesome. It works without fanfare.

                                                                                              1. 8

                                                                                                As someone who dislikes Go quite a lot, and is really enjoying Rust: I think they are different enough they serve different use cases.

                                                                                                Go is much faster to learn, doesn’t require you to think about memory management or ownership (sometimes good, sometimes very very bad, Go code tends to be full of thread-safety bugs), is usually fast enough. For standard web app that is CPU-bound it’ll work just fine, without having to teach your team a vast new language scope.

                                                                                                On the flip side, I’m working on a LD_PRELOADed profiler that hooks malloc(), and that’s basically impossible with Go, and I also need to extract every bit of performance I can. So Rust is (mostly) great for that—in practice I need a little C too because of Reasons. More broadly, anytime you want to write an extension for another language Go is not your friend.

                                                                                                1. 4

                                                                                                  Go comes with a built-in race detector. What sources do you have for “tends to be full of thread-safety bugs”?

                                                                                                  1. 11

                                                                                                    As someone with a fair bit of go experience: the race detector only works on races you can reproduce locally, as it’s too slow to run in production.

                                                                                                    Rust stops you from doing things that can race; go gives you a flashlight to look for one after you accidentally introduce it.

                                                                                                    1. 1

                                                                                                      That is a good point, but once a developer has encountered the type of code that makes the race detector unhappy, will they not write better code in the future? Is this really a problem for most popular Go projects on GitHub, for instance? Also, the “staticcheck” utility helps a lot.

                                                                                                      1. 2

                                                                                                        Unfortunately, there’s still, even among some experienced programmers, a notion that “a race will happen so rarely we don’t have to worry about this one case”. Also I’ve seen an expectation that e.g. “I’m on amd64 so any uint64 access is atomic”, with no understanding of out-of-order execution etc. I assume in Rust this would be a harder sell… though the recent drama with a popular web framework (can’t recall the name now) in Rust seems to show to me that using unsafe is probably kinda similar approach/cop-out (“I just use unsafe in this simple case, it really does nothing wrong”, or “because speed”).

                                                                                                2. 3

                                                                                                  I think (hope) the world settles into Rust and Go. I don’t see the Bell Labs folks not going 3 for 3 with Go. Rust I enjoyed playing with many months ago and it was a bonus I guess (at the time) that it was backed by Mozilla. Misplaced loyalties abound. Maybe (in a decade or so) lots of things once written in C will Rust over and tcp/http stuff will be written in Go.

                                                                                                1. -3

                                                                                                  It seems to be a common theme of prog-lang-started-by-child-prodigy projects that they adopt features where I simply can’t fathom how they are going to maintain and develop them in the mid-to-long-term.

                                                                                                  Perhaps I’m the only one who is concerned by the complexity these party-trick features seem to involve?

                                                                                                  (The other option is that this stuff is really that easy and all the hundreds of full-time C/C++ compiler engineers are just idiots for not doing it.)

                                                                                                  1. 33

                                                                                                    There are more details on why and how this works here: zig cc: a Powerful Drop-In Replacement for GCC/Clang

                                                                                                    The other full time C/C++ compiler engineers are not idiots; they just have different goals since they work for companies trying to turn a profit by exploiting open source software rather than working for a non-profit just trying to make things nice for everyone.

                                                                                                    1. 6

                                                                                                      The other full time C/C++ compiler engineers are not idiots; they just have different goals since they work for companies trying to turn a profit by exploiting open source software rather than working for a non-profit just trying to make things nice for everyone.

                                                                                                      This feels like a big statement, and that’s fine, but would you mind elaborating? Which companies do you mean? What goals do they have that are incompatible with something like zig cc?

                                                                                                      1. 5

                                                                                                        I think the point there was just that e.g. Apple has no particular interest in making using clang to cross-compile Windows binaries easy. They wouldn’t necessarily be against it, but it’s not something that aligns with their business interests whatsoever, so they’re very unlikely to spend any money on it. (Microsoft actually does value cross-compilation very highly, and has been doing some stuff in that area with clang, and so is almost a counterexample. But even there, they focus on cross-compilation in the context of Visual Studio, in which case, improving the CLI UI of clang again does not actually do anything for them.)

                                                                                                    2. 40

                                                                                                      Am I the only one who is concerned by the complexity these party-trick features seem to involve?

                                                                                                      (The other option is that this stuff is really that easy and all the hundreds of full-time C/C++ compiler engineers are just idiots for not doing it.)

                                                                                                      This mindset is one of the major causes why modern software sucks so much. The amount of tools that can be improved is humongous and this learned helplessness is why we keep having +N layer solutions to problems that would require re-thinking the existing toolchains.

                                                                                                      I encourage you to read the Handmade Manifesto and to dive deeper into how Zig works. Maybe you’re right, maybe this is a party trick, but the reality is that you don’t know (otherwise you would take issue with specific approaches Zig employs) and you’re just choosing the safe approach of reinforcing your understanding of the status quo.

                                                                                                      Yes, there are a lot of snake oil sellers out there, but software is not a solved problem and blanket statements like this one are frankly not helping anybody.

                                                                                                      1. 1

                                                                                                        I think you are wrong and the exact opposite is the case:

                                                                                                        We can’t have nice things because people don’t learn from their predecessors.

                                                                                                        Instead they go out to reinvent flashy new stuff and make grandiose claims until it turns out they ignored the inconvenient last 20% of work that would make their new code reliable and complete – oh, and their stuff takes 200% more resources for no good reason.

                                                                                                        So yeah, if people don’t want to get suspected of selling snake oil, then they need to be straight-forward and transparent, instead of having these self-congratulatory blog articles.

                                                                                                        Build trust by telling me what doesn’t work, and what will never work.

                                                                                                        1. 17

                                                                                                          Here’s what doesn’t work https://github.com/ziglang/zig/labels/zig%20cc.

                                                                                                      2. 7

                                                                                                        Clang could provide the same trivial cross compilation if it were a priority. Zig is mostly just using existing clang/llvm features and packaging them up in a way that is easier for the end user.

                                                                                                        1. 21

                                                                                                          “just”

                                                                                                          1. 4

                                                                                                            Perhaps not obvious, but I meant the “just” to be restricted to “mostly just using existing clang/llvm features”. I’m in no way denegrating Andrew’s efforts or the value of good UX.

                                                                                                        2. 5

                                                                                                          Another option is that it’s easy if you build it in at the start and much more difficult to add it later. It’s like the python 2 to 3 migration. It wasn’t worth it for some projects, but creating a new python 3 project is easy. Path dependence is a thing.

                                                                                                          1. 2

                                                                                                            I think the hard part is adding these kinds of features after the fact. But assuming it’s already in place, I feel like this is actually not a very hard thing to maintain?

                                                                                                            I think a lot of complexity with existing tools is around “oh we’re going to have this be global/implicit” and that permeating everywhere, so then when you want to parametrize it you have to play a bunch of tricks or rewrite everything in the stack to get it to work.

                                                                                                            But if you get it right out of the door, so to speak, or do the legwork with some of the dependencies… then it might just become a parameter passed around at the top level (and the lower levels already had logic to handle this, so they don’t actually change that much).

                                                                                                            case in point: if you have some translation framework relying on a global, your low-level will read that value and do a lookup, and the high level will not handle it. If you parameterize it, now your high-level stuff has to pass around a bunch of translation state, but the low-level (I guess the hard part, so to speak?) will stay basically the same. At least in theory

                                                                                                            I do kinda share your skepticism with the whole “let’s rewrite LLVM” thing… but cross compilation? Having a build system that is “just zig code” instead of some separate config lang? These seem good and almost simpler to maintain. I don’t think C compiler engineers are idiots for not doing X, just like… less incentivised to do that, since CMake isn’t a problem for someone who has spent years doing it.

                                                                                                            1. 2

                                                                                                              I agree with you. This doesn’t make any sense for Zig to take on. Andrew shared it with me as he was working on it and I thought the same thing then: what? Why does a compiler for one language go to this much trouble to integrate a toolchain for another language? Besides being severely out of scope, the problem space is fraught with pitfalls, for example with managing sysroots and dependencies, maintaining patched forks of libcs, etc. What a huge time sink for a group who should ostensibly have their hands full with, you know, inventing an entire new programming language.

                                                                                                              The idea of making cross-compilation easier in C and C++ is quite meritous. See Plan 9 for how this was done well back in the naughts. The idea that it should live in the zig build tool, however, is preposterous, and speaks rather ill of the language and its maintainers priorities. To invoke big corporate compiler engineers killing open source as the motivation is… what the fuck?

                                                                                                              Sorry Andrew. We don’t always see eye to eye, but this one is particularly egregious.

                                                                                                              1. 7

                                                                                                                No, this makes a lot of sense. Going back to the article, Go’s toolchain (like Plan 9’s) is good at cross-compilation, but “I recommend, if you need cgo, to compile natively”. This sort-of works for Go because cgo use is low. But Zig wants to encourage C interoperability. Then, Zig’s toolchain being good at cross-compilation is useless without solving C’s cross-compilation, because most of Zig will fail to cross-compile because of C dependency somewhere. By the way, most of Rust fails to cross-compile because of C dependency somewhere. This is a real problem.

                                                                                                                Once you solved the problem, it is just a good etiquette to expose it as CLI, aka zig cc, so that others can use it. The article gives an example of Go using it, and mentions Rust using it in passing.

                                                                                                                I mean, yes, zig cc should be a separate project collaboratively maintained by Go, Rust, and Zig developers. Humanity is bad at coordination. Big companies are part of that problem. Do you disagree?

                                                                                                                1. 2

                                                                                                                  The best way, in my opinion, to achieve good C interop is by leveraging the tools of the C ecosystem correctly. Use the system linker, identify dependencies with pkg-config, link to system libraries, and so on. Be prepared to use sysroots for cross-compiling, and unafraid to meet the system where it’s at to do so. Pulling the concerns of the system into zig - libc, the C toolchain, statically building and linking to dependencies - is pulling a lot of scope into zig which really has no right to be there. Is the state of the art for cross-compiling C programs any good? Well, no, not really. But that doesn’t mean that those problems can jump domains into Zig’s scope.

                                                                                                                  I am a believer that your dependency’s problems are your problems. But that definitely doesn’t mean that the solution should be implemented in your domain. If you don’t like the C ecosystem’s approach to cross compiling, and you want to interoperate with the C ecosystem, the correct solution involves going to the C ecosystem and improve it there, not to pull the responsibilities of the C ecosystem into your domain.

                                                                                                                  Yes, other languages - Go, Rust, etc - should also be interested in this effort, and should work together. And yes, humanity is bad at cooperation, and yes, companies are part of that problem - but applying it here doesn’t make sense. It’s as if I were talking about poaching as contributing to mass extinction, and climate change for also contributing to mass extinction, and large corporations for contributing to climate change, and then conclude that large corporations are responsible for poaching.

                                                                                                                  1. 3

                                                                                                                    There’s another path to achieving C interop, which is by using whatever feels more convenient but staying true to whatever ABI boundaries. In terms of Zig, this is achieved in a few ways: It uses its own linker (currently LLD) which is useful when you don’t have a local system linker (pure linux/windows install) and still works with existing C code out there. It uses paths for dependencies, leaving it up to the user to specify how they’re found (e.g. pkg-config). It links to system libraries only if told explicitly but still works without them - this is also useful when building statically linked binaries which still work with existing C code.

                                                                                                                    For cross-compiling, sysroot is a GCC concept. This doesn’t apply to other environments like clang (the C compiler Zig uses), or the defaults of Mac/Windows. Zig instead uses LLVM to emit any supported machine code (something which requires having multiple compilers for in GCC), bundled the build environment needed (lib files on windows, static libc on linux if specified, nothing if dynamically linking), and finally links them together to the appropriate output using LLD’s cross-linking ability.

                                                                                                                    Having this all work seamlessly from whatever supported system is what makes it appealing. For example, andrew (creator of Zig) has showcased in the past cross-compiling the compiler on an x86 machine to aarch64, then using qemu to cross-compile the compiler again from the aarch64 vm back to x86, and it works. This applies also to other operating systems, which is a feature that isn’t present in current cross compiling tools, even clang.

                                                                                                                    For the issue of problem domains, this is not something you could address by trying to fix existing C tools. Those already have a defined structure as andrew noted above given they have different goals and are unlikely to change it. This could be why Zig takes upon solving these problems locally, and pulls the responsibility of what it wishes to provide, not the entire C ecosystem. I believe its partially of similar sub-reasons why Go has its own build system but also claims to compile to different environments.

                                                                                                                    I also agree that different ecosystems could pitch in for what seems to be a universally helpful tool, but as its been going on today, maybe they have different design goals. Where another path such as using the existing C ecosystem (for various situational reasons) makes more sense than the idealistic one Zig has chose to burden.

                                                                                                                    1. 1

                                                                                                                      It links to system libraries only if told explicitly but still works without them - this is also useful when building statically linked binaries which still work with existing C code.

                                                                                                                      System libraries can also be static libraries, and there’s lots of reasons to link to them instead. We do build statically linked programs without the Zig tools, you know!

                                                                                                                      For cross-compiling, sysroot is a GCC concept. This doesn’t apply to other environments like clang

                                                                                                                      Clang definitely uses sysroots. Where does it find the static libs you were referring to? Or their headers? The answer is in a sysroot. Zig may manage the sysroot, but it’s a sysroot all the same.

                                                                                                                      There’s more to take apart here, but on the whole this is a pretty bad take which seems to come from a lack of understanding about how Linux distributions (and other Unicies, save for macOS perhaps) work. That ignorance also, I think, drove the design of this tool in the first place, and imbued it with frustrating limitations which are nigh-unsolvable as a consequence of its design.

                                                                                                                      1. 3

                                                                                                                        The explicitly provided system libraries is not about dynamic vs static linking, its about linking them at all. Even if you have the option to statically link libc, you may not want to given you can do its job sometimes better for your use case on platforms that don’t require it (e.g. linux). The closest alternative for C land seems to be -ffreestanding (correct me if i’m wrong)? This is also an option in zig, but it also gives the option to compile for platforms without having to link to any normal platform libraries.

                                                                                                                        Clang has the option to use sysroots, but it doesn’t seem to be required. In zig’s case, it uses whatever static libs you need by you explicitly linking to them rather than assuming they exist upon a given folder structure in the same directory. Zig does at least provide some methods of finding where they are on the system if you don’t know there they reside given the different configurations out there. I’d say this differs from a sysroot as its more modular than “system library directory”.

                                                                                                                        Without a proper explanation, the idea that this approach “stems from lack of understanding” or has “frustrating limitations which are nigh-unsolvable” don’t seem make such sense. As we’re both guilty of prejudice here, i’d relate your response to one of willfully ignorant to unknown systems and gate-keeping.

                                                                                                                        1. 1

                                                                                                                          Clang has the option to use sysroots, but it doesn’t seem to be required.

                                                                                                                          Your link does not support your statement. I don’t think you understand how cross-compiling or sysroots actually work.

                                                                                                                          Again, it’s the same with the rest of your comments. There are basic errors throughout. You have a deep ignorance or misunderstanding of how the C toolchain, linking, and Unix distributions work in practice.

                                                                                                                          1. 4

                                                                                                                            Given you haven’t actually rebutted any of my claims yet, nor looked into how clang supports using sysroots, we probably won’t be getting anywhere with this. Hope you’re able to do more than troll in the future.

                                                                                                                            1. 1

                                                                                                                              Clang totally uses sysroot, see here. (Long time ago, I wrote the beginning of Clang’s driver code.) I don’t know where to begin, but in fact, ddevault is correct about all technical points and you really are demonstrating your ignorance. Please think about it.

                                                                                                                              1. 3

                                                                                                                                Please re-read by post above which literally says “clang supports using sysroots”, a claim that agrees with yours. My original point a few messages back was about how clang doesn’t need sysroot in order to cross-compile, which still stands to be disproved, as its just short for a bunch of includes.

                                                                                                                                Once again, just as ddevault, you enjoy making claims about others without specifying why in an attempt to prove some point or boost your ego. Either ways, if this is your mindset, there’s no point in further discussion with you as well.

                                                                                                              2. 1

                                                                                                                What features in this case?

                                                                                                                1. 1
                                                                                                              1. 3

                                                                                                                Is this for Optane only or are there other NVRAM technology dimms available to consumers?

                                                                                                                1. 1

                                                                                                                  There are enclosures for traditional DIMMs with backup battery. Not sure if they provide the same “API”, though.

                                                                                                                  1. 2

                                                                                                                    Can you only have one instantiation in a compilation unit?

                                                                                                                    1. 9

                                                                                                                      As many as you like, just like C++. An A* pathfinder, for instance: https://github.com/glouw/ctl/blob/master/examples/astar.c#L84

                                                                                                                      1. 2

                                                                                                                        Signs point to no, as per the examples on this page.

                                                                                                                      1. 6

                                                                                                                        Krste is a strong believer in macro-op fusion but I remain unconvinced. It requires decoder complexity (power and complexity), more i-cache space (power), trace caches if you want to avoid having it on the hot path in loops (power and complexity), weird performance anomalies when the macro-ops span a fetch granule and so the fusion doesn’t happen (software pain). And, in exchange for all of this, you get something that you could have got for free from a well-designed instruction set.

                                                                                                                        1. 3

                                                                                                                          Why is more instruction cache needed? Is this lumping in the uop cache with L1i, or…?

                                                                                                                          1. 12

                                                                                                                            Imagine you need to clear the upper word of a variable (var &= 0xFFFFFFFF). This is very common when doing arithmetic on size smaller than the word size (e.g. 32-bit on a 64-bit computer). On RISC-V it is done with two instructions: a shift left followed by a shift right. For example, consider a 16-bit unsigned addition x8 += x9:

                                                                                                                            add x8, x8, x9
                                                                                                                            slli x8, x8, 16
                                                                                                                            srli x8, x8, 16
                                                                                                                            

                                                                                                                            In the base instruction set, each instruction would require 4 bytes, for a total of 12 bytes. Of course, the compressed instruction set allows these instructions to be encoded as 2 bytes each, for a total of 6 bytes. As a point of comparison, on x86 this would be ADD ax, bx, which is encoded as 3 bytes.

                                                                                                                            Separate to the encoding size issue is that on a typical architecture, these instructions will take at least 3 cycles to complete. They cannot be rearranged or completed in parallel because of the dependency of each instruction on the result of the previous instruction. One way around this is to just create a new instruction to zero-out the top 16-bits of a register. However, the RISC-V people decided against doing this. Instead, they suggest that CPUs implement macro-op fusion. So the instruction decoder has to recognize that a left shift followed by a right shift of the same amount can be done in one cycle. This is tricky to get right (and no one has done it in silicon AFAIK). It also introduces a lot of complexity to optimization, similar to in-order superscalar cores (see e.g. the pentium 1 section of this guide).

                                                                                                                            However, there is also the problem that while the above code could execute in 2 cycles with macro-op fusion, it still takes 6 bytes to encode. This takes up extra space in the instruction cache when compared to a dedicated “clear upper word” instruction (4 bytes) or more compact/irregular instruction sets like x86 (3 bytes).

                                                                                                                            (A way around this could be to cache the decoded instructions. This is not an uncommon technique, but decoded instructions are usually larger than encoded instructions. So it’s unlikely that this can be used to improve instruction cache density.)

                                                                                                                            1. 4

                                                                                                                              Thanks for the elaboration. I have been enlightened. :)

                                                                                                                              So that particular sequence of instructions would be longer, but assuming that macro-op fusion can be made to work well, is it possible to increase the code density this way overall? For example, although there is not a specific instruction to clear the upper bits, since there are fewer instructions to encode it potentially allows more common instructions to be shorter?

                                                                                                                              1. 6

                                                                                                                                So that particular sequence of instructions would be longer, but assuming that macro-op fusion can be made to work well, is it possible to increase the code density this way overall?

                                                                                                                                Yes (hopefully). In general, the RISC-V authors have chosen to make RISC-V very RISC-y when compared to other RISC ISAs. This manifests in few instructions, 3-operands for everything, and a very regular encoding in the base ISA. This has been criticized as wasting space in the base ISA, though I don’t have a link to such criticism on hand.

                                                                                                                                For example, although there is not a specific instruction to clear the upper bits, since there are fewer instructions to encode it potentially allows more common instructions to be shorter?

                                                                                                                                This is effectively what compressed instructions are for.

                                                                                                                          2. 1

                                                                                                                            The paper linked in the article appears to show RV64GC, the compressed variant of RV64G, results in smaller program sizes than x86_64. If that’s true, wouldn’t that mean you would need less i-cache space? This isn’t my area of expertise, but I find it fascinating.

                                                                                                                            1. 3

                                                                                                                              There are a lot of variables here. One is the input corpus. As I recall, that particular paper evaluated almost exclusively C code. The generated code for C++ will use a slightly different instruction mix, for other languages the difference is even greater. To give a concrete example, C/C++ do not have (in the standard) any checking for integer overflow. It either wraps for unsigned arithmetic or is undefined for signed. This means that a+b on any C integer type up to [u]int64_t is a single RISC-V instruction. A lot of other languages (including Rust, I believe, and the implementations of most dynamic languages) depend on overflow-checked arithmetic on their fast paths. With Arm or x86 (32- or 64-bit variants), the add instructions set a condition code that you can then branch on, accumulate in a GPR, or use in a conditional move instruction. If you want to have a good fast path, you accumulate the condition code after each arithmetic op in a hot path then branch at the end and hit a slow path if any of the calculations overflowed. This is very dense on x86 or Arm.

                                                                                                                              RISC-V does not have condition codes. This is great for microarchitects. Condition code registers are somewhat painful because they’re an implicit data dependency from any arithmetic instruction to a load of others. In spite of this, Arm kept them with AArch64 (though dramatically reduced the number of predicated instructions, to simplify the microarchitecture) because they did a lot of measurement and found that a carefully optimised compiler made significant use of them.

                                                                                                                              RISC-V also doesn’t have a conditional move instruction. Krste likes to cite a paper by the Alpha authors regretting their choice of a conditional move, because it required one extra read port on the register file. These days, conditional moves are typically folded into the register rename engine and so are quite cheap in the microarchitecture of anything doing a non-trivial amount of register rename (they’re just an update in the rename directory telling subsequent instructions which value to use). Compilers have become really good at if-conversion, turning small if blocks into a path that does both versions and selects the results. This is so common that LLVM has a select instruction in the IR. To do the equivalent with RISC-V, you need to have logic in decode that recognises a small branch forward and converts it into a predicated sequence. That’s a lot more difficult to do than simply having a conditional move instruction and reduces code density.

                                                                                                                              I had a student try adding a conditional move to a small RISC-V processor a few years ago and they reproduced the result that Arm used in making this decision: Without conditional moves, you need roughly four times as much branch predictor state to get the same overall performance.

                                                                                                                              Note, also, that these results predate any vector extensions for RISC-V. They are not comparing autovectorised code with SSE, AVX, Neon, or SVE. RISC-V has used up all of its 16-bit and most of its 32-bit instruction space and so can’t add the instructions that other architectures have introduced to improve code density without going into the larger 48-bit encoding space.

                                                                                                                              1. 1

                                                                                                                                The paper linked in the article appears to show RV64GC, the compressed variant of RV64G, results in smaller program sizes than x86_64

                                                                                                                                x86-64 has pretty large binary sizes; if your compressed instruction set doesn’t have smaller binaries than it should be redesigned.

                                                                                                                                I would take the measurements in that paper with a grain of salt; they aren’t comparing like-to-like.
                                                                                                                                The Cortex A-15 Core benchmarks, for example, should have also been run in Thumb-2 mode.
                                                                                                                                Thumb-2 causes substantial reductions in code size; it’s dubious to compare your compressed ISA to a competitor’s uncompressed ISA.

                                                                                                                                1. 1

                                                                                                                                  This paper has some size comparisons against thumb (along with Huawei’s custom extensions). This page has sone as well.

                                                                                                                              1. 3

                                                                                                                                My memory of the Catan rules is hazy but shouldn’t it be possible to just store every possible option when something invisible happens? There is very little information to store per player and there shouldn’t be that much branching because invisible actions are rare.

                                                                                                                                1. 1

                                                                                                                                  Yeah, the entire possibility state would not take much ram; even if it did, when new evidence arises you could trim the state space to eliminate incompatible ones.

                                                                                                                                  1. 1

                                                                                                                                    He’s essentially doing that, but in a compressed form that makes it easy to trim invalid states.

                                                                                                                                  1. 1

                                                                                                                                    Perhaps homebrew should do this for you?

                                                                                                                                    1. 4

                                                                                                                                      Looks interesting, primarily because of the first-class datalog integration. It will be interesting to see if there’s significant benefit to having first class support rather than a high-quality library.

                                                                                                                                      1. 1

                                                                                                                                        I wonder if performance could be improved significantly by moving to a higher dimensional binary space. Then you just need to calculate the Hamming distance using XOR and POPCOUNT. It sounds like AVX2 doesn’t have a POPCOUNT, but you can implement one using a lookup table: https://stackoverflow.com/questions/50081465/counting-1-bits-population-count-on-large-data-using-avx-512-or-avx-2

                                                                                                                                        Or use the AVX-512 POPCOUNT.