Threads for atilaneves

  1. 25

    I laughed at the headline since if you look at my open source libraries you’ll find a few files bigger than that: https://github.com/adamdruppe/arsd/ simpledisplay is 21.8k, nanovega.d is 15.1k, minigui.d is 14.4k, cgi is 11.1k…

    I find larger files easier to work with than collections of smaller files, all other things equal, and I like having complete units of functionality.

    But “all other things equal” does a lot there - the article’s description of “It looked like the entire file would execute through from top to bottom”… That’s what is really scary: not the size of the file, but rather that this appears to be a large single function. See, if I open simpledisplay.d, I’m not looking at the whole file. I’m just interested in SimpleWindow.close() or EventLoop.impl. The rest of the file isn’t terribly important; I open it, jump straight to the function I want to work on, and do what needs to be done. Then the individual function is pretty ordinary.

    So I push back against file size by itself as mattering - a file is just a container. You actually reason about functions or classes or whatever so that’s what you want to keep easy to follow (and note that smaller is not necessarily easier to follow, I also prefer long, simple code to short, clever code, and I’d prefer direct local use to indirect things through multiple layers).

    1. 9

      I find larger files easier to work with than collections of smaller files, all other things equal

      That’s interesting, I’m very much the opposite: I try to keep my source files under 500 lines, and each file has a specific purpose like one class, or a set of constants used in one module. Makes it a lot easier to jump to a specific part of the code, by just clicking on a filename or tab. And when I search I’m limited to the appropriate context, like the specific class I’m working on.

      What is it you prefer about single big files?

      1. 8

        I’m also team big files. I hate how many subjective decisions you have to make when you split things across files “Hey we split up foo and bar, now where do we put this function that was used by both foo and bar? Into foo.js, bar.js, or helpers.js?” or “Hey do we group models together and controllers together, or do we group by feature?”.

        Whatever organizational decisions you make, some of them will ultimately prove unsatisfying as your codebase evolves and you’ll face the constant temptation to spend a bunch of energy reorganizing, but reorganizing your code across files doesn’t actually make your code more modular or more adaptable or improve it in any way other than maybe making it a little bit more navigable for somebody whose muscle memory for navigating codebases revolves around filenames.

        I default to large files because it requires the least energy to maintain.

        1. 7

          I’ve had this idea for a while - why can’t a filesystem or text editor support “views,” or different groupings of the same underlying files. Example: view code by business function, or view code by technical concern e.g. “show me all web controllers.”

          1. 2

            It seems what you really want is to store all “source code” in a database instead.

            When you think about it, how can it be possible that storing program code in a bunch of plain text files (ASCII, UTF-8) is in any way optimal for comprehension and modification? Text files are very much a least-common-denominator representation. We continue to use it because of the ecosystem of operating systems, version control, text editors and such allow us to use and interchange this information. So there is a very good reason why they persist to this day.

            But I can imagine some kind of wacky Matrix-y (in the William Gibson, Vernor Vinge sense) 3D representation of programs, which makes great use of colored arrows, shapes and more to represent program operations and flow.

            Do I have the slightest idea of where to start making such a programming “language”, and what exactly it looks like? No, I do not. Until we have better 3D systems (something akin to a floating hologram in front of me), that allows me to easily grab and manipulate objects, I don’t think I’d want to use such a system anyway. But this is the direction I think things will go in… eventually. That will likely take a long time.

            Also, do we want to design a programming system that is optimized for human comprehension? Or something that is optimized for AI to use?

            1. 2

              Also, do we want to design a programming system that is optimized for human comprehension? Or something that is optimized for AI to use?

              Well, my vote is always for humans. I have no stock in AI-produced code every being a good thing.

              It seems what you really want is to store all “source code” in a database instead.

              Actually now that I think about it, NDepend is pretty similar to this. Warning: that page autoplays a video with sound.

              1. 2

                It seems what you really want is to store all “source code” in a database instead.

                So…Smalltalk? Lucid Common Lisp?

                1. 1

                  Those are heading in the right direction, but I’m thinking about something more comprehensive. The link from /u/amw-zero about NDepend is very interesting.

            2. 3

              I’m not sure all or even most of these decisions are subjective - I think that ideally one would want to reduce coupling throughout, and to limit the visibility of implementation details.

              I tend to think in terms of build system DAGs - it profoundly annoys me when everything needs to get rebuilt for no actual good reason. Which is another reason to prefer smaller files, I think - less rebuilding of code.

              1. 3

                I agree. For one, I don’t think the decisions are subjective or objective in itself . But the fact that I do have to spend time thinking about code before I start is a clear benefit to me, not a disadvantage. Yes, sometimes it’s annoying, but mostly in pays off in, as you say, simpler code, clearer boundaries, less stuff tied into one big implementation.

                1. 2

                  Splitting things into classes or modules can limit coupling/visibility. Some programming languages enforce one-module-per-file, but many (golang, ruby) don’t and in these languages there is no encapsulation benefit to putting things in different files.

                  Optimizing the build is a good point, though. If that’s the criterion for file divisions, and not subjective developer feelings about what belongs where, that eliminates the problem for me…

                2. 1

                  At some point in my career, I realized I had crossed a threshold where the writing & testing of the code was no longer the hard part: the hard part is organizing the code so it “makes sense” for what it’s doing, and I’ll be able to figure it out in 5 years when I come back to it.

                  My mental line is 500 lines, too. Once I hit that, and I don’t immediately know how to break it up, it’s usually a sign that I need to take a hike and think about the structure at a higher level. Most of the time, this mental refactoring unlocks a lot of future features, and the invested “thinking” time pays itself off multiple times over.

                  (None of this is a new insight, btw. I think it’s been written about since before the transistor.)

                3. 2

                  What is it you prefer about single big files?

                  Basically everything. It is easier to find what I’m looking for, since I can just search inside the file rather than having to find more. Perhaps if I used an IDE and project files I’d feel differently, but I don’t. And even if I did, sometimes I browse projects online, and online, you often click links that lead to single files to see in the web browser. So it is easier to work with here and easier to browse online. It is also easier for users (which includes me doing a quick ad-hoc build or test run): I can say “download cgi.d and list it on your build command” and it just works for them, no complication in mirroring the directory and figuring out which list of files belong in the build, etc. D’s encapsulation is set at the file layer too, so I can actually define smaller, better defined API surfaces this way than with a variety of files, since the implementations are all wrapped up, without being tempted to expose package protection or similar to share bits that need to be shared across files (or worse yet, marking something public because I need it from a different file in my own project instead of committing to supporting it long term to the users, which is what I think public SHOULD me).

                  When working with other people’s projects, I find the file organization almost never helps. I can never guess where something is actually found from the directory/file organization and just have to grep -R it.

                  So I believe in the old gmail slogan: don’t organize, search! And then you start working on a more abstract level - classes, functions, etc. - instead of files anyway, so just take one more complication out of the way.

                  1. 0

                    Perhaps if I used an IDE and project files I’d feel differently, but I don’t.

                    It’s not just in IDEs. Any good programming editor ought to offer a file hierarchy view and support multi-file search. I can’t even imagine doing nontrivial coding without that, just like I can’t imagine not having Undo.

                    I don’t mean to sound condescending, but it really is worth it to look into more powerful tools.

                    1. 1

                      What possible benefit could I get from merging my file browser into my editor? I have a file browser, I have an editor, they know how to talk to each other.

                      1. 0

                        In the magic land where your file browser works as well with your editor as in an IDE, I’m sure you are correct. 🦄

                  2. 2

                    I generally like the large file as well, if I don’t have really good navigation in my IDE, because it puts that much more within easy reach of the search function of my editor. And since I spend a lot more time looking for and reading code than jumping confidently to places in a codebase that biases me to wanting longer files.

                    1. 4

                      Does your IDE not have good ‘search this entire project’ support?

                      1. 1

                        Yes, thus my statement “when I have good IDE navigation support.” The logical conclusion of that is Smalltalk where there are no files. But if I am sitting in a terminal with cli tools and vi, large files are easier since I don’t have to keep Ctrl+z’ing to run grep and fg’ing again.

                    2. 2

                      There’s no right / wrong answer here, because obviously people’s individual brains are different. But lots of small files absolutely kills my productivity, because working on large pieces of functionality ends up requiring changing 5-10 files. There’s no good way to look at 10 files simultaneously, I don’t care how large your monitor is. So the benefit of larger files is that all of the code that I need to understand something is localized within something that’s on my screen right now, and it’s pretty easy to use something like the sidebar in Sublime or VSCode to scroll quickly to the part of the file that you need. Or text search for a specific function name to jump right to it. The benefit is not really needing to do anything to find the code that I need.

                      Also:

                      Makes it a lot easier to jump to a specific part of the code, by just clicking on a filename or tab

                      This isn’t unique to the small-file approach, if your IDE / editor has “jump to definition” support, it works just as well within a file. Like I said, while this can be subjective and you may prefer one way or the other, I find this often with people who prefer small files - there’s no actual tangible reason or benefit, it just feels more organized (to proponents) when code is factored into small pieces.

                      It may be a limitation of our tools, but I find too many files to be a cognitive cost. After a while I have too many editor tabs open, I can’t get to the part of code that I wanted without backtracking, etc. And, what is the downside of larger files? “Large things are bad” is not an axiom. “Large files are bad because they are large” is circular logic.

                      All that being said, I don’t care all that much. I can navigate around most codebases whatever the structure.

                      1. 2

                        There’s no good way to look at 10 files simultaneously

                        There’s also no good way to look at 11k lines simultaneously :)

                    3. 7

                      The problem is not, in my experience, large files, but the lack of separation of concerns. Large file sizes can be a symptom of a lack of separation of concerns but they’re the symptom, not the problem. I started work on clang in 2008 because I was working on GNUstep and wanted to use the shiny new Objective-C features that Apple had shipped. Apple had their own fork of GCC and no one merged their changes into the main branch[1] and so Objective-C on non-Apple platforms had a NeXT-era feature set.

                      I looked at GCC to see how much effort it would be to update it. All of the Objective-C code in GCC was contained in a single file, objc-act.c, which was around 10K lines. It didn’t have any clear separation between the compiler stages and it was littered with if (next_runtime) everywhere. Some of the new features needed a new runtime, so all of those would need auditing and extending to provide a different implementation and become exciting switch cases.

                      At the time, clang had mostly working codegen for C (it miscompiled printf implementations, but a lot of C code worked). It also had parsing and semantic analysis support for Objective-C, but no code generation. I started by adding an abstraction layer separating the language-specific parts from the runtime-specific parts. That’s still there: there is an abstract CGObjCRuntime class with a bunch of different subclasses (Apple has two significantly different runtimes and a bunch of different variants of the newer one, so has made a lot of use of this abstraction). For a while, clang had better support for Objective-C on non-Apple platforms than on macOS.

                      Clang now has a bunch of source files that are larger than objc-act.c, but they’re cleanly layered. Parsing, semantic analysis, and IR generation are all in separate files. Objective-C runtime-agnostic IR generation is mostly in one file, Apple runtimes in another, non-Apple runtimes in a third. If you want to navigate the codebase and modify the Objective-C support, it’s easy to find the right place.

                      [1] The FSF used to point to Objective-C as a big win for the GPL. I consider it a great example of failure. NeXT was forced to open source their GCC changes but not their runtime, which made the changes useless in isolation. Worse, the NeXT code was truly awful. If NeXT had offered to contribute it to GCC, I strongly suspect that it would have been rejected, but because the FSF had made such a big deal about forcing NeXT to release it, it was merged.

                      1. 4
                        andy@ark ~/d/zig (master)> wc -l (find src/ -name '*.zig') | sort -nr
                         186923 total
                          23226 src/Sema.zig
                          11146 src/AstGen.zig
                           7805 src/codegen/llvm.zig
                           6745 src/clang_options_data.zig
                           6716 src/link/MachO.zig
                           6586 src/translate_c.zig
                           6359 src/arch/x86_64/CodeGen.zig
                           6187 src/type.zig
                           5554 src/Module.zig
                           5247 src/value.zig
                           5181 src/Compilation.zig
                           5171 src/arch/arm/CodeGen.zig
                           5134 src/main.zig
                        ...
                        

                        shrug

                        1. 1

                          Maybe some of those source files are too big and ought to be broken up into smaller subcomponents?

                      1. 1

                        I worked on a C project with a function with 10kSLOC and 46 parameters. Then someone decided to add another parameter because at that point, I guess why not.

                        1. 2

                          It’s only a problem when you get to 52 parameters and run out of letters. Then you have to add… long options.

                        1. 2

                          Tests should be at the bottom for me. For multiple reasons, but one I’d like to highlight is they handle the API semantics anyway.

                          1. 2

                            it has some serious, even prohibitive limitations when it comes to applications that require short run-times and low latency … Julia also suffers from another limitation it shares with many other languages: garbage collection

                            Short runtime + low latency requirements would indicate to me that a language compiled to native code with a GC would be the best choice - both C++ and Rust, if used properly, would “waste” time deallocating at the end just to hand back the memory to the OS anyway. With a GC, you get the benefits of never deallocating unless you use up too much memory, which is unlikely in a short-lived program.

                            On Zig:

                            Instead, there are structs dedicated to managing memory known as allocators. One intriguing consequence of this is that different programs are free to take completely different approaches to memory management

                            One can use allocators if one wants to even in C, C++ has them as template parameters in its own standard library, and D even has pre-built ones.

                            1. 10

                              This was from 2007. Do people still see this problem nowadays? Admittedly all the interviewing I’ve done recently has been somewhat removed from the front lines, and I’m hiring for positions where people would be working in a somewhat obscure language (Clojure) but I’ve never seen anything like this.

                              1. 5

                                In 2014 i had a group project at uni. The group had 12 people, and the top-grade student wasn’t able to implement a histogram computation for a grayscale image. Which is basically “foreach(pixel) array[pixel] += 1” He also wasn’t able to ask for help in the two days he struggeled with a solution. So my N=1 sample here is that it was necessary 7 years later, and it probably still is

                                1. 5

                                  I have had my first experience at hiring programmers this year and although I have not had the issue described in the article (we do not advertise publicly and are not hiring entry level people at this time), I have also had issues with people who won’t ask for help. For me this is the worst possible failure. If a person came and said “I actually don’t know how to write a loop in this language” I would be patient enough to evaluate how fast they can learn, knowing they would need to do a lot of it. But people who get stuck on things that there is no way they could possibly know (e.g. internal company design decisions we forgot to inform them about) and then just waste time trying to puzzle it out without the required information… I can’t work with people like that. You send them some work with a two week deadline, and they say they are making progress, and then 2 hours before the deadline you find out they have done nothing at all.

                                  1. 2

                                    Yeah, exactly. Just openly saying that one cannot do something, but is willing to learn (and is able to) is way more worth than people telling you they can do everything and then fail doing so.

                                  2. 2

                                    Not to defend the student, who should have asked for help, I suspect that the idea you can do array[pixel] probably just didn’t occur to him, so he got stuck. Similarly for fizzbuzz, I think many coders just don’t know about % because you don’t use it very often. A problem for junior/poor coders is you get stuck in certain places, and then there’s nowhere to go but to try some xyproblem.info nonsense.

                                    1. 4

                                      I think many coders just don’t know about % because you don’t use it very often.

                                      Maybe it depends on the level of abstraction, but I use % (or, more often, bit-twiddling special cases of it) quite a lot in C/C++. That said, you don’t actually need % for fizzbuzz. The pattern repeats every 15 elements, so if you don’t know about modulo arithmetic (which is surprising, since it’s a pretty important part of a computer science degree, even if you don’t know that there’s a built-in operator for it) then you can write something like this:

                                      long i=1;
                                      while (1)
                                      {
                                        printf("%ld\n", i++);
                                        printf("%ld\n", i++);
                                        printf("Fizz\n"); i++
                                        printf("%ld\n", i++);
                                        printf("Buzz\n"); i++;
                                        printf("Fizz\n"); i++;
                                        printf("%ld\n", i++);
                                        printf("%ld\n", i++);
                                        printf("Fizz\n"); i++;
                                        printf("Buzz\n", i++);
                                        printf("%ld\n", i++);
                                        printf("Fizz\n"); i++;
                                        printf("%ld\n", i++);
                                        printf("%ld\n", i++);
                                        printf("FizzBuzz\n"); i++;
                                      }
                                      

                                      This implementation doesn’t require modulo arithmetic at all. If I saw this in an interview, I’d then ask how to do it in a single printf call in the loop. This would then hit an interesting corner case in C, because argument order is undefined and so a correct solution couldn’t use i++ in the arguments and would have to look like this:

                                      long i=1;
                                      while (1)
                                      {
                                        printf("%ld\n%ld\nFizz\n%ld\nBuzz\nFizz\%ld\n%ld\Fizz\nBuzz\n%ld\nFizz%ld\n%ld\nFizzBuzz\n",
                                           i+1,
                                           i+2,
                                           i+4,
                                           i+7,
                                           i+8,
                                           i+11,
                                           i+13,
                                           i+14);
                                        i += 15;
                                      }
                                      

                                      So you can actually write C FizzBuzz implementation with a single statement in the loop body and no arithmetic other than addition.

                                      This isn’t a particularly optimised FizzBuzz, but it’s very simple and the required knowledge to build it in any language is:

                                      • Loops
                                      • Addition
                                      • Outputting strings and integers
                                      • The rules of FizzBuzz
                                      • The ability to spot a repeating pattern in a trivial sequence
                                  3. 3

                                    Yes. I regularly interview candidates for dev roles. Somebody handed in this:

                                    def oh_god_why():
                                        return MyClass.__init__()
                                    
                                    class MyClass(object):
                                        def __init__(self):
                                            # notice the lack of `pass` here
                                    

                                    I don’t even dare guess what they meant.

                                    That’s one example. I’ve… seen things.

                                    1. 1

                                      You’re misjudging “this person has spent about 5 minutes to google for python syntax, a language they probably never used before” for “this person has never programmed”. Sure it could be true, but it’s not a string indicator - especially the pass is something that is very weird to someone being used to C-like languages. void foo() {} is perfectly fine.

                                      1. 4

                                        I’m not sure I can agree with this. An average developer proficient in C should not be so confused that they would submit something like that above (assuming that’s a verbatim submission).

                                        I wouldn’t expect perfection, or even idiomatic usage or the language, but if you’re sending in code to apply for a job the code should at least make sense. I would not want to hire someone willing to send in code that doesn’t make sense - though I would also accept submissions in nearly any language that a candidate is familiar with.

                                        1. 3

                                          I interpreted it as a whiteboard/realtime thing. Of course that changes if it was actually submitted with someone being able to test it.

                                          1. 2

                                            Oh, yeah if we’re talking whiteboard then that all goes out the window.

                                            1. 2

                                              It wasn’t in realtime, they had to refactor a tiny codebase in their own time and submitted that.

                                              1. 2

                                                Well…then…I think I would ask why submit it at all?

                                                1. 2

                                                  I would ask the same… but the person in question emailed that in.

                                          2. 1

                                            I’d expect someone who says they know Python in their CV to well… know Python. And I’m not claiming this is proof of “has never programmed”, I’m saying it’s proof that people that can’t program apply for programming jobs.

                                      1. 9

                                        The nice thing about Go is that when it is verbose, like for the list deletion example, it highlights that the computer is doing more work. If you are constantly deleting items in the middle of an array (or if you’re doing this at all), it might not be the best choice of data structure.

                                        1. 31

                                          This sounds like a post hoc rationalization to me. You can always hide arbitrary computations behind a function call.

                                          I believe this is explained by the lack of generics, and I predict that, if/when generics are implemented, go stdlib will gain a bunch of slice-manipulating functions a-la reverse.

                                          1. 2

                                            Which is probably one of the stronger arguments for genetics.

                                            1. 2

                                              This sounds like a post hoc rationalization to me.

                                              This was always pretty reliably cited as the motivation for a lot of design decisions, back when Go was just released.

                                              You can always hide arbitrary computations behind a function call.

                                              Yes, but one nice thing about Go is that functions are essentially the only way to hide arbitrary computation. (And, relatedly, the only mechanism that Go has to build abstraction.) When you’re reading code, you know that a + b isn’t hiding some O(n²) bugbear. That’s valuable.

                                              1. 3

                                                Yup, I agree with the general notion that reducing expressive power is good, because it makes the overall ecosystem simpler.

                                                But the specific example with list manipulation is “a wrong proof of the right theorem”, and this is what i object to.

                                                Incidentally, a + b example feels like a wrong proof to me as well, but for a different reason. In Go, + can also mean string concatenation, so it can be costly, and doing + in a linear loop can be accidentally quadratic. (I don’t write Go, might be wrong about this one).

                                                1. 1

                                                  In Go, + can also mean string concatenation, so it can be costly

                                                  How do you mean? String concatenation is O(1)…

                                                  doing + in a linear loop can be accidentally quadratic. (I don’t write Go, might be wrong about this one).

                                                  How’s that?

                                                  1. 2

                                                    String concatenation is O(1)…

                                                    Hm, I don’t think that’s the case, seem to be O(N) here:

                                                    
                                                    14:48:36|~/tmp
                                                    λ bat -p main.go 
                                                    package main
                                                    
                                                    import (
                                                        "fmt"
                                                        "time"
                                                    )
                                                    
                                                    func main() {
                                                        for p := 0; p < 10; p++ {
                                                            l := 1000 * (1 << p)
                                                            start := time.Now()
                                                            s := ""
                                                            for i := 0; i < l; i++ {
                                                                s += "a"
                                                            }
                                                            elapsed := time.Since(start)
                                                            fmt.Println(len(s), elapsed)
                                                        }
                                                    }
                                                    
                                                    14:48:40|~/tmp
                                                    λ go build main.go && ./main 
                                                    1000 199.1µs
                                                    2000 505.49µs
                                                    4000 1.77099ms
                                                    8000 3.914871ms
                                                    16000 14.675162ms
                                                    32000 49.782358ms
                                                    64000 182.127808ms
                                                    128000 661.137303ms
                                                    256000 2.707553408s
                                                    512000 11.147772027s
                                                    
                                                    1. 3

                                                      You’re right! O(N). Mea culpa.

                                            2. 41

                                              it highlights that the computer is doing more work

                                              It seems strange to me to trust in the ritual of doing strange and obnoxius, bug-prone, programming to reflect the toil the computer will be burdened with. Doubly strange when append() is generic magical go function built-in which can’t be implemented in Go so it’s hard to even say, without disassembly and studying the compiler, what the code would even end up doing at runtime; I guess we can make very good educated guesses.

                                              1. 20

                                                IMHO, no performance concerns are valid* until a profiler is run. If it wasn’t run, the program was fast enough to begin with, and if it was, you’ll find out where you actually have to optimise.

                                                • Exception: if there are two equally readable/maintanable ways to write code, and one is faster than the other, prefer the faster version. Otherwise, write readable code and optimise when/if needed.
                                                1. 4

                                                  I feel this statement is a bit too generic, even with the exception. Especially when you’re talking about generic stdlib(-ish) functions that may be used in a wide variety of use cases, I think it makes sense to preëmptively think about performance.

                                                  This doesn’t mean you should always go for the fastest possible performance, but I think it’s reasonable to assume that sooner or later (and probably sooner) someone is going to hit some performance issues if you just write easy/slow implementations that may be “fast enough” for some cases, but not for others.

                                                2. 18

                                                  But I want the computer to be doing the work, not me as human source code reader or, worse, source code writer.

                                                  1. 1

                                                    Sure, but a language that hides what is computationally intensive or not, is worse.

                                                    1. 24

                                                      Source code verbosity is poorly correlated with runtime cost, e.g. bubble sort code is shorter and simpler than other sorting algorithms.

                                                      Even in the case of removing items from arrays, if you have many items to remove, you can write a tiny loop with quadratic performance, or write longer, more complex code that does extra bookkeeping to filter items in one pass.

                                                      1. 6

                                                        Then don’t build your language to hide what’s computationally expensive. Fortunately, most languages don’t do any sort of hiding; folding something like list deletion into a single function call is not hiding it - so this statement isn’t relevant to the discussion about Go (even if true in a vacuum). Any function or language construct can contain any arbitrary amount of computational work - the only way to know what’s actually going on is to read the docs, look at the compiler, and/or inspect the compiled code. And, like @kornel stated, “Source code verbosity is poorly correlated with runtime cost”.

                                                      2. 1

                                                        As a user, I don’t want the computer to be doing the work. It reduces battery life.

                                                        1. 10

                                                          If you’re referring to compilation or code-generation - that work is trivial. If you’re referring to unnecessary computation hidden in function calls - writing the code by hand is an incredibly bad solution for this. The correct solutions include writing efficient code, writing good docs on code to include performance characteristics, and reading the function source code and/or documentation to check on the performance characteristics.

                                                          As @matklad said, “You can always hide arbitrary computations behind a function call.”e

                                                    1. 49

                                                      Good lord, how is it elegant to need to turn your code inside-out to accomplish the basic error handling available in pretty much every other comparable language from the last two decades? So much of Go is well-marketed Stockholm Syndrome.

                                                      1. 16

                                                        I don’t think that responding with a 404 if there are no rows in the database is that any language supports out of the box. Some frameworks do, and they all have code similar to this for it.

                                                        1. 3

                                                          And sadly so often error handling is often done in a poor manner in the name of abstraction, though really bad one that effectively boils down to ignore that things can go wrong, meaning that one ends up digging through many layers when they actually do go wrong.

                                                          People eventually give up and copy paste StackOverflow[1] solutions in the hopes that one of them will work, even when the fix is more accidental and doesn’t fix the root cause.

                                                          The pinnacle was once checking code that supposedly could not fail. The reason was that every statement was wrapped in a try with an empty catch.

                                                          But back to the topic. Out of the box is all good and nice until you want to do something different which in my experience happens more often than one would think. People sometimes create workarounds. In the example of non existing rows, for example doing a count before fetch, so doing two queries instead of one, just to avoid cases where a no rows error would otherwise be thrown.

                                                          Now i am certainly not against (good) abstractions or automation, but seeing people fighting against those in many instances makes me prefer systems where they can be easily be added and can easily be reasoned about, like in this example.

                                                          [1] Nothing against StackOverflow, just blindly copy pasting things, one doesn’t even bother to understand.

                                                        2. 10

                                                          In what way is Go’s error handling turning my code inside out?

                                                          1. 6

                                                            Pike has set PLT back at least a decade or two.

                                                            1. 7

                                                              It is possible to improve the state of the art while also having a language like Go that is practical, compiles unusually fast and is designed specifically to solve what Google found problematic with their larger C++ projects.

                                                              1. 8

                                                                compiles unusually fast

                                                                There is nothing unusual about it. It’s only C++ and Rust that are slow to compile. Pascal, OCaml, Zig and the upcoming Jai are decent. It’s not that Go is incredible, it’s that C++ is really terrible in this regard (not a single, but a lot of different language design decisions made it this way).

                                                                1. 3

                                                                  For single files, I agree. But outright disallowing unused dependencies, and designing the language so that it can be parsed in a single pass, really helps for larger projects. I agree on Zig and maybe Pascal too, but in my experience, OCaml projects can be slow to compile.

                                                                  1. 2

                                                                    I’m enjoying tinkering with Zig but I do wonder how compile times will change as people do more and more sophisticated things with comptime.

                                                                    1. 2

                                                                      My impression from hanging out in #zig is that the stage 1 compiler is known to be slow and inefficient, and is intended as a stepping-stone to the stage 2 compiler, which is shaping up to be a lot faster and more efficient.

                                                                      Also there’s the in-place binary patching that would allow for very fast incremental debug builds, if it pans out.

                                                                  2. 2

                                                                    Don’t forget D.

                                                                  3. 1

                                                                    My experience with Go is that it’s actually very slow to compile. A whole project clean build might be unusually fast, but it’s not so fast that the build takes an insignificant amount of time; it’s just better than many other languages. An incremental build, however, is slower than in most other languages I use; my C++ and C build/run/modify cycle is usually significantly faster than in Go, because its incremental builds are less precise.

                                                                    In Go, incremental builds are on the package level, not the source level. A package is recompiled when either a file in the same package changes, or when a package it depends on changes. This means, most of the time, that even small changes require recompiling quite a lot of code. Contrast with C, where most of the time I’m working on just a single source file, where a recompile means compiling a single file and re-linking.

                                                                    C’s compilation model is pretty bad and often results in unnecessary work, especially as it’s used in C++, but it means that you can just work on an implementation by just recompiling a single file every build.

                                                                    1. 1

                                                                      I have not encountered many packages that take more than one second to compile. And the Go compiler typically parallelizes compilation at the package level, further improving things. I’m curious to see any counter examples, if you have them.

                                                                  4. 4

                                                                    I don’t remember anyone in POPL publishing a PLT ordering, partial or total. Could you show me according to what PLT has been set back a decade?

                                                                  5. -2

                                                                    Srsly, I was looking for simpler, and was disappointed by the false promise.

                                                                  1. 44

                                                                    Most IDEs create entire worlds where developers can create, but creating requires configuration. It takes time to adjust that world, to play god, to create shortcuts and hotkeys, to get used to different command structures and UI.

                                                                    The authors seem to be describing Emacs.

                                                                    With code completion, Git control, and even automatic deployment systems, modern IDEs are a Swiss Army Knife of features

                                                                    Do people just not know what can be done in Emacs? Git control???

                                                                    1. 6

                                                                      Emacs is amazing, but has a horrible issue with discoverability. “Distros” like Doom or Spacemacs have tried to address that, but, regardless of my personal feelings on them, they haven’t really broken into the mainstream enough to be noticed. (And as someone who used Emacs for years, I think the discoverability bit is very real.)

                                                                      1. 4

                                                                        Spacemacs

                                                                        I have used Spacemacs for a while, but as someone who frequently opens and closes the editor completely - it just boots way too slow, even IntelliJ is faster.

                                                                        1. 3

                                                                          emacsclient --alternate-editor=''

                                                                          Really, Emacs in daemon mode is faster than anything else. Even vim is slow to start in comparison. I don’t understand why Emacs is the only editor which does this. I guess, you need to be reeeealy slow for someone to actually implement this feature :-)

                                                                          1. 5

                                                                            I’ve tried emacs daemon mode, I never found it very pleasing. There were a number of papercuts I was running into… I need to figure out again what that was.

                                                                            1. 2

                                                                              Yeah, I actually have the same experience, getting useful setup of daemon mode took me several tries. Like, you need to pass an empty string as an argument to --alternate-editor to have it auto-spawn the daemon if it isn’t running yet. That’s right next to sending SIGUSR1 to du in my chart of horrible CLI interfaces :)

                                                                          2. 1

                                                                            People who aren’t used to having a fully-featured editor boot, over SSH, in the blink of an eye don’t really know what they’re missing. One less interruption in the flow of thought.

                                                                            Never really got used to emacs, but boot time is one reason i switched to vim.

                                                                          3. 2

                                                                            I think the discoverability bit is very real.

                                                                            Yes, this is the word I should have used. Good GUIs can really help with discoverability.

                                                                            1. 1

                                                                              Emacs is amazing, but has a horrible issue with discoverability.

                                                                              I do want to specifically point out that the parent comment emphasized the mention of “Git control”, when one of the aspects of the Emacs ecosystem a bystander is most likely to have heard praise for is magit (another being org-mode). So it’s not so much a matter of whether it’s possible to discover magit’s existence in a vacuum – it’s that having access to one of the best interfaces to Git, period, is something often evangelized to non-Emacs-users as a point in Emacs’ favor.

                                                                              1. 1

                                                                                What do you mean by discoverability? Shoudln’t the help-system address that? C-h m to see what commands are bounds, C-h f to see what they do, M-x customize-group to see what you can change.

                                                                                1. 3

                                                                                  C-h and friends are great, but you kind of have to know what the options are to get the help you want. For example, you can do a lot of Git manipulation in dired, but you need to know that exists, or at least to think of maybe dired might have some stuff in that area, to even know what you should be asking for. Experienced users, sure, I think this is sufficiently discoverable, but new users (and I mean that fairly broadly), it’s trickier.

                                                                            1. 4

                                                                              Nim is roughly in the same space as Rust, but with somewhat better ergonomics in the language.

                                                                              1. 7

                                                                                Yes, and zig and D-lang. All three of these deserved a mention if Ada gets a spot, in my opinion.

                                                                                1. 2

                                                                                  Just as my own sanity check, D is not memory safe: https://run.dlang.io/is/FxoDZr. I haven’t tried this, but I believe Ada rejects such code.

                                                                                  1. 2

                                                                                    Neither is Zig. Nim is memory safe thanks to GC, and possibly thanks to a borrowing system once they get it polished.

                                                                                    Incidentally, Araq just hours ago posted that async seems to be working with it ORC: https://forum.nim-lang.org/t/6549#42774

                                                                                    1. 2

                                                                                      That won’t compile if you add @safe: to the top of the module or individually to each function: https://run.dlang.io/is/G9fIev

                                                                                  2. 4

                                                                                    Nim has a garbage collector… I wouldn’t call it the same space as Rust. More like same space as Go?

                                                                                    1. 4

                                                                                      People are using Rust for a lot of things where a GC would be just fine. So the space is kinda wide :)

                                                                                      1. 2

                                                                                        Maybe a GC would be fine for those cases, but by choosing rust one is choosing to avoid a GC. That’s the thing that makes rust special and different and worth talking about at all. Rust with a GC would just not be a big conversation buzz. It would be just another language.

                                                                                        1. 2

                                                                                          Agreed. Having a deterministic destructor is a feature not found in many languages.

                                                                                      2. 1

                                                                                        Nim has a garbage collector

                                                                                        It actually has a ref counting and borrow system similar to rust now as a compiler flag.

                                                                                    1. 5

                                                                                      Editors are not a group you can reasonably compare. A JPEG editor and a WAV editor have nothing in common and exactly the same applies to “text” editors. You don’t want to edit HTML and Java and Latex and ToDos and emails in the same way.

                                                                                      If you squint a little you probably would want to edit all programs in the same way but that unfortunately doesn’t work because you just cannot refactor Python like you refactor Java. So even trying to do such a comparison for IDEs is tricky.

                                                                                      All of these discussions about how far away the arrow keys - or your mouse - are, are completely pointless in comparison. This article almost caught on to this fact and gives a list of editors with their strength but doesn’t make it to the conclusion that text editors are just not a reasonable thing to discuss anymore. Nobody edits “text”. And the people who do use Word.

                                                                                      1. 9

                                                                                        All of these discussions about how far away the arrow keys - or your mouse - are, are completely pointless in comparison. This article almost caught on to this fact and gives a list of editors with their strength but doesn’t make it to the conclusion that text editors are just not a reasonable thing to discuss anymore. Nobody edits “text”. And the people who do use Word.

                                                                                        That’s amusing. Just last night my roommate had a problem where all the cells in her document were set to “£0.00” after saving, it would have taken about 8 hours for her to recover it.

                                                                                        Luckily she already had the list, she just wanted it in a Libreoffice table. I opened vim (Because raw sed is line-based), and just used the expressions: :%s/$/"/, :%s/ \r/, "/. I saved it as a .csv and opened it in Libreoffice Calc, copied the cells, then imported it into Writer by doing C-V and selecting “RTF”.

                                                                                        It took five minutes, not 8 hours. A handful of small adjustments that added 2 minutes on to it, and the table was exactly as before.

                                                                                        You are severely underestimating the power and use of the tools programmers have been using for the last 60 years. The arrogance on display is absolutely staggering.

                                                                                        The purpose of a tool like Vim, is to make your editing commands become muscle memory, so you do not have to think about what you want to do, and then do it. You can simply do it reflexively. I’ve been using it for 8 years on all kinds of information, and it’s extremely powerful.

                                                                                        I wonder, when I hit d} in vim, am I still editing text? The operation itself runs over a paragraph. What about di) which deletes the contents of a set of brackets, do you think you can find a use for that in programming? If your answer here is “paragraphs and argument lists are all text”, that means you’re coming close to the realization that “XML is text”, “S-Expressions are text”, and pretty much every format you can think of is a form of text.

                                                                                        Is ParEdit, a mode for editing the structure of Lisp documents in Emacs, still editing text? What about rope, a refactoring tool for Python with both Emacs and Vim plugins, when you use those plugins, are you still editing text? Vim provides an interface for “text objects”, which allow you to manipulate XML rather easily, are you editing text, or the objects? Of course you are editing text. Because everything that isn’t a binary format is text.

                                                                                        In addition, both Emacs and Vim have binary editors built in. They both have extensive sets of plugins for editing programming languages while retaining the power of each environment, and the benefit of a uniform interface that you have experience and will log 100s of hours in.

                                                                                        You don’t want to edit HTML and Java and Latex and ToDos and emails in the same way.

                                                                                        You’re right, however, that you don’t want to edit all programs the same way. However, most modern text editors have functionality specific to what you’re editing. Hell, the exact commands used for alphabetically sorting the contents of CSS directives in Vim (:%g/{/ .+1,/}/-1 sort) dates back to before Vim existed – it was a feature of ed(1) – which isn’t even visual!

                                                                                        1. 2

                                                                                          So, I think your argument is: we are not editing text we are editing all kinds of things in a text representation. Which is also what I was trying to say.

                                                                                          I think we just disagree about the effectiveness of this approach. I agree that the editors we have are often better than nothing but I wouldn’t advertise vim for it’s ability to edit tabular data using :s. Neither do I think the incantation in your last line is something that I should have to know. I just want a CSS editor and then press the sort button.

                                                                                          #editorflamewar2020

                                                                                          1. 5

                                                                                            So, I think your argument is: we are not editing text we are editing all kinds of things in a text representation.

                                                                                            No, my point was that we are all editing text, we just smooth it over so we don’t have to care about the specifics. Text editors do this too, except they provide more power if you wish it. From your comment here and your other comments about this, you seem grossly uneducated about the capabilities of text editors as tools. I’ve linked you articles in another comment – please read them.

                                                                                            I just want a CSS editor and then press the sort button.

                                                                                            You don’t have to know it.

                                                                                            nnoremap <leader> s :%g/{/ .+1,/}/-1 sort
                                                                                            

                                                                                            My leader key is set to f. So I can just type f s and the document is sorted. In emacs, I can set a mode switch, so that this keybinding is only available for CSS documents.

                                                                                            1. 1

                                                                                              In emacs, I can set a mode switch, so that this keybinding is only available for CSS documents.

                                                                                              FYI, you can do the same in Vim with ftplugins.

                                                                                              1. 1

                                                                                                No need for anything complex, just use autocmd

                                                                                            2. 2

                                                                                              You are not expected to know this incantation. It is not a word, it is a sentence. Do you «know» the sentence «Neither do I think the incantation in your last line is something that I should have to know.» ?

                                                                                              The reason we prefer to use a single editor with a language to think about editing in [without agreeing which one] is that things that are just out of reach of the functionality supplied by the editor/plugins/etc. are actually within reach if you express them in the slang of the editor.

                                                                                              Learning such a language probably doesn’t pay off if handling the things typically represented as text is not an activity taking up a large share of your computer use. And maybe not if you just need five representations and the single-use options available for these representations are good enough from your point of view.

                                                                                              But as there are more (and slightly less popular) things to handle, no thanks, I will take a small set of tools I know well and can push one step beyond over sorting through special-case solutions, with many of them missing «trivial» features I have come to depend on.

                                                                                          2. 5

                                                                                            You don’t want to edit HTML and Java and Latex and ToDos and emails in the same way.

                                                                                            Why not? As an Emacs user, I appreciate that different modes interpret concepts such as paragraphs, words, symbols, top level structures in their own appropriate ways, so that keybindings and other commands trivially adapt to the kind of file I’m working with. Sure, not everything fits perfectly into the abstract, primitive categories, but it’s still preferable to having n different editors and work environments for n different problems.

                                                                                            1. 1

                                                                                              “Why not?”

                                                                                              Because having a dedicated editor is more efficient. PyCharm is better for Python and IntelliJ better for Java and Kile better for Latex and Trello better for Todos and Thunderbird better for Mails.

                                                                                              The reason you can use Emacs reasonably well for all of those is that it actually tries to be n different editors for n different problems. org-mode barely has anything in common with other modes. Paredit as well, etc…

                                                                                              1. 5

                                                                                                Because having a dedicated editor is more efficient

                                                                                                Having a dedicated editor may be more efficient for a single thing, but most of the projects I work on involve ‘text’ files with at least half a dozen different languages (programming languages, build system scriting, markup languages). If I had to switch between editors for each one then each editor would have to be a lot more efficient to offset the cost of the cognitive load from having different tooling for each file type.

                                                                                                1. 3

                                                                                                  Well I always feel a lot less productive in IntelliJ editors, and find the Emacs equivalents a lot more comfortable. I guess that’s a difference in attitude, but I think it should be considered as a factor.

                                                                                                  The reason you can use Emacs reasonably well for all of those is that it actually tries to be n different editors for n different problems

                                                                                                  Major modes are interfaces, of sorts, connecting and implementing the files to existing subsystems (completion-at-point, imenu, xref, …), not their own Editors. Sure, some major modes have more additional keybindints, such as Org mode, but it’s not unrelated: I can still use isearch, avy, highlight-symbol-at-point, etc. or any other function that wasn’t written for a specific mode in mind. Paredit is a minor mode for S-Expression based file formats, so I don’t quite get your point here…

                                                                                              2. 4

                                                                                                You don’t want to edit HTML and Java and Latex and ToDos and emails in the same way.

                                                                                                I most certainly want to do exactly that.

                                                                                                1. 1

                                                                                                  Nobody edits “text”. And the people who do use Word.

                                                                                                  Thanks for making this point clear. I think programmers (usually systems programmers who post to Lobsters or The Orange Site) have basically boxed themselves into a “fake reality” where talking about plain text email and using vi for prose actually matter. I’m basically the only systems programmer on my team (excluding maybe one or two others), and talking to clients (who almost are all non-technical industries) or even other team members - they have big HTML signatures with images (and use formatting in their messages!), reply on top, use IDEs or maybe Notepad++ on Windows. They don’t tweak their window manager or Emacs config, they’re busy whatever line-of-business application.

                                                                                                  1. 2

                                                                                                    I’m not sure that applies to everyone. I have proselint jacked into vim and I use it frequently. The ability to perform operations on sentences or paragraphs without having to move your hand to the mouse is underappreciated.

                                                                                                1. 4

                                                                                                  I had never heard of or thought of using colours to highlight which parens close which and now it seems so obvious in hindsight I want this yesterday.

                                                                                                  1. 4

                                                                                                    It’s been a thing (at least in Emacs) for a while: https://www.emacswiki.org/emacs/RainbowDelimiters

                                                                                                  1. -1

                                                                                                    Can anyone recommend some material describing concrete motivations for adding generics to Go? I’m aware of the abstract idea that you need generics in order to build data structures that work with different types, but is there a real-world setting where this is actually better than just copying code and doing s/type1/type2/g? My sense is that projects that use complex data structures almost always customize the data structures in a way that depends on the data type being stored.

                                                                                                    1. 19

                                                                                                      I hope it’s not that hard to imagine wanting different data structures than hash maps; maybe your problem fits better into a binary search tree for example.

                                                                                                      Well, I for one usually don’t feel like implementing my own red-black tree, so I would like to just grab a library. That library will be much nicer to use if I can just make an RBTree<string, MyFoo>. I certainly wouldn’t want to copy/paste an int->string red-black tree into some file(s) and judiciously search/replace until I have a string->MyFoo tree (and then do the same when I need an int->float tree).

                                                                                                      1. 0

                                                                                                        That makes sense, but I am still looking for some grounding in actual programming practice. Is there a use of a red-black tree that would not warrant customizing it for the storage type? Or one where it would make sense to add a library dependency rather than copying the RB tree code?

                                                                                                        1. 8

                                                                                                          How do you write a library that provides a Red-Black tree that can in principle work with many different client types without generics? This isn’t a rhetorical question, I don’t know Go and I genuinely don’t know how you would implement this kind of library in Go without generics.

                                                                                                          1. 6

                                                                                                            Go’s sync.Map (concurrent hashmap) is an actual real world example of this, and it uses interface{}, akin to Java’s Object.

                                                                                                            1. 25

                                                                                                              Right, that’s a great example. Because it uses interface{}, it:

                                                                                                              • Requires all keys and values to be heap allocated, leading to worse performance, worse memory usage, and worse memory fragmentation. Requiring two heap-allocated ints to store one value in an int->int concurrent hash map is unacceptable for many uses.
                                                                                                              • Is less ergonomic, requiring a cast every time you want to use a value.
                                                                                                              • Provides no type safety. (I imagine this one will be the least convincing to Go programmers, since Go generally expects the programmer to just not make mistakes)
                                                                                                              1. 4

                                                                                                                This brings me back to C++Builder 3 back in the 90s. To use a list, you had to create a class derived from a kind of TItem class to be able to store things. Why anyone would want to go back to that in productive code boggles my mind.

                                                                                                                1. 1

                                                                                                                  I’m using a sync.Map (for its concurrency support - I have many goroutines writing map entries, and another goroutine periodically ranging over the entire map to dump it to a json file).

                                                                                                                  However I know the types I write to the map, I have no need for interface{}.

                                                                                                                  Am I better off with a real typed map + using sync.RWLock/mutex/etc. directly (in a custom struct)? Performance-wise.

                                                                                                                  1. 1

                                                                                                                    I don’t know, you would have to benchmark or measure CPU or memory usage. The sync.Map documentation suggests that using a regular map + mutexes could be better though: https://golang.org/pkg/sync/#Map

                                                                                                                    The Map type is specialized. Most code should use a plain Go map instead, with separate locking or coordination, for better type safety and to make it easier to maintain other invariants along with the map content.

                                                                                                                    The Map type is optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys. In these two cases, use of a Map may significantly reduce lock contention compared to a Go map paired with a separate Mutex or RWMutex.

                                                                                                                    If your usage falls outside of the two use cases which sync.Map is optimized for, it would absolutely be worth looking into replacing your sync.Map with a regular map and a mutex.

                                                                                                                    I suppose it becomes a question of which has the biggest performance penalty for you, heap allocation + indirection with sync.Map or lock contention with regular map + mutex?

                                                                                                                    (Also, in most cases, this probably doesn’t matter; make sure you’re not spending a long time improving performance in a part of your code which isn’t actually a performance issue :p)

                                                                                                                    1. 1

                                                                                                                      Right - the code “just works(TM)” and it takes around 0.5 seconds to render the JSON file every minute (which I track with metrics just to be safe) so it should be fine to keep as is. This is just a for-fun conversation.

                                                                                                                      or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys. In these two cases, use of a Map may significantly reduce lock contention compared to a Go map paired with a separate Mutex or RWMutex.

                                                                                                                      I definitely remember reading this sentence and it made me choose sync.Map because it sounds like my usecase. But like you say if I don’t measure it’ll be hard to tell.

                                                                                                              2. -1

                                                                                                                I don’t know and I didn’t think you could. I’m asking for an example use of an RB tree where using a library would make sense.

                                                                                                                1. 6

                                                                                                                  Here is a popular Go RB tree implementation https://github.com/emirpasic/gods/ that uses Interface{} for the key and value types. Just search github for uses of it… With generics, users of this library would get greater typesafety.

                                                                                                                  https://github.com/search?q=%22github.com%2Femirpasic%2Fgods%2Ftrees%2Fredblacktree%22&type=Code

                                                                                                                  1. -2

                                                                                                                    okay. except i don’t know how to search github for uses of it and your search link brings me to a login page :(

                                                                                                                    1. 5

                                                                                                                      To short-circuit this:

                                                                                                                      At a previous job, I worked on a tool that started various services. The services had different dependencies, each of which needed to be started before the service. We wanted to be able to bring them up with as much parallelism as possible, or have a flag to launch them serially.

                                                                                                                      A simple approach to doing this correctly is modeling the dependencies as an acyclic graph (if it’s a cyclic graph, you’ve got a problem — you can never bring the services up, because they all depend on each other). To launch them in parallel, launch each one that has its dependencies met. To launch them serially, topologically sort the graph into an array/list/whatever and launch them one by one.

                                                                                                                      A generic graph implementation would be very useful, as would a topological sort that worked on generic graphs. With Go, you can’t have one that’s type-safe.

                                                                                                                      Another great use case for graphs: visualizing dependency graphs! You can have an open source graph visualization library, build a graph of whatever it is you’re trying to visualize, and pass it to the library and get a nice visualization of the data.

                                                                                                                      Graph data structures can be quite useful. Supporting generics makes them type-safe, so you catch errors at compile time instead of runtime. Some other examples of the usefulness of graphs:

                                                                                                                      • Graphs of friends at a social network (I currently work at one, and we use generic graph data structures all over the place — graphs of people to people, graphs connecting people and photos they’re tagged in, etc)
                                                                                                                      • Network topology graphs
                                                                                                                      • Graphs of links between documents

                                                                                                                      etc.

                                                                                                                      And it’s not just graphs. How do you write a type-safe function that takes in a list of possibly-null items, and returns a new list with the nulls stripped out? How about a function that takes a map and returns the list of its keys? In Golang, the answer is always copy-paste or give up type safety. In languages with generics, you can trivially write these functions yourself if they’re not in the standard library.

                                                                                                                      1. 1

                                                                                                                        thanks, this is a good motivating example.

                                                                                                                        1. 1

                                                                                                                          Huh. It had not occurred to me that github search would require a login.

                                                                                                              3. 11

                                                                                                                To turn the question around, why would you want to manually copy/paste code all over the place when the compiler can do it for you? And while I personally think “DRY” can be over done, not having the same (or very similar) code copy/pasted all over the place seems like a big practical win.

                                                                                                                As far as customizing specific data structure and type combinations, most languages with generics have a way to do that, and I’d bet the Go designers thought of it.

                                                                                                                1. 2

                                                                                                                  Copy / paste has got it’s own problems, but it lets you avoid a ton of complexity in the toolchain.

                                                                                                                  Toolchain development is all about tradeoffs. For instance, I use Typescript; the reference implementation is featureful, but slow to boot, so it keeps a background process alive to cache the heavy lifting, which accumulates state and introduces subtle confusions (eg type errors that don’t exist) until it’s restarted.

                                                                                                                  For some problem spaces, the problems introduced by copy/paste pale in comparison to the problems introduced by slow, stateful compilers.

                                                                                                                  1. 7

                                                                                                                    Copy/paste vs generics is unrelated to compiler bugginess.

                                                                                                                    If you carefully pick TypeScript as the comparison point, you can make the case that a buggy toolchain is bad (not that most users care, they just restart the compile process when it starts to go bad).

                                                                                                                    But if you were to pick say ReasonML for comparison, you could say that it’s possible to have a solid generics implementation (much less copy-pasting) and a super-fast, accurate compiler.

                                                                                                                    I.e. you can have both buggy and non-buggy compilers supporting generics. Hence, unrelated.

                                                                                                                    1. 2

                                                                                                                      ReasonML is great!

                                                                                                                      That said, while the relationship is indirect, it’s there. Adding complexity is never free. It didn’t cost ReasonML speed or reliability, but it costs maintainers time and makes every other feature more difficult to add in an orthogonal way.

                                                                                                                      1. 2

                                                                                                                        In the scheme of things, is it more important to have a super-simple compiler codebase, or is it more important to put more power and expressiveness in the hands of users? Note that every mainstream language that started without generics, has now added it.

                                                                                                                        1. 1

                                                                                                                          IMO, there’s such a thing as a right time to do it.

                                                                                                                          In the early years it’s more important to keep the simplicity - there aren’t that many users and you’re still figuring out what you want the language to be (not every feature is compatible with every approach to generics).

                                                                                                                          Once you’re ready to start generics you need to answer questions like - do you want monomorphisation or lookup tables? Is boxing an acceptable overhead for the improved debugging ergonomics?

                                                                                                                          1. 1

                                                                                                                            It seems like Go has been going through exactly the process you’re describing.

                                                                                                                        2. 2

                                                                                                                          I think these comparisons are a bit unfair: isn’t Typescript self hosted, whereas ReasonML is written in OCaml? It seems like Typescript would have a very hard time competing.

                                                                                                                          1. 3

                                                                                                                            Being able to use lots of existing OCaml bits is a huge advantage.

                                                                                                                            Typescript has been able to compete due to the sheer number of contributors - MS pays quite a large team to work on it (and related stuff like the excellent Language Server Protocol, VScode integration).

                                                                                                                            However, large teams tend to produce more complex software (IMO due to the added communications overhead - it becomes easier to add a new thing than find out what existing thing solves the same problem).

                                                                                                                            1. 1

                                                                                                                              I should clarify my comment was more about comparing performance of the two languages.

                                                                                                                              OCaml is a well optimized language that targets native machine code so tooling built in OCaml should be more performant than tooling built in Typescript. As a result, it’s hard to compare the complexity of either tool by the performance of the tool. It’s apples and oranges.

                                                                                                                            2. 2

                                                                                                                              isn’t Typescript self hosted, whereas ReasonML is written in OCaml? It seems like Typescript would have a very hard time competing.

                                                                                                                              That’s a strange argument. If it were very hard for them to compete why would they not use OCaml as well, especially since its contemporary alternative, Flow, was written in OCaml too? Or why would they not make TypeScript as good as a language for writing TypeScript in as OCaml is?

                                                                                                                              1. 1

                                                                                                                                My comment was more about performance, but it wasn’t very clear. It’s hard for Typescript, which is compiled to Javascript and then interpreted/JITed, to create tooling that’s as fast as a language that builds optimized native code.

                                                                                                                                Given that Typescript is self hosted it has the advantage that community involvement is more seamless and I don’t want to downplay the power that brings.

                                                                                                                          2. 0

                                                                                                                            But compilers that support generics are more likely to be buggy. That’s a relation.

                                                                                                                            1. 2

                                                                                                                              Any source for this rather surprising assertion?

                                                                                                                              1. 0

                                                                                                                                generics are feature that requires code to implement; code can contain bugs.

                                                                                                                                1. 1

                                                                                                                                  But a self-hosting compiler with generics is likely to be less verbose (because generics) than one without, so it should be less buggy.

                                                                                                                                  1. 1

                                                                                                                                    i guess you can’t prove it either way but IME the complexity of algorithms is more likely to cause bugs than verbosity.

                                                                                                                          3. 5

                                                                                                                            I think Typescript is a straw man. Does this Go implementation of generics slow down the compiler a noticeable amount? There’s nothing inherent to generics that would make compiling them slow.

                                                                                                                            On the other hand, copy/pasted code is an ever increasing burden on developer and compile time.

                                                                                                                          4. -2

                                                                                                                            You are imagining a code base where the same complex data structure is instantiated with two different types. Is that realistic?

                                                                                                                            1. 6

                                                                                                                              You are imagining a code base where the same complex data structure is instantiated with two different types. Is that realistic?

                                                                                                                              Realistic enough that the Linux kernel developers went through the hassle of developing generic associative arrays, circular buffers, and other generic data structures using void*.

                                                                                                                              And so did Gnome with GLib, which provides generic lists, hash tables, and trees, along with several others structures, also using void*.

                                                                                                                              And the standard libraries of most modern languages include reusable and generic sequence and associative data types, and some times significantly more than that.

                                                                                                                              For most data structures, though, focusing on a single code base gives too narrow of a view. Generics allow libraries of data structures to be created, so even though a single code base only use one R* tree (or whatever), that R* tree library can be used as-is by any number of projects.

                                                                                                                          5. 8

                                                                                                                            The Abstract and Background sections of the draft design doc touch on the motivations. Additionally, each section describing a dimension of the design usually mentions, at least briefly, the motivation for that feature.

                                                                                                                            1. 8

                                                                                                                              Here is an example that I’ve wanted for ever, and can finally do. Higher order combinators that you can leverage first class functions with!

                                                                                                                              Generic map, in go

                                                                                                                              1. 1

                                                                                                                                That’s the type of thing I have seen as a justification, but I don’t get why that’s so important. Can’t you just use a for loop?

                                                                                                                                1. 22

                                                                                                                                  “Can’t you just …” goes forever. “Can’t you just write your for loop with labels and jumps in assembly?”^^

                                                                                                                                  For me, it’s all about abstraction. Having low level combinators, like this, that I can compose to build higher level abstractions in a generic way is wonderful.

                                                                                                                                  ^^: See also whataboutism.

                                                                                                                                  1. 3

                                                                                                                                    I’m not sure that composing from higher level abstractions is always such a good idea. I like both Go (hobby projects) and Rust (work!) but I still fell that most of the time I prefer this level of abstraction:

                                                                                                                                    type Server struct {
                                                                                                                                    ...
                                                                                                                                        Handler Handler // handler to invoke, http.DefaultServeMux if nil
                                                                                                                                    ...
                                                                                                                                    }
                                                                                                                                    type Handler interface {
                                                                                                                                        ServeHTTP(ResponseWriter, *Request)
                                                                                                                                    }
                                                                                                                                    

                                                                                                                                    from this:

                                                                                                                                     pub fn serve<S, B>(self, new_service: S) -> Server<I, S, E>
                                                                                                                                        where
                                                                                                                                            I: Accept,
                                                                                                                                            I::Error: Into<Box<dyn StdError + Send + Sync>>,
                                                                                                                                            I::Conn: AsyncRead + AsyncWrite + Unpin + Send + 'static,
                                                                                                                                            S: MakeServiceRef<I::Conn, Body, ResBody = B>,
                                                                                                                                            S::Error: Into<Box<dyn StdError + Send + Sync>>,
                                                                                                                                            B: HttpBody + 'static,
                                                                                                                                            B::Error: Into<Box<dyn StdError + Send + Sync>>,
                                                                                                                                            E: NewSvcExec<I::Conn, S::Future, S::Service, E, NoopWatcher>,
                                                                                                                                            E: H2Exec<<S::Service as HttpService<Body>>::Future, B>,
                                                                                                                                        {
                                                                                                                                        ...
                                                                                                                                    

                                                                                                                                    Don’t get me wrong, i like type level guarantees and I can see flexibility here, but in my experience with c++, rust and haskell is that generic programming often ends up complicating things to a degree that I personally don’t like.

                                                                                                                                    1. 1

                                                                                                                                      I think this is going to be a balance that the community has to find. I don’t regularly program in rust, but I’d be quite surprised if it wasn’t possible to get something close to the Go http API in it. The example you pasted seems complicated for the sake of being complicated. In theory, the Go community has been drilled into thinking in terms of the littlest abstraction that’ll work, which maybe makes it possible to generally avoid generic APIs that don’t actually need to be?

                                                                                                                                    2. 3

                                                                                                                                      “Can’t you just” does not go forever. It is a simpler way to say that the alternative is not significantly harder than what’s proposed. Is there some type of task that would be doable using a generic map but unreasonably hard using for loops?

                                                                                                                                      I feel like Go was designed from the ground up to be written in an imperative style, and composing first order functions is more of a functional style of coding. If I understand, without generics you would be nesting for loops rather than composing map functions, which is no more difficult to understand or write.

                                                                                                                                      I don’t follow the connection to whataboutism.

                                                                                                                                      1. 2

                                                                                                                                        I think it’s fine for your style of writing code to be to use loops and conditionals instead of map and filter. I think it’s a fine way to code that makes more sense in an imperative language. Straight for loops and while loops with if statements inside them is just better, more easily understandable code in an imperative language, in my opinion, than .map(...).filter(...).map(...) etc.

                                                                                                                                        1. -1

                                                                                                                                          Incidentally there is a repo wherein Rob Pike expresses his attitude towards this style of coding:

                                                                                                                                          https://github.com/robpike/filter/

                                                                                                                                          I wanted to see how hard it was to implement this sort of thing in Go, with as nice an API as I could manage. It wasn’t hard.

                                                                                                                                          Having written it a couple of years ago, I haven’t had occasion to use it once. Instead, I just use “for” loops.

                                                                                                                                          You shouldn’t use it either.

                                                                                                                                          1. 2

                                                                                                                                            I mean… that’s like … one man’s opinion… man. See also.

                                                                                                                                            Generics are going to create a divide in the Go community, and it’s going to be popcorn worthy. There’s no point of adding Generics to the language if this filter thing “shouldn’t be used,” and the community rejects the abstractions that Generics provide.

                                                                                                                                            This divide is easily already seen in the community as it relates to test helpers. On the one hand, there’s a set of developers that say “stdlib testing is more than enough.” On the other hand, there are people who want the full testing facilities of junit, with matchers, lots of assert style helpers, etc. Who is right? They all are, because those things work for their respective teams and projects.

                                                                                                                                            This general dogmatic approach to language idioms is why I call it “idiotmatic” Go.

                                                                                                                                            1. -1

                                                                                                                                              I suppose if Ken and Rob wanted generics they would’ve put them in the original language, and there wouldn’t be this controversy. Time to go back to learning Erlang which seems old and dusty enough to not have big language changes and drama.

                                                                                                                                        2. 16

                                                                                                                                          You can’t pass a for loop to anything, you can only write it where you need it. Sure, toy examples look like toy examples, but the fact remains that Go has first-class functions, which should be a nice thing, but it doesn’t actually have a type system rich enough to express 90% of the things that make first-class functions worth having.

                                                                                                                                          1. -1

                                                                                                                                            You can’t pass a for loop to anything, you can only write it where you need it.

                                                                                                                                            right, so the example code could be done with a for loop no problem. is there a more motivating example?

                                                                                                                                            it doesn’t actually have a type system rich enough to express 90% of the things that make first-class functions worth having.

                                                                                                                                            how do you mean?

                                                                                                                                          2. 3

                                                                                                                                            Consider composing multiple transformations and filters together. With multiple for loops you have to iterate over the array each time, while by composing maps you only need to iterate once.

                                                                                                                                            1. 3

                                                                                                                                              Just compose the operations inside the loop.

                                                                                                                                              for x in y:
                                                                                                                                                  ...f(g(x))...
                                                                                                                                              
                                                                                                                                              1. 2

                                                                                                                                                That works in some cases, but it’s pretty easy to find a counter example, too.

                                                                                                                                        3. 7

                                                                                                                                          In terms of collections, the truth is most of the time a map/slice is a good option. Here’s my top two favorite use cases for generics in go:

                                                                                                                                          1. Result<T> and functions that compose over them.
                                                                                                                                          2. Typesafe versions of sync.Map, sync.Pool, atomic.Value, even a rust like Mutex
                                                                                                                                          1. 5

                                                                                                                                            Oh man. I hadn’t even considered a better way to do error handling, eg. a Result type. People are gonna get so mad.

                                                                                                                                            1. 5

                                                                                                                                              Generics isn’t enough to do what people want to do with error handling 99.99% of the time, which is to return early. For that, you either need a macro, such as the aborted try proposal, or syntactic sugar for chaining such “functions that compose over them” (like Haskell’s do notation).

                                                                                                                                              Otherwise you end up with callback hell à la JavaScript, and I think nobody wants that in Go.

                                                                                                                                              1. 4

                                                                                                                                                I was more thinking of something where the if err pattern is enforced via the type system. You’re still not getting there 100%, you could get reasonably close, with a generic Result type that panics when the wrong thing is accessed, forcing you to check always or risk a panic.

                                                                                                                                                r := thing()
                                                                                                                                                if r.HasError() { handleError(r.Err()) }
                                                                                                                                                else v := r.Val() { handleSuccess(v) }
                                                                                                                                                

                                                                                                                                                And of course it’s easy to question why this is interesting until you do chaining of things, and get a full on, type safe Result monad.

                                                                                                                                                r := thing().andThen(func(i int) { ... }).andThen(func(i int) { ... })
                                                                                                                                                if r.IsErr() {
                                                                                                                                                   handleErrForWholeComputation(r.Err())
                                                                                                                                                } else {
                                                                                                                                                   handleSuccessForWholeComputation(r.Val())
                                                                                                                                                }
                                                                                                                                                

                                                                                                                                                The alternative can be seen in things like this where you skirt around the fact that you can’t generically accept a value in one of those called functions. This is also why I said people are going to get so mad. These things are confusing to people who haven’t dealt with them before, and will make Go much more expressive, but less easy to grok without effort.

                                                                                                                                          2. 5

                                                                                                                                            but is there a real-world setting where this is actually better than just copying code and doing s/type1/type2/g

                                                                                                                                            All of them. Copying code manually is one of the worst things you can do in software development. If it weren’t, why even bother writing functions, ever?

                                                                                                                                            My sense is that projects that use complex data structures almost always customize the data structures in a way that depends on the data type being stored.

                                                                                                                                            The fact that libraries exist that don’t customise in such a way in languages with generics would disprove that notion.

                                                                                                                                            1. 6

                                                                                                                                              one of the worst things you can do in software development

                                                                                                                                              For me that’s “making things unreadable for whoever comes after you”. And sometimes copying a bit of code is the optimal solution for avoid that.

                                                                                                                                              1. 0

                                                                                                                                                but is there a real-world setting where this is actually better than just copying code and doing s/type1/type2/g

                                                                                                                                                All of them. Copying code manually is one of the worst things you can do in software development. If it weren’t, why even bother writing functions, ever?

                                                                                                                                                I disagree with your implication that the use of functions means code should never be copied. For example if you want to use strlcpy() in a portable C program, it makes more sense to put a copy in your source tree rather than relying on an external library. An extra dependency would add more headache than just copying the code.

                                                                                                                                                My sense is that projects that use complex data structures almost always customize the data structures in a way that depends on the data type being stored.

                                                                                                                                                The fact that libraries exist that don’t customise in such a way in languages with generics would disprove that notion.

                                                                                                                                                That’s why I said “almost always.” And remember that the existence of a library doesn’t mean it is used with any frequency.

                                                                                                                                              2. 3

                                                                                                                                                Suppose you have a structure parametrizable by types T1, T2. You’re writing it in Go, so you assume that it’s ok if T1=string, T2=int. Also, in some of the places, you were using int for purpose unrelated to T2 (ie. if T2=foo, then there are still ints left in the source). Another programmer wants to copy-paste the code and change some types. How does he do it?

                                                                                                                                                1. 2

                                                                                                                                                  I think “this would make copy-pasting code harder” is not so compelling an argument. One of the major points of introducing generics is that it would eliminate much of the present need for copy/paste in Go.

                                                                                                                                                  1. 2

                                                                                                                                                    Yes it would be harder than a search-and-replace, but that is still abstract and unrelated to any real-world use case.

                                                                                                                                                    Yes, I’m just counterpointing the parent commenter’s argument. I know the value of generic structures.

                                                                                                                                                  2. -1

                                                                                                                                                    Yes it would be harder than a search-and-replace, but that is still abstract and unrelated to any real-world use case.

                                                                                                                                                1. 1

                                                                                                                                                  Defer was unique to Go but Swift has since adopted it as well, although it is subtly different.

                                                                                                                                                  1. 8

                                                                                                                                                    defer wasn’t unique to Go even when it was introduced given that D has had scope(exit) for longer.

                                                                                                                                                    1. 1

                                                                                                                                                      Not arguing, just reminded of a gotcha people should know: go’s defer is not scope exit, it’s function exit.

                                                                                                                                                    2. 4

                                                                                                                                                      GCC has __cleanup__ attribute for ages.

                                                                                                                                                      1. 2

                                                                                                                                                        ZIg also has defer, and errdefer, which I haven’t seen elsewhere.

                                                                                                                                                        1. 4

                                                                                                                                                          And unlike Go, Zig gets the scoping of defer correct.

                                                                                                                                                          1. 2

                                                                                                                                                            Sorry to mention D again in this thread, but scope(failure) is pretty much similar to errdefer.

                                                                                                                                                        1. 12

                                                                                                                                                          This is interesting, and I think I agree with many arguments when it comes to the reasons java, OCaml, Haskell, Go, etc. haven’t replaced C. However the author cites rust only briefly (and C++ not at all?) and doesn’t really convince me why rust isn’t the replacement he awaits: almost all you can do in C, you can do in rust (using “unsafe”, but that’s what rust is designed for after all), or in C++; you still have a higherer-level, safer (by default — can still write unsafe code where needed), unmanaged language that can speak the C ABI. Some projects have started replacing their C code with rust in an incremental (librsvg, I think? And of course firefox) because rust can speak the C ABI and use foreign memory like a good citizen of the systems world. Even C compilers are written in C++ these days.

                                                                                                                                                          To me that’s more “some were meant for no-gc, C ABI speaking, unsafe-able languages” than “some were meant for C”. :-)

                                                                                                                                                          1. 17

                                                                                                                                                            Besides Rust, I think Zig, Nim, and D are strong contenders. Nothing against Rust, of course, but I’m not convinced it’s the best C replacement for every use case. It’s good to have options!

                                                                                                                                                            Nonetheless, I imagine C will linger on for decades to come, just due to network effects and economics. Legacy codebases, especially low-level ones, often receive little maintenance effort relative to usage, and C code is incredibly widespread.

                                                                                                                                                            1. 15

                                                                                                                                                              I love Rust, but I think Zig and D (in the ‘better C’ mode and hopefully their new borrow checker) are closer to the simplicity and low-level functionality of C. Rust is a much nicer C++, with simpler (hah!) semantics and more room to improve. C++ is, unfortunately, a Frankenstein monster of a language that requires a 2000 page manual just to describe all the hidden weird things objects are doing behind your back. Every time I have to re-learn move semantics for a tiny project, I want to throw up.

                                                                                                                                                              1. 3

                                                                                                                                                                i was also wondering, while reading the article, how well ada would fit the author’s use case (i’m not at all familiar with the langauge, i’ve just heard it praised as a safe low-level language)

                                                                                                                                                                1. 1

                                                                                                                                                                  The lot of them! It was kind of a large gap between C and Python/Perl/Ruby/Java.

                                                                                                                                                                  1. 12

                                                                                                                                                                    Maybe I’m the archetype of a C-programmer not going for Rust. I appreciate Rust and as a Mathematician, I like the idea of hard guarantees that aren’t a given in C. However, Rust annoys me for three main reasons, and these are deal-breakers for me:

                                                                                                                                                                    • Compile time: This is not merely the language’s fault, and has more to do with how LLVM is used, but there doesn’t seem to be much push to improve the situation, either. It annoys me as a developer, but it also really annoys me as a Gentoo user when a Firefox compilation takes longer and longer with each subsequent Rust release. Golang is a shining example for how you can actually improve compilation times over C. Admittedly, Rust has more static analysis, but damn is it slow to compile! I like efficiency, who doesn’t? Rust really drops the ball there.
                                                                                                                                                                    • Standard library/external libraries: By trying to please everyone and not mandating certain solutions, one is constantly referred to this or that library on GitHub that is “usually used” and “recommended”. In other cases, there are two competing implementations. Sure, Rust is a young language, but for that reason alone I would never choose it to build anything serious on top of it, as one needs to be able to rely on interfaces. The Rust developers should stop trying to please everybody and come up with standard interfaces that also get shipped with the standard install.
                                                                                                                                                                    • Cargo/Package management: This point is really close to the one before it: Cargo is an interesting system, but really ends up becoming a “monosolution” for Rust setups. Call me old-fashioned, but I like package managers (especially Gentoo’s) and Cargo just works around it. When installing a Rust package, you end up having to be connected to the internet and often end up downloading dozens of small crates from some shady GitHub repos. I won’t make the comparison with node.js, given Cargo can be “tagged” to a certain version, but I could imagine a similar scenario to leftpad in the future. Rust really needs a better standard library so you don’t have to pull in so much stuff from other people.

                                                                                                                                                                    To put it shortly: What I like about C is its simplicity and self-reliance. You don’t need Cargo to babysit it, you don’t need dozens of external crates to do basic stuff and it doesn’t get in the way of the package manager. I actually like Rust’s ownership system, but hate almost anything around it.

                                                                                                                                                                    1. 16

                                                                                                                                                                      C doesn’t even have a hash table. It needs more external libraries to do basic stuffs, not less.

                                                                                                                                                                      1. 5

                                                                                                                                                                        See, I feel the exact opposite when it comes to Cargo vs. system’s package manager: managing versions of libraries using your system’s package manager is a royal pain in the ass or outright impossible when you have multiple projects requiring different versions of a library. In my experience with C and C++, you’ll end up using CMake or Meson to build exactly the same functionality that Cargo deploys for you, at a much higher cost than just adding one line in a configuration file.

                                                                                                                                                                        In fact, my biggest gripe with C and C++ is that they still depend on a 3-step build system (preprocessing, compiling, linking) each of which requires you to specify the location of a group of files. I get why having header files was attractive in the 1970s when you counted your computer’s memory in KBs, but it makes writing and maintaining code such a pain in the ass when compared with a modern module system.

                                                                                                                                                                        The funniest bit is I used to consider all these things as ‘easy to deal with’ when all I did was write C/C++ code 15 years ago. Nowadays, having to switch from Go, Rust or any other language to C for a small project makes me want to cry because I know I’ll spend about 20% of the time managing bullshit that has nothing to do with the code I care about.

                                                                                                                                                                        1. 9

                                                                                                                                                                          Build systems in C give a feeling of craftsmanship. It takes a skill to write a Makefile that correctly supports parallel builds, exact dependencies, interruptions and cleanups, etc. And so much work into making it work across platforms, and compilers.

                                                                                                                                                                          And then Cargo just makes it pointless. It’s like you were king’s best messenger trained to ride the fastest stallions, and Cargo’s like “thanks, but we’ve got e-mail”.

                                                                                                                                                                          1. 2

                                                                                                                                                                            LOL, I guess it’s a matter of age. When I first started programming, I’d love all that stuff. I can’t count how many libraries and tools I re-implemented or extended because they didn’t do something exactly the way I wanted it. Or the nights I spent configuring my Linux machine to work just right. Or the CPU time I spent re-encoding all of my MP3 collection to VBR because it’d save 5% of storage.

                                                                                                                                                                            Now, I learned Cargo for a tiny project and I keep swearing every time I have to start a Python virtualenv because it’s just not easy enough, goddammit!.

                                                                                                                                                                        2. 2

                                                                                                                                                                          This is a fair criticism of C, personally I would love to see a set commonly used data structures added to the C standard library. However, currently in the C world you either write your own or use something like glib, neither of these cases require the equivalent of Cargo.

                                                                                                                                                                          1. 4

                                                                                                                                                                            However, currently in the C world you either write your own or use something like glib, neither of these cases require the equivalent of Cargo.

                                                                                                                                                                            Neither does using the Rust standard library, which also has a hash table implementation (and many other useful data structures). You can just use std and compile your project with rustc.

                                                                                                                                                                            1. 1

                                                                                                                                                                              We’re talking about dependencies in general, not just hash tables. FRIGN’s point is that the Rust standard library is lacking, so you end up needing crates.

                                                                                                                                                                              1. 3

                                                                                                                                                                                But you and FRIGN are complaining about the Rust standard library compared to C. The Rust standard library is much more comprehensive than the C standard library or the C standard library + glib. So, the whole point seems to be void if C is the point of comparison.

                                                                                                                                                                                If you are comparing to the Java standard library, sure!

                                                                                                                                                                                1. 1

                                                                                                                                                                                  But you and FRIGN are complaining about the Rust standard library compared to C.

                                                                                                                                                                                  Not really. The point being made is that a typical Rust application has to download a bunch of stuff from Github (crates), where as a typical C application does not.

                                                                                                                                                                                  1. 8

                                                                                                                                                                                    That’s just because it’s convenient and most people don’t really care that it happens. But it’s not inherent to the tooling:

                                                                                                                                                                                    $ git clone -b ag/vendor-example https://github.com/BurntSushi/ripgrep
                                                                                                                                                                                    $ cd ripgrep
                                                                                                                                                                                    $ cargo build --release
                                                                                                                                                                                    

                                                                                                                                                                                    Other than the initial clone (obviously), nothing should be talking to GitHub or crates.io. You can even do cargo build --release --offline if you’re paranoid.

                                                                                                                                                                                    I set that up in about 3 minutes. All I did was run cargo vendor, setup a .cargo/config to tell it to use the vendor directory, committed everything to a branch and pushed it. Easy peasy. If this were something a lot of people really cared about, you’d see this kind of setup more frequently. But people don’t really care as far as I can tell.

                                                                                                                                                                                    where as a typical C application does not

                                                                                                                                                                                    When was that last time you built a GNU C application? Last time I tried to build GNU grep, its build tooling downloaded a whole bunch of extra goop.

                                                                                                                                                                                    1. -2

                                                                                                                                                                                      Nice strawman, I said a typical C application, not a typical GNU C application.

                                                                                                                                                                                      1. 5

                                                                                                                                                                                        TIL that a GNU C application is not a “typical” C application. Lol.

                                                                                                                                                                                        1. -1

                                                                                                                                                                                          None of the C code I’ve worked on was written by GNU, and most of the C code out in the real world wasn’t written by GNU either. I find it frankly bizarre that you are seriously trying to suggest that GNU’s practices are somehow representative of all projects written in C.

                                                                                                                                                                                          1. 3

                                                                                                                                                                                            You said “a typical C application.” Now you’re saying “representative” and “what I’ve worked on.”

                                                                                                                                                                                            If the implementation of coreutils for one of the most popular operating systems in history doesn’t constitute what’s “typical,” then I don’t know what does.

                                                                                                                                                                                            Talk about bizarre.

                                                                                                                                                                                            Moreover, you didn’t even bother to respond to the substance of my response, which was to point out that the tooling supports exactly what you want. People just don’t care. Instead, you’ve decided to double down on your own imprecise statement and have continued to shift the goal posts.

                                                                                                                                                                                            1. 0

                                                                                                                                                                                              and most of the C code out in the real world wasn’t written by GNU either.

                                                                                                                                                                                              ^ typical

                                                                                                                                                                                              I don’t have the time or patience to debate semantics though.

                                                                                                                                                                                              As for your other point, see FRIGN’s comment for my response. (It doesn’t matter what’s possible when the reality is random crates get pulled from github repos)

                                                                                                                                                                          2. 1

                                                                                                                                                                            C doesn’t even have a hash table.

                                                                                                                                                                            Why do you say “even”? There are many hash table implementations in C, with different compromises. It would be untoward if any of them made its way into the base language. There are other things missing in C which are arguably more fundamental (to me) before hash tables. It is only fair if all of these things are kept out of the language, lest the people whose favorite feature has not been included feel alienated by the changes.

                                                                                                                                                                          3. 15

                                                                                                                                                                            but there doesn’t seem to be much push to improve the situation

                                                                                                                                                                            Definitely not true. There are people working on this and there has been quite a bit of progress:

                                                                                                                                                                            $ git clone https://github.com/BurntSushi/ripgrep
                                                                                                                                                                            $ cd ripgrep
                                                                                                                                                                            $ git checkout 0.4.0
                                                                                                                                                                            $ time cargo +1.12.0 build --release
                                                                                                                                                                            
                                                                                                                                                                            real    1:04.05
                                                                                                                                                                            user    1:51.42
                                                                                                                                                                            sys     2.282
                                                                                                                                                                            maxmem  360 MB
                                                                                                                                                                            faults  736
                                                                                                                                                                            $ time cargo +1.43.1 build --release
                                                                                                                                                                            
                                                                                                                                                                            real    19.065
                                                                                                                                                                            user    2:34.51
                                                                                                                                                                            sys     3.101
                                                                                                                                                                            maxmem  740 MB
                                                                                                                                                                            faults  0
                                                                                                                                                                            

                                                                                                                                                                            That’s 30% of what it once was a few years ago. Pretty big improvement from my perspective. The compilation time improvements come from all around too. Whether it’s improving the efficiency of parallelism or micro-optimizing rustc itself: here, here, here, here, here, here or here.

                                                                                                                                                                            People care.

                                                                                                                                                                            The Rust developers should stop trying to please everybody and come up with standard interfaces that also get shipped with the standard install.

                                                                                                                                                                            That’s one of std’s primary objectives. It has tons of interfaces in it.

                                                                                                                                                                            This criticism is just so weird, given that your alternative is C. I mean, if you want the C experience of “simplicity and self-reliance,” then std alone is probably pretty close to sufficient. And if you want the full POSIX experience, bring in libc and code like its C. (Or maybe use a safe interface that somebody else has thoughtfully designed.)

                                                                                                                                                                            When installing a Rust package, you end up having to be connected to the internet

                                                                                                                                                                            You do not, at least, no more than you are with a normal Linux distro package manager. This was a hard requirement. Debian for example requires the ability to use Cargo without connecting to the Internet.

                                                                                                                                                                            and often end up downloading dozens of small crates from some shady GitHub repos.

                                                                                                                                                                            Yup, the way the crates.io model works means the burden of doing due diligence is placed on each person developing a Rust project. But if you’re fine with the spartan nature of C’s standard library, then you should be just fine using a pretty small set of well established crates that aren’t shady. Happy to see counter examples though!

                                                                                                                                                                            but I could imagine a similar scenario to leftpad in the future.

                                                                                                                                                                            The leftpad disaster was specifically caused by someone removing their package from the repository. You can’t do that with crates.io. You can “yank” crates, but they remain available. Yanking a crate just prevents new dependents from being published.

                                                                                                                                                                            Rust really needs a better standard library so you don’t have to pull in so much stuff from other people.

                                                                                                                                                                            … like C? o_0

                                                                                                                                                                            1. 10

                                                                                                                                                                              This point is really close to the one before it: Cargo is an interesting system, but really ends up becoming a “monosolution” for Rust setups. Call me old-fashioned, but I like package managers (especially Gentoo’s) and Cargo just works around it.

                                                                                                                                                                              C has just been in the luxurious position that its package managers have been the default system package managers. Most Linux package managers are effectively a C package managers. Of course, over time packages for other languages have been added, but they have mostly been second-class citizens.

                                                                                                                                                                              It is logical that Cargo works around those package managers. Most of them are a mismatch for Rust/Go/node.js packages, because they are centered around distributing C libraries, headers, and binaries.

                                                                                                                                                                              but I could imagine a similar scenario to leftpad in the future.

                                                                                                                                                                              Rust et al. certainly have a much higher risk, since anyone can upload anything to crates.io. However, I think it is also an illusion that distribution maintainers are actually vetting code. In many cases maintainers will just bump versions and update hashes. Of course, there is some gatekeeping in that distributions usually only provide packages from better-known projects.

                                                                                                                                                                              Rust really needs a better standard library so you don’t have to pull in so much stuff from other people.

                                                                                                                                                                              You mean a large standard library like… C?

                                                                                                                                                                              1. 3

                                                                                                                                                                                C has just been in the luxurious position that its package managers have been the default system package managers.

                                                                                                                                                                                This just isn’t true, if you look at how packages are built for Debian for example you will find that languages such as Python and Perl are just as well supported as C. No, the system package managers are for the most part language agnostic.

                                                                                                                                                                                1. 5

                                                                                                                                                                                  This just isn’t true, if you look at how packages are built for Debian for example you will find that languages such as Python and Perl are just as well supported as C.

                                                                                                                                                                                  Most distributions only have a small subset of popular packages and usually only a limited number of versions (if multiple at all).

                                                                                                                                                                                  The fact that most Python development happens in virtual environments with pip-installed packages, even on personal machines, shows that most package managers and package sets are severely lacking for Python development.

                                                                                                                                                                                  s/Python/most non-C languages/

                                                                                                                                                                                  No, the system package managers are for the most part language agnostic.

                                                                                                                                                                                  Well if you define language agnostic as can dump files in a global namespace, because that’s typically enough for C libraries, sure. However, that does not work for many other languages, for various reasons, such as: no guaranteed ABI stability (so, any change down the chain of dependencies needs to trigger builds of all dependents, but there is no automated way to detect this, because packages are built in isolation), no strong tradition of ABI stability (various downstream users need different versions of a package), etc.

                                                                                                                                                                                  1. 5

                                                                                                                                                                                    No, most development happens in virtualenv because python packaging is so broken that if you install a package you cannot reliably uninstall it.

                                                                                                                                                                                    If we didn’t have a package manager for each language then the packages maintained by the OS would be more comprehensive, by necessity. Basically having a different packaging system for each programming language was a mistake in my view. I have some hope that Nix will remedy the situation somewhat.

                                                                                                                                                                                    edit: it’s also difficult to reply to your comments if you substantially edit them by adding entirely new sections after posting…

                                                                                                                                                                                    1. 5

                                                                                                                                                                                      No, most development happens in virtualenv because python packaging is so broken that if you install a package you cannot reliably uninstall it.

                                                                                                                                                                                      I have no idea what you mean here. Can’t you use dpkg/APT or rpm/DNF to uninstall a Python package?

                                                                                                                                                                                      If we didn’t have a package manager for each language then the packages maintained by the OS would be more comprehensive, by necessity.

                                                                                                                                                                                      We are going in circles. Why do you think languages have package managers? Technical reasons: the distribution package managers are too limited to handle what languages need. Social/political reasons: having the distributions as gatekeepers slows down the evolution of language ecosystems.

                                                                                                                                                                                      I have some hope that Nix will remedy the situation somewhat.

                                                                                                                                                                                      Nix (and Guix) can handle this, because it is powerful enough to implement the necessary language-specific packaging logic. In fact, Nix’ buildRustCrate is more or less an implementation of Cargo in Nix + shell script. It does not use Cargo. Moreover, Nix can handle a lot of the concerns that I mentioned upthread: it can easily handle multiple different versions of a package and ABI-instability. E.g. if in Nix the derivation of say the Rust compiler is updated, all packages of which Rust is a transitive dependency are rebuilt.

                                                                                                                                                                                      As I said, traditional package managers are built for a C world. Not a Rust, Python, Go, or whatever world.

                                                                                                                                                                              2. 4

                                                                                                                                                                                The first problem is a technical one, unless Rust is doing things such that it can’t be compiled efficiently, but the latter two are cultural ones which point up differences in what language designers and implementers are expected to provide then versus now: In short, Rust tries to provide the total system, everything you need to build random Rust code you find online, whereas C doesn’t and never did. Rust is therefore in with JS, as you mention, but also Perl, Python, Ruby, and even Common Lisp now that Quicklisp and ASDF exist.

                                                                                                                                                                                I was going to “blame” Perl and CPAN for this notion that language implementations should come with package management, but apparently CPAN was made in imitation of CTAN, the Comprehensive TeX Archive Network, so I guess this goes back even further. However, the blame isn’t with the language implementers at all: Packaging stuff is one of those things which has been re-invented so many times it’s bound to be re-invented a few more, simply because nobody can decide on a single way of doing it. Therefore, since language implementers can’t rely on OS package repos to have a rich selection up-to-date library versions, and rightly balk at the idea of making n different OS-specific packages for each version of each library, it’s only natural each language would reinvent that wheel. It makes even more sense when you consider people using old LTS OS releases, which won’t get newer library versions at this point, and consider longstanding practice from the days before OSes tended to have package management at all.

                                                                                                                                                                                1. 7

                                                                                                                                                                                  Therefore, since language implementers can’t rely on OS package repos to have a rich selection up-to-date library versions, and rightly balk at the idea of making n different OS-specific packages for each version of each library, it’s only natural each language would reinvent that wheel.

                                                                                                                                                                                  This is right on the mark.

                                                                                                                                                                                  Sorry if this is a bit of a tangent, but I think it is not just a failing of package sets – from the distributor’s perspective it is impossible to package every Rust crate and rust crate version manually – but especially of package managers themselves. There is nothing that prevents a powerful package management system to generate package sets from Cargo.lock files. But most package managers were not built for generating package definitions programmatically and most package managers do not allow allow installing multiple package versions in parallel (e.g. ndarray 0.11.0 and ndarray 0.12.0).

                                                                                                                                                                                  Nix shows that this is definitely feasible, e.g. the combo of crate2nix and buildRustCrate can create Nix derivations for every dependency in a Cargo.lock file. It does not use cargo at all, compiles every crate into a separate Nix store path. As a result, Rust crates are not really different from any other package provided through nixpkgs.

                                                                                                                                                                                  I am not that familiar with Guix, but I bet it could do the same.

                                                                                                                                                                                2. 3

                                                                                                                                                                                  Build times are important, and you’re right, Rust takes a while to compile. Given the choice between waiting for rustc to finish and spending a lot longer debugging a C program after the fact, I choose the former. Or, better yet, use D and get the best of both worlds.

                                                                                                                                                                                  1. 3

                                                                                                                                                                                    rust can be damn fast to compile, most rust library authors just exercise fairly poor taste in my opinion and tend not to care how bad their build times get. sled.rs compiles in 6 seconds on my laptop, and most other embedded databases take a LOT longer (usually minutes), despite sled being Rust and them being C or C++.

                                                                                                                                                                                    rust is a complex tool, and as such, you need to exercise judgement (which, admittedly, is rare, but that’s no different from anything else). you can avoid unnecessary genericism, proc macros, and trivial dependencies to get compile times that are extremely zippy.

                                                                                                                                                                                    1. 2

                                                                                                                                                                                      Thanks for your insights! I’ll keep it in mind the next time I try out Rust.

                                                                                                                                                                                      1. 1

                                                                                                                                                                                        Feel free to reach out if you hit any friction, I’m happy to point folks in the direction they want to go with Rust :)

                                                                                                                                                                                  2. 5

                                                                                                                                                                                    almost all you can do in C, you can do in rust

                                                                                                                                                                                    As an anecdata, I‘ve immediately recognized the snippet with elf header from the article, because I used one of the same tricks (just memcpying a repr(C) struct) for writing elf files in Rust a couple of months ago.

                                                                                                                                                                                    1. 2

                                                                                                                                                                                      I though about using rust to implement a byte-code compiler / vm for a gc’d language project, but I assumed that this would require too much fighting to escape rust’s ownership restrictions. Do you have any insight into how well suited rust is for vm implementation? I haven’t used the language much but I’d love to pick it up if I though I could make it work for my needs.

                                                                                                                                                                                      (I see that there’s a python interpreter written in rust, but I’m having trouble locating its gc implementation)

                                                                                                                                                                                      1. 5

                                                                                                                                                                                        I honestly don’t know, I have never written an interpreter. You probably can fall back on unsafe for some things anyway, and still benefit from the move semantics, sum types, syntax, and friendly error messages. I’m doing a bit of exploring symbolic computations with rust and there are also some design space exploration to be done there.

                                                                                                                                                                                        1. 2

                                                                                                                                                                                          IMO, Rust is just as good as C and C++ for projects like this, if not better thanks to pattern matching and a focus on safety (which goes far beyond the borrow checker). Don’t be afraid to use raw pointers and NonNull pointers when they are appropriate.

                                                                                                                                                                                          1. 2

                                                                                                                                                                                            Also, just saw this GC for Rust on the front page: https://github.com/zesterer/broom/blob/master/README.md

                                                                                                                                                                                            Looks like it’s designed specificly for writing dynamic languages in Rust.

                                                                                                                                                                                            1. 1

                                                                                                                                                                                              Oh cool! This looks super useful

                                                                                                                                                                                          2. 2

                                                                                                                                                                                            I wrote a ST80 VM in rust just to play around; the result was beautifully simple and didn’t require any unsafe code, though it doesn’t currently have any optimization at all. The result was still reasonably snappy, but I suspect that a big part of that is that the code I was running on it was designed in, well, 1980.

                                                                                                                                                                                            1. 2

                                                                                                                                                                                              I recently did a simple lisp interpreter in rust. Eventually decided to re-do it in c because of shared mutation and garbage collection.

                                                                                                                                                                                              1. 2

                                                                                                                                                                                                Rust works well for VM implementation. For GC, you may want to look at https://github.com/zesterer/broom.

                                                                                                                                                                                            1. 2

                                                                                                                                                                                              I never understood the advantages of ninja with respect to make. It seems to boil down to things like that the makefiles do not use tab characters with semantic value, that the -j option is given by default, or that the syntax is simpler and slightly better. But apart from that, what are the essential improvements that would justify a change from make to ninja? If ninja is slightly better than GNU make, I tend to prefer GNU make that I know and that it is ubiquitous and it avoids a new build dependency.

                                                                                                                                                                                              1. 14

                                                                                                                                                                                                The article discusses how it’s really a low-level execution engine for build systems like CMake, Meson, and the Chrome build system (formerly gyp, now GN).

                                                                                                                                                                                                So it’s much simpler than Make, faster than Make, and overlapping with the “bottom half” of Make. This sentence is a good summary of the problems with Make:

                                                                                                                                                                                                Ninja’s closest relative is Make, which attempts to encompass all of this programmer-facing functionality (with globbing, variable expansions, substringing, functions, etc.) that resulted in a programming language that was too weak to express all the needed features (witness autotools) but still strong enough to let people write slow Makefiles. This is vaguely Greenspun’s tenth rule, which I strongly attempted to avoid in Ninja.

                                                                                                                                                                                                FWIW as he also mentions in the article, Ninja is for big build problems, not necessarily small ones. The Android platform build system used to be written in 250K lines of GNU Make, using the “GNU Make Standard Library” (a third-party library), which as far as I remember used a Lisp-like encoding of Peano numbers for arithmetic …

                                                                                                                                                                                                1. 6
                                                                                                                                                                                                  # ###########################################################################
                                                                                                                                                                                                  # ARITHMETIC LIBRARY
                                                                                                                                                                                                  # ###########################################################################
                                                                                                                                                                                                  
                                                                                                                                                                                                  # Integers a represented by lists with the equivalent number of x's.
                                                                                                                                                                                                  # For example the number 4 is x x x x. 
                                                                                                                                                                                                  
                                                                                                                                                                                                  # ----------------------------------------------------------------------------
                                                                                                                                                                                                  # Function:  int_decode
                                                                                                                                                                                                  # Arguments: 1: A number of x's representation
                                                                                                                                                                                                  # Returns:   Returns the integer for human consumption that is represented
                                                                                                                                                                                                  #            by the string of x's
                                                                                                                                                                                                  # ----------------------------------------------------------------------------
                                                                                                                                                                                                  int_decode = $(__gmsl_tr1)$(if $1,$(if $(call seq,$(word 1,$1),x),$(words $1),$1),0)
                                                                                                                                                                                                  
                                                                                                                                                                                                  # ----------------------------------------------------------------------------
                                                                                                                                                                                                  # Function:  int_encode
                                                                                                                                                                                                  # Arguments: 1: A number in human-readable integer form
                                                                                                                                                                                                  # Returns:   Returns the integer encoded as a string of x's
                                                                                                                                                                                                  # ----------------------------------------------------------------------------
                                                                                                                                                                                                  __int_encode = $(if $1,$(if $(call seq,$(words $(wordlist 1,$1,$2)),$1),$(wordlist 1,$1,$2),$(call __int_encode,$1,$(if $2,$2 $2,x))))
                                                                                                                                                                                                  __strip_leading_zero = $(if $1,$(if $(call seq,$(patsubst 0%,%,$1),$1),$1,$(call __strip_leading_zero,$(patsubst 0%,%,$1))),0)
                                                                                                                                                                                                  int_encode = $(__gmsl_tr1)$(call __int_encode,$(call __strip_leading_zero,$1))
                                                                                                                                                                                                  
                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                    Source, please? I can’t wait to see what other awful things it does.

                                                                                                                                                                                                    1. 1

                                                                                                                                                                                                      Yup exactly, although the representation looks flat, it uses recursion to turn 4 into x x x x! The __int_encode function is recursive.

                                                                                                                                                                                                      It’s what you would do in Lisp if you didn’t have integers. You would make integers out of cons cells, and traverse them recursively.

                                                                                                                                                                                                      So it’s more like literally Greenspun’s tenth rule, rather than “vaguely” !!!

                                                                                                                                                                                                    2. 1

                                                                                                                                                                                                      Yes, so I guess its main advantage is that it is really scalable. This is not a problem that I have ever experience, my largest project having two hundred files that compiled in a few seconds, and the time spent by make itself was negligible. On the other hand, for such a small project you may get to enjoy the ad-hoc GNU make features, like the implicit compilation .c -> .o, the usage of CFLAGS and LDFLAGS variables, and so on. You can often write a makefile in three or four lines that compiles your project; I guess with ninja you should be much more verbose and explicit.

                                                                                                                                                                                                      1. 5

                                                                                                                                                                                                        He mentions that the readme explicitly discourages people with small projects from using it.

                                                                                                                                                                                                        I suspect it’s more that ninja could help you avoid having to add the whole disaster that is autotools to a make-based build rather than replacing make itself.

                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                          I suspect it’s more that ninja could help you avoid having to add the whole disaster that is autotools to a make-based build rather than replacing make itself.

                                                                                                                                                                                                          Sure; autotools is a complete disaster and a really sad thing (and the same thing can be said about cmake). For small projects with few and non-configurable dependencies, it is actually feasible to write a makefile that will compile seamlessly the same code on linux and macos. And, if you don’t care that windows users can compile it themselves, you can even cross-compile a binary for windows as a target for a compiler in linux.

                                                                                                                                                                                                        2. 2

                                                                                                                                                                                                          You don’t (or better, shouldn’t!) write Ninja build descriptions by hand. The whole idea is something like CMake generates what Ninja actually parses. I’ve written maybe 3 ninja backends by now.

                                                                                                                                                                                                      2. 4

                                                                                                                                                                                                        we use ninja in pytype, where we need to create a dependency tree of a project, and then process the files leaves-upwards with each node depending on the output of processing its children as inputs. this was originally done within pytype by traversing the tree a node at a time; when we wanted to parallelise it we decided to instead generate a ninja file and have it invoke a process on each file, figuring out what could be done in parallel.

                                                                                                                                                                                                        we could doubtless have done the same thing in make with a bit of time and trouble, but ninja’s design decisions of separating the action graph out cleanly and of having the build files be easy to machine generate made the process painless.

                                                                                                                                                                                                        1. 4

                                                                                                                                                                                                          It’s faster. I am (often) on Windows, where the difference can feel substantial. The Meson site has some performance comparisons and mentions: “On desktop machines Ninja based build systems are 10-20% faster than Make based ones”.

                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                            I use it in all my recent projects because it can parse cl.exe /showincludes

                                                                                                                                                                                                            But generally like andyc already said, it’s just a really good implementation of make

                                                                                                                                                                                                          1. 3

                                                                                                                                                                                                            Fearless Concurrency actually delivers.

                                                                                                                                                                                                            Until you write a deadlock, which I managed to do within 30min of writing Rust code.

                                                                                                                                                                                                            1. 16

                                                                                                                                                                                                              Deadlocks are easy compared to data races. You just connect a debugger, and you see exactly what is deadlocked, and which stack traces lead to this situation. That is a walk in the park compared to some bytes somewhere sometimes briefly having wrong value, which might not even be reproducible with a debugger attached.

                                                                                                                                                                                                              I’ve had my share of deadlocks in Rust, and I’ve been able to solve them all. OTOH in my last C project I’ve had a data race, and the best I could do was to give up and remove all non-trivial OpenMP uses. Was it my bug? Was it compiler bug? I couldn’t know. I couldn’t even reproduce the data race myself.

                                                                                                                                                                                                              1. 3

                                                                                                                                                                                                                Three years ago I fought with a deadlock in a distributed system. Never learned of a good way to debug this. My problem magically disappeared due to some seemingly unrelated change.

                                                                                                                                                                                                                1. 1

                                                                                                                                                                                                                  Deadlocks are easy compared to data races. You just connect a debugger, and you see exactly what is deadlocked, and which stack traces lead to this situation. That is a walk in the park compared to some bytes somewhere sometimes briefly having wrong value, which might not even be reproducible with a debugger attached.

                                                                                                                                                                                                                  I’ve had my share of deadlocks in Rust, and I’ve been able to solve them all. OTOH in my last C project I’ve had a data race, and the best I could do was to give up and remove all non-trivial OpenMP uses. Was it my bug? Was it compiler bug? I couldn’t know. I couldn’t even reproduce the data race myself.

                                                                                                                                                                                                                  distributed system? Everything is more complicated with micro-services and other distributed systems. Rust can’t save you there.

                                                                                                                                                                                                                2. 3

                                                                                                                                                                                                                  While I’d agree with your overall point that data races are more insidious and typically harder to debug, deadlocks can certainly be nondeterministic and arbitrarily difficult to reproduce with a debugger attached too.

                                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                                    The nice thing about deadlocks is that they’re eminently fixable but still sufficiently annoying that after the third time you start asking “why am I getting deadlocks all the time?” and rekajigger your design so that it can’t happen any more. In my experience it’s another case of confused ownership, just expressed differently.

                                                                                                                                                                                                                    1. 2

                                                                                                                                                                                                                      Did you try clang thread sanitizer? Last time I used it was over 5 years ago and it very effectively told me exactly which threads and exactly what structures / bytes had a race.

                                                                                                                                                                                                                      I use address sanitizer every time I code C and C++ now and it has changed how I code… I would call it “fearless memory management”

                                                                                                                                                                                                                      1. 3

                                                                                                                                                                                                                        That’s the experience you get in Rust, but at compile time. The important improvement is that you don’t need to make the race (or use-after-free) happen for it to be caught. This helps ensure that also rare situations you haven’t thought about testing are correct too.

                                                                                                                                                                                                                        1. 3

                                                                                                                                                                                                                          I would call it “fearless memory management”

                                                                                                                                                                                                                          I wouldn’t. I’ve used valgrind since the early 2000s and asan since I knew it existed (probably around 2010/2011 but I’m not sure). I’ve lost count of how many memory management bugs went undetected until a segfault led me to investigate.

                                                                                                                                                                                                                          asan is huge for C and C++. But it’s only as good as your test suite and its coverage, and even then…

                                                                                                                                                                                                                        2. -1

                                                                                                                                                                                                                          Deadlocks are easy compared to data races

                                                                                                                                                                                                                          Even assuming I agree, I don’t see how that changes my point that if I can write deadlocks it’s not fearless.

                                                                                                                                                                                                                          You just connect a debugger, and you see exactly what is deadlocked, and which stack traces lead to this situation

                                                                                                                                                                                                                          That’s not been my experience. And that’s if the bug is easily reproducible, which tends to not be the case with concurrency.

                                                                                                                                                                                                                          OTOH in my last C project I’ve had a data race

                                                                                                                                                                                                                          Why compare to C? C is terrible at writing concurrent programs. If the bar for Rust is “better than C”…

                                                                                                                                                                                                                          1. 6

                                                                                                                                                                                                                            Compile-time verification that the whole program (and dependencies!) are free from data races is without a doubt a major improvement over C, C++ and Go. It is a way stronger guarantee than most other languages offer with “just be careful” and runtime sanitizers (and Rust also supports thread sanitizer). On top of that Rust has a rich ecosystem of libraries with robust concurrency primitives, so you can make programs multi-threaded without even touching any lock yourself.

                                                                                                                                                                                                                            But your entire argument hangs only on a literal interpretation of an advertising slogan. Of course a slogan doesn’t contain every possible caveat. And arguing about it is arguing about who fears what, which is unconstructive (there’s always someone afraid of everything, but this doesn’t change what Rust does).

                                                                                                                                                                                                                            The reasonable interpretation is that we all know data races are a major source of hard-to-fix bugs that are commonly feared, and Rust improves in this area so much that addition of concurrency to programs is no longer a major risk (but Rust does not guarantee bug-free programs, and never claimed that).

                                                                                                                                                                                                                            1. 0

                                                                                                                                                                                                                              major improvement over C, C++ and Go

                                                                                                                                                                                                                              I don’t think Go is even close to being in the same category as C or C++ in this regard.

                                                                                                                                                                                                                              rich ecosystem of libraries

                                                                                                                                                                                                                              I don’t think any existed at the time I wrote the deadlock (I looked). I ended up using mio directly. Things certainly seem to have change for the better since then.

                                                                                                                                                                                                                              we all know data races are a major source of hard-to-fix bugs that are commonly feared,

                                                                                                                                                                                                                              In certain languages, sure. I’ve written more concurrency bugs than I can count in C++, and I’m glad that if I write Rust those wouldn’t be possible. But there are other languages which make such bugs rare or non-existent. Rust is literally the only language I’ve every written a deadlock in and I did it on my first day.

                                                                                                                                                                                                                              1. 6

                                                                                                                                                                                                                                Again, you’re hung up on that one deadlock, as if it invalidated everything else. “Fearless” doesn’t mean “can be totally careless” or “guaranteed bug-free”.

                                                                                                                                                                                                                                I don’t think you will find any language that can use low-level mio/libuv that can do better here, so let’s not pit real shipping Rust against an imaginary ideal.

                                                                                                                                                                                                                                And you don’t have to use mio. You can use Rust with data-bound parallelism where you can’t make that bug. You can use Rust with an actor framework. Probably you could even hook it up to the Pony runtime.

                                                                                                                                                                                                                                BTW, Pony is deadlock-free on a technicality. It doesn’t solve the problem, but merely reshapes it to “two actors will send a message only after receiving a message from each other”.

                                                                                                                                                                                                                        3. 8

                                                                                                                                                                                                                          I always assume that Rust’s “fearless” is more about “your thread will not accidentally use data that it shared with someone else and it was freed in the meantime” than preventing you from writing deadlocks – just like its regular safety features will prevent you from breaking the memory-related stuff, not from actual logic bugs.

                                                                                                                                                                                                                          1. 1

                                                                                                                                                                                                                            I don’t think that’s a fair comparison. I can trust Rust to not let me violate memory safety. I can’t trust it to prevent me from writing deadlocks. In one case, it’s fearless memory management, in the other it’s fearless concurrency except for deadlocks.

                                                                                                                                                                                                                            Plenty of languages make it easy to not accidentally used data that is shared with someone else.

                                                                                                                                                                                                                            1. 4

                                                                                                                                                                                                                              Really? How many languages provide shared-memory concurrency with the promise that you won’t have data races? I can’t think of any other than rust. Erlang is another case but it doesn’t really share memory.

                                                                                                                                                                                                                              1. 1

                                                                                                                                                                                                                                How many languages Erlang

                                                                                                                                                                                                                                I think you partially answered your own question. From my personal experience, also Haskell and D. From reading the tutorial, Pony. The fact that Erlang doesn’t share memory is a good thing.

                                                                                                                                                                                                                                1. 5
                                                                                                                                                                                                                                  • Does D protect you from data races if you use mutex?
                                                                                                                                                                                                                                  • Does pony have locks? It seems to me it’s only about channels and sharing immutable references (which you can also do in rust, no need for locks if you share & Foo, just have to ensure it lives long enough)
                                                                                                                                                                                                                                  • Erlang doesn’t share memory at all, which is good, except when you try to write something very efficient I guess, like a shared cache. Besides you can get deadlocks with actors too, if two actors are waiting for messages from one another.
                                                                                                                                                                                                                                  • Haskell provides shared memory abstractions, but apart from the STM I don’t think it does anything special for the deadlock issue.

                                                                                                                                                                                                                                  In this list, I still don’t see anything that offers what rust does (race-free shared memory concurrency, statically guaranteed, where multiple threads can still modify the shared structures concurrently). Channels don’t protect from deadlocks at a higher level, and even then, rust has them too if they fit your needs.

                                                                                                                                                                                                                                  1. 0

                                                                                                                                                                                                                                    Does D protect you from data races if you use mutex?

                                                                                                                                                                                                                                    That’s not the usual way of writing concurrent code, which is to send messages between threads. You can only send immutable or shared data. The former guarantees a lack of data races, the latter can’t really be used without locks. That’s an oversimplified explanation though.

                                                                                                                                                                                                                                    Does pony have locks?

                                                                                                                                                                                                                                    No.

                                                                                                                                                                                                                                    but apart from the STM

                                                                                                                                                                                                                                    I think STM is their secret weapon when it comes to concurrency.

                                                                                                                                                                                                                                    1. 3

                                                                                                                                                                                                                                      That’s not the usual way of writing concurrent code, which is to send messages between threads.

                                                                                                                                                                                                                                      But in rust you can do that too, and the move semantics even allows you to send mutable things since you give up their ownership. And you can still get deadlocks. I fail to see what D does that rust doesn’t.

                                                                                                                                                                                                                          2. 5

                                                                                                                                                                                                                            Detecting deadlocks statically is equivalent to solving the halting (heh) problem, so it’s never gonna be 100%.

                                                                                                                                                                                                                            That being said, it is possible to detect deadlocks once they’ve occurred or are about to…how does Rust handle a deadlock situation? Does it do runtime deadlock detection and panic, or just lock up?

                                                                                                                                                                                                                            (Forgive me if I sound pedantic. Not my goal. Low on coffee and it’s a pain to buy more right now, what with the apocalypse and all…)

                                                                                                                                                                                                                            1. 4

                                                                                                                                                                                                                              Rust as a language doesn’t even know that locks exist. They’re purely a library feature. Data-race safety of the locked data is expressed via Send/Sync traits, but there isn’t more to it. The stdlib just wraps pthread mutexes, but you’re free to use whatever implementation you want. parking_lot from WebKit is a popular alternative.

                                                                                                                                                                                                                              1. 2

                                                                                                                                                                                                                                You can constraint the programmer such that deadlocks are impossible by construction. X10 started like this. Later they introduced old-fashioned locks via library so the guarantee can be broken (like unsafe in Rust).

                                                                                                                                                                                                                                X10 has async { } to spawn of an activity (think thread). There is “finish { }” which waits for all (recursively) spawned activities inside the block at the end of the block. You also get some barrier synchronization with a smart API design. If that is all you use, then you cannot have deadlocks.

                                                                                                                                                                                                                                1. 0

                                                                                                                                                                                                                                  Detecting deadlocks statically is equivalent to solving the halting (heh) problem, so it’s never gonna be 100%.

                                                                                                                                                                                                                                  The Pony programming language guarantees absence of deadlocks.

                                                                                                                                                                                                                                  That being said, it is possible to detect deadlocks once they’ve occurred or are about to

                                                                                                                                                                                                                                  Not deterministically.

                                                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                                                    The Pony programming language guarantees absence of deadlocks.

                                                                                                                                                                                                                                    Right, I was referring to classic free-for-all threads-with-locks models; there are ways to prevent deadlocks by construction or restriction.

                                                                                                                                                                                                                                    Not deterministically.

                                                                                                                                                                                                                                    I’m rapidly approaching the edge of my knowledge, so absolutely correct me if I’m wrong, but at least for traditional threads-with-locks, you can generate resource-acquisition graphs to determine if a deadlock has occurred, right?

                                                                                                                                                                                                                                    You can’t determine a priori if a deadlock will occur by static analysis of an arbitrary threads-and-locks, at least not in the general case, though, you’re right; that’s what I was alluding to above.

                                                                                                                                                                                                                                2. 3

                                                                                                                                                                                                                                  Most schemes to tame concurrency still seem to be able to deadlock and livelock. My favorite, though, is still SCOOP. Heck, I”ll throw in a submission on it since SCOOP appears portable across languages.

                                                                                                                                                                                                                                  1. 5

                                                                                                                                                                                                                                    Eiffel is the best of all the OOP languages. It’s got so much going for it, it’s sad that it isn’t more popular.

                                                                                                                                                                                                                                    1. 3

                                                                                                                                                                                                                                      Seemed like a great language for maintainable, business applications. Took a while for me to understand the non-technical reasons languages like it don’t get much adoption.

                                                                                                                                                                                                                                      1. 6

                                                                                                                                                                                                                                        For Eiffel the two big ones were

                                                                                                                                                                                                                                        1. The compiler was really expensive for a long time
                                                                                                                                                                                                                                        2. Bertrand Meyer did a great job alienating the entire OOP community. Like to the point of even refusing to use common, widespread terminology any of his papers.
                                                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                                                          Carl Sassenrath, is that you?!?

                                                                                                                                                                                                                                          stares wistfully at REBOL

                                                                                                                                                                                                                                      2. 1

                                                                                                                                                                                                                                        I tried to learn it, because I’m a huge fan of contracts, only to find out you couldn’t get it up and running without purchase, so I binned the plan.

                                                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                                                          There were a few open source compilers over the years but, as far as I know, none are still under much active development.

                                                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                                                            D has contracts.

                                                                                                                                                                                                                                          2. 1

                                                                                                                                                                                                                                            I know it was still cutting edge 25 years ago, but has it kept that up since? I remember reviewing it and being pretty disappointed. Even its contract system doesn’t feel that great now that we have Racket and Dafny.

                                                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                                                              You know me, I’m a technical Luddite. Anything newer than 25 years is inexcusable newfangled decadence.

                                                                                                                                                                                                                                              But, to answer your question, no, I don’t think it’s kept up. Last time I looked at the “official” Eiffel implementation, they were mainly focusing on interop and compiler speedups, though that was several years ago.

                                                                                                                                                                                                                                          3. 1

                                                                                                                                                                                                                                            Pony doesn’t allow deadlocks to happen.

                                                                                                                                                                                                                                            1. 2

                                                                                                                                                                                                                                              Well, it uses the actor model instead of multithreading with shared state that I recall. Great that it stops deadlocks, though.

                                                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                                                          I started with Vanilla Emacs 20 years ago and have gradually built up a config by adding modes and packages I needed + working around annoyances.

                                                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                                                            Agree completely. I wrote something similar in my blog.

                                                                                                                                                                                                                                            1. 5

                                                                                                                                                                                                                                              You have to explicitly use smart pointers in C++

                                                                                                                                                                                                                                              This is not a good thing.

                                                                                                                                                                                                                                              The memory management with std::unique_ptr has by design no overhead in performance or memory compared to a raw pointer

                                                                                                                                                                                                                                              This is not true: https://www.youtube.com/watch?v=rHIkrotSwcc

                                                                                                                                                                                                                                              Most of the article then talks about RAII, which isn’t exclusive to C++.

                                                                                                                                                                                                                                              1. 3

                                                                                                                                                                                                                                                Specifically, unique_ptr can’t be passed in registers, but has to be on stack, due to ABI backwards compatibility. Weirdly, C++ wasn’t supposed to have a stable ABI, but it also refuses to change the one it has.

                                                                                                                                                                                                                                                1. 1

                                                                                                                                                                                                                                                  C++ doesn’t have an ABI, let alone a stable one. C++ implementations have ABIs, and it’s those implementations that refuse to break backwards binary compatibility. The C++ standards committee doesn’t require any ABI stability, they just don’t add features that force vendors to break compatibility. There’s a big difference there.

                                                                                                                                                                                                                                                  I have never seen any hard evidence that unique_ptr being unable to be passed in registers actually causes performance degradation. That doesn’t mean it doesn’t, of course, but it would be nice to see some evidence given how often this is brought up as an issue. All he shows in that video, if I remember it correctly, is that the generated assembly is larger, but not that this actually necessarily causes performance issues. For all we know it might actually be more performant due to less register pressure. The stack is certainly in L1 cache anyway so it’s not much slower than a register. If you’re passing a unique_ptr to the function, the performance cost of actually dereferencing it is likely to be much larger than the cost of passing it on the stack instead of in registers.

                                                                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                                                                    C++ doesn’t have an ABI, let alone a stable one.

                                                                                                                                                                                                                                                    The thing is, the C++ committee (WG21) has decided in Prague this year that C++ won’t break ABI (explained in the article link in prev post). So C++ is in a funny situation where it doesn’t guarantee a stable ABI, but also all standard library fixes and performance improvements that would require breaking ABI stability have been explicitly ruled out.

                                                                                                                                                                                                                                                    1. 1

                                                                                                                                                                                                                                                      Which is exactly what I said: the committee will not force vendors to break ABI, but that doesn’t mean they’re forbidding them from doing so.

                                                                                                                                                                                                                                                      This is also nothing new.

                                                                                                                                                                                                                                                2. 2

                                                                                                                                                                                                                                                  This is not a good thing.

                                                                                                                                                                                                                                                  That’s pretty subjective. I think a lot of people would consider it a good thing. Having the ownership graph encoded explicitly in the types of the pointers gives some big benefits in terms of understanding that ownership graph over the GC-languages model of just having a big bag of objects all pointing at each other. I personally find it very easy to get lost in a C# or Java codebase, always asking the question: what code is actually responsible for this object? Who owns this object? Who is allowed to modify it? When should it no longer be touched? Having garbage collection doesn’t stop you from having to, for example, close a file at a well-defined time and then make sure not to touch the file object after that.

                                                                                                                                                                                                                                                  1. 5

                                                                                                                                                                                                                                                    I understood it differently: explicit is fine, which is why C++’s ambiguously-owned raw pointers are a bad default. In contrast Rust is very explicit with ownership, but the default basic syntax is given to single ownership and moves, and other cases need noisier Rc or raw pointers.

                                                                                                                                                                                                                                                    1. 3

                                                                                                                                                                                                                                                      A clear ownership graph is a good thing indeed, when one exists, but requiring that one exists is not good.

                                                                                                                                                                                                                                                      GC permits some styles of programming where the ownership graph isn’t complete. For example, I have a complicated data structure (a function expressed as SSA nodes) where the entire tree is owned, but individual nodes in the tree don’t really have owners, and none of the models I considered ensured that owners were created before the nodes they own.

                                                                                                                                                                                                                                                      Most of the program uses ownership. I ended up using a lot of time on ownership for that data structure that I could have spend on business logic.

                                                                                                                                                                                                                                                      1. 1

                                                                                                                                                                                                                                                        the entire tree is owned, but individual nodes in the tree don’t really have owners

                                                                                                                                                                                                                                                        In those cases the graph owns the nodes, presumably storing them in a vector, while the nodes should just have non-owning raw pointers to each other.

                                                                                                                                                                                                                                                        none of the models I considered ensured that owners were created before the nodes they own.

                                                                                                                                                                                                                                                        That’s pretty trivial, isn’t it? The owner should be the thing creating the nodes it owns. Something cannot create before it itself exists.

                                                                                                                                                                                                                                                        1. 4

                                                                                                                                                                                                                                                          I thought it was trivial too. I only realised my naïvete when Valgrind showed me that one of those raw pointers was dereferenced too late, and the root problem behind that late deferencing wasn’t easily fixed.

                                                                                                                                                                                                                                                          Mixing raw pointers and std::shared_ptr/unique_ptr is partial RAII. Partial RAII relies too much on developer smarts for me.

                                                                                                                                                                                                                                                      2. 1

                                                                                                                                                                                                                                                        what code is actually responsible for this object? Who owns this object?

                                                                                                                                                                                                                                                        The runtime.

                                                                                                                                                                                                                                                        close a file at a well-defined time and then make sure not to touch the file object after that.

                                                                                                                                                                                                                                                        I don’t think this is a very good example. Both C# and Java have ways of opening a file and ensuring it gets closed when you’re done with it, and I struggle to think why anyone would do anything else like return it from a function instead.

                                                                                                                                                                                                                                                    1. 7

                                                                                                                                                                                                                                                      Great write-up.

                                                                                                                                                                                                                                                      static linking makes binaries easy to deploy

                                                                                                                                                                                                                                                      I always wonder why Go gets the fame for this as if it’s the only language that can do this. You can statically link a C or C++ program. It’s not the default, but you can. There’s definitely something to be said for good defaults and how they encourage behaviour, but I’m still puzzled as to how Go managed to get this particular slice of the marketing pie.

                                                                                                                                                                                                                                                      when you make something simple, you move complexity elsewhere

                                                                                                                                                                                                                                                      This is a very good point, and one I’ve made before. Whatever logic your program needs to do is still complicated. Making the language simple just makes user code have to do more of the work. There’s a sweet spot, but that lies on different points in the spectrum for different people. Haskell is a step too far for me, and I like Haskell.

                                                                                                                                                                                                                                                      1. 13

                                                                                                                                                                                                                                                        You can statically link a C or C++ program. It’s not the default, but you can.

                                                                                                                                                                                                                                                        Defaults matter, especially in cases where you want to use others’ libraries, because there’s a decent chance that the non-default configuration doesn’t work. For example, glibc doesn’t support static linking at all, and more than a few other libraries only work with glibc.

                                                                                                                                                                                                                                                        1. 4

                                                                                                                                                                                                                                                          Defaults matter

                                                                                                                                                                                                                                                          I agree, which is why I mentioned how good defaults encourage good behaviour.

                                                                                                                                                                                                                                                          1. 6

                                                                                                                                                                                                                                                            I agree that defaults matter. That’s why I don’t use Go. The fact that print still exists, but prints to stderr is craptastic.

                                                                                                                                                                                                                                                            Imagine if you asked for a knife to chop things for a salad, and someone handed over a simple kitchen knife. As you start to slice the tomato, you realize something is terribly wrong. You feel an edge blunter than a butter knife. You turn around to your friend confused, and they smack their forehead and go “ohhhhh, for a tomato! Hold on.” And they wander off to another room and bring back a chef’s block with a nearly identical set of actually sharp knives. “You just have to go get these from the pantry when you want actual sharp knives. We don’t keep them in the kitchen, for obvious reasons.”

                                                                                                                                                                                                                                                            As a noob to the language, I was tinkering with some code. At one point, flipping between python and Go, I put a print() statement in Go. I then spent the next couple of minutes confused about why the next program in the pipe couldn’t see the output from my Go.

                                                                                                                                                                                                                                                            For those that have never used Go or are just starting out: you need to import “fmt”, and then use fmt.Print. Yes, it’s in all the tutorials. Yes, you will likely still beans it up and use print() by accident, especially if you come from other languages. No, there aren’t any warnings when you do this and then go run or go build. Go just does something different for the concept of “print” and “println” in silence. I don’t need that kind “tee hee, GOTCHA!” in my life.

                                                                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                                                                              Go just does something different for the concept of “print” and “println” in silence. I don’t need that kind [of] “tee hee, GOTCHA!” in my life.

                                                                                                                                                                                                                                                              Maybe Go could use some of Ruby’s POLA philosophy.

                                                                                                                                                                                                                                                          2. 11

                                                                                                                                                                                                                                                            I always wonder why Go gets the fame for this as if it’s the only language that can do this. You can statically link a C or C++ program. It’s not the default, but you can. There’s definitely something to be said for good defaults and how they encourage behaviour, but I’m still puzzled as to how Go managed to get this particular slice of the marketing pie.

                                                                                                                                                                                                                                                            My theory is that it’s because Ruby, Python, and Node lowered the bar so tremendously on this that anything else is seen as a huge leap forward.

                                                                                                                                                                                                                                                            1. 5

                                                                                                                                                                                                                                                              and the c/c++ toolchains are not exactly easy to handle. i won’t even try to build something static that wasn’t intended to. that’s why many things which are distributed as binary use linker magic and ship their own set of libs.

                                                                                                                                                                                                                                                            2. 2

                                                                                                                                                                                                                                                              You can statically link C and C++, yes.

                                                                                                                                                                                                                                                              But with Go you can also cross-compile that static executable with zero effort to multiple platforms.

                                                                                                                                                                                                                                                              1. 2

                                                                                                                                                                                                                                                                What does Cross compilation have anything to do with static linking or what I said?