1. 2

    In this specific case, I believe the model only fails if you pick the constants appropriately: spawn more goroutines than buffered slots. But that’s basically the bug in the first place, so it leaves me with the feeling that it helps you find the bug if you know what the bug is already. That said, I can see how writing the model might help focus the author on the choice of constants and help them discover the bug.

    I’d be interested to hear the authors thoughts on that. Does that feeling sound fair? Are there tools that would find choices for the constants that fail for you?

    1. 6

      In this case I modeled it to closely match the code, but normally you’d design the spec so that it “sweeps” through a space of possible initial values. It’d look something like

      variables
        Routines \in {1..x: x \in 1..MaxRoutines};
        NumTokens \in 1..MaxTokens;
        channels = [limitCh |-> 0, found |-> {}];
        buffered = [limitCh |-> NumTokens];
        initialized = [w \in Routines |-> FALSE];
      

      Then it would try 1 routine and 1 token, 1 routine and 2 tokens, etc.

      1. 1

        Here I believe the NumRoutines constant exists to ensure the model is bounded– TLC will check every number of potential goroutines from 1 to whatever NumRoutines ends up being. In addition, there are actually only two buffered channels to begin with (“limitCh” and “found”). As long as NumRoutines >= 3, then, TLA will find this issue.

        In my experience writing the model helps you better understand the problem and thus pick appropriate constants (as you mentioned). But even if it didn’t, it wouldn’t be unreasonable with this spec to plug in 10 for NumRoutines and see what happens.

        With TLA+ models I find that I wind up with an emotion similar to writing untested code: if it 100% works the first time I get immediately suspicious and start poking at things until they break as a means to convince myself it was actually working.

        1. 1

          In addition, there are actually only two buffered channels to begin with (“limitCh” and “found”). As long as NumRoutines >= 3, then, TLA will find this issue.

          I disagree. Found is unbuffered, and limitCh is buffered. I believe it will only find the bug if NumRoutines >= the buffer size of limitCh.

          1. 1

            Oh beans, in my pre-meeting rush I misread the spec. Which is definitely a separate problem!

            So it looks like limitCh has a buffer the size of NumTokens and you’d want to ensure you pick NumRoutines > NumTokens during one of the runs, right? I’m not sure there’s a tool that checks your work there.

          2. 1

            Here I believe the NumRoutines constant exists to ensure the model is bounded– TLC will check every number of potential goroutines from 1 to whatever NumRoutines ends up being. In addition, there are actually only two buffered channels to begin with (“limitCh” and “found”). As long as NumRoutines >= 3, then, TLA will find this issue.

            While that’s best practice, here I fixed the number of routines at NumRoutines for simplicity. It will find also find a bug with 2 routines and 1 token.

        1. 21

          This is hands down the best article I have ever read on NAT traversal. I’ve spent years of my life dealing with all of the fun issues that come up (for example, some UPnP implementations care about the Case-Sensitivity of the headers sent in the faux-HTTP request, and they don’t all agree!), and I still learned things reading it.

          1. 10

            Thank you! In return, I’ve just learned that some UPnP implementations care about header case. Thanks, I hate it!

            But seriously, that’s great intel and TIL :). If you have references for what empirical behaviors you discovered, I’m interested!

            1. 5

              I don’t have any references, but I can say that I’ve also seen:

              1. Routers may or may not like SOAPAction header value double quoted.
              2. Routers may reject have multiple mappings to the same internal port, even with distinct destination IPs. For example, if you have a mapping for external port 8080 to internal address 192.168.1.100:7070, it will reject adding a mapping for external port 8181 to internal address 192.168.1.200:7070 because the internal port “collides”.

              I think that’s all I can remember, and I don’t remember any data about how often these things occurred. Consumer routers are amazing creatures.

              1. 5

                Amazing creatures indeed. Thanks for the tips! So far, it seems that, thankfully, most routers these days that offer UPnP IGD also offer NAT-PMP or PCP. UPnP does grab a teeny bit more of the long tail, but you can sort of get away with ignoring UPnP in a lot of cases.

          1. 5

            Been waiting for this release, very nice! Hope binary sizes keep shrinking.

            1. 2

              Provided they don’t become absurd, why do you care about binary sizes?

              1. 3

                I’ve got some pretty slow internet - this morning I was downloading terraform 0.13 at ~70kb/s, haha. Granted, it’s a particularly large example.

                Go hello world is ~10 MB. I understand there’s a lot of runtime packed in there (or something else? Go binaries compress particularly well, so there’s not a lot of entropy somewhere), but for some perspective I’ll beat the dead horse and say statically including musl libc weighs ~1 MB at most. So, it would be pretty cool to have like, 5 MB go hello world. I’d guess the go linker could be more smart, I never actually bothered to look at what’s taking up space.

                In the end, it’s not really an issue for me personally. Definitely being idealistic here. I’d guess the engineering effort involved would just not be worth it, at least not to Google.

                1. 1

                  10MB seemed outrageous to me, so I checked. Using

                  package main
                  import "fmt"
                  func main() { fmt.Println("Hello, world!") }
                  

                  the binary output is 2MB for both 1.14 and 1.15. So I guess it’s pretty cool?

                  Also, since this release does shrink binary sizes, it shows that people are spending the engineering effort involved to shrink them.

                  1. 2

                    Ah sorry, I didn’t actually bother to verify my poor memory there. Thanks.

              2. 1

                Yes. I wish there was something like Go that could produce relly tiny ELF files.

                1. 3

                  Does tinygo help?

                  1. 2

                    Have you looked at upx? Shrinks my Go binary by 50%.

                    -rwxr-xr-x   1 mikeperham  staff   5910544 Aug 12 12:32 faktory
                    -rwxr-xr-x   1 mikeperham  staff  11506756 Aug 12 12:32 faktory-big
                    

                    https://upx.github.io

                    1. 3

                      It helps, but I’m thinking that an “hello world” ELF file making one system call to print the message and one to exit the program should take less than two kilobytes, not close to two megabytes (or more).

                      What is the Go compiler storing in a “hello world” executable to make it that large, and more importantly: why are unused bytes not automatically removed?

                      1. 3

                        Compared to C, there’s the addition of a garbage collector and a unicode database, at minimum.

                        1. 1

                          But there already is a Unicode database in your system?

                          1. 2

                            Can you provide a list of all of the locations for all of the unicode databases on every one of these systems: https://github.com/golang/go/blob/master/src/go/build/syslist.go#L10

                            Additionally, provide the methods of interfacing with them, what their differences are, and what their commonalities are? Please be sure to do that for every version of Go for the last 10 years.

                            Too much work? I agree.

                            1. 1

                              Not one designed for fast in-memory access using, say, a paged array data structure. You really don’t want to parse and convert the UnicodeData.txt files provided by the unicode consortium into a reasonable query structure every time you start a binary that does text processing.

                          2. 1

                            You want Rust? Go always includes a managed runtime for GC, et al.

                            1. 1

                              No, I get that Go and Rust aims for different things, but I don’t understand why the runtime is included if it’s not being used.

                              1. 2

                                Because for any non-trivial program, the runtime will be included. Why care about optimizing binary size of a program that is useless?

                                1. 1

                                  Embedded platforms often require small sizes. And I would love to use Go for writing a 4k demoscene demo, if it had been possible.

                                  Regardless of the above, why include unused bytes in the first place?

                                  1. 2

                                    Because there’s engineering effort involved both in initial implementation and long term maintenance. Approximately zero programs would benefit from that engineering effort.

                                    1. 1

                                      I agree that there is effort required to implement it in the first place, but fewer bytes emitted usually results in a lower maintenance cost. That’s my intuition, at least.

                                      All it takes is one dedicated developer that decides to send a pull request.

                                      1. 2

                                        I think you are misestimating multiple things.

                                        1. The set of useful programs that do not use the runtime. You cannot allocate, use goroutines, use channels, use maps, use interfaces, or large portions of the reflect library, just to get started.
                                        2. The effort involved to make the runtime removed from the small programs that don’t use it. The dead code elimination required is a non-trivial problem due to things like package initialization, generating tracebacks on crashes, etc.
                                        3. The ongoing effort involved in maintaining that code. It would have to remain under test, work on all of the environments that support it, etc. That the compiler outputs less bytes in some specialized programs that can basically only add numbers does not imply that there would be less things to maintain going forward.

                                        There is no calculus that makes this a useful optimization. The developers consistently have been putting in large efforts (like rewriting the linker, for example) to get even marginal (1-5%) reductions in binary size. Not every idea is a good one.

                            2. 1

                              should take less than two kilobytes

                              Compiled a simple “hello world” in both C and Go to static binaries and stripped them.

                              -rwxr-xr-x 1 rjp rjp 715912 Aug 14 08:42 hw-c
                              -rw-r--r-- 1 rjp rjp     97 Aug 14 08:41 hw.c
                              -rwxr-xr-x 1 rjp rjp 849976 Aug 14 08:42 hw-go
                              -rw-r--r-- 1 rjp rjp     52 Aug 14 08:41 hw.go
                              

                              This is GCC 10.1.0, Go 1.14.6 on Linux 5.7.5-arch1-1, x86_64.

                              If I upx them, I get

                              -rwxr-xr-x 1 rjp rjp 283608 Aug 14 08:42 hw-c
                              -rwxr-xr-x 1 rjp rjp 346748 Aug 14 08:42 hw-go
                              
                              1. 1

                                Yes, I dont get why those has to be that large either. Very few assembly instructions are needed.

                      1. 1

                        I believe

                        var _ I = (*A)(nil)
                        

                        would save you an alloc all together.

                        1. 2

                          While that is theoretically correct, it turns out that

                          var _ I = new(A)
                          

                          is compiled away, so there is no alloc to remove.

                        1. 1

                          But is it Turing complete?

                          I’ll see myself out.

                          1. 5

                            Not sure how much to trust an author who gets the name of the language wrong…

                            1. 1

                              Unkind flag was me. Honestly, come on. By a couple sentences in, I have a sense that the author’s first language is not english, and by the referenced link at the end (with a sketchy looking URL, but, eh), it’s clear that this was adapted from an article written in Chinese. Also, the author, pxlet, is a member here. Maybe at least point out the correct way the language should be named?

                              (I honestly don’t even know if you’re correcting the casing to “Golang” or the spelling to “Go”)

                              I’ve been messing around with Go, and hadn’t come across unsafe yet. From a very quick scan, it doesn’t look like spam or marketing copy to me… Is it incorrect? Is it bad? I honestly would like to know, because I’m curious to give it a read. I’m curious if Go’s unsafe is conceptually similar or different from Rust’s, which I’m more familiar with.

                              1. 5

                                The author’s contributions to lobste.rs have been primarily links to their own blog.

                                1. 2

                                  Worth pointing out, thank you. If I’d clicked through, I would have seen that they have made zero comments in their 11 months on this site, which does change the framing a little.

                                2. 2

                                  As far as incorrectness goes:

                                  • The section on accessing unexported fields is subtly wrong: it doesn’t account for padding between struct fields. In the example, there happens to be no padding, so it’s fine, but if you change the age field to a boolean, it would not be right (https://play.golang.org/p/enZ9FuEfK1t).

                                  • Both of the usages of reflect.SliceHeader and reflect.StringHeader are incorrect. Because the reflect package defines the pointer fields as type uintptr, they are not recognized as containing pointers to the garbage collector. Since the usage of the bytes/string is dead after creating the bh or sh variables, they could be collected at that moment. (https://play.golang.org/p/_lUYicIFUzW)

                                  There’s a strict set of rules and valid things to do with unsafe documented at https://golang.org/pkg/unsafe/#Pointer and it is very very easy to get them wrong.

                                  1. 1

                                    Thank you!

                                  2. 1

                                    Go is very popular in China, which I attribute to some mix of the strong unicode support the fact that Baidu’s heavy usage of it has spurred many Chinese universities to teach it to undergrads.

                                    Unlike Rust, there’s no ‘unsafe’ keyword in Go; there’s just a package with that name.

                                    1. 3

                                      I guess my point about language was that your comment boiled down to a cheap shot at how they wrote, and one that I think reflects a real, harmful cultural bias against folks whose first language isn’t English. I couldn’t tell if your comment was based on more than the title, and I really would value your opinion of the content as trustworthy or not.

                                      unsafe— thanks yeah. I have gotten that far :) As far the comparison, what I meant was more, given that they both let you be dangerous with pointers, what are the allowances and limits of those abilities, whether designed as part of the language or the standard library. But that’s just my own motivating curiosity about this post.

                                      (ps. I undid my downvote on your comment. I was grumpy this morning, and that action wasn’t called for since I was making a reply to your comment anyway. Sorry.)

                                      1. 1

                                        There’s also no safe in Go. The Go type system does nothing to prevent two goroutines from concurrently modifying the same object. That doesn’t have to be unsafe, but Go also has large value types, and so assigning from one goroutine and reading from the other can result in tearing. That isn’t intrinsically unsafe, except that this also applies to some built-in types, such as slice (unless this has been fixed recently). You can read the base of one slice and the bounds of another, spuriously pass the bounds check, and then access memory out of bounds. If you’re careful, you can avoid this kind of subtle bug in Go, but the language doesn’t help you at all.

                                        1. 1

                                          The language helps by providing channels. But you have to use them.

                                          1. 1

                                            Channels don’t help. If you pass a pointer over a channel, now you and the receiver have pointers to the object and you get into exactly this problem. Design patterns built over channels to guarantee unique ownership may work, but the language doesn’t help.

                                  1. 15

                                    Isn’t this a complaint about the lack of free support from distros? Am I misunderstanding?

                                    Perhaps a group of people would like to start a paid support service for Python 2?

                                    1. 9

                                      RHEL will be supporting a Python 2 interpreter until at least June of 2024. Potentially longer if they think there’s enough money in offering another “extended lifecycle” (which got RHEL 6 up to a whopping 14 years of total support from its initial release date).

                                      1. 2

                                        Alternately something can be “done” and never need to be touched again. Expiring toolchains breaks a lot of “done” code.

                                        1. 20

                                          In the current security landscape? Are you serious? No code is perfect. New flaws in old code are being found and exploited all the time.

                                          1. 1

                                            Obviously python is large enough to be a security problem, but take e.g. boltdb in the golang world. It doesn’t need more commits, unless golang shifts under it. I believe it’s possible to have code that’s useful and not a possible security problem.

                                            1. 15

                                              I don’t understand where you’re coming from here. I’m not a Golang fan, but looking at the boltdb repo on github I see that it’s explicitly unmaintained.

                                              You’re saying that you don’t think boltdb will ever have any serious security flaws that need addressing?

                                              I don’t mean to be combative here, but I have a hard time swallowing this notion. Complex software requires maintenance in a world where the ingenuity of threat actors is ever on the increase.

                                              1. 3

                                                Maybe it wouldn’t need any more commits in terms of features, which is most likely true in case of Bolt as the README states that it focusses on simplicity and doing one thing. But there’s no way to prove that something is secure, you can’t know if there’s a certain edge-case which will result in a security vulnerability. And in that sense, it does require maitainence. Because we can’t prove that something is secure, we can only prove that something is insecure.

                                                1. 2

                                                  In fact, the most recent Go 1.14 release adds a -d=checkptr compiler flag to look for invalid uses of unsafe that is enabled for -race and -msan builds by default, and because it does invalid unsafe pointer casts all over the place, it causes fatal errors if you like to run with -race in CI, for example.

                                                  So yeah, Go indeed did shift from under it very recently.

                                              2. 6

                                                Some things might be able to. I do not personally believe a sendmail milter is one of those things that can be “done” and never need to be touched again. Unless email itself becomes “done”, I suppose.

                                            1. 30

                                              The article has a technical inaccuracy with respect to Go’s path printing. It claims that Go prints the “wrong path”, but it prints exactly the bytes in the file name. This can be confirmed with strace -xx:

                                              write(1, "\x20\x20\x20\x20\x20\x20\xbd\xb2\x3d\xbc\x20\xe2\x8c\x98\x0a", 15)
                                              

                                              It’s unclear what they mean by “wrong” but I guess they don’t want to it to be so easy to write non-utf-8 encoded bytes to standard out?

                                              I also disagree with the characterization of Go saying “don’t worry about encodings! things are probably utf-8”. There’s exactly two spots in the entire language that tie strings with utf-8, and that’s when you range over a string or do conversions with []rune. In every other spot, they are byte sequences.

                                              1. 39

                                                This is not a fair comparison. Go 1.9.2 was released over 2 years ago. In that time they have fixed a lot of the GC stutter issues. Comparing rust nightly to a 2 year old compiler is unfair.

                                                1. 17

                                                  I am very confused why they do not mention why they didn’t upgrade. The Go compiler is intentionally (and will probably always be) backwards compatible. At Google we dogfood the new compiler throughout the fleet before releasing to the public. I have never seen a rollback. The GC has consistently gotten faster through versions.

                                                  I’m not saying what Discord did was wrong (I simply don’t know enough about the underpinnings of GC) but that they didn’t address such obvious low hanging fruit is strange.

                                                  1. 34

                                                    From /u/DiscordJesse on reddit:

                                                    We tried upgrading a few times. 1.8, 1.9, and 1.10. None of it helped. We made this change in May 2019. Just getting around to the blog post now since we’ve been busy.

                                                    https://www.reddit.com/r/programming/comments/eyuebc/why_discord_is_switching_from_go_to_rust/fgjsjxd/

                                                    1. 26

                                                      It’s worth noting that go1.12 was released before May 2019, and the release notes include lines like this:

                                                      Go 1.12 significantly improves the performance of sweeping when a large fraction of the heap remains live. This reduces allocation latency immediately following a garbage collection.

                                                      I believe that directly addresses what they were observing.

                                                      1. 14

                                                        Seems like the release didn’t come in time for their needs, considering they probably started the work on the Rust port after trying 1.10.

                                                        1. 5

                                                          Sure, but the Rust version wasn’t complete or deployed for multiple more months, implying they had that long to upgrade to go1.12, let alone go1.11 or the betas and release candidates. I can count on one finger the number of times upgrading a version of Go caused any problems for me, and the one time it did, it was because I had a bunch of my own cgo bugs to fix, so it’s hard for me to imagine a reason a compiler upgrade would be anything harder than a normal deploy of any change, which they claim they did many of (tuning, etc.) Deciding not to do that because you were already underway on a Rust port is just a sunk cost fallacy, and it’s not like it’s much work or unreasonable to expect people with production Go services to keep up with the two major releases they do per year.

                                                          That said, I’m operating on incomplete information. I’d like to give the benefit of the doubt so I expect they made a good decision here. They seem happy with the outcome and the end result is better for them on a number of different metrics, so that’s great.

                                                          It’s just unfortunate that we’ll probably never know if go1.12 would have solved their latency issues for not only them, but perhaps many others. A program reproducibly showing failures (and latency spikes like that are a failure for Go’s GC) is a valuable thing.

                                                          1. 8

                                                            It’s likely that they weren’t aware of the fixes, and if they were, the rewrite might have solved other pain points and simplified the code in other ways. The issues they faced with the GC would then just be a catalyst for the rewrite. Not the cause itself.

                                                        2. 5

                                                          So in 2016, Pusher switched from Haskell to Go because Haskell had poor GC latency due to a large working set.

                                                          The fundamental problem was that GHC’s pause times were proportional to the size of the working set (that is, the number of objects in memory). In our case, we have many objects in memory, which led to pause times of hundreds of milliseconds. This is a problem with any GC that blocks the program while it completes a collection.

                                                          Enter 2020, Discord switches from Go to Rust because of poor GC latency due to a large working set.

                                                          We kept digging and learned the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references.

                                                          So, I guess it’s only a matter of time before Pusher rewrite their service in Rust?

                                                          1. 2

                                                            And with the recent improvements to garbage collection in GHC, back to Haskell after that?

                                                            http://www.well-typed.com/blog/2019/10/nonmoving-gc-merge/

                                                  1. 3

                                                    Shouldn’t nonce contain unique value? I am no Go expert but it seems that it will always be null string.

                                                    1. 1

                                                      That is indeed being passed zero bytes. I think this is only dangerous if you reuse the same nonce for different messages with the same key. It’s a bit subtle, but all the usages of that function I can find are given a unique key every time (like the scrypt recipient has a unique salt, and the x25519 recipient uses emphemeral keys).

                                                      The document at https://age-encryption.org/v1 contains some language about making sure that property is true (e.g. A new salt MUST be generated for every new file key.)

                                                      1. 10

                                                        Indeed, this is intentional and safe, but worth a comment in the code. Coming up, thank you.

                                                        1. 4

                                                          Is there a reason not to randomize the nonce? This seems like a “we could use a static nonce, but a random nonce costs us nothing and it may save us if other things go wrong” situation.

                                                          1. 3

                                                            IMO better to not randomize the nonce and leave a comment explicitly stating why. While I appreciate “defensive engineering” I think code that that has no rational justification rots a codebase over the long term. E.g. anything that eventually results in comments like “XXX: not sure why this needed” makes it really hard to modify the code.

                                                            1. 2

                                                              Not really, it’s just superfluous overhead. It wouldn’t be unreasonable to randomise it, but I’m fairly confident in the randomness of the key here.

                                                      1. 14

                                                        I’m all for Rust, but let’s not shit on Go like that. Lack of deterministic destruction may make resource management a bit harder/more manual, but calling it “unacceptably crippled” is a hyperbole.

                                                        1. 6

                                                          Yeah, that last paragraph really ruined the article for me. Not only does it make me question the author’s character, it also casts doubt on the earlier technical parts; I’d recommend they delete it.

                                                          1. 6

                                                            I don’t think he’s talking about Go’s GC. I think he’s talking about how Spartan and locked down the language is (by design). I wouldn’t describe it as unacceptably crippled, in general, but for some use cases it really is.

                                                            1. 5

                                                              What use cases in particular? I can’t think of any that wouldn’t also lead you to say that the vast majority of languages are “unacceptably crippled”, which makes the phrasing less technically useful and more emotionally charged for no good reason.

                                                              1. 2

                                                                Fair point. I almost certainly would not have chosen those words, myself. Most languages would be disqualified for the use cases that Go is not a good choice for.

                                                              2. 1

                                                                Yeah there’s other ways in which Go is deliberately crippled besides the lack of deterministic destruction. Whether or not that crippling counts as unacceptable depends on what you want to do and what you’re used to, but I personally agree with the author on that point, and I hope they don’t take it out of the blog post in response to criticism.

                                                            1. 6

                                                              Why does anyone trust Keybase? They’ve been untrustworthy since they originally suggested uploading private GPG keys for convenience

                                                              1. 4

                                                                Exactly! Like hell I’m giving them my private key!

                                                                IIRC, I created a key pair just for Keybase. Signed it with my key pair. That worked fine since nobody on Keybase checked it that I remember. That just inspires more confidence, right? ;)

                                                                1. 3

                                                                  It’s the same as uploading an encrypted key to Dropbox, Google drive, etc. Yes, in theory you lose a tiny bit of security, but realistically your attacker needs to break AES to use your key, and such attacker capabilities usually aren’t included in most threat models.

                                                                  1. 1

                                                                    The keys weren’t encrypted with a passphrase for the web stuff to work seamlessly.

                                                                    1. 1

                                                                      IIRC web stuff connects to keybase service on your computer to work

                                                                      1. 5

                                                                        It originally didn’t at the launch of Keybase. You had the option of cli tools (secure, you control the key) or uploading to their web servers for convenience

                                                                      2. 1

                                                                        Odd. The web app does scrypt (even says that on the login button) on the password, I’d be surprised if the derived key wasn’t used to encrypt the keys used for messaging.

                                                                        1. 2

                                                                          Unless you have a time machine you won’t be able to see what they used to do with uploaded GPG keys

                                                                          1. 1

                                                                            Indeed, because the backend is closed source.

                                                                            1. 2

                                                                              And even if it was open, because you can’t know that’s what they were actually running. (This is why E2E encryption and an open client is important, and an open backend is a security red-herring.)

                                                                    2. 3

                                                                      This is one of those situations where if you’re a hardcore crypto-head, and have been managing your own PGP/GPG keys for years? You probably shouldn’t, but then it’s not FOR you.

                                                                      It’s for people who want a reasonably secure, convenient way to use crypto to send/receive email, store files, and chat.

                                                                      There’s no requirement that you upload your existing keys to them, you can always have them generate a fresh key and use it that way.

                                                                      1. 1

                                                                        Yes true but it is misleading to the non-technical users. Compromise of the Keybase servers meant compromise of their private keys, and as there was no forward secrecy in use…

                                                                        1. 3

                                                                          I disagree. I don’t think they ever claimed that users keys wouldn’t be compromised if they (Keybase) were.

                                                                          This is a perfect example of the perfect being the enemy of the good.

                                                                    1. 1

                                                                      They say that as operators of the server, they can’t get your ssh keys. That would be true if they didn’t also build and distribute the binaries for their open source client.

                                                                      I’m not saying they’d do something nefarious, yadda yadda, what if a bad actor takes over their build infrastructure, look at node.js, yadda yadda. I think this side of the topic is well understood.

                                                                      1. 1

                                                                        I guess there is no way around that, if you want to do what they set out to do. There’s always a risk, and there is no mitigation against certain types of risk if you can’t or don’t want to compromise on features (they can’t, the feature is their service in this case). All you can do is make the risk understood (which I think they do a good job of, I certainly don’t feel they are not honest about that).

                                                                        Ubuntu has their client packaged btw, as far as I know, I’m sure other distributions have too.

                                                                        I’d be interested in what you would you suggest as a viable alternative approach?

                                                                        1. 2

                                                                          Ubuntu builds the client from source, then? (One would hope they’d always do that..) Let’s see… Hm, I can’t find a keybase package in Ubuntu. https://packages.ubuntu.com/search?keywords=keybase&searchon=names&suite=all&section=all

                                                                          But yes, if/when distros build and publish the keybase client and put it under their respective security umbrellas, AND if some independent analysts declare that the client’s E2E works as advertised (server can’t read ssh keys nor messages etc), then that would solve some of my complaints. You see, I do not particularly worry about the upstream author of bash putting a back door in, because distros build it and put their reputations on the line. Same deal.

                                                                          frkl, in general, keybase just needs to release their source code for the server and whatever related source code. That would solve all of my complaints eventually, as the community evaluates the code, breaks it, improves it, etc. (Or maybe we’d decide the hype wasn’t deserved. How should I know, yet?) If it’s legit, then I’m confident that most users will use the official instance anyway. I’m sure I would.

                                                                          I just want them to pull that left hand out from behind their back. This is a security product. Show me your hands. Duh.

                                                                          I might even be willing to accept keybase if they (or somebody) releases a free software alternative server that the client can connect to. Even if it has less features? (Am I negotiating with capitalists?) I guess this is called a reference implementation.

                                                                          I just will NOT tell all my friends and family to use a closed source non-free communications platform again. (I brought big G a hundred innocent souls before I realized what was going on there… My heart is broken. Never again.)

                                                                          1. 2

                                                                            How does releasing the server source help audit the security of the system? You can’t be sure that the source they release is the source their running, and the whole point of E2E encryption is that you only need to audit the source of the endpoints, and in this case, that source is open. Additionally there has been a (paid for by keybase) 3rd party security audit: https://keybase.io/docs-assets/blog/NCC_Group_Keybase_KB2018_Public_Report_2019-02-27_v1.3.pdf

                                                                            I do agree that it would help with the “keybase shuts down” scenario, though.

                                                                            1. 1

                                                                              Releasing the server source would be an act of good will, it would be morally correct (to me), and it would be ethically correct (among certain reputable segments of the internet population).

                                                                              If we have faith in E2E, I guess we don’t have to trust the servers? Yes, that’s the whole point.

                                                                              Of course, what if that’s not the source code that’s actually running?

                                                                              Well, see https://signal.org/blog/private-contact-discovery/, specifically under the heading Trust But Verify. As you can see, the problem isn’t ‘solved’ but at least one group has taken a whack at it.

                                                                              The signal server is open source. btw.. But as you know I’m sure, keybase is far more ambitious. kbfs makes me daydream…

                                                                              1. 1

                                                                                Economic and moral arguments are fine. I don’t intend to argue with those.

                                                                                I’ve read that signal blog before, and it leaves me with some questions. For example, even if you’re sure you’re contacting the secure enclave, and that it’s running the software it’s claiming it is, the enclave has to have some sort of key to decrypt the messages you’re sending it, otherwise anyone could read them. How do you know someone else doesn’t have that key? Indeed, it seems like those first assumptions about correctly contacting the enclave are dubious: how do you know that they don’t just have the “secure enclave’s” key and are pretending to be one? Have you physically inspected their hardware and agree that the packets are being routed to the correct server? I’d love to learn why these things aren’t actually problems.

                                                                                But until then, it seems like it has just put up some smoke and mirrors to distract you from just trusting them. It probably helps against attackers compromising their servers, but it shouldn’t be possible for that to be a threat in the first place. It’s fundamentally worse even if the source is open, because it’s strictly better when the server source is irrelevant. I can run keybase with zero trust in their servers, but I’ll always have to have some amount of trust in signal.

                                                                                1. 1

                                                                                  Okay. I don’t know this part… What is the difference between the Keybase client and the Signal client that makes server trust unnecessary for one and necessary for the other?

                                                                                  1. 1

                                                                                    The contacts you send to signal being disguised requires you to trust their correct implementation of the secure enclave.

                                                                            2. 1

                                                                              Ubuntu builds the client from source, then? (One would hope they’d always do that..) Let’s see… Hm, I can’t find a keybase package in Ubuntu. https://packages.ubuntu.com/search?keywords=keybase&searchon=names&suite=all&section=all

                                                                              Forget I said that, I’m an idiot. I did an ‘apt search’ on one of my machines and forgot I installed their ppa…

                                                                              And yes, fair points…

                                                                        1. 6

                                                                          I am considering switching to other languages for my next project or migrating slowly. Will have to wait and see I suppose.

                                                                          The main strengths of go (from my point of view) are good libraries and no annoying sync/async code barrier.

                                                                          The main weaknesses are a runtime that makes using C libraries hard, and some feeling of kludgyness because users can’t define things like try or w.e. themselves. ‘Go 2’ doesn’t really change anything.

                                                                          1. 4

                                                                            I consider myself relatively neutral when it comes to Go as a language. What really keeps me from investing much time or attention in it is how its primary implementation isolates itself so completely as if to compel almost every library to be rewritten in Go. In the short term this means it will boost the growth of a vibrant ecosystem but I fear a longer term world where the only reasonable way to interoperate between new languages and systems which don’t fit into Go’s model is to open a socket.

                                                                            While I don’t think we need to be alarmist about bloated electron apps but in general, we’re talking about many orders of magnitude in cost increase for language interoperation. This is not the direction we should be going and I fear Go has set a bad precedent with its answer to this problem. Languages will evolve and systems will too but if we have to climb higher and higher walls every time we want to try something new, we’ll eventually be stuck in some local optimum.

                                                                            I’d like to see more investment in languages, systems, and runtimes sitting between them that can respond to new ideas in the future w/o involving entirely new revisions of a language with specific features responding to specific problems. Perhaps some version of Go 2 will get there but at the moment it seems almost stuck on optimizing for today’s problems rather than looking at where things are going. Somewhere in there is a better balance and I hope they find it.

                                                                            1. 4

                                                                              Yeah - I really want to use either GNU guile or Janet to write http handlers for Go, with the current system it is not really possible to do it well.

                                                                              There are multiple implementations of Lua in Go for the same reasons, poor interop if you aren’t written in Go and want two way calls.

                                                                              1. 3

                                                                                A crucial part of this is that go was explicitly, deliberately created as a language to write network servers in.

                                                                                In that context, of course the obvious way to interop with a go program is to open a socket.

                                                                                1. 2

                                                                                  Sure. Priorities make RPC look like their main goal but the cost of an RPC call is on an entirely different level than a function call and comes with a lot of complexity from accidental distributed system is now required to call some logic written in another language.

                                                                                  At a company where everything is already big and complex, this may seem like a small price but it’s becoming a cost today so we see people opting to write pure Go libraries and pass on shareable libraries or duplicating effort. In many cases this becomes a driver to kill diversity in technical choices that I talk about in my original comment above.

                                                                                  It’s an obvious problem but the Go team would rather drive people away from systems level interoperability for Go’s short term gains. They claim that it’s be too hard to support a real FFI option or that they are short on resources but other runtimes do a better job of this so it’s possible and secondarily, Go supposedly isn’t a Google project but a community, yet we see it clearly being managed from one side of this coin.

                                                                                  1. 1

                                                                                    In my experience, it’s the quality of the golang tools driving this.

                                                                                    For instance: I found it easier to port a (smallish) library to go and cross-compile the resulting go code, than to cross-compile the original c library.

                                                                                    I initially considered porting to rust, which is imo a delightful language, but even though cross-compilation is much easier in rust than in c (thanks to rustup), it doesn’t compare to go.

                                                                                    The process for c:

                                                                                    • For each target arch, research the available compiler implementations; packages are often unavailable or broken, so you’ll be trying to build at least a few from source, probably on an unsupported platform.

                                                                                    The process for rust:

                                                                                    • For each target arch, ask rustup to fetch the toolchain. It’ll tell you to install a bunch of stuff yourself first, but at least it tends to work after you do that.

                                                                                    The process for go:

                                                                                    • Set an environment variable before running the compiler.
                                                                                    1. 1

                                                                                      … so we see people opting to write pure Go libraries and pass on shareable libraries or duplicating effort. In many cases this becomes a driver to kill diversity in technical choices that I talk about in my original comment above.

                                                                                      It’s unclear to me why having another implementation of something instead of reusing a library reduces diversity rather than increasing it.

                                                                                      They claim that it’s be too hard to support a real FFI option or that they are short on resources but other runtimes do a better job of this so it’s possible

                                                                                      I’m personally a maintainer of one of the most used openssl bindings for Go, and I’ve found the FFI to be a very real option. That said, every runtime has it’s own constraints and difficulties. Are you aware of any ways to do the FFI better that would work in the context of the Go runtime? If not, can you explain why not? If the answer to both of those is no, then your statements are just unfounded implications and fear mongering.

                                                                                      Go supposedly isn’t a Google project but a community, yet we see it clearly being managed from one side of this coin.

                                                                                      And yet, I’m able to have changes included in the compiler and standard library, provide feedback on proposals, and have stewards of the language directly engage with my suggestions. My perspective is that they do a great job of listening to the community. Of course they don’t always agree with everything I say, and sometimes I feel like that’s the unattainable bar that people hold them to in order to say that it’s a community project.

                                                                                      1. 1

                                                                                        The specific issues with Go FFI interop is usually around dealing with structured data rather than buffers of bytes and integers. Data layout ABI options would be a big plus. Pinning shared data would also help tremendously in avoiding extra copying or marshaling that is required in many of these cases. On the other side, calling into Go could be made faster in a number of ways, particularly in being able to cache thread local contexts for threads Go doesn’t manage (these are currently setup and torn down for every call in this direction).

                                                                                        There are also plenty of cases where construction of movable types could be supported with proper callbacks provided but instead Go opts to disallow sharing any of these data types entirely.

                                                                                      2. 1

                                                                                        They claim that it’s be too hard to support a real FFI option or that they are short on resources but other runtimes do a better job of this so it’s possible and secondarily,

                                                                                        It’s been a while since I actively used and followed Go, isn’t the problem that they have to forgo the ‘run a gazillion Goroutines’ if they wanted to support real FFI? To support an extremely large number of goroutines, they need small, but growable stacks, which means that they have to do stack switching when calling C code. Plus it doesn’t have the same call conventions.

                                                                                        In many respects have designed themselves into a corner that it is hard to get out of, without upsetting users and/or breaking backwards compat. Of course, they may be happy with the corner that they are in.

                                                                                        That said, Go is not alone here, e.g. native function calls in Java are also expensive. It seems that someone has made an FFI benchmark ;):

                                                                                        https://github.com/dyu/ffi-overhead

                                                                                    2. 2

                                                                                      I generally symphatize with your main argument (personally, I also miss easier C interop, esp. given that it was advertised as one of the initial goals of the language) - but on the other hand, I don’t think you’re giving justice to the language in this regard.

                                                                                      Specifically, AFAIK the situation with Go is not really much different from other languages with a garbage collector - e.g. Java, C#, OCaml, etc, etc. Every one of them has some kind of a (more or less tricky to use) FFI interface to C; in case of Go it’s just called cgo. Based on your claim, I would currently assume you don’t plan to invest much time in any other GCed language either, is that right?

                                                                                      1. 2

                                                                                        I can’t speak to modern JVMs but OCaml and C# (.Net Cote and Mono) both have much better FFIs both in support for passing data around and in terms of performance costs. It’s hard to understate this but CGo is terribly slow compared to other managed language interop systems and is getting slower not faster over time.

                                                                                        I’ll let folks draw their own conclusions on whether this is intentional or just a limitation of resources but the outcome is a very serious problem for long term investments in a language.

                                                                                        1. 1

                                                                                          I think it’s important to quantify what “terribly slow” is. It’s on the order of ~100ns. That is more than sufficient for a large variety of applications.

                                                                                          It also appears from the implication that you believe it’s intentionally being slow. Do you have any engineering evidence that this is happening? In other words, are you aware of any ways to make the FFI go faster?

                                                                                          1. 1

                                                                                            Not in my experience. Other than trivial call with no args and return nothing, it is closer to 1 microsecond for Go calling out in many cases because of how argument handling has to be done and around 10 microseconds for non-Go code calling Go.

                                                                                            1. 1

                                                                                              It is indeed slower to call from C into Go for various reasons. Go to C calls can also be slower depending on how many arguments contain pointers because it has safety checks to ensure that you’re handling garbage collected memory correctly (these checks can be disabled). I don’t think I’ve ever seen any benchmarks place it at the microsecond level, though, and I’d be interested if you could provide one. There’s a lot of evidence on the issue tracker (here or here for example) that show that there is interest in making cgo faster, and that good benchmarks would be happily accepted.

                                                                                        2. 2

                                                                                          Every one of them has some kind of a (more or less tricky to use) FFI interface to C; in case of Go it’s just called cgo. Based on your claim, I would currently assume you don’t plan to invest much time in any other GCed language either, is that right?

                                                                                          LuaJIT C function calls are apparently as fast as from C (and under some circumstances faster):

                                                                                          https://nullprogram.com/blog/2018/05/27/

                                                                                    1. 6

                                                                                      Mostly the same old same old, but I do have to agree the inability to reflect on non-public fields can lead to some pretty stupid serialization code.

                                                                                      1. 3

                                                                                        you can reflect on unexported fields, but it’s generally a bad idea. The json package doesn’t do it as a design choice for the json package, but it’s not a limitation of the language in general.

                                                                                        Here’s an example of reflecting on unexported struct fields, which I do not recommend doing for anything other than debug logging: https://play.golang.org/p/swNXd266OVL

                                                                                        If the json marshaler serialized unexported fields, then any consumers of your library that serialized your data would see their serialized data change when you’ve made changes to unexported fields. That breaks the design philosophy of unexported vs exported identifiers: you pledge to those that import your code that exported identifiers will continue to exist and be supported, and you retain for yourself the right to alter any unexported identifiers as you see fit, without having to coordinate with people who import your code. The json marshaler serializing unexported fields would break the second principle.

                                                                                        In fact I had a struct with 74 private fields that I wanted to serialize with json and was forced to make all fields public and update all uses throughout the large app.

                                                                                        Look I’m gonna go out on a limb here and guess that a struct type with 74 fields is not a particularly great design to begin with, or that such code would be easy to work with in any language.

                                                                                        1. 1

                                                                                          The consumers of my library don’t serialize its data; they ask it to serialize its own data in a few well-specified ways. Is it actually an expected contract that I can take any arbitrary struct returned from any library, serialize/deserialize it myself, and expect it to work? That seems…a bit too much to expect.

                                                                                          Conflating code visibility with serialization visibility seems like a non sequitur to me. If I have a library function func (*thing) GenerateJSON() string that’s supposed to generate the JSON for a thing, and that function uses a struct whose only purpose is to drive the JSON library to generate the right JSON, there’s no Go code that needs source-level visibility into that struct. So it just seems goofy that I have to mark those fields “public”.

                                                                                          1. 1

                                                                                            Is it actually an expected contract that I can take any arbitrary struct returned from any library, serialize/deserialize it myself, and expect it to work? That seems…a bit too much to expect.

                                                                                            are you suggesting that json.Marshal, when called from package A, should be incapable of serializing a value of a type defined in package B?

                                                                                            Conflating code visibility with serialization visibility seems like a non sequitur to me.

                                                                                            If package A exports A.Thing, and package B imports package A and uses an A.Thing and wants to serialize that to json for whatever reason (let’s say package B defines a type that has a field of type A.Thing in an API response, for example), the author of package A should be able to rename the unexported fields of A.Thing without the author of package B having the output of their program changed. If you want to change the json name of a field you can do so with a struct tag, and if you want to mark an exported field as being hidden from the json, you can also do that with a struct tag. The behavior of json.Marshal is such that a struct defined with no json tags at all will serialize all of its exported fields and none of its unexported fields. The default has to be something, and that seems like a reasonable default. I think it would be more confusing if unexported fields were serialized by default.

                                                                                            If I have a library function func (*thing) GenerateJSON() string that’s supposed to generate the JSON for a thing

                                                                                            I’m not sure where you’re going with this. encoding/json defines the json.Marshaler interface for this purpose. If the default doesn’t suit the output that you want, you can readily change the output to be any arbitrary output you want by satisfying the json.Marshaler interface.

                                                                                            please stop saying public. They’re exported or unexported identifiers. The exported/unexported concept in Go is different than the public/private concept in Java/C++/etc, and the naming reflects that.

                                                                                            1. 1

                                                                                              First, to get a terminology problem out of the way, s/public/exported/g in what I said.

                                                                                              The problem is not with a “default”. It would be fine if the default was to ignore unexported fields. As far as I can tell, it’s impossible to serialize unexported fields using the json package. You just get “struct field xxx has json tag but is not exported”. [0]

                                                                                              So when marshaling/unmarshaling is not simply a matter of copying exported fields, to use json internally you end up making another unexported struct with exported fields and copying data into that, or some such workaround.

                                                                                              [0] https://play.golang.org/p/FHvX6x8_61k

                                                                                              1. 1

                                                                                                and if you’re serializing something to json, you’re doing so … why? To make the data available externally. The very act of serializing the data to json is an act of exporting; to make it available beyond the confines of the current binary, perhaps to other programs written by yourself, things written by other people, or to even future versions of the same application. That’s the fundamental theory of what exported fields are: things that are a part of public APIs.

                                                                                                You either have an exported or an unexported struct type. If it’s an unexported struct type, what difference does it make whether the fields are exported or not? If it’s an exported struct type, why would you want to serialize an unexported field? It’s not going to show up in docs for your package, your consumers will be totally confused as to where it comes from, and again, once it gets serialized, you’re no longer free to make unannounced changes to that field.

                                                                                                This really seems like grasping at straws. I have literally never found this to actually impede my work in seven years of using Go.

                                                                                                1. 2

                                                                                                  This is kind of getting into the weeds so it should probably move to some esoteric mailing list. :)

                                                                                                  My library writes JSON that only it reads. The JSON itself isn’t a public API; its contents are not documented. The fact my code wants to reveal fields to itself through JSON doesn’t mean it wants to reveal those fields to other Go code. I’m totally free to make changes to the JSON and the code, as long as they’re backward/forward compatible to my code.

                                                                                                  “Exported” does not have an identical meaning in code and in serialization. Consider that I can serialize a gzip.Writer into JSON, because it has some exported fields. I’m pretty sure that is purely accidental as far as the authors of the gzip package are concerned, and obviously if you unmarshal that JSON you don’t get a working gzip.Writer. I don’t think what they meant by exporting the Header field was “please serialize this”; they just meant it’s OK to use it directly in your code at runtime.

                                                                                                  Anyway, I didn’t say this was a deal-breaker, just that it’s goofy. An unexported struct type with exported fields — to me the only non-goofy reason for that would be when the struct is implementing an interface with those fields in it. Instead, I’m doing it because otherwise it would be arbitrarily restricted from being serialized in the most convenient way, that is, by using the json package.

                                                                                        2. 1

                                                                                          Just in case you want to upset anyone who looks at your code, you can use https://github.com/zeebo/sudo to remove the read-only restriction on a reflect value.

                                                                                        1. 1

                                                                                          I think that this looks quite nice. I am annoyed by chat options that are not truly multi-device. I do like that the client is open-source.

                                                                                          Why can’t old chats be synced to new devices, though? You could sync them via other devices, couldn’t you?

                                                                                          1. 1

                                                                                            Non-ephemeral chats are synced to new devices as described in the FAQ:

                                                                                            Non-ephemeral messages persist until the user explicitly deletes them and are E2E synced to new devices. This produces a Slack-like experience, only encrypted! So when you add someone to a team, or add a new device for yourself, the messages are unlocked.

                                                                                            1. 1

                                                                                              But why only non-ephemeral ones? The article mentioned forward security but it makes no real sense to me. You could use new forward secure keys for the new communication…

                                                                                              1. 2

                                                                                                I thought that don’t-sync-to-new-devices was a feature. I thought the ephemeral option was for those messages that you want to send but not keep. Messages that don’t have value in the future (or do have liability in the future).

                                                                                                1. 1

                                                                                                  ah, alright. that does make sense. For me, a new trusted device is a relatively arbitrary boundary to bounce ephemeral messages on, though.

                                                                                          1. 5

                                                                                            Cross comment from HN:

                                                                                            Show us the server: https://github.com/keybase/client/issues/6374

                                                                                            1. 7

                                                                                              this is not really relevant to the security claims in the article IMO. It is important separately of course.

                                                                                              1. 6

                                                                                                I would love it if drive-by passive aggression like that comment from HN stayed on HN.

                                                                                                1. 12

                                                                                                  That was painful to read. Such entitled attitudes.

                                                                                                  1. 2

                                                                                                    yep, and unfortunately some of them come from lobsters.

                                                                                                  2. 7

                                                                                                    This is important, IMHO. At this point keybase is yet another walled garden. Investing in them means that you are subjected 100% to their whims and future success (or lack thereof). It’s painful when your communication is shut down because a company decided to go do something else, or close up shop.

                                                                                                    1. 4

                                                                                                      It isn’t. If the claims are true, that things are encrypted on the devices, which they seem they are (and the source is open source) then it doesn’t matter what happens on the server from a security perspective.

                                                                                                      1. 1

                                                                                                        Nope, it definitely is. You may be able to recover your data, but you’ll then be searching quite urgently for a service to replace it since the proprietary server stuff is unavailable.

                                                                                                    2. 2

                                                                                                      How about you spend 8 hours a day and make a great library that Keybase will really want to use in their backend, and make your library use GPLv3. Then they will have to open source.

                                                                                                      That’s not true. If the backend is not distributed to anyone then it would not need to be open source.

                                                                                                      1. 1

                                                                                                        Is this also true of the AGPL?

                                                                                                      2. 0

                                                                                                        I don’t have a clear understanding of how keybase server works. Can someone provide details.

                                                                                                        1. 2

                                                                                                          https://keybase.io/docs/server_security has details about what the server is responsible for, and what clients trust and verify from them.

                                                                                                      1. 4

                                                                                                        I’ve always found myself overly interested in the busy beaver function: simple formulation, but it outpaces any computable function, no matter how many factorials and exponents you throw in!

                                                                                                        Scott Aaronson has a great write-up: https://www.scottaaronson.com/writings/bignumbers.html

                                                                                                        1. 2

                                                                                                          I totally agree. Having a M.S. in math, the things that would interest me the most about CS would be things like the halting problem, busy beaver function, etc. For example, another Scott Aaronson link (https://www.scottaaronson.com/busybeaver.pdf) gives explicit descriptions of small Turing machines that cannot be proven to halt using ZFC (the axioms underlying most of modern mathematics), or proving they halt is equivalent in difficulty to proving Goldbach’s conjecture or the Riemann hypothesis.

                                                                                                        1. 9

                                                                                                          Bit too late for me. Doing wonderfully with Keybase git for private repos.

                                                                                                          1. 5

                                                                                                            Indeed. I don’t understand the appeal for private repos that aren’t also private to the host. I won’t be using them even though they’re free.

                                                                                                            1. 1

                                                                                                              Are such repos as easy for people to start using as the gitlab way?

                                                                                                              1. 1

                                                                                                                It’s really easy to use, but does not compare to gitlab in terms of features. But since I just need secure, distributed place to access my repos, it’s a good match.

                                                                                                            1. 7

                                                                                                              Using polymorphic variants you can strengthen the return type. For example:

                                                                                                              match foo 2 with 
                                                                                                                | `Y y -> y 
                                                                                                                | `Z -> 0
                                                                                                              

                                                                                                              will continue to compile with both

                                                                                                              let foo y = if y == 0 then `Z else `Y (y + 1)
                                                                                                              

                                                                                                              and the strengthened

                                                                                                              let foo y = `Y (y + 1)