Threads for tubbo

  1. 16

    I think this needs to be solved at the OS level, not the language level.

    It’s a problem for every language – packages you download can write to any file or make network connections. It might be a little worse in NPM because of the culture, but the problem is pervasive.

    Personally I’m interested in the direction of lightweight containers that behave like executables, and that have composable dependencies. Developing in Docker-like containers can work but there are a bunch of downsides to be mitigated.

    1. 11

      It’s a problem for every language – packages you download can write to any file or make network connections.

      No, it’s only a problem for capability-unsafe languages. By analogy, if I said “packages you download can corrupt arbitrary memory,” a reasonable objection might be that it’s only a problem for memory-unsafe languages.

      1. 6

        So you mean the languages that approximately 100% of software is written in?

        I’m familiar with various efforts / ideas around making Python, Go, and JavaScript capability safe, going back ~20 years.

        Language-based capabilities have an inherent O(N^2) interoperability problem. The problem is that security is systems issue, and systems are written in more than one language.

        A capability in JS means nothing to Go – you need some kind of bridge, and N^2 bridges in general. The bridges themselves will also have usability and security issues.

        It seems like you’re confusing research and practice … if not I would like to read about the deployed systems you’re thinking about.

        1. 7

          The analogy to memory-unsafety is rather deep. There was a time when all software was memory-unsafe, all languages were memory-unsafe, and garbage collection was considered too expensive to ever deploy in production.

          Your objection to transporting capabilities between languages can also be analogized, although it comes in two cases, depending on whether the communicating processes are colocated. When the processes are next to each other, then they can share memory through various shared-memory schemes; analogously, file descriptors are capabilities which can be communicated from process to process. For processes which are remote to each other, they cannot truly share memory or capabilities, but they can capture snapshots of memory and they can export capabilities with cryptography.

          The flavor of E I worked on, Monte, supports loading untrusted packages safely. The code is freely available; I built a generic computation server and a Warner-style spellserver. But, it needs an explanation in order to be usable. I can tell you exactly which language features make this possible.

          First, in any flavor of E, loading code does not execute it. Instead, code objects are created which represent the loaded code. In Monte, compilation to bytecode preserves the structure of code objects, so that loading a package is like creating a literal AST value. Evaluation is always relative to a specified environment. Even the “safe scope” of standard objects is not available to evaluators by default.

          Second, Monte implements E-style auditors, and in particular, it is possible for an object to prove itself transitively immutable to anybody who asks. Modules are annotated by the compiler so that the module and its exports are proven transitively immutable at runtime.

          Note that I used AMP with JSON payloads to transport capabilities, but Capn Proto RPC would be a more serious choice for production deployments, with explicit support for representing and sending messages to capabilities.

          1. 2

            I’m not sure the n^2 is really a problem if we’re talking about most practical software. In almost all cases it looks like (for example) JS with a native library for “very specific things” like the event loop or some file/service interface. At that point you can use a single bridge, or just still gain lots from capabilities by treating anything past FFI as a black box. (Or even isolate it separately through seccomp/selinux context)

            We’ve got lots to gain even if we ignore the bridges.

          2. 5

            Yeah, this can’t ONLY be solved at the OS level. I should be able to build a web application where an npm package gives me a better Button component but isn’t allowed to mine bitcoin and access the network. But some other npm package in my application might be an ORM that talks to my database and needs network access.

            Like Corbin says, your language needs to be able to define and enforce capabilities. The OS can only really help with enforcement.

            1. 1

              I’d argue that there’s little value in a button component as buttons as is are fine… and this is a part of the JS eco problem—gotta have a package/component to solve every thing. I came into a project using some React video player, and the thing injected a CDN script at runtime without my permission.

              1. 3

                Sorry you didn’t like my completely arbitrary example.

                Thinking everything is so simple that you never need packages or abstractions is a really tired meme. A Button component could typed in a way that prevents bad (eg poor accessibility) usage while still being a <button> under the covers. Is that so bad?

                1. 1

                  I just gave you a concrete example of something that literally could have been a <video> tag and instead injects arbitrary scripts. I spend 90% of my time around the NPM ecosystem and no, it’s not meme, and does lead to problems like auditing and these side effects as mentioned about giving too many permissions to untrusted packages to npm install is-array.

                  I still don’t like this example either. Abstracting over an element often obsures just as much as it helps. What is inaccessible about the built-in button? I see people putting onclick on random elements without role=button and no hover/focus states all the time because they don’t understand that <button> is already accessible.

                  1. 4

                    Icon-only buttons need a label or a screen reader can’t announce them properly and you can’t have a tooltip when mousing over them. That’s why we have tools like Lighthouse that can audit your site and tell you when you’ve screwed that up. <button>s are not always accessible just by existing.

                    That said, unless your thesis is that all packages are inherently bad, this is an extremely off-topic argument. If my hypothetical package is not to your liking, imagine another.

            2. 4

              Given the behaviours of almost all standard libraries and standard package managers that are popular these days, it’s pretty clear that the incentives of OSS don’t line up with this being done at the language level, so yes I’d say there’s a very strong case to make that it should be done at the OS level.

              1. 3

                At which level would you say that Nix operates? While it is leaky, Nix and similar package managers attempt to treat package references as capabilities, and also attempt to prevent untrusted packages from exploiting builders by limiting their ambient authority.

                1. 1

                  I’m not really qualified to answer, I know of Nix and generally feel it seems more sensible than most package managers, but I haven’t used it.

            3. 4

              I think this needs to be solved at the OS level, not the language level.

              This is possible only if your language uses OS abstractions for isolating modules. This is not common and in something like NodeJS would require a separate V8 instance per module, which would be prohibitive (imagine if the function to trim whitespace needed 300 MiB of RAM for a JS JIT, all by itself).

              For this to work well, you need cooperation between a capability-secure set of OS abstraction (which, currently, means that your choice of operating systems is narrowed down to FreeBSD or Fuchsia) and capability-secure language abstractions. Within a type-safe language, you can rely on type safety to preserve the capabilities. Across languages you need to use OS abstractions. Ideally, you want a Multics-style library model, where a single process has isolated memory regions for each library.

              Hmm, it’s almost as if that’s what I’ve spent the last decade building…

              1. 2

                What’s the best intro to that work?

                I read papers on Capsicum ~10 years ago but otherwise haven’t kept up with it. When I look at the page now it seems like it’s partially deployed ? I’d be interested in some kind of retrospective.

                https://www.cl.cam.ac.uk/research/security/capsicum/

                Or maybe the problems with it are what led to the desire for hardware and language support?

                Either way I think this supports my point … Adding capabilities just to node.js is of limited use – it’s a systemic issue and all layers of the stack are affected.


                One reason I’m interested in shell is because it does use OS abstractions for isolating modules :) From my POV an entire node.js program is a “module”. There were a lot of motivations for this, but one is that at Google, the security reviews for services would frequently recommend or even require that certain libraries be run in separate processes. For example I recall some huge PDF and or internationalization libraries handling user input. They were simply too risky to link into processes running at a high privilege level.

                1. 2

                  This blog, which was posted on lobste.rs is probably the best overview of the things that I’m working on. I regard Capsicum as basically a solved problem at this point - it’s a retrofit of ideas from ‘80s capability systems to POSIX and it does that well. If you don’t need POSIX then you can design APIs better (and Zircon / Fuchsia does). Extending that into something that can work at a library level and isolate C/C++/assembly code that is used in a typesafe language is the more difficult part of the problem.

                  1. 2

                    I remember reading about Capsicum when it was brand new and being pretty excited about it. Unfortunately, AFAIK it still hasn’t really made any headway beyond FreeBSD and experimental changes to some programs. I read somewhere (but might be wrong) that the Chromium team didn’t manage to keep Capsicum support maintained, probably because it doesn’t exist on Linux.

                2. 2

                  It might be a little worse in NPM because of the culture, but the problem is pervasive.

                  I think it’s perceived as “worse” in NPM because it’s so widely used, and tends to lean on the side of smaller packages with more focused goals, therefore you need more of them in order to complete an implementation. This is in contrast to Python packages like NumPy or SciPy, which try to do “everything related to math” or “everything related to science” and are relatively large libraries in comparison. RubyGems has this problem to a lesser extent because it’s a middle-ground between huge Python packages and tiny JS packages. But in reality, all languages are vulnerable to this kind of attack, so I agree with you that this might be better implemented on the OS level.

                  But good luck getting all the OSes to agree on a standard!

                  Personally I’m interested in the direction of lightweight containers that behave like executables, and that have composable dependencies.

                  Can you elaborate on how this might help? I’m a big fan of containerized applications but mostly because it prevents polluting global namespaces and causing problems that are difficult to diagnose. You still need some kind of system for defining what capabilities a program has, and it can’t just be a “yes/no” thing like macOS has. Ideally, we’d take some inspiration from OAuth2 and implement some kind of scope-based access that allows the program to tell the user exactly the kinds of permissions it needs. This would stop a lot of the more annoying supply-chain attacks, and would require a much more targeted approach, which I think would alleviate the problem for most people since they are not targeted.

                  1. 2

                    Right, the culture of fine-grained libraries is a difference in severity in JS, but not a difference in principle.

                    I think for dev tools, Linux is the main system that needs to be changed / upgraded. For example git started out as Linux-only and spread to OS X and Windows. And same with Docker – when you run Docker on OS X or Windows, you’re also running Linux.

                    node.js is used mainly for servers and CLIs, which mostly run on Linux, especially in production. So I think a solution there will go a long way.


                    The way I’m thinking of it is that we need coarse-grained principle of least privilege in two main contexts:

                    1. At build time. For example, the TypeScript compiler is a huge JS blob, which I believe 99% of people run unsandboxed.

                    When I mean “containers that behave like executables”, I mean that you should be able to run:

                    tsc-sandboxed myinput myoutput
                    

                    And it will only be given the capability to read myinput and write myoutput. It should not be able to write ~/.bashrc! So this is a form of coarse-grained security that I believe a shell can help with.

                    Note that https://oilshell.org is also growing a declarative part that is supposed to be better “YAML”. It will evaluate to JSON, so you could use it for the JSON spec in Docker/OCI containers, package.json, etc.

                    1. At runtime. Say you are building a JavaScript server like https://stackeditpro.com or something. You might invoke it as:

                      stackedit-sandboxed –port 8080 –data-root ~/stackedit

                    It should be able to listen to that port, but not any others. And it should be able to read and write data under its root dir, but no others. That could be implemented with a file system namespace.

                    Again, it shouldn’t be able to write to ~/.bashrc, /etc/passwd, or exfiltrate data via UDP or DNS and so forth.

                    Again I believe maybe 90% of people run such servers unsandboxed in their dev environments. Docker is good for production but there’s a bit too much friction for development. Some people do it though.


                    So that is the general idea I’m interested in. Capabilities should be explicit and specified with flags or configuration. The benefit of having it in a shell-like language is that it composes. Systems are composed out of many parts and you use language to compose them.

                    I also think this is not very far from standard practice. It doesn’t require drastic rewriting or rethinking of how we develop systems. You don’t need to rewrite your software in a different language or use a different operating system.

                    There are a lot of other details, always happy to chat more, e.g. on https://oilshell.zulipchat.com/

                    1. 2

                      So you basically want Deno’s permissions model applied to the entire OS?

                      https://deno.land/manual/getting_started/permissions

                      Not a bad idea. It takes a little getting used to when you first start out with Deno, but the idea that you can whitelist files to read, subprocesses to execute, and network endpoints to call and ensure that the program cannot do anything past that. It’s funny because Ryan Dahl referred to the way Deno handled security as “Web-style sandboxing” in the initial talk he gave introducing the project, since he was very much influenced by the security model of browsers when building it.

                      Other than out-of-the-box TypeScript compliation (which is an absolute bear on NodeJS given all the different configuration possibilities…), I think the security model is the most compelling feature of Deno, and I wish it was available in any language I chose to use.

                      1. 2

                        It would be default-deny without ambient authority, so yes it looks similar. The web started out as mostly default-deny, although browser vendors have been poking some questionable holes in it … Still better than app stores though.

                        FWIW I’m keeping track of related threads here: https://oilshell.zulipchat.com/#narrow/stream/266575-blog-ideas/topic/Dumpling.20container.20format

                        I don’t know much about Deno, but subresource integrity also came up, and that is related to the “composable dependencies” I mentioned. They should be pinned by hashes / content. Except they’re not just JavaScript files; they’re git-like subtrees of runnable code.

                        I think OS X has been moving to a stricter default-deny model as well. There are a lot of UI problems there but what I’m envisioning is more for server-side distributed computing.

                1. 5

                  phone calls over the internet taste good and go down smooth. just take a SIP.

                  1. 6

                    I wouldn’t consider SIP a small protocol, especially if you include the extensions.

                    1. 5

                      As someone who works in SIP daily and pays my bills, I agree wholeheartedly. It is far from being small.

                      1. 4

                        As someone who will never work with SIP voluntarily again, I completely agree.

                    2. 3

                      This works as a dad joke too. Cheers

                    1. 3

                      I was thinking more about k8s for desktop usage (as a qubes-os like mechanics) when I saw the title, but then:

                      >rancher desktop
                      >desktop
                      *looks at github sidebar*
                      >typescript 35%, js 18%, vue 18%
                      

                      Don’t do that, don’t give me hope.

                      1. 1

                        Yeah, I think this is more interesting for developers that need to package for Kubernetes, like myself sometimes. And yes, for the reason you point out it’s probably not something you’ll want to have running all the time.

                        It also remains to be seen if this actually adds that much convenience over just using K3s on your machine, and opening up Lens (also an Electron monstrosity) only when needed.

                        In fact, I kind of wonder why people like to go for Electron for these kinds of tools, over running a web service inside the local Kubernetes solution that you can open in a better browser.

                        1. 3

                          Sensing some confusion on what this is (but by all means let me know if I am wrong!):

                          This is for replacing Docker Desktop (eg run docker/k8s locally on your Mac/Windows box, usually for development purposes) and not really for running production workloads. It has a UI because Docker Desktop has a UI, but really what this does is implement a specialized Linux VM (using Qemu/Lima) for running containers locally.

                          But at $DAYJOB this was not considered ‘production’ ready and so they are ponying up the License fees for Docker this year.

                          Me? I’ve been meaning to experiment with Nix on Mac as a way of doing isolated envs, since even though we run k8s locally, there are substantial differences between dev and prod anyways and docker/k8s for dev seem to not work all that well in practice , requiring all sorts of effort and drama around working around the limitations they present (hence: devspace, Telepresence, skaffold, etc).

                          1. 2

                            But at $DAYJOB this was not considered ‘production’ ready and so they are ponying up the License fees for Docker this year.

                            I got an email about that as well, and I just laughed wondering how they planned to enforce this…I guess they could just troll large companies for the biggest payouts, but it seemed like a real “nickel-and-dime” thing to do.

                            1. 2

                              They can learn from oracle and their license audits

                      1. 35

                        Why did GitHub remove his account/projects?

                        1. 44

                          That’s the part that bothers me.

                          I understand it wasn’t a nice thing to do, and that people are upset, but it’s his own code in his own repos. He even announced ahead of time he would do something like this, so “buyer” beware.

                          I realize GitHub TOS covers them to remove accounts and repos at their discretion, but it’s a little unsettling that they’ll actually do so arbitrarily without a clear TOS violation. It might be time I move everything to Sourcehut and treat GitHub as a mirror…

                          1. 24

                            It might be time I move everything to Sourcehut…

                            The Sourcehut guy has always seemed a little unstable to me (didn’t he get banned from this site, in fact?) So, why would I trust him any more than I trust GitHub?

                            1. 33

                              I banned him and I would not call him unstable. Not just because that kind of insult is inappropriate here, but because it obviously doesn’t apply. He writes inflammatory hyperbole that’s not a good fit for this site, but he’s a skilled, accomplished professional who looks like he’s seeing a lot of success in his life.

                              1. 11

                                I didn’t mean to insult him. Maybe “erratic” would have been a better word without any mental health connotations (which I absolutely didn’t intend)? Poor word choice on my part, I’m sorry for that.

                                …but he’s a skilled, accomplished professional who looks like he’s seeing a lot of success in his life.

                                Sure, same goes for the GitHub guys. A person who can’t tone it down enough to keep a Lobsters account just isn’t someone I feel I can trust to host my code, particularly given that he’s in charge of the whole operation. Obviously everyone is free to decide who to trust and for what reasons.

                                1. 9

                                  A person who can’t tone it down enough to keep a Lobsters account just isn’t someone I feel I can trust to host my code

                                  Bear in mind, Linus Torvalds would also probably have been banned from here multiple times in the past.

                                  I’d be perfectly happy to trust someone that volatile a lot (and I guess I do, running Linux since 1996 :) ). But I would be careful which groups and forums I invited them to :)

                                  1. 2

                                    …I guess I do, running Linux since 1996

                                    Very different, at least to me. If Linux was a service, control would have been taken away from Linus a long time ago (I mean, as it is they made him step back for awhile to work on his issues). But it’s not, it’s just code that other people then build into something, often applying patches in the process. If Linus had a meltdown there is already sufficient infrastructure in place that the vast majority of us wouldn’t even notice.

                                    I wouldn’t trust a code hosting service Linus ran by himself either.

                                    1. 1

                                      Nobody made Linus step back. He recognized that he had issues and took a sabbatical to deal with them himself. Are you saying you wouldn’t trust a service by a guy who has been diligently working on the same project for 30 years? Not to mention the guy who invented the base of all of the services discussed in this thread.

                                      Why do people assume that “Bigger is better” when it comes to web services? The two most reliable services I use are Pinboard, run by an insanely opinionated and outspoken developer, and NewsBlur, who was, and may still be, a one man shop that just quietly does his own thing. In the same time as those services have been faithfully up and running, Google has shut down more services than I can count, because “It didn’t fit with their corporate vision”

                                      It’s misguided, and harmful.

                                      1. 1

                                        Nobody made Linus step back.

                                        We’ll probably never know for sure, but the subtext (well, and the text) of his announcement email sure makes it sound like his hand was forced, at least to me.

                                        Are you saying you wouldn’t trust a service by a guy who has been diligently working on the same project for 30 years?

                                        No, I’m saying I wouldn’t trust a service run by a guy who randomly goes off on people in totally inappropriate ways (his admission). Or, as is the case here, a guy who can’t even behave himself well enough to keep a Lobsters account.

                                        Not to mention the guy who invented the base of all of the services discussed in this thread.

                                        That has literally nothing to do with anything. A person can be productive or brilliant and also have other, undesirable, qualities.

                                        Why do people assume that “Bigger is better” when it comes to web services?

                                        I don’t, so I can’t answer that.

                                        Google has shut down more services than I can count…

                                        Agree with you there! I don’t trust Google for anything but search (I don’t even use Gmail), because that’s the one thing I don’t think they’ll ever kill (or break). I don’t think GitHub is going anywhere either, the worst case scenario is that Microsoft sells it.

                                        It’s misguided, and harmful.

                                        If there was a person who had the views you seem to ascribe to me, then I might agree!

                              2. 30

                                That’s unfair to Drew. He’s passionate, and rude, and opinionated, and submissions here from his site generally stirred up giant flamewars. But I do believe he’s got what it takes to keep sourcehut running.

                                1. 18

                                  GitHub will keep running, too. I’m not sure we’ve answered the question of

                                  why would I trust him any more than I trust GitHub?

                                  1. 8

                                    Not only is the sourcehut software available under the AGPL, the issue trackers and such give you export and import functions to pull your data into another instance easily. The software itself is not trivial to host, but it’s not prohibitively hard either. If I needed to eject because Drew became untrustworthy, I am very comfortable that I could do that.

                                    Even though that’s a non-zero amount of work, GitHub gives me no comparable ability. That’s a good reason to trust him more than I trust GitHub, in my opinion.

                                    1. 3

                                      GitHub gives me no comparable ability.

                                      The GitHub command line client provides this functionality, as does the API. Obviously, the data formats are specific to the way GH works, but there are ways to extract most if not all of the relevant data (I use this heavily with my team to script up our findings workflow, for example).

                                      1. 5

                                        Interesting. Unless I’m missing something, you can’t stand up your own self-hosted instance of github, and import that, can you? The ability to stand up my own instance of the forge and import my data, to use on a self-hosted site, is what I meant by “comparable”. (That’s the angle I was coming from… if Drew won’t let me use his hosted service, I can just set up my own copy on any host I want since upstream is AGPL, then import my data from the sr.ht export since sr.ht exposes those functions.)

                                        1. 2

                                          GitLab supports importing to a self-hosted instance from GitHub [1], although I’m sure it’s not perfect, so it may or may not be useful. It also isn’t clear to me based on a 15 second review whether you can import from some kind of local data dump or raw GitHub API responses or if your GitHub account needs to be currently active.

                                          [1] https://docs.gitlab.com/ee/user/project/import/github.html

                                          1. 2

                                            That looks much better than I thought, particularly if it turns out to work off saved data/responses. And it’s nice that Gitlab enable that for all their tiers.

                                          2. 1

                                            Unless I’m missing something, you can’t stand up your own self-hosted instance of github, and import that, can you?

                                            GitHub Enterprise can be bought as a GitHub-hosted or self-hosted thing. These support (most of) the same APIs as the public GitHub, so you can run your own instance if you are willing to pay.

                                            1. 2

                                              It would be an interesting experiment to see whether they would sell an enterprise installation to someone whose account they forcibly closed. I was sort of assuming that if they won’t let you be a customer of their public service, they won’t sell you the private one, but that is uninformed speculation.

                                      2. 3

                                        Because sourcehut is open source so nothing is lost when I leave. More than that chances are if sourcehut goes a bad route there would likely be others jumping in.

                                      3. 2

                                        Not that you exactly claim otherwise, but Drew also makes some nice things and has created a business structure that enables at least one other developer to make some nice things.

                                        Quite apart from that, though, and similarly quite apart from whether he has what it takes to keep sourcehut running, he’s given me an out so that I don’t, strictly speaking, need him to. He’s released the software that runs the forge under the AGPL, here. And it exposes ways for me to export the hosted stuff and import it into a self-hosted instance.

                                        So regardless of whether I trust Drew personally, he’s made it so I don’t need to for this purpose.

                                        If Drew got angry and decided I couldn’t be his customer anymore, I could stand up my own instance or pay someone to do that for me and import my data. My repos wouldn’t be down at all, my tickets, docs, etc. would be down for a day or so, and my mailing lists might see a bit more disruption than that. If github decided that I shouldn’t be their customer anymore, I’d have my repos. For the rest, I’d kind of be out of luck. (I think this last paragraph is more responsive to @glesica ‘s comment than the one I’m replying to, and I’m too lazy to split it to another reply.)

                                      4. 17

                                        Because “more than I trust Microsoft” is a damn low bar.

                                        1. 7

                                          It’s like a little devil hovering over my right shoulder, and a slightly less predictable devil hovering over the other.

                                      5. 6

                                        From other options there’s also fediverse approach with Gitea, and p2p approach will be available soon with Radicle.

                                        1. 11

                                          It might be time I move everything to Sourcehut and treat GitHub as a mirror…

                                          That time was years ago, but hey, better late than never.

                                          1. 5

                                            Consider hosting your own, instead. I published a blog post with a list of defunct code hosting sites which I update occasionally. Maybe that list is a good reminder. Remember, it’s not just code that goes away with such sites, it’s also issue queues and in some cases, wikis and mailing lists too.

                                            1. 4

                                              Are you also start hosting a list of defunct private websites that used to host Git repos that are gone forever and where the disappearence came completely unexpected? I would trust Github more with staying online since that’s their job than a developer running a Gitweb on some VPS with some domain name that requires regular payment to stay online.

                                              Kinda like I registered callcc.org after it lapsed to make sure the links to the CHICKEN website don’t break and it doesn’t get domain-squatted and I’m redirecting to the official website these days.

                                              1. 1

                                                Are you also start hosting a list of defunct private websites that used to host Git repos that are gone forever and where the disappearence came completely unexpected?

                                                I can’t think of anything offhand where I’ve taken a dependency that’s done that. But when I do take a dependency on something, I generally mirror the SCM repo if there is one. And I am very reluctant to take dependencies on things I can’t have the source to. Since the things I depend on generally haven’t gone away, I haven’t bothered to publish my mirrors, but I would if the license permits it.

                                                1. 3

                                                  But when I do take a dependency on something, I generally mirror the SCM repo if there is one.

                                                  I learned that the hard way when Rubyforge went down, a few employers ago. We weren’t that active in the Ruby community anymore, so we missed the notice. When the site went away and I had to do some small maintenance tasks on a legacy project, all the third party svn subtrees from Rubyforge were no longer working (and, more painfully, another project of ours was completely gone too). Note that Rubyforge was huge in the Ruby community back in the day.

                                                2. 1

                                                  I would trust Github more with staying online since that’s their job than a developer running a Gitweb on some VPS with some domain name that requires regular payment to stay online.

                                                  Like I said, history has shown these hosting sites are not as trustworthy as people like to believe they are. The GitHub company can get sold to an untrustworthy partner (har har, like that’d ever happen… oh wait) or go out of business (what if MS decides to sell the company to, I dunno, Oracle or something because it’s not making a profit?), or there might be some other new VCS that comes out that completely blows git out of the water. I’m sure nobody saw coming what happened to Bitbucket - it started out as a Mercurial hosting site, then started offering git and finally dropped Mercurial after Atlassian took over. Its founders probably never would have let that happen if it were still in their power.

                                                  From my own perspective, I’ve personally ran into at least five hosting sites who were hosting projects I started or heavily contributed to that are no longer available now (Rubyforge, Dutch govt OSOSS’ uitwisselplatform, Berlios, Bitbucket and Google Code). And then there’s Sourceforge which at least still hosts some of my defunct projects, but had for a while been injecting malware into downloads. If I or my employers (as the case may be) had hosted our own projects from the start, this pain would’ve been completely avoided. These are projects in which I had a stake, and it was in my interest to not let them die.

                                                  Now, having said that, I agree that from a third party perspective (someone who is using the hosted code) that’s different. I understand your point saying you don’t want to rely on some random developer’s VPS being up, and neither would I. But then, people change repositories on code hosting sites all the time, too. They move to other hosting sites, or rename repositories etc. Links rot and die, which is unfortunate and something we all have to live with.

                                                  Case in point:

                                                  Kinda like I registered callcc.org after it lapsed to make sure the links to the CHICKEN website don’t break and it doesn’t get domain-squatted and I’m redirecting to the official website these days.

                                                  Thanks for doing that. AFAIK this domain was never communicated as being official, but I might be wrong.

                                            2. 8

                                              I don’t know what the GitHub rationale was, but the ‘limitation of liability’ bit in most open source licenses only goes so far. If I intentionally introduce malicious behaviour into one of my open source projects, knowing that it would damage downstream consumers, then I’d probably be liable under the Computer Misuse Act in the UK and similar legislation elsewhere. GitHub’s T&C’s don’t explicitly prohibit using their service for criminal purposes but that’s normally implicit: if GitHub didn’t act then they might end up being liable as an accessory (at least as an accessory after the fact). Their distribution channel (NPM) is being used by a malicious actor to attack other users.

                                              It’s normally difficult to prove malicious intent in this kind of thing (incompetence and malice look similar) but it seems pretty clear here from the author’s own statements.

                                              1. 12

                                                I don’t know what the GitHub rationale was, but the ‘limitation of liability’ bit in most open source licenses only goes so far.

                                                This is disturbing. Software is provided as is, with no liability whatsoever, but the author should still be liable for what happens when other people use it, because it broke things? What if the author decided to completely change the library’s API, or recycle it to just print squares of color, because they liked the name?

                                                If find what the author did pretty stupid, but frankly, suggesting it falls into criminal behavior call for some stepping back and put things in perspective.

                                                1. 8

                                                  There is a difference, and it’s not subtle at all, between making a possibly unwanted change in software that is provided without any warranty, and deliberately making a crippling change with the express intent of breaking other people’s applications.

                                                  To put it another way: if you accidentally commit an integer overflow bug that causes batteries to blow up, that is, presumably, just bad engineering. But if you deliberately commit a clever hack that causes people’s batteries to blow up, with the express intent of getting people injured, or at least destroying their phones, I think it makes a little sense to not put it under “well, it did say no warranty of any kind on the box, didn’t it?”.

                                                  Obviously, this didn’t kill anyone, so I’m obviously not thinking it ought to be treated as murder. But “no warranty” is not a license to do anything.

                                                  It’s not like software is being given special treatment here, it’s how warranties work everywhere. If you sell boats with two years’ servicing warranty and they break down after three years, that’s one thing, but if you fit them with bombs that go off after two years and one day, with the express intent of killing anyone on them, that still falls under “murder”, not “what happens after two years isn’t our problem, it says so on the purchase contract”.

                                                  (Edit: obviously IANAL and this is not legal advice, either, I’m only parroting second-hand, non-lawyer advice about how insurance works for some high-stakes software projects)

                                                  1. 5

                                                    I guess that makes sense, when you put it that way :)

                                                  2. 3

                                                    I am not a lawyer, this is not legal advice:

                                                    My understanding is that it comes down to intent. If I upload a buggy piece of crap to GitHub with an open source license, and you use it, then it sucks to be you. If I upload something to GitHub, wait for you to deploy it and then intentionally introduce a vulnerability or other malicious behaviour in it then legally dubious. Normally it’s very difficult to prove intent. If I submit a patch to the Linux kernel that introduces a vulnerability, if you wanted to prosecute me then you’d have to prove that I did so knowing that the bug was there and with the intent to cause harm. That’s very difficult to do in the general case (the NSA null-pointer dereference bugs are a great case in point here: people suspect that the NSA knew about that vulnerability class and introduced it deliberately, but no one can prove it and there’s enough reasonable doubt that it would never stick in court unless there was some corroborating evidence - it could easily have been accidental). If, before I submit the patch, I post publicly about how I am going to intentionally break things for the people using my code and then I push a new version out to public repositories then it’s likely to be easy to prove malicious intent. The author of these packages did exactly that: posted saying that he was going to break things for people if they didn’t pay him and then, when they didn’t pay him, broke things. That may (again, not a lawyer) count as blackmail, as well as computer misuse.

                                                    1. 3
                                                      1. Code license != Github TOS.
                                                      2. Liability could only be disclaimed to the extent permitted by law. You cannot put a sign “free food, no liability whatsoever” and then put poison inside and expect that disclaimer to save you from prison. E.g., GPL states “THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.”
                                                  3. 7

                                                    I think until they make a statement about it, nobody knows but them. But my assumption is that this happened on a weekend, and whoever was on call figured that the easiest thing to do to minimize disruption till Monday was to suspend the account and hard revert the content until more people could be brought in. I’m also assuming suspending the account just automatically says that you violated the ToS.

                                                    1. 3

                                                      I could imagine that somebody identified this as a possible account hack and thus locked it.

                                                      1. 2

                                                        They didn’t, they suspended his account so he can’t log in. You are still free to troll him on GitHub without any recourse whatsoever.

                                                      1. 2

                                                        If you include a language with a single implementation within your definition of portability, then this definitely is more portable than C.

                                                        Still, this goes without saying that we should simplify build-systems in general. A simple Makefile and a config.mk file for optional tweaks that don’t need to be done on 99% of systems suffices in most cases.

                                                        1. 25

                                                          Have you noticed that there no portable way to print “Hello World” in C?

                                                          There is a solid standard for source code that passively promises it could do it when built, but you can’t run this source code in a standard, portable way. I see people treat gcc on POSIX as this sort of standard, but it’s not. There are other compilers, other platforms, and even within gcc-flag-copying compilers on close-enough-to-POSIX systems it’s endless papercuts.

                                                          I had a simple Makefile with config.mk, and ended up in a situation where I couldn’t build my own project on my own machine. After days of debugging I think the simplest solution would be to recompile GCC from scratch with -mno-outline-atomics… but I just don’t want to deal with these things. In C everything is theoretically possible, but nothing is easy, nothing is reliable.

                                                          I’m completely unmoved by the theoretical possibility of porting my project to DEC Alpha or a DSP with 12-bit bytes, when I can’t even run it on macOS and Windows.

                                                          1. 1

                                                            I know about C’s inconsistencies and quirks, but it’s reasonable to just target a POSIX compliant system, which is a standard and clearly defines interfaces you can use to print “Hello World”.

                                                            I do not depend on the mercy of the Rust developers to support a certain platform,and when I check the Rust officially supported platforms I don’t see that many. Even Microsoft itself acknowledges that by offering WSL.

                                                            1. 15

                                                              Your position is odd. On one hand you seem to value support for all platforms, even ones that are too obscure too be in LLVM. But at the same time you say it’s reasonable to just drop the biggest desktop platform.

                                                              Porting Rust needs an LLVM back-end and libc, which is pretty much the same as porting Clang. You don’t need a permission from an ISO committee to do this. LLVM and Rust are open. In practice the list includes every modern platform people care to support. There’s also a GCC back-end in the works, so a few dead platforms will get support too.

                                                              1. 2

                                                                I think FRIGN is arguing that targeting POSIX does not mean dropping Windows, because WSL exists. I don’t agree, but it is an arguable position.

                                                                1. 7

                                                                  WSL is Linux. I would not call that targeting the Windows platform.

                                                                  1. 1

                                                                    Why not? WSL is part of Windows. What do you gain by targeting non-WSL Windows? (I think there is a gain, I am just curious what you think it is.) Is it that WSL is an additional install? (JVM is an additional install too, and frankly WSL is easier to install than JVM.) Is it support for versions older than Windows 10? Is it overhead, which I think is tiny?

                                                                    Would you stop targeting Windows if all of the following happens: 1) WSL becomes part of default install 2) all versions of Windows not supporting WSL become irrelevant 3) performance overhead improves enough to be immaterial? What else would you want?

                                                                    1. 13

                                                                      Windows can also run Web apps and Android applications, or even emulate an old Mac, but in a discussion of portability I think it’s important to make a distinction between getting software to run somehow vs being able to directly use the native vendor-recommended APIs of the platform.

                                                                      Is Visual Basic 6 portable and can target Linux? There is WINE after all, and it’s probably as easy to install as WSL.

                                                                      The WSL tangent feels like I’m saying “I can’t eat soup with a fork”, and you “You can! If you freeze it solid or if you spin the fork really really fast!”

                                                                      1. 2

                                                                        I also think depending on Wine instead of porting to Linux a defensible position.

                                                                      2. 4

                                                                        Why not? WSL is part of Windows. What do you gain by targeting non-WSL Windows?

                                                                        I worked at a place that wouldn’t allow WSL because the security team couldn’t install the network monitoring tool they used. Ultimately, it is an optional component and can be disabled.

                                                                        1. 3

                                                                          Why not? WSL is part of Windows. What do you gain by targeting non-WSL Windows?

                                                                          WSL is a brand name covering two things:

                                                                          • WSL1 uses picoprocesses in the NT kernel with a Linux system call compatibility layer (similar in concept to the *BSD Linux compat layers)
                                                                          • WSL2 just runs a Linux VM with Hyper-V.

                                                                          There is some integration with the host system, a WSL application in either version can access the host filesystem (with WSL1, there are filter drivers over NTFS that present POSIX semantics, with WSL2 it uses 9p over VMBus, both are slow). They can create pipes with Windows processes. But they can’t use any of the Win32 API surface. This means no GUI interaction, no linking with any Windows DLLs. For something like an image library (as in the article), providing a Linux .so is of no help to someone writing a Windows application. The most that they could do with it is write a Linux command-line tool that decoded / encoded an image over a pipe, and the run that win WSL, on the subset of Windows machines that have enabled WSL (unless it’s changed recently, neither WSL1 or 2 is enabled by default).

                                                                  2. 3

                                                                    I think it does boil down to the question whether targeting POSIX is reasonable or not. Many people, including myself and the author of this article, find it unreasonable. But I admit it is a defensible position.

                                                                2. 7

                                                                  That works unless it involves one of threading, Windows, or cross-compilation. All three works out of the box with Rust. C is more capable, but Rust is more convenient.

                                                                  1. 1

                                                                    That’s a fair point, but this only works because there is one single Rust implementation that is ported, and that’s what you depend on.

                                                                    I’m not arguing about the convenience.

                                                                  2. 4

                                                                    A simple Makefile and a config.mk file for optional tweaks that don’t need to be done on 99% of systems suffices in most cases.

                                                                    I’d argue the opposite, that cargo new and cargo build suffice in most cases, and you don’t need the capabilities of C most of the time unless you’re doing something weird, or something with hardware.

                                                                    1. 1

                                                                      But think about the massive complexity behind cargo. Why does anyone think it’s reasonable to pull in dozens of online (!) dependencies for trivial code (node.js all over again), and with Rust you can’t get around this bloat.

                                                                      And I’m not even saying that this was about C vs. Rust. I am aware of the advantages Rust offers.

                                                                      But the bloat is unnecessary. Consider Ada for example, which is way more elegant and provides even more security than Rust.

                                                                      1. 14

                                                                        Cargo is simple technically. Probably simpler than cmake and dpkg.

                                                                        Rust uses many dependencies, but that isn’t necessarily bloat. They’re small focused libraries. Slicing a pizza into 100 slices instead of 6 doesn’t make it any larger. I have 45 transitive dependencies in libimagequant, and together they are smaller than 1 OpenMP dependency I’ve had before.

                                                                        Cargo uses many small deps, because it’s so easy and reliable, that even trivial deps are worth using. I don’t think pain should be the driving factor technical decisions — deps in C are a pain whether they’re used justifiably or not. Even despite dependency friction C has, applications still use many deps. Here’s a recent amusing example of dependency bloat in vim.

                                                                        I’ve considered Ada, but it offers no safety in presence of dynamic memory allocation without a GC. The usual recommendation is “then just don’t allocate memory dynamically”. That’s what I’m aiming for in C and Rust too, but obviously sometimes I do need to allocate, and then Rust offers safety where Ada doesn’t.

                                                                        1. 6

                                                                          The usual recommendation is “then just don’t allocate memory dynamically”

                                                                          Ada allows you to declare anything pretty much in any declaration block, including run-time sized arrays, functions, variables, tasks, etc. so a lot of time you’re logically “allocating” for work, but it’s usually stack allocated though the compiler can implicitly allocate/free from the heap (IIRC). Being able to return VLAs on the secondary stack also simplifies things a bit like returning strings. The standard library is pretty extensive too, so you usually just use stuff from there which will be RAII controlled. Between RAII via “Controlled types” and limits on where access types (pointers) can be used, I don’t think I’ve ever actually seen raw allocations being passed around. Implicitly passing by reference when needed, and allowing declaration of different pointer types to the same thing, but declared in different places seems to really cut down on pointer abuse.

                                                                          One of my Ada projects is ~7000 LoC (actual, not comments or blanks) and I have one explicit allocation where I needed polymorphism for something, wrapped in a RAII smart pointer which also allocates a control block.

                                                                          Looking at Alire, ~28600 LoC (actual, not comments or blanks), shows only 1 allocation which doesn’t go through the program lifetime, which is inside an RAII type. If you include all 1554 files of it and its dependencies, Unchecked_Dellocation only appears in 60 of them.

                                                                          I understand the disbelief. I get that it’s weird.

                                                                          (shrug), I dunno, it’s sort of hard to explain, but the language and standard library means you just often don’t need to allocate explicitly.

                                                                        2. 9

                                                                          This is a strawman argument, because Cargo’s package management features aren’t what we were talking about. But you can use Rust without needing to depend on any other packages, in fact you can use it without Cargo at all if you want, just create a Makefile that calls rustc. Saying that you “can’t get around” this bloat is not really factual.

                                                                          1. 1

                                                                            How far is this going to get you when the standard library doesn’t even include the most basic of things?

                                                                            Packagers (rightfully) won’t touch rust-packaging as the majority uses cargo and everything is so scattered. When you are basically forced to pull in external dependencies, it’s reckless, on the other hand, not to rely on cargo.

                                                                            Ada does it much better and actually reflects these things, and C benefits from a very good integration into package management and systems overall.

                                                                            Rust could’ve had the chance to become easily packagable, but it isn’t. So while the “move fast”-philosophy definitely helped shape the language, it will lead to long-term instability.

                                                                            1. 12

                                                                              As someone who occasionally packages rust things (for Guix) I don’t really know why everyone thinks it’s so hard. Cargo metadata for every project means you can often automate much of the packaging process that in C I do by hand.

                                                                              1. 2

                                                                                Given you seem to have experience with this, how simple is it to create a self-contained tarball of a program source with all dependencies (as source or binary) that does not require an internet connection to install?

                                                                                1. 14
                                                                                  1. 2

                                                                                    Nice, I didn’t know about that one. Thanks!

                                                                                  2. 6

                                                                                    Note that this isn’t a requirement for all packaging systems. The FreeBSD ports tree, for example, requires only that you be able to download the sources as a separate step (so that the package builders don’t need network access). There’s infrastructure in the ports tree for getting the source tarballs for cargo packages from crates.io. These are all backed up on the distfiles mirror and the package build machines have access to that cache. All of the FreeBSD packages for Rust things use this infrastructure and are built on machines without an Internet connection.

                                                                                    1. 1

                                                                                      Very interesting, thanks for pointing that out!

                                                                                2. 5

                                                                                  Just because you like C and Ada doesn’t mean every other language is terrible.

                                                                                  1. 1

                                                                                    I totally agree. For instance I love Julia for numerics.

                                                                        1. 17

                                                                          Modern browsers don’t cache requests to CDNs across multiple domains, since that can be used to track users — this means that even if someone has already downloaded the library you’re including from the CDN on one website, they’ll have to download it again when they visit your website.

                                                                          This is still news to a lot of developers. It bears repeating.

                                                                          What I do is just download the library that I want and include the files in my repo, just like any other source file. I usually do this in a directory called vendored or 3p, and be sure to include the version number of the package that I downloaded in the filename. This takes maybe 60 seconds more work than including the CDN version, which seems worth it to me for the privacy, robustness, and speed benefits.

                                                                          I’m with you up to here. I’ve heard the treat-dependencies-as-source approach before. I don’t think it’s realistic for a couple of reasons:

                                                                          1. JavaScript dependencies listed on npm are a common malware vector. Expecting everyone to read all of the source of all their dependencies themselves and understand it before they copy/paste it into their source is not going to be as effective as an npm audit report.
                                                                          2. I’ve inherited a couple of projects that are set up like this. It tends to be fairly error-prone. If someone leaves off a version number or forgets a sub-dependency then you spend a day tearing your hair out trying to fix a version conflict or just look up the right version of the API documentation. Tools like LibMan for .NET automate this process for a marginally better developer experience, but it isn’t as robust as npm, which will make sure the right versions of sub-dependencies and peer dependencies are installed.

                                                                          npm can be a frustrating tool to use to be sure, but your proposal strikes me as only marginally more secure than using a CDN and with some significant drawbacks.

                                                                          1. 5

                                                                            I agree that vendoring dependencies isn’t always the right way to go about things, but as an alternative to pulling things in from a CDN, it’s usually what people want. If I’m pulling in more than a couple dependencies, or dealing with a dependency graph, then I use a real package manager. But for something like pulling in jQuery or lodash and building on top of it, NPM doesn’t really buy you anything but complexity.

                                                                            I don’t think there are people out there loading a giant dependency graph from CDNs — typically that’s something people only do for small projects, and so my suggestion here was implicitly assuming that scale — it’s “what you should do instead of pulling from a CDN”, not “how you should manage your dependencies in general” :)

                                                                            1. 4

                                                                              Installing directly from NPM can also improve your JavaScript dev experience since TypeScript can load type definitions automatically for many packages. If those packages are not in a standardized place, then you likely won’t get as much help from your other tools.

                                                                              I have a tendency to use a bundling tool and then just specify the packages part of the vendored chunk. It usually has very similar benefits with caching, but it can be overall fewer bytes, fewer open concurrent connections, and it works with more standardized tooling.

                                                                              1. 2

                                                                                I don’t think there are people out there loading a giant dependency graph from CDNs

                                                                                They do, unfortunately. WordPress and ecommerce sites are especially liable to ballooning JS dependencies. What starts out as just “jQuery or lodash” multiplies with plugins and other common dependencies like Bootstrap. It’s what jsdelivr’s combine fearure is for.

                                                                                Context is everything. If it’s just you or you and a team of people who have agreed to JavaScript austerity, it makes sense to do as you say. But as general advice, I still disagree.

                                                                                1. 2

                                                                                  NPM doesn’t really buy you anything but complexity.

                                                                                  It just does the thing you’re manually doing. The problem with NPM/Yarn IMHO is that it has a shitty default: it always uses ^ for the version constraint rather than allowing you to configure it or (a way better strategy) using ~. Caret means “give me whatever the latest of this is other than the next major because everyone uses SemVer properly and nothing ever goes wrong”. Tilde means “I want patch updates but not major or minor versions because I’m a realist and people don’t use SemVer correctly and/or cannot predict every possible breaking change”

                                                                                  Why can’t I do this?

                                                                                  yarn add jquery --tilde
                                                                                  

                                                                                  and then it could end up like

                                                                                  {
                                                                                    "dependencies": {
                                                                                      "jquery": "~2.0.0"
                                                                                    }
                                                                                  }
                                                                                  

                                                                                  The JS ecosystem makes package management harder than it needs to be. Even Bundler does a better job at this.

                                                                                  1. 1

                                                                                    Totally. You can use the @ syntax, but you still have to look up the version number on https://semver.npmjs.com and then:

                                                                                    npm install --save jquery@~2.0.0
                                                                                    

                                                                                    Would love to be able to:

                                                                                    npm install --save jquery@~latest
                                                                                    

                                                                                    …or something like it to install the latest version and peg it to the latest patch. Another crappy default is not adding an engines field automatically. I’ve seen so many newcomers to npm waste time on npm installation errors that turn out to be caused by environment incompatibility.

                                                                                    I cringe that I’m recommending npm at all, but I have yet to encounter anything that’s substantially better. Bower found a middle road of complexity long ago before they got shamed for the overlap with npm and fell into disuse.

                                                                              1. 22

                                                                                Help me. I am not a Ruby person and I probably never will be. I just cannot figure out what Hotwire is. I have read this post, I have read the Hotwire homepage, I have googled it, I cannot for the life of me figure out what it actually is.

                                                                                I keep reading “HTML over the Wire” but that is how normal websites work. What is different?

                                                                                1. 45

                                                                                  you know how HTML is usually transferred over HTTP? well, Hotwire just transfers that same HTML over a different protocol named WebSockets.

                                                                                  that’s it. that’s the different.

                                                                                  1. 9

                                                                                    Thanks. Your explanation saves me countless hours.

                                                                                    1. 9

                                                                                      what on earth

                                                                                      1. 28

                                                                                        For others who may be confused: This is dynamic HTML over web sockets. The idea is that the client and server are part of a single application which cooperate. Clients request chunks of HTML and use a small amount of (usually framework-provided) JS to swap it into the DOM, instead of requesting JSON data and re-rendering templates client-side. From the perspective of Rails, the advantage of this is that you don’t need to define an API – you can simply use Ruby/ActiveRecord to fetch and render data directly as you’d do for a non-interactive page. The disadvantage is that it’s less natural to express optimistic or client-side-only behaviors.

                                                                                        1. 1

                                                                                          Ah. That … sort of makes sense, honestly?

                                                                                          1. 7

                                                                                            it’s not really a new idea, they stole it from phoenix liveview. https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html

                                                                                            1. 6

                                                                                              … which I guess in turn is the spiritual successor to TurboLinks

                                                                                              1. 5

                                                                                                It’s worth noting that idea isn’t new with Phoenix. Smalltalk’s Seaside web framework had this back in 2005 or so (powered by Scriptaculous), and WebObjects was heading down that path before Apple killed it.

                                                                                                Phoenix LiveView looks great, and is likely the most polished version of the concept I’ve seen, don’t get me wrong. But I don’t think DHH is “stealing” it from them, either.

                                                                                                1. 5

                                                                                                  There’s a lot of implementations of it, here’s a good list: https://github.com/dbohdan/liveviews

                                                                                                  1. 4

                                                                                                    Not unsurprising, given that Phoenix is designed by a prolific Rails contributor. There’s a healthy exchange.

                                                                                                    https://contributors.rubyonrails.org/contributors/jose-valim/commits

                                                                                                    1. 5

                                                                                                      Moreover, DHH has been experimenting with these techniques since roughly the same time that Elixir (not even Phoenix) first appeared: https://signalvnoise.com/posts/3697-server-generated-javascript-responses

                                                                                                  2. 1

                                                                                                    In addition to brandonbloom’s excellent points, I personally liken it to the Apple/Oxide push for hardware and software being signed together. This type of tech makes it much easier to keep frontend and backend designs coherent. It is technically possible to do with SPAs and the APIs they rely on but the SPA tech makes it too easy (at an organizational level) to lose track of the value of joint design. This tech lowers the cost of that joint design and adds friction to letting business processes throw dev teams in different directions.

                                                                                              2. 4

                                                                                                Right. These days, a lot of normal websites transfers JSON over WebSockers and piece the HTML together on the client side with JavaScript, and Hotwire is a reaction against that, bringing it back to transfering HTML.

                                                                                                1. 3

                                                                                                  Really? How is that the answer to all of life’s problems, the way DHH is carrying on?

                                                                                                  1. 10

                                                                                                    because now your (data) -> html transformations all exist in one place, so you don’t have a server-side templating language and a client-side templating language that you then have to unify. also the performance of client-side rendering differs more substantially between devices, but whether that’s a concern depends on your project and its audience. just different strategies with different tradeoffs.

                                                                                                    1. 4

                                                                                                      Yes, these are good points; another is that you have basically no client-side state to keep track of, which seems to be the thing people have the most trouble with in SPAs.

                                                                                                      1. 1

                                                                                                        Great for mail clients and web stores, the worst for browser games.

                                                                                                        1. 1

                                                                                                          depends on the game. A game that runs entirely in the browser makes no sense as a Hotwire candidate. But a for a game that stores its state in the server’s memory it’s probably fine. Multiplayer games can’t trust the state in the client anyway.

                                                                                                          1. 1

                                                                                                            If you genuinely need offline behavior, or are actually building a browser-based application (e.g., a game, or a photo editor, etc.), something like Hotwire/Liveview makes a great deal of sense.

                                                                                                            At least until you get to a certain scale, at which point you probably don’t want to maintain a websocket if you can help it, and if you are, it’s probably specialized to notifications. By that time, you can also afford the headcount to maintain all of that. :)

                                                                                                1. 1

                                                                                                  the web itself is an easter egg

                                                                                                  1. 4

                                                                                                    can y’all do me a favor and specify which Linus you’re talking about?

                                                                                                    1. 4

                                                                                                      File contents are stored as a database of de-duplicated chunks using buzhash, similar to common backup utilities.

                                                                                                      Interesting, I always wondered how they did that…whether there was a “standard solution” or did people just wing it?

                                                                                                      Is this also how GridFS splits files into 16mb chunks?

                                                                                                      1. 6

                                                                                                        The standard method is to split the file using a “rolling hash.” Rsync also uses this method for synchronizing files remotely. Interestingly, the whole area of rolling hashes is a bit of a software patent minefield; it’s probably better to make some random thing up than to try to use an existing rolling hash, since the existing one might well fall under some overly-broad software patents.

                                                                                                        1. 2

                                                                                                          I don’t believe that rsync uses variable-length chunks based on a rolling hash signature like leading/trailing zeroes, unless it’s changed since I looked at it many years ago. I believe it computes hash signatures (for both rolling and crypto hashes) over fixed-size chunks, so it can’t benefit from the optimization that “content-based chunking” affords for changes that don’t change the contents of a chunk, only its displacement within the file.

                                                                                                          1. 1

                                                                                                            It’s in the wikipedia page for rsync. I haven’t actually looked at how it works though.

                                                                                                            1. 1

                                                                                                              Ah right, I’d forgotten that detail, thanks for the pointer. The algorithm still uses fixed-width chunks though, not “content-based chunking” (which, again, determines chunk boundaries based on bit patterns in the rolling hash like leading/trailing zeroes). Content-based chunking doesn’t produce chunks of fixed width, just fixed average width.

                                                                                                          2. 2

                                                                                                            It’s also not particularly difficult, I built myself one a few years ago just to make sure I understood the concept. It’s super-useful.

                                                                                                          3. 4

                                                                                                            The GridFS page you linked to says “By default, GridFS uses a default chunk size of 255 kB; that is, GridFS divides a file into chunks of 255 kB with the exception of the last chunk. The last chunk is only as large as necessary.” So it doesn’t sound like there’s any rolling hash going on. The page does mention the size 16MB, but only as the size limit(!) of the BSON format.

                                                                                                            1. 2

                                                                                                              ah ok, makes sense.

                                                                                                              size limit(!)

                                                                                                              yeah, BSON does indeed have a size limit. but it’s per-document, not per-field. as most single documents don’t get anywhere near that size limit, it’s rare to find GridFS users, at least in my experience. at a past job we tried to store images on GridFS but it offers zero additional benefits over just uploading to GCS or S3.

                                                                                                          1. 1

                                                                                                            the controls really messed me up lol

                                                                                                            1. 14

                                                                                                              Neither post mentions what the actual bug is; I think it’s this one; that putByte() function calls String.fromCharCode(), and the call does as well. Why does this matter?

                                                                                                              >>> String.fromCharCode(65)
                                                                                                              "A"
                                                                                                              
                                                                                                              >>> String.fromCharCode(String.fromCharCode(65))
                                                                                                              "\u0000"
                                                                                                              

                                                                                                              So that’s not going to be very random 🙃

                                                                                                              If only JavaScript had some sort mechanism to signal unusual situations when the programmer almost certainly made a mistake by passing the wrong type…

                                                                                                              Looks like this bug was already present in the first commit from 2013.

                                                                                                              1. 3

                                                                                                                i love all the TC39 proposals, but i do honestly feel like type annotations are an important feature that needs to be in the language. it’s pretty clear from typescript’s adoption that type defs are something JS devs want, and i don’t think most of us also want to be subservient to microsoft’s whims either.

                                                                                                                1. 3

                                                                                                                  Type hints won’t really help here, because you’ll still have those function from the standard library that accept $anything and do $weird_stuff. They can be fixed without type hinting quite easily: if (typeof(arg) != 'number') thow Exception('not a number').

                                                                                                                  This is really a core problem that’s not easily fixable with a new feature or whatnot. You really need to break things in a backwards-incompatible way, although that can be guarded with a use stricter similar to the already-existing use strict. They didn’t want to do that “because it would fork the web”, because clearly writing bug-prone software for decades to come is better, and as if the TypeScript thing isn’t a fork either.

                                                                                                                  These mistakes compounds and leak in to “modern” JS too; what happens if you try to send a plain ol’ Object with the “modern” fetch API? Seems like that would work no? Nope! It will send [object Object] to the server as POST data. Yes, really. This isn’t a modern API; throw an exception, return an error, don’t send a request, anything else makes more sense than sending [object Object].

                                                                                                                  So people use millions upon millions of lines of tools to work around this stuff… TypeScript isn’t wholly bad, but an entire ecosystem to compile, manage, and debug all of it is a lot of overhead, never mind having to know two languages (kind-of). And even with all of that I find the daily dev-experience suboptimal as I can’t just take a few lines of TypeScript and run it in the browser’s REPL. It’s one step forward and two steps back.

                                                                                                                  All these other TC39 things are unimportant until this is fixed. “Modern” JavaScript isn’t better; it’s just the old broken JavaScript with more features (which doesn’t necessarily make it better) and layers upon layers on top on things to manage the broken foundations.

                                                                                                                  1. 1

                                                                                                                    And even with all of that I find the daily dev-experience suboptimal as I can’t just take a few lines of TypeScript and run it in the browser’s REPL.

                                                                                                                    Google tried to do this with Dart, like 10 years ago, but they gave up before getting very far. Which is, perhaps, fair since, at that point in time, Dart was a pretty “meh” language itself and people hadn’t really gotten on board the type train yet (CoffeeScript was still cool).

                                                                                                                    1. 1

                                                                                                                      The last time I used Dart (a long time ago now), I was pretty taken aback by how big the bundle ended up being after you compiled to JS. I think the value add of type-safety was not enough to justify the major drawback of your code taking 4x the time to download. When TypeScript was introduced, the only reason I even looked at it was its promise that it would compile down to JS that is easy to read and not filled with extraneous garbage you’re not using.

                                                                                                                      1. 1

                                                                                                                        Yeah, TypeScript is definitely the way to go (IMHO) for JS development. The original promise of Dart was that there would be a VM in the browser, so you wouldn’t need to compile to JS at all. That didn’t pan out, though.

                                                                                                                        1. 2

                                                                                                                          I mean… ;-)

                                                                                                                          One day, perhaps!

                                                                                                                    2. 1

                                                                                                                      Type hints won’t really help here, because you’ll still have those function from the standard library that accept $anything and do $weird_stuff.

                                                                                                                      I do get that, and I think if JS ever adopts some level of type-safety it might have to be “except the DOM APIs” for that reason, at least in the very beginning. What I really want is for TypeScript’s type-safety guarantees to apply after I compile, honestly. Having to worry about types seems like a lot of work if you don’t even get those benefits when your code runs in the real world. It feels like training wheels, not a better bicycle.

                                                                                                                      1. 1

                                                                                                                        If you would add type hints everywhere you’ll still have stuff like 1 + '1' = '11', '' + {} = '[object Object], etc. Your function is constrained to just a number but if you don’t change the automatic casting rules then it’s still a huge footgun. You can’t really fix that with type hints, as it would mean you’d have two different “flavours” of numbers: “regular numbers” and “strict numbers”, which behave different depending on how the function was declared. Copy/pasting code from one function to the other could potentially break it, or make it behave different. That would be even more confusing.

                                                                                                                        Actually, if you fix this then a lot of these APIs will probably be fixed automatically as well; the reason these APIs behave weird is because JavaScript in general behaves weird with all its automatic casting.

                                                                                                                        And you can do this in a compatible way with 'use strict 2';, 'use stricter';, 'strict me harder';, or whatnot. Perl has been doing this for a long time, and it works quite well.

                                                                                                                        Once this is fixed you have a decent (dynamic) type system to work off, and you can start thinking about type hints to constrain what functions can accept.

                                                                                                                1. 9

                                                                                                                  I like the concept, but do you really think they will let you circumvent their paywall forever? This seems like a project with a built-in expiration date.

                                                                                                                  1. 18

                                                                                                                    Thanks! And yeah, this could stop working any moment 😂 I still got a lot of value out of making it and launching it though. It’s also possible this doesn’t make a big enough dent in their profit for them to care. Nitter, Invidious, and the like all continue to work well.

                                                                                                                    1. 4

                                                                                                                      Don’t let that discourage you! I love these types of front ends. I currently used the Privacy Redirect extension in Firefox for Invidious mirrors, nitter, libredd.it and others. They’re all amazing, and contributes have kept many of them functional as well as providing several alternative hosts!

                                                                                                                      Most of them are way better than using the actual sites as far as usability goes as well. Thank you for your effort. These types of frontends are really important.

                                                                                                                    2. 18

                                                                                                                      You get the first 2 or 3 articles “free”; if you don’t store cookies in your browser then you always get “free” articles. If the scribe.rip service gets banned then you can host your own version – it’ll be hard to block that as long as they keep the “first n articles for free”-scheme.

                                                                                                                      But even if you get all of it for free this is great, because the entire thing is just so incredible annoying with all their stupid JavaScript that prevents you from copying text (instead, you can tweet about it because, apparently, tweeting something is more important than copying).

                                                                                                                      1. 1

                                                                                                                        In practice, it’s more than 2 or 3 because whether or not articles go behind the paywall is a decision made by individual authors (though it’s now enabled by default, rather than disabled by default as it was up until about a year ago). This proxy seems to remove the intrusive pop-overs asking non-logged-in users to register, which people repeatedly mistake for the very visually similar paywall pop-overs.

                                                                                                                        I’m not sure what you mean about “javascript that prevents you from copying text” – maybe some extension is removing it for me, or maybe some extension on your end is causing the social-highlighting mechanism to interfere with copy & paste? I copy text from medium all the time.

                                                                                                                    1. 23

                                                                                                                      Waiting for ““Static Linking Considered Harmful” Considered Harmful” Considered Harmful

                                                                                                                      1. 7

                                                                                                                        it’s just harmful considerations all the way down

                                                                                                                        1. 1

                                                                                                                          Allthough if we’re to keep the cadence, the next harmful consideration is due in September 2036 or something.

                                                                                                                        2. 2

                                                                                                                          I think that would be hilarious!

                                                                                                                          1. 2

                                                                                                                            Rule of Threes. After ““Static Linking Considered Harmful” Considered Harmful” Considered Harmful, we’ll get tired of the joke. So, still one more to go lol

                                                                                                                            1. 1

                                                                                                                              We need a British person beyond that point. Because we all know that they have a good feeling for overdoing jokes to the point where they get funny again.

                                                                                                                              1. 2

                                                                                                                                But you have to listen very carefully, because they will say it only once.

                                                                                                                            2. 2

                                                                                                                              Reminds me of a notice I saw that said (translated) all posting except posting about posting forbidden forbidden.

                                                                                                                            1. 0

                                                                                                                              bloggymcblogface.blog

                                                                                                                              1. 7

                                                                                                                                When they say “built on the same technologies as other popular chat apps,” what do they mean? The web page mentions “privacy” a lot but doesn’t say anything specific about it. Neither do the GitHub repos’ READMEs, unless I missed something.

                                                                                                                                To me, “privacy” means my messages aren’t going to be stored on the admin’s server, given that a lot of the time I’d probably not have any particular trust in said admin or their competence at setting up e.g. a MongoDB server without public access open to the world.

                                                                                                                                1. 11

                                                                                                                                  Looks like its not really private at all. There’s no encryption. Its just self hosted.

                                                                                                                                  So the privacy they’re talking about is the de facto gains of hosting it yourself or by a person you trust and not a big company like discord or guilded that sells your data

                                                                                                                                  If a malicious actor was watching your traffic somehow, like over coffeehouse WiFi or w/e then your messages being sent aren’t any less secret or safe. That’s no better than discord is today though.

                                                                                                                                  If you want encrypted rooms I guess you gotta stick with matrix but what appeals to me about revolt is my friends might actually try it because it feels like discord. Unlike matrix

                                                                                                                                  1. 11

                                                                                                                                    That’s no better than discord is today though.

                                                                                                                                    If I run discord from the browser it’s a https site, so plaintext should not be accessible from the wire.

                                                                                                                                    I can’t imagine the “native” app works any differently.

                                                                                                                                    There’s degrees of privacy. A self-hoster will probably not sell traffic/usage data to advertisers, but there’s absolutely not guarantees they won’t snoop on private messages/images. This can happen in a big org like Discord but there’s a higher chance that such behavior is monitored and caught - if nothing else for the massive reputational risk of it being exposed.

                                                                                                                                    1. 1

                                                                                                                                      It’s also a practical thing. Matrix per default also doesn’t doesn’t e2e encrypt everything at all, depending on some circumstances, same goes for IRC. And that’s fine, it allows for very performant full text search and sync (which is also certainly eco friendlier). Telegram (the chat program) didn’t win because of its encryption, it won because of its features, usability and speed. And if I look at day to day usage of discord I’m certain you don’t need e2e here, you may tell people upfront that for e2e worthy content they should choose something else than a glorified gaming slack with video + audio and nice icons, but that’s it.

                                                                                                                                      I’m currently adding some online sync to my own f-droid app, it also won’t have any e2e features. TLS it is, and if you’re distrusting me, you can host your own instance as easy as running some binary on your own VPS.

                                                                                                                                      1. 6

                                                                                                                                        Matrix per default also doesn’t e2e encrypt anything

                                                                                                                                        That’s wrong, Matrix (more specifically Element, the reference client implementation, previously called Riot) has been encrypting private chats by default for over a year now.

                                                                                                                                        1. 1

                                                                                                                                          Ok then my work instance simply doesn’t use E2E in their rooms. But we’re using it longer than matrix has e2e and we’ve not adopted it for that.

                                                                                                                                    2. 3

                                                                                                                                      Looks like its not really private at all. There’s no encryption. Its just self hosted.

                                                                                                                                      The roadmap at least does mention E2EE as a possibility in future, s. https://revolt.chat/roadmap#features-4, potentially scroll down to see:

                                                                                                                                      (draft) 0.6.0: E2EE Chat

                                                                                                                                      This is the drafted version for e2ee.

                                                                                                                                      1. 2

                                                                                                                                        I asked in the beta instance (devs are in there) and it looks like that e2ee roadmap item is for DMs and small group chats, not the discord-like ““servers”” within your instance

                                                                                                                                    3. 6

                                                                                                                                      To each their own but IMHO this can be okay depending on the circumstances.

                                                                                                                                      Example: I hang out a lot in a Slack where there’s ≈2000 persons in #general. So it’s basically as private as Twitter. To me that’s fine – but I would perhaps not tell all the secrets of my heart in that Slack. But having a discussion about Go or Python or whatever is cool with me, I don’t consider those topics private.

                                                                                                                                      Even if that chat room was end-to-end-encrypted I still would not consider it private. Anyone of those members could copy my messages or take screenshots and spread it to God-knows-where.

                                                                                                                                      1. 1

                                                                                                                                        Mostly WebSockets. Which is pretty much a given for any browser-based chat product. But I suppose with Rustaceans that might not be the case, since in theory you could create your own socket protocol and communicate over a Wasm client… :)

                                                                                                                                      1. 12

                                                                                                                                        I don’t think “winning” is the right choice of words here. Rather, I believe the world has moved on from UNIX.

                                                                                                                                        Considering that UNIX was an operating system developed in the 1970s, this seems pretty reasonable to me. There have been many developments in both hardware/software technology as well as development practices that have obviated many of the operating systems and programs of yesteryear. UNIX wasn’t designed to be run on embedded devices, or laptops, or TV set-top boxes, because none of those things existed yet. We’ve been living beyond the constraints that UNIX was built within for a while now, and there have been some great developments in recent years that improve the experience on a UNIX-based OS.

                                                                                                                                        Personally, after years of using Upstart on Ubuntu, dealing with all the issues of legacy init scripts interacting with Upstart jobs, and things being quite hard to manage on multiple OSes, I’m a huge fan of how systemd has made a lot of that work much simpler. Its documentation is better than the myriad of tools that only exist in manpages and developer’s brains, it’s easier to get the information you need out of systemd’s machinery, and it’s a lot easier to work with than launchd because you don’t have to keep esoteric XML syntax in mind.

                                                                                                                                        All in all, I’m a big fan of systemd and the changes it’s introduced to the world of Linux. I wish BSD and other UNIX-based OSes would adopt it or something like it, even if they feel like it should be rewritten to support native OS features, because it’s truly a great idea and helps Linux stay competitive in a world that free software could have never predicted.

                                                                                                                                        1. 5

                                                                                                                                          Lodash is super mad that they didn’t just name their library _ on NPM.

                                                                                                                                          1. 2

                                                                                                                                            “Why not use lodash? You might already have it as a dependency!”

                                                                                                                                            1. 1

                                                                                                                                              That’d cause a lot of confusion with Underscore though… “which fork am I installing again?”

                                                                                                                                              But it would still be pretty funny :D

                                                                                                                                            1. 7

                                                                                                                                              I had a few attempts to learn it, but I failed every time. It is fair to say that I don’t understand it, so I cannot even rant about its features, shortcomings, or flaws.

                                                                                                                                              Nice 👍🏻

                                                                                                                                              1. 3

                                                                                                                                                to be fair I’m using it for one hobby project (not a webdev at day) and still feel overwhelmed by the amount of additional plugins (Redux, how do you do global state for login + user-menu that is decoupled, hooks…). Add the whole JS stack on top (with its 40k modules) and you are starting to feel like every major rust dependency graph is irrelevant in comparison. Vue felt way easier but lacked one major component for one thing I needed (QR code display) and often it’s not obvious how you use any external JS libraries together with these self-managing frameworks.

                                                                                                                                                1. 10

                                                                                                                                                  Don’t do Redux. Only big sites warrant using it. Start with minimal hooks (useState) passing the state and setters as params. everywhere. Once you understand this, work your way up using contexts and effects. You don’t need more than this for 80% complex sites out there.

                                                                                                                                                  1. 3

                                                                                                                                                    thanks for the explanation, will look into that

                                                                                                                                                    1. 2

                                                                                                                                                      contexts and hooks are probably the best thing to happen to react since JSX

                                                                                                                                                1. 4

                                                                                                                                                  If you are Facebook, using React makes sense. For a plain business website, React just introduce a shit ton of complexity and security issues while providing very little, if any, advantages. Why people use that crap for normal websites is beyond me.

                                                                                                                                                  1. 4

                                                                                                                                                    If you are Facebook, using React makes sense.

                                                                                                                                                    It’s been a while since I last used Facebook (and it was mostly for the “marketplace” stuff anyway), but when I last used it about a year ago it was slow, buggy, and generally just not a very pleasant experience.

                                                                                                                                                    I don’t know the causes for this, but it’s not like React is necessarily working out all that great for them.

                                                                                                                                                    1. 7

                                                                                                                                                      I don’t work at FB, but having worked on many large web apps using many different frameworks, React likely isn’t the bottleneck. The poor performance is likely due to how much tracking they do. Tracking adds tons of excess events, http requests, and general bloat.

                                                                                                                                                      1. 2

                                                                                                                                                        We were in the middle of rolling out the new website one year ago. Check it out now. The new site (no blue bar at the top) is quite a bit faster, at least compared to the old one.

                                                                                                                                                        1. 2

                                                                                                                                                          It is even worse now than it used to be. Frequently freezes up entirely while typing and clicking buttons often does nothing at all. If react is supposed to prevent bugs, Facebook is about the worst advertisement I could imagine.

                                                                                                                                                          1. 2

                                                                                                                                                            While your personal experience may differ I promise that performance in general is MUCH better on the new site (individual interactions, the initial page load, incremental page loads, average and median, slower machines faster machines, etc). We measure extensively.

                                                                                                                                                      2. 3

                                                                                                                                                        If you are Facebook, using React makes sense.

                                                                                                                                                        It makes sense if you’re building web applications, basically. Not that it’s the only technology anyone should ever use, it definitely competes with a few other amazing tools I can think of, but it’s still a viable choice.

                                                                                                                                                        1. 3

                                                                                                                                                          React just introduce a shit ton of complexity

                                                                                                                                                          That’s true, as soon as you pull in React you’ll also need a bundler, something to strip the JSX, and all the complexity that comes from managing those tools. Not to mention how React components and hooks are both their own DSL that has its own separate rules from JS.

                                                                                                                                                          and security issues

                                                                                                                                                          Care to elaborate? Do you mean security issues from using tons of JS libraries? You can actually use React without pulling in tens of libraries to do everything, and if you’re still concerned about bundle/code size you can also use Preact in production.

                                                                                                                                                          while providing very little, if any, advantages.

                                                                                                                                                          This is simply not true. Keeping UI and application state in sync starts out simple but quickly becomes tremendously complex, so having a library that takes care of that for me is invaluable. Making reusable components is also hard and clunky and React also makes that incredibly easy.

                                                                                                                                                          1. 2

                                                                                                                                                            I don’t consider the jsx parser or bundler, which come preconfigured for most projects via create-react-app, as complexity. If we are considering that complexity, then anything that comes with dependencies is automatically complex even if you have no need to configure anything.

                                                                                                                                                            The complexity of react is really based around understanding of hooks and lifecycle. If you’re building a webapp, you need to deal with state somewhere so that’s universal. I guess you could say jsx is complex but it’s basically html instead of a bunch of function calls, which should be pretty easy to understand.

                                                                                                                                                            1. 2

                                                                                                                                                              I would argue that hooks made JSX more complex because now there are extra things to consider when rendering your components. I only say this because I had a coworker who was learning React get thoroughly confused by when and how they could use hooks in relation to JSX syntax. Additionally, JSX isn’t just html but an additional hidden function call. It used to be a simpler h('div', attributes, children) but it’s gotten a little more complex under the hood with what that function does now.

                                                                                                                                                              I’d also be tempted to argue that JSX is a poorer version of how to structure html because it only accepts attributes and not properties, so it actually adds extra complexity around how to set properties on real dom nodes. Which in turn complicates how one uses web components within React.

                                                                                                                                                            2. 1

                                                                                                                                                              Regarding security issues, I’m primarly thinking about, yes, dependencies and the use of JavaScript on the server side. When you first go down that Node.js rabbit hole, the libraries tend to stack up pretty fast. I’ve seen pretty simple webpages passing 100.000 files in node_modules - with God knows how many people behind them. And why JS are all kinds of bad ideas, is a whole discussion in itself.

                                                                                                                                                              Add in the issues complexity in itself introduces and you end up with a system which, for a normal webpage, is totally overkill, hard to maintain, expensive to host and stable if you’re lucky.

                                                                                                                                                              In my honest opinion, that whole Node craze, was a big mistake. There are numerous alternatives out there, but if I were to pick one, I would go with Elixir and Phoenix LiveView. That stuff is just on a whole other level.