1. 4

    These days it seems woefully quaint to have this discussion without factoring in how browsers work, but as far as I know you don’t even have the luxury of being able to refer to a book when it comes to that. (the article is from 2012)

    1. 3

      Yeah one thing i see a lot is toy operating systems. But somehow nobody dares to try a toy browser :)

      I looked and found only a couple, probably 2 orders of magnitude less than OSes:

      https://limpet.net/mbrubeck/2014/08/08/toy-layout-engine-1.html

      https://www.udacity.com/course/programming-languages--cs262

      Although I have to update that to say the recent OS project here is really impressive and includes some HTML rendering:

      https://lobste.rs/s/2pnnel/serenityos_from_zero_html_year

      If anyone knows of any more, let me know!

      1. 6

        This is an interesting HTML5 subset that looks like it aims to be production quality:

        https://cobalt.googlesource.com/cobalt/+/master/src/README.md

        1. 8

          [Disclaimer: I’ve worked on Cobalt code and with the Cobalt team briefly. However I didn’t work with it that much, so I’m mostly clueless.]

          I think Cobalt (and its team) is really awesome. It is used in production by YouTube TV which runs on a lot of smart TVs and other related devices, so definitely production-level. I am not sure how effective it is as a learning tool in general for browsers given that it is production code and not a toy, but I did get a sense of how a browser could be structured from working with it.

          https://cobalt.foo/ has some useful additional context, and I found it interesting to poke through the following directories when working with it:

          1. dom/: Various data models representing how the browser thinks about the DOM. HTMLElement is nice to examine to get a sense of what stuff is common between different element types, e.g. the layout system.

          2. rasterizer/: Various implementations for rendering out the nodes from the render_tree. e.g. here’s the EGL render_tree_node_visitor, it’s responsible for drawing a lot of stuff.

          3. script/ contains the interface for the Javascript engine and two implementations.

          4. cssom/: The CSS object model tree and various node types.

          I also thought that browsing Starboard’s code (their way of porting Cobalt to multiple platforms) was super insightful for understanding different platform APIs. I’ve found it hard to find good documentation on how to write production-ready code for window creation etc., so this code can be nice to look at.

          Anyways mostly talking out of my bum here, please take this with a grain of salt.

          1. 1

            Thanks for the pointers! It looks pretty awesome, although yeah maybe a bit big to learn from. I would like for someone to take this and make some sort of alternative browser, sort of like like v8 was taken for node.js!

        2. 4

          WeasyPrint is a production work, not a toy, but it is a relatively small, self-contained, and rather complete CSS layout engine completely written in Python.

          1. 4

            Of note, Servo’s CSS parser (hence now Firefox’s CSS parser!) was written by one of main developers of WeasyPrint. This should give an idea of its quality, despite its lack of resource.

            1. 2

              seconding weasyprint. It’s maintainers are also very good stewards of the project, and convinced me to throw a couple bucks of month their way.

              It’s also probably the most readable layout code you’ll find out there, especially compared to the very line-noise-y C++ you’ll find in browser layout (if only from the accumulation of cruft over the years).

              1. 1

                Wow cool! It’s a lot smaller than I would have thought, even taking into account the html5lib dependency, which is also small. I thought browser rendering engines were of similar size to a JS engine, e.g. 500K or 1M lines of code.

                My experience is that Python can be 5-8x shorter than C++, which would still be ~100K lines.

                I think supporting only the modern stuff saves a lot of space. I think all browsers have to include XHTML parsers which is basically a dead-end.

                ~/git/other/WeasyPrint$ find . -name '*.py'|grep -v test | xargs wc -l|sort -n
                
                   968 ./weasyprint/css/__init__.py
                  1130 ./weasyprint/draw.py
                  1285 ./weasyprint/text.py
                  1286 ./weasyprint/layout/inlines.py
                  1421 ./weasyprint/css/validation/properties.py
                  1579 ./weasyprint/formatting_structure/build.py
                 21944 total
                
                  ...
                   917 ./html5lib/_inputstream.py
                  1721 ./html5lib/_tokenizer.py
                  2791 ./html5lib/html5parser.py
                  2947 ./html5lib/constants.py
                 16204 total
                
                1. 2

                  The largest saving is from not supporting dynamic HTML. Making C++ scriptable from JavaScript takes a lot of code: usually “script” part is as large as “layout” part. WeasyPrint also does not implement any incremental layout or dirty tracking because it is not interactive and that also simplifies layout a lot.

              2. 1

                Well, your first link recognizes WebWhirr as a fellow toy engine (and I guess NetSurf and Links are not «toy» enough?)

                On the other hand, by now I actually consume the majority of WWWeb content by scraping HTML files, parsing them and running some (custom) code to convert them to text I read in Vim. I don’t yet have any plan for script support, though.

            1. 8

              Re: format choice — I would recommend considering minimalistic HTML file with a single table. Not much worse than CSV w.r.t. serialisation overhead and simplicity, contains its own encoding header, can be opened at any device capable of downloading the export file from the website in the first place, opens in a sane way in many spreadsheet and word processor applications.

              Not always the best option, but probably worth using more than it is used now.

              1. 4

                CSV is a terrible format: almost zero cross-platform consistency outside of plain us-ascii 6bit less the comma. Excel still thinks an interchange format should use the machines charset; BoM and embedded newline corner cases, multiple quoting “standards” and so on.

                I’ve probably written four xlsx parsers/generators at this point. Takes an afternoon— about as long to read some random unknown-quality “library” api and you’ll at least know the performance characteristics but look: if you can’t be bothered, at least consider using JSON and using a JS-Zip-based xlsx writer. Most users will be fine with that (and real api consumers will prefer JSON to guessing what particular flavour of csv this is)

                1. 1

                  These were exactly our pain points with CSV - BOM, formatting and quoting, delimiters, unicode issues etc. Some of the fields in the data are user generated so we had to deal with how emojies and new lines are encoded… huge pain.

                  Generating xlsx in the client is probably a good solution but it required some time we didn’t have back then.

                2. 2

                  Without thinking of all the potential corner cases, that looks like quiet a good idea.

                  Moreover, one could ship the HTML file with a JS library (linked or inlined) to produce an excel file on the client side

                  1. 2

                    Agree it’s great.

                    Biggest issue with this approach (and it’s not that big an issue) is that large enough HTML tables will hang/crash most browsers.

                  1. 7

                    Great article.

                    A question I don’t hear asked often enough in discussions about lock-in and avoiding it is, what failure modes are you concerned about? And I find this kind of puzzling. “What failure modes are you concerned about?” is, in basically every context except lock-in, one of the most basic inputs into architectural decisions.

                    For example, if you’re talking about going multi-cloud instead of AWS-only:

                    • Are you concerned about Amazon raising prices? What is the probability of that, and what is the likely range of increases? How much of an increase would there have to be to exceed the development cost of going multi-cloud?
                    • Are you concerned about Amazon discontinuing a proprietary service? What is the probability of that, and is the probability-adjusted cost of switching to something else later on, given the likely amount of lead time you’d have based on looking at previous instances of Amazon discontinuing things, significantly greater than the up-front cost of rolling your own solution?
                    • Are you concerned about reliability? How sure are you that the alternatives you’re considering would be more reliable? How much will that reliability cost you, and will the cost be less than the business value of the added reliability?
                    • Are you concerned about Amazon completely going out of business with little enough warning that you would be unable to move your stuff somewhere else? What is the probability of that?

                    The answers will vary; I’m not here to suggest there’s a single universally-applicable correct approach. But if you’re not asking these kinds of questions, in my opinion you’re not thoroughly thinking through your lock-in decisions.

                    1. 1

                      Speaking of Amazon prices, it is not a question whether Amazon changes the pricing, it is the question whether your service evolves and some of the tasks become much cheaper with leases in terms of dedicated-server-years.

                    1. 10

                      End of an era, I suppose. There’s no such thing as a healthy monoculture, just monocultures that haven’t found their blight yet.

                      1. 33

                        According to that Stack Overflow survey there are plenty of popular alternatives, such as “ZIP file back-ups”, “Copying and pasting files to network shares”, and “I don’t use version control”.

                        1. 18

                          Did the ever popular “Final FINAL Copy 2” make it in? That’s got to be the number one version control system, right?

                          1. 9

                            My first programming job was like that. I was working as a repair tech at a computer shop, at some point they needed someone to clean up the intranet, and that someone was me.

                            First thing I did was set up svn and get rid of all the “foo.orig”, “foo.orig2”, etc directories. This was trickier as it might seem as some of the projects were being served from those .orig2 dirs.

                            All was going well, and then half a year later the guy who had been doing this asked me if knew what happened to the “cms.orig” directory. After I told him I deleted it he said he had been storing the company outings photos there for the last 15 years. By the time we discovered it was too late to recover from backup, so … all lost.

                            I still don’t understand why you would store pictures in a deeply nested subdir of an otherwise unused cms.orig …. 🤨 from what I heard he ditched svn and went back to his “system” after I left.

                        2. 13

                          Well, just to play a devil’s advocate… Some things are so good exactly because they are ubiquitous. Like Unicode, for example. Or POSIX. They have their flaws for sure, but they made writing interoperable software much easier.

                          There are already other tools that work with the same repository format as the Torvalds’ git. Maybe git format becoming the standard repo format is a good thing after all. No one has to use the reference implementation if they prefer different UI and abstractions.

                          1. 14

                            Maybe git format becoming the standard repo format is a good thing after all.

                            No, it’s definitely not. It doesn’t scale. The git “API” is literally the local filesystem. Microsoft has valiantly hacked the format into functioning at scale with VFS for Git, but the approach is totally bananas.

                            How does it work?

                            VFS for Git virtualizes the filesystem beneath your Git repository so that Git tools see what appears to be a normal repository when, in fact, the files are not actually present on disk. VFS for Git only downloads files as they are needed.

                            VFS for Git also manages Git’s internal state so that it only considers the files you have accessed, instead of having to examine every file in the repository. This ensures that operations like status and checkout are as fast as possible.

                            - vfsforgit.org

                            Microsoft had to implement an entire virtual filesystem that, through smoke and mirrors, tricks git to behave sanely. More details in this GVFS architecture overview.

                            1. 4

                              Isn’t the same true for Mercurial and every other DVCS in existence?

                              1. 17

                                No. Git’s remote repository API is nothing more than a specialized rsync implementation (git-send-pack and git-receive-pack).

                                Mercurial uses a semantic API for exchanging changes with the server. It doesn’t need local files in the same way git does. That opens up a lot of doors for scaling large repositories, because you can implement optimizations in the client, protocol, and server.

                                For git repos, where local filesystem operations are the protocol, there really is no alternative to Microsoft’s smoke and mirrors, virtualize the world approach. You’d have to just reimplement git, which defeats the point.

                                1. 1

                                  Ah, that is interesting. Thanks for the information, I should look into the way mercurial actually works.

                                  1. 3

                                    If you’re curious about the actual on-disk formats (which should be irrelevant, as hg tries to compartmentalise them), you can read about Mercurial’s internals.

                              2. 4

                                I don’t see anything wrong with git using the local file system API.

                                There are multiple implementations of such file systems – Linux, FreeBSD, OS X, Minix, etc. git works fine on all those systems, and the code is portable AFAICT.

                                1. 8

                                  So, I personally love how git transparently exposes its internal data structures for direct manipulation by the user. It gives you tons of power and freedom. Back when I used git, I considered it just as much a development tool as my editor.

                                  But that transparency is problematic for scaling. To the point where you really do need to implement a virtual remote filesystem tailored for git to support huge repos. Whether you like git or not, that’s bananas.

                                  1. 5

                                    There’s nothing bananas about that: scaling is a feature and it’s not surprising that you need more code/engineering to scale. It would be surprising if you didn’t!

                                    To make a very close analogy, two companies I worked at used Perforce (the proprietary VCS). At one company we used it out of the box, and it worked great. Thousands of companies use Perforce like this, and Perforce is a very profitable company because as a result.

                                    The second company (Google) also used Perforce out of the box. Then we needed to scale more, so we wrote a FUSE-based VFS (which I imagine the git VFS is very similar to). That doesn’t mean Perforce is “bananas”. It works for 99% of companies.

                                    It’s just designed for a certain scale, just like git is. Scale requires a lot of tradeoffs, often higher latency, decreased throughput, etc. git seems to have made all the right tradeoffs for its targeted design space. That it succeeded beyond the initial use case is a success story, not an indication of problems with its initial design.


                                    Also, I don’t see any evidence that Mercurial reached the same scale. It probably has different problems – you don’t really know until you try it. I heard some teams were working on scaling Mercurial quite awhile ago [1], but I’m not sure what happened.

                                    https://engineering.fb.com/core-data/scaling-mercurial-at-facebook/

                                    1. 4

                                      Then we needed to scale more, so we wrote a FUSE-based VFS

                                      I currently work at Google. CitC has nothing to do with Piper performance, it’s more about the utility of sharing your workspace, both between dev machines and tools (desktop, cloudtop, cider, critique), as well as blaze.

                                      (which I imagine the git VFS is very similar to).

                                      Not at all. The git “protocol” is filesystem operations. Microsoft made VFS for Git because they need to intercept filesystem operations to interface with the git toolchain. Perforce and Mercurial have actual remote APIs, git does not.

                                      That doesn’t mean Perforce is “bananas”. It works for 99% of companies.

                                      I don’t think Perforce is bananas. I don’t think git is bananas either. I specifically think “git format becoming the standard repo format” is NOT a good thing. The git toolchain and the repo format are inseparable, leading to Microsoft’s bananas implementation of a scalable git server. Clever and impressive, but bananas.

                                      1. 2

                                        What I’m reading from your comments is: “If only git had decoupled its repo format and push/pull protocol, then it would be more scalable”.

                                        I don’t think that’s true. You would just run into DIFFERENT scalability limits with different design decisions. For example: Perforce and Mercurial don’t share that design decision, as you say, but they still have scalability limits. Those designs just have different bottlenecks.

                                        Designing for scale you don’t have is an antipattern. If literally the only company that has to use a git VFS is Microsoft, then that’s a fantastic tradeoff!!!

                                        IMO Google’s dev tools are a great example of the tradeoff. They suffer from scalability. They scale and solve unique problems, but are slow as molasses in the common case (speaking from my experience as someone who worked both on the dev tools team and was a user of those tools for 11 years)

                                        1. 2

                                          I don’t think that’s true. You would just run into DIFFERENT scalability limits with different design decisions.

                                          Git was probably strongly tied to the filesystem because it was made in 2005 (Pentium 4 era) for a lower-performance scenario by someone who understood the Linux filesystem better than high-performance, distributed applications. It worked for his and their purposes of managing their one project at their pace. Then, wider adoption and design inertia followed.

                                          It’s 2019. Deploying new capabilities backwards compatible with the 2005 design requires higher, crazier efforts with less-exciting results delivered than better or more modern designs.

                                          1. 1

                                            “If only git had decoupled its repo format and push/pull protocol, then it would be more scalable”.

                                            It would be easier to scale. When the simplest and easiest way to scale genuinely is implementing a client-side virtual filesystem to intercept actions performed by git clients, that’s bananas. To be clear, VFS for Git is more than a simple git-aware network filesystem, there’s some gnarly smoke and mirrors trickery to make it actually work. The git core code is so tightly coupled to the file format, there’s little else you could do, especially if don’t want to break other tooling using libraries like libgit2 or jgit.

                                            Designing for scale you don’t have is an antipattern.

                                            Designing tightly coupled components with leaky abstractions is an antipattern. Mercurial supports Piper at Google through a plugin. Doing the same with git just isn’t possible, there’s no API boundary to work with.

                                    2. 3

                                      To the best of my knowledge it still uses the intricate knowledge of filesystem behaviour to avoid (most?) fsync calls — and the behaviour it expects is ext3 (which is better at preserving operation order in case of crash than most other filesystems).

                                      I actually had a hard poweroff during/just after commit corrupt a git repository.

                                      So, in a way, the API it actually expects is often not provided…

                                      1. 1

                                        Do you mean its reliance on atomic rename when managing refs? Or some other behavior?

                                        1. 3

                                          I would hope that atomic renames are actually expected from a fullu-featured POSIX FS (promised by rename manual page etc).

                                          But Git also assumes some ordering of file content writes without using fsync.

                                          1. 2

                                            Renames are only atomic with respect to visibility in a running filesystem, not crash safety, though. So I guess it’s not surprising you’ve seen corruption on crash.

                                            I’ve been trying to find a way to run a small-scale HA Git server at work — as far as I can tell the only viable option is to replace the whole backend as a unit (e.g., GitHub DGit/Spokes, GitLab’s Gitaly plans). Both GitHub and GitLab started by replicating at a lower level (drbd and NFS, respectively), but moved on. I can say from experience that GlusterFS doesn’t provide whatever Git requires, either.

                                            1. 1

                                              There is at least some pressure to provide metadata atomicity on crash (you cannot work around needing that, and journalled filesystems have a natural way to provide it), but obviously data consistency is often the same as without rename. And indeed there are no persistency order guarantees.

                                              Systems like Monotone or Fossil outsources the consistency problem to SQLite — which uses carefully ordered fsync calls to make sure things are consistent — but Git prioritised speed over portability outside ext3. (Mercurial is also not doing fsync, though)

                                              And if ext4 doesn’t provide what Git needs, of course GlusterFS won’t…

                              1. 1

                                AppEngine is the tee for deployment.

                                1. 2

                                  AppEngine and Glitch for hosted, sure, but I think that the equivalent for native software is a bit missing at the moment. The old answer would’ve been Visual Basic or HyperCard, but I’m not aware of anything filling those niches right now. PowerPoint with VBScript might honestly be closest.

                                  1. 1

                                    I would say that for a little GUI that just has to work somehow, Lazarus (Free Pascal Compiler based IDE following most of the Delphi design decisions) exists and works and is pretty simple to start using. It can also be used in more advanced ways, but that’s optional.

                                1. 4

                                  If one wants to learn concepts, isn’t it better to learn DVCS concepts first?

                                  (And for that I would recommend reading arbitrary Git, Mercurial, Fossil and Monotone tutorials in parallel, so that the differences and similarities are more visible)

                                  1. 4

                                    It looks like Bochs also has x86_64 emulation; and if you cannot use JIT or direct execution on the host CPU, maybe it is an option to look into.

                                      1. 1

                                        Can’t an attacker just replace the hash with their malicious hash?

                                        1. 1

                                          There’s only one hash. Most curl attacks use the user-agent, timing attacks, etc., so if the returned script is malformed or malicious, the hash would not match whatever’s advertised on the website. This is only applicable when you read the script before piping it to sh. If you pipe scripts without reading, it’s a lost case and there’s no way to stop anybody.

                                        2. 1

                                          Is there any threat model where curl-hashpipe-sh is safer than straight curl-sh (with HTTPS and basic partial-content precautions)?

                                          1. 1

                                            It makes sense when your browser’s connection to the package website is trustworthy but the connection you’re curling from isn’t trustworthy.

                                            Which, like, when does that happen? I put my browsers on jank WiFi more often than my servers, and if I can’t trust my server’s upstream why do I trust the install image or RAM?

                                          2. 1

                                            I started writing something similar a while ago but never finished it: https://github.com/zimbatm/curlsh

                                            The tricky bit is that because curl and bash are available almost everywhere, they are being used for bootstrapping. So that tool would also have to be distributed widely.

                                          1. 4

                                            You gave examples of “bad” commands, which is good, but I didn’t see any examples of “good” commands. This would make it difficult for the offenders to fix if they don’t know how to fix it.

                                            1. 5

                                              Safe, trustworthy software distribution is hard, and the mechanisms to do it right are diverse across platforms (unlike curl | sh …).

                                              The Right Way is usually something along the lines of…

                                              • Provide a self-contained package appropriate for the user’s platform
                                              • Along with a hash that the user can use to check the download
                                              • Or better yet, signed with a key which is verifiably owned by the developer
                                              • And in the best case, distributed and vetted by the platform’s official package manager or App Store

                                              All of which greatly improve’s user confidence in your software, but is a ton of work and probably a lot slower than just curling that shell script.

                                              (fwiw, I’m the kind of person who will build and sign an rpm to distribute a one-file script, but I get why people find the whole process tedious…)

                                              1. 2

                                                But then, hashes hosted on the same server as the file to download do not protect against any malicious action, RPM or DEB is less likely to be inspected before installation (that already runs arbitrary code) than a plain curl-to-sh script that can be downloaded, and fraction of users that will meaningfully verify signing key identity via independent means is low enough that Yarn providing signatures at all more or less saturates the real benefits.

                                                Oh, and by the way Yarn provides Debian packages, which is actually worse than their curl-to-sh option because adding their repository makes you vulnerable to future infrastructure intrusions, not just currently ongoing ones.

                                                Judging from «it has green padlock, it is safe!» around HTTPS, things like co-hosted hash can make the overall situation slightly worse.

                                              2. 1

                                                I don’t think there is a good command of the form curl ... | bash. It’s always a way to get a short term boost in downloads while polluting the commons for everyone.

                                                https://en.wikipedia.org/wiki/Tragedy_of_the_commons

                                                1. 0

                                                  You engage with trustworthy distributions to have your software packaged.

                                                1. 7

                                                  OK, I’ll get the cite training rolling with something easy to trump. Here’s Haskell two years ago. But I bet someone here can find an article about doing this in COBOL in 1964 or something.

                                                  For the details of the headline’s claim, looks like that starts around 28:10 in the video.

                                                  1. 3

                                                    Kitsune did it for C in OOPSLA 2012 and cites other “recent” systems, at least one from 2005. No doubt that one cites older works in turn.

                                                    1. 3

                                                      Initially, I was also kinda surprised by the “first” claim, given that I remember something like that in MSVC++ in some olden times, and given that I find his achievement super interesting and valuable even without the “first” claim. (In particular, I believe “hot code reloading” is sorely missing among “recent” non-interpreted/VM languages, and even in C++ it didn’t seem to spread outside MSVC++.)

                                                      But after viewing the presentation, and seeing that he does in fact reference the /hotpatch option of MSVC++, I’m starting to think he may be trying to highlight the ease of use of the feature, with close to no “special support” needed in one’s app, as well as its extensiveness/depth (as in “fully supporting all features of the language”). Notably, glancing through the Haskell link you provided, I seem to understand it requires some special code at pre-determined “decoupling points”, as well as that it has some important limitations (e.g.: “The main binary cannot persist any reference to code or data originating from the shared object, because that will prevent the GC from unloading the object.”). He doesn’t mention specifics, but I believe he claims that the /hotpatch was also limited in some important ways. In contrast, I believe he suggests his solution implemented for Nim is much easier to use and not limited in any important ways.

                                                      Personally, I think it might be worth considering if his achievement is not unlike how Dropbox was “not much more” than FTP ;)

                                                      1. 3

                                                        There are some approaches in Wikipedia’s Dynamic software updating and Hot swapping. Their references might have some older approaches.

                                                        Common Lisp and Smalltalk were obvious counterpoints if they allowed for bytecode. I couldn’t remember if CL was native or bytecode for hot swapping. The CL programmers on HN are saying it could do native hot swapping.

                                                        1. 2

                                                          I think CMUCL has been able to produce native code for decades (and Common Lisp specification requires hot swapping). Right now there are definitely multiple Common Lisp implementations that produce native code.

                                                      1. 2

                                                        Are these specifically made for this competition or are they found in the wild?

                                                        1. 3

                                                          Well, the expectation is that the submissions are written for the competition; and the code has to solve the posted task (which is not hard, but reduces the likely number of preexisting eligible code).

                                                          1. 3

                                                            The expectation is that they are made for the competition. We do hope that the teardowns can provide educational value, however, since we try to pick as winners code that is “bad” in a way we could imagine seeing for real (not just obfuscated, golf, etc)

                                                            1. 1

                                                              Oh, cool. If someone did commented braces for real, I would be so terrified lol

                                                          1. 7

                                                            Could somebody who has watched the video explain how «natively» is defined? I would consider some implementation of Common Lisp natively compiled, and they have code reloading in runtime.

                                                            1. 8

                                                              Seems to be used in the sense of “there are some hacks to accomplish this with c++ but they aren’t well integrated”.

                                                            1. 8

                                                              The attackers did not get in through a security flaw in Matrix itself, but via an outdated Jenkins.

                                                              The nice thing about Matrix is that it is federated (much like e-mail is); there’s no reason to use the matrix.org server. Instead, you can use your own trusted home server. Another reason to use your own is that the Matrix main server tends to be pretty overloaded so everything is quite slow.

                                                              1. 3

                                                                I mean, it doesn’t saying anything about the quality of the Matrix codebase as such, but some things do make you wonder about the level of understanding that the people working on it bring to it:

                                                                Attacker gains access to production infrastructure by hijacking a forwarded SSH agent logging into the compromised Jenkins worker

                                                                … and the corresponding Issues stop just short of overtly saying that that account had root access on all servers. The picture painted by that combination of a handful of facts makes me really wary…

                                                                1. 0

                                                                  It looks like the core parts of the protocol (including E2E encryption) could now be under takeover-in-progress by French national security agencies.

                                                                  New Vector doesn’t look likely to say no to one more reasonably-looking paid change request from France, and also not likely to find security implications if there is a nation-state effort to hide them. Some of the incentives are good, though (it was promised that French goverment agencies will have an option of limited external federation with mainline installations; the current public story even claims that the agencies will run the mainline server code).

                                                                  For some bugfixes in Synapse, it is worrying the the pre-fix state was widely deployed for months…

                                                                  1. 3

                                                                    So, speaking as the project lead for Matrix, this really isn’t true. ANSSI (the french cybersecurity agency) have not made any code contributions to the E2E crypto or the protocol itself. If they did make contributions, we’d audit them ourselves incredibly carefully, if we even accepted them at all. So far we’ve minimised the number of contributors to libolm (the E2E library) to a very small set of individuals.

                                                                    To be clear: we would be utterly stupid to sabotage Matrix by somehow giving any entity (whether that’s France, or New Vector, or Ericsson or whoever) a competitive advantage, no matter how much they offered to pay us. We aren’t building Matrix out of short-termist greed, but long-term altruism - to build an open global standard for comms. And we are not naive, and are very aware that some parties might try to sneak in trojan horses, and will do everything to fight them. We’ve spent a lot of time trying to codify this into the governance of the Matrix.org Foundation over at https://github.com/matrix-org/matrix-doc/blob/matthew/msc1779/proposals/1779-open-governance.md.

                                                                    Now, in terms of some of Synapse’s long-lived pre-1.0 bugs being concerning (e.g. state resets; fungible event IDs)… this is true. But we were fixing these independently of the French deployment as part of our existing 1.0 roadmap. The only difference is that we knew we had to have them fixed in time for France to go live. The actual design and solution and code was written entirely by us, and France does run off the same public synapse tree as everyone else and so got them at the same time with no privileged access.

                                                                    So, TL;DR: there is categorically not a takeover-in-progress, and it’s very unfortunate if it seems that way from the outside. Instead, DINSIC have been amazingly good open source citizens, and I only wish all government ministries operated like this.

                                                                    1. 1

                                                                      I am indeed surprised that none of the bugs that looked like hotfixes (and persisted for months) were from ANSSI audit. Interesting, thanks for commenting about that.

                                                                      I only consider the threat model of them managing to find and explain a real problem in a way that leads to a natural fix that has unintended complicated implications — but hopefully they don’t want to create a risk for something they will also be expected to run.

                                                                      I actually hoped they already started pushing the bug reports — a fresh set of qualified reviewers who have an actual motivation to find out if the protocol works has a good chance of being useful.

                                                                      Sorry for not wording some parts of my comment well, and thanks for the clarifications that they haven’t yet gave you the results of their (hopefully already ongoing) audit.

                                                                      1. 3

                                                                        So to be clear - there have been at least 3 audits, but primarily around the infrastructure of the deployment rather than the E2EE in matrix, mainly because Matrix is obviously still developing rapidly. Once there’s a full audit on the Matrix side I’d push for it to be public, just like our US-government-funded E2EE audit from 2016.

                                                                  1. 2

                                                                    Ugh. I’ve thought about opening up the Jenkins server on one of my open source projects to the world, but hesitate due to crap like this. Even though I update it regularly, the risk seems high just to have public build statuses (which I could probably just proxy through something else).

                                                                    I also hate how you have you mount the underlying machine’s docker socket to allow docker build agents. Surely there’s got to be a better user-mode docker solution.

                                                                    1. 2

                                                                      Yeah, that’s unfortunate. Maybe you could script it to generate some static artifacts and then drop them somewhere web accessible?

                                                                1. 2

                                                                  I guess the oldest code I have written lives in my .vimrc and provides some bindings. Argh, now that I am reminded I am tempted to recompile a litle puzzle game I wrote approximately at the same time (clone of a preexisting one, but with undo for misclicks) and play it a bit… That should be 2005 at the latest.

                                                                  A tool for customisable data acquisition from an ADC system that still works is probably ~2007 version, ~2003 version got obsoleted by ISA interfaces being hard to come by — but I think some code is shared between the two.

                                                                  A web system (coordinating education events, and processing the records, mostly) with a database name ending in 2010 (a rewrite sharing some code with 2009 version) is definitely still running. And handles hundreds of things per year, with a lot of participants (and reports). I know, I sometimes chat with the current maintainer and tell her what was the logic of the code nobody normally has to touch nowadays. But some functionality seems to have become «dark functionality» — it requires some knowledge to reach (only useful for a few qualified people anyway, and one can request a report of load-creating size), so either it entered the workflow (maybe as a yearly task), or people forgot it is even there. Sometimes I remind people it exists (in the context of something they want to do).

                                                                  1. 6

                                                                    There is a comparison to Plan9 and Hurd, but not to GenodeOS (which superficially seems more similar).

                                                                    I am not sure if Fuchsia brings anything actually good compared to Genode (and it would be interesting to see); of course, the adoption question is drivers, and Fuchsia will get adoption if Google forces manufacturers to support it, now without any problems around closed-source drivers.

                                                                    1. 8

                                                                      I heard more BeOS influence, due to the Be alumni working on it.

                                                                      1. 6

                                                                        And we don’t get a BeOS comparison, either.

                                                                        A bit like explaining Git while being careful to never compare it to preexisting DVCSes, only to Monotone — on the other hand, that approach usually works…

                                                                        (And I guess running Android apps on Fuchsia also means that security changes will matter more in terms of protecting manufacturer from consumers actually owning the device, than in providing consumers with securities, as most of the user-facing security problems are apps being happy to leak their entire state — although indeed it might also make it simpler for a custm build to provide fake sensitive data)

                                                                        1. 1

                                                                          Well, with no global file system, it should be easier for apps to have their own private storage to protect the customer from greedy data scouting.

                                                                          1. 2

                                                                            Hard to say how many vulnerabilities are exploited as single-app takeovers (these will probably stay in Fuchsia).

                                                                            On the other hand, cross-application interoperation becomes harder and users lose the ability to have file managers…

                                                                            (And some kind of cross-app message-passing will probably formally exist — inherited from intents — but will continue to require too much opt-in from apps to be widely useful — like OpenIntents that apparently didn’t even take off among apps explicitly wishing to ship in F-Droid)

                                                                          2. 1

                                                                            The speaker isn’t actually working on the OS so perhaps wasn’t aware of those comparisons to be made.

                                                                          3. 2

                                                                            If Fushia internals are remotely comparable to Be’s KernelKit then it’s a great architecture. I wrote an embedded kernel in grad school using Be’s documented Kernel API, and it’s extremely well designed. The VFS is a piece of art*; I still have fond memories implementing vfs_walk(). It’s a shame BeOS’ architecture is not better studied.

                                                                            *not technically part of the kernel and well detailed in Practical Filesystem Design de D. Giampaolo.

                                                                            1. 4

                                                                              It’s not really like BeOS, no; BeOS was a monokernel, and Fuchsia is a microkernel. There are some vague similarities, but not anything too major.

                                                                              On the other hand, Haiku is a complete re-implementation of BeOS including the kernel, and IMO our APIs are even more beautiful than Be’s … but I am biased, of course. :)

                                                                            2. 1

                                                                              And we don’t get a BeOS comparison, either.

                                                                              A bit like explaining Git while being careful to never compare it to preexisting DVCSes, only to Monotone — on the other hand, that approach usually works…

                                                                              (And I guess running Android apps on Fuchsia also means that security changes will matter more in terms of protecting manufacturer from consumers actually owning the device, than in providing consumers with securities, as most of the user-facing security problems are apps being happy to leak their entire state — although indeed it might also make it simpler for a custm build to provide fake sensitive data)

                                                                          1. 4

                                                                            It is time we did away with “publish or perish” and replace it with “publish and perish.” Nothing will be more blasphemous than writing a textbook that anyone can go out and buy. Those who commit such a crime will face expulsion and reprisals.

                                                                            Hppasus, a Pythagorean, was executed by his fellows for publicly revealing the existence of irrational numbers.

                                                                            (Okay, so it’s an apocryphal story, but since we don’t really know what happened, I choose to believe the more entertaining version.)

                                                                            1. 2

                                                                              If it happenned, the fact that irrational numbers are problematic for Pythagorean religious teachings might have played a role — it’s not just a random discovery he let out, it’s a very inconvenient one.

                                                                            1. 1

                                                                              Phone: K9

                                                                              Laptop: fdm → (custom scripts) → PostgreSQL → (custom FUSE tool — QueryFS) → Vim → mimedecode.py & ripmime & lynx –dump as needed

                                                                              My preferred protocol is POP3 (because I do not want syncing the «read» status between phone and desktop), I do use webmail from time to time, mainly to mark SPAM/not-SPAM for on-server spam filters.

                                                                              (considering migration from lynx –dump to a custom HTML dumper that I now have)

                                                                              1. 6

                                                                                https://i.imgur.com/9BK7QwL.png

                                                                                StumpWM. On the right: Vim with a grabbed textified copy of this page. I do read more of the web like that than in the browser. I indeed skimmed the list of the descriptions there and only started the browser (on the left) to upload an image to Imgur and write a comment here. I use Vim (and do not use Emacs).

                                                                                On the left — my Firefox setup. Minimal UI, tab bar auto-hidden even with multiple tabs (but there are just Lobsters and Imgur because I start a separate ns-isolated Firefox instance for every browsing tabs). Orange stripes in the address bar show that Marionette is enabled in this instance (I always enable it for script control of the browser — enabling Javascript as needed, dumping the pages that need some manual Javascript navigation, etc).

                                                                                On the bottom — my system information xterm. WiFi state, email count (fresh/deferred/long-term-deferred), web stream status (fresh/backlog), IP adresses, free RAM (free/free+caches), battery state, date, UTC and non-UTC time. (there are sometimes more timezones, and they are sometimes marked with 3-letter place codes). Email and web streams are fetched in the background and listed in a database. Then they are read in Vim via QueryFS that represents an SQL query to list the fresh entries as a directory of symlinks (deleting a symlink marks the entry as read).

                                                                                1. 4

                                                                                  Hi5, StumpWM buddy!

                                                                                  Here’s my desktop at work: https://i.imgur.com/cFUfqlG.png although it isn’t really showing work stuff but most of my virtual desktops have the same layout. I’ve got an ultrawide screen which I love but I could really use another monitor for keeping chat always visible.

                                                                                  Void Linux, StumpWM, Emacs + Slime + Evil (Vim keybindings), rxvt with unicode, Firefox (I wish it wasn’t bright white but it’s just too much of a hassle). Working on a Common Lisp version of my ex-boss’ @ahu Hello DNS.

                                                                                  As can be seen in the upper-right two sync services are running: pCloud and Dropbox. I transferred to pCloud a good while ago because of its price but had to revert to Dropbox for really important files like the password manager because I got too many sync conflicts. Still pCloud is really nice as a remote NAS.

                                                                                  1. 1

                                                                                    How would an extra monitor help? If you have an ultrawide monitor, you probably do have some space you could allocate for the IM (or not?). In StumpWM group switching is global anyway. That’s why I just use tagging instead…

                                                                                    1. 2

                                                                                      Oh darn, that’s true. I forgot about the group switching being global since I was still using i3 when on multiple monitors. Well, I’ll find a way.

                                                                                      Anyway, I’m pretty set in my ways on where I want my chat and it’s on a separate monitor :-)

                                                                                1. 9

                                                                                  pushcx desktop

                                                                                  Basically the same as 2015: I no longer have the bookmarks bar turned on in Firefox; I added caffiene-indicator to the systray for disabling the screensaver; I toned down my vim colorscheme to one based on jcs’s but with the LightItalic from Operator Mono for constants and comments. Still thinking of moving to xmonad, especially since July when awesomewm made some breaking changes to config files.

                                                                                  1. 3

                                                                                    DWM is the bestest… Awesome is a fork of it, but breaks things :D

                                                                                    1. 2

                                                                                      I recommend Xmonad, ive been using it for a lot of years and it never lets me down. make sure to add xmobar as well for a nice info bar.

                                                                                      Ill post my setup later today.

                                                                                      1. 2

                                                                                        What’s the extension on the left of firefox?

                                                                                        1. 1

                                                                                          Tree-Style Tab. I find the grouping of tabs by parent and being able to open and close folders very useful.

                                                                                          1. 1

                                                                                            How do you shrink down the main horizontal firefox tab bar?

                                                                                            1. 1

                                                                                              Either Tree-Style Tab removes it or I dragged it around in the usual Firefox appearance customizer.

                                                                                              1. 1

                                                                                                Hmm. Tree-Style Tabs doesn’t seem to change the main tab bar for me, and I can’t figure out a way to change it in the customizer.

                                                                                        2. 2

                                                                                          We had a nice chat about fonts in #lobsters in response to this thread and there were a bunch of mono italics I hadn’t heard of before: @zpojqwfejwfhiunz shared Dank Mono, @flyingfisch shared Plex, and @zdsmith shared PragmataPro. None cursive like Operator, but very nice fonts (except for the f in Dank mono).

                                                                                          1. 1

                                                                                            I can’t find any way to view the full size image of any of these imgur posts… and the page is full of random memes.

                                                                                            1. 1

                                                                                              If you load them in the browser, open the link twice (first time works as a redirect to the image page, but once you load the image page, further requests for the image itself give you indeed the image, regardless of the Referer: field). I use wget and Geeqie, so for me they load fine the first time.

                                                                                          1. 1

                                                                                            Many of the first run behaviours of Firefox can be disabled by pre-seeding preferences, in fact. Then, I wouldn’t trust nothing to leak through $HOME — previously, Flash plugin saved data did leak through a completely different path than Firefox profile.

                                                                                            I prefer to pre-seed configuratoin, run Firefox once to prepare a profile (second start is faster than the first), then run new instances with no access to the true /home/ using fresh copies of the prepared profile.