Threads for adrien

  1. 11

    Generating C++ is an underrated strategy! C++ has great debugging and performance tools like uftrace, perf, CLion, etc. And you can link with anything.

    It can be annoying to write, e.g. writing all the redundant headers, but generating code from a different language solves that.

    Oil is doing something similar: Brief Descriptions of a Python to C++ Translator

    It gives us much better performance, as well as memory safety. There were several reactions like “Why C++? It’s not memory safe”. But the idea is to generate code with that property, as SerenityOS is doing. If you never write free() or delete in your source language, then there are no double-free / double-delete bugs.

    (The boundary between manually written C++ and generated C++ is still an issue .. I’ll be interested see their API. But it should be nicer and more direct than the interface between two totally separate languages.)


    There are also some interesting ways to generate fast code with C++ templates: https://souffle-lang.github.io/translate

    i.e. you get a fairly nice language for writing a templatized “runtime”, and then you generate very high level typed code that targets it. Then the C++ compiler mushes it all together into fast code.

    Oil’s scheme is simpler, but it has this flavor. The garbage collector and templated types interact in an interesting way, e.g. consider tracing Dict<int, int> vs. Dict<int, Str*>. (I’m working on a blog post about this.)

    1. 5

      There were several reactions like “Why C++? It’s not memory safe”.

      As if assembly directly were any safer.

      1. 3

        D with the -betterC mode could also be a good target for transpilers. It’s got most of the good stuff from C++ and the imperative/inspection-based meta programming model could be easier to target, I think. Also has 3 compilers and all of them compile faster than C++ (no preprocessor etc). The only problem in Serenity’s case would be binding to existing C++ code seamlessly.

        1. 3

          Yeah D is nice because it’s garbage collected already! It is a shame that people seem to shy away for certain things because it is garbage collected, but I view that as a big benefit.

          I’m guessing SerenityOS has a good reason for ARC though … So yeah that is a problem – GC’s are hard to tune and aren’t universal across applications. There was a thread here recently with Chris Lattner on GC vs ARC.

          I think Nim lets you choose the allocator, but to me a big downside is that the generated code isn’t readable. It is a huge benefit to be able to read it, step through it in the debugger, pretty print data structures, etc.

          1. 7

            The -betterC subset is not GCed. It doesn’t have a runtime. However, the good stuff (RAII, meta-programming) is all there… it can be tedious to use by hand but as a transpiler target I think it is a good choice.

        2. 1

          Generating C++ is an underrated strategy

          I have a suspicion that the reason it’s underrated is that common old wisdom says “since I’m going to all the effort to generate code at all, maybe I generate C instead to get faster compile times.”

          I wonder to what extent that is no longer relevant? My belief is that there’s not an interesting difference now because most time goes into optimisers rather than parsing.

          C++ totally makes sense for this context of course because they already have a C++ codebase which they want to link with. :)

          1. 3

            Well I definitely ran into some template bloat problems, so it’s a double-edged sword. I think it basically comes down to whether you want to use templates, classes, and exceptions, etc. My post mentions that:

            http://www.oilshell.org/blog/2022/05/mycpp.html#why-generate-c-and-not-c

            Templates increase compile times but they can generate faster code, as the Souffle Datalog example shows. I would think of it as a very expressive and high level language for monomorphization.


            Also, when you generate code you have more control over the translation units, so the nonlinear explosion in compile times can be mitigated. Right now Oil has 24 translation units, and I see the redundancy in build times. I even preprocessed each one and counted the 10x blowup in line count:

            https://oilshell.zulipchat.com/#narrow/stream/121539-oil-dev/topic/Preprocessed.20Output.20.28ninja.20is.20fun.29 (requires login)

            So less than 100K lines of code blows up to about 1M once all the template headers are included. But I could probably cut it down to 6 or 8 translation units, and compile a linear amount of code. I think it would end up at 200K or less.

            But yeah the point is that once you generate code you can do stuff like that, whereas manually writing C++ you often get “stuck”. Most big C++ projects have bad build systems and bad code structure …

            1. 2

              Thanks for weighing in. <3

              Templates increase compile times but

              I was kinda thinking you wouldn’t use them in a code generator, since you can generate exactly the needed monomorphic versions yourself? but that might not take advantage of the C++ compiler’s ability to coalesce identical implementations sometimes.

              Also, when you generate code you have more control over the translation units…

              I was thinking that perhaps it might be possible to generate code with absolutely minimal headers / forward declarations too, which could help some more?

              1. 2

                Yeah I think it’s mainly about the implementation effort … Trying to recreate what C++ is doing could be the difference between a 1 year project and a 5 year one. I should know :)

                e.g. I liken it to one of the “wrong turns” I mentioned on the blog – to “just” write a Python bytecode interpreter. Well it turns out that is a huge project, and there’s no way to make it fast with a reasonable amount of effort.

                You can get something slow working, but getting good performance is a whole different story!

        1. 2

          As far as I’m concerned, I find there are few alternatives because of the software support which is often poor with other boards.

          I’ve recently found out about https://rpilocator.com (well, this website is quite recent, that’s why).

          Obviously this website doesn’t create any board but it’s very helpful to find some. When going that way, it’s good to prepare and check several shops along with their shipping costs because stocks don’t last very long.

          1. 4

            Rather than remove the package, wouldn’t it make more sense to replace it in order to poison it? That way every build pulling it could get a build error and a notice about the issue.

            1. 9

              Removing the package will also cause build errors for the (apparently very few) projects relying on it. Someone investigating why their build is failing will likely come across the info from the crates.io maintainers about this.

            1. 4

              So many cool Raspberry Pi projects; virtually no Raspberry PI boards available for purchase. I hope the stock situation gets better soon, it’s been dire for a long time. I’d sure hate to be a company that was producing a CM-based design right now…

              1. 1

                You can have a look at https://rpilocator.com to make it easier to find boards.

                As for companies, they actually get boards. See https://www.raspberrypi.com/news/production-and-supply-chain-update :

                Right now we feel the right thing to do is to prioritise commercial and industrial customers – the people who need Raspberry Pis to run their businesses – we’re acutely aware that people’s livelihoods are at stake. There is currently enough supply to meet the needs of those customers. (Read to the end if you’re in this position and are struggling.) Unfortunately this comes at the cost of constrained supply for individual customer, who might be looking to buy a small number for home projects or for prototyping.

                1. 1

                  I looked at rpilocator before making that post. I look at it every day. In the current state of play, it does not make it easier to find boards, it just makes it easier to confirm that there are no boards available.

                  1. 1

                    I got one thanks to that website and its RSS feed (bought in the UK for France, with hopefully no crazy taxes) but I agree that it’s very difficult to get boards. It took me at least a couple weeks to get a single one. Still it’s the best thing that we have for that.

              1. 2

                There is a linked blog post that contains more information: https://grafana.com/blog/2022/03/30/qa-with-our-ceo-about-grafana-mimir/ .

                TOC of the Q&A is below:

                • What is Grafana Mimir?
                • Why was this change made now?
                • So is Mimir a fork of Cortex? Why didn’t you add these features to Cortex?
                • Why is it called Grafana Mimir? How do you pronounce Mimir?
                • Why is Grafana Mimir launching with 2.0?
                • Can I run Grafana Mimir myself? Can others use Grafana Mimir to offer Prometheus as a Service?
                • How will the choice of the AGPLv3 license affect Mimir adoption?
                • What will happen to the Cortex project?
                • What does this mean for the Prometheus project?
                • Can other projects (such as Prometheus or Thanos) reuse code from Grafana Mimir?
                • What is dskit, and how does it relate to Grafana Mimir?
                • What does this mean for users of Grafana Cloud and Grafana Enterprise Metrics?
                • How do we choose when to add a feature to Grafana Mimir versus adding it to Grafana Enterprise Metrics?
                • Will we accept outside contributors or even outside maintainers for Grafana Mimir?
                • If I choose Grafana Mimir, will I be locked in? What options do I have to migrate away from Grafana Mimir in the future?
                1. 20

                  After I learned about “ci” in vim I got hooked. All of the sudden replacing text in quotes became as simple as ci” and now I’m having a hard time to use other editors. Sometimes a little detail is all that it takes.

                  1. 8

                    This was extremely helpful thanks.

                    Just to clarify to others. In vim if you are on a word “c” starts a change and the next keystroke determines what will be changed. For example, “c$” removes text from where the cursor is to the end of the line.

                    Now what is new for me is vim has a concept of “inner text”. Such as things in quotes, or inbetween any two symmetric symbols. The text between those two things are the “inner text”.

                    For example, in this line, we want to change the “tag stuff” to “anything”.

                    <tag style="tag stuff">Stuff</tag>
                    

                    Move the cursor anywhere between the quotes and type ci then a quote and you are left with

                    <tag style="">Stuff</tag>
                    
                    1. 8

                      This is a good example of why to me learning vi is not worth the trouble. In my normal editor, which does things the normal way, and does not have weird modes that require pressing a key before you are allowed to start typing and about which there are no memes for how saving and quitting is hard, I would remove the stuff in the quotes by doing cmd-shift-space backspace. Yes, that technically is twice as many key presses as Vi. No, there is no circumstance where that would matter. Pretty much every neat Vi trick I see online is like “oh if you do xvC14; it will remove all characters up to the semicolon” and then I say, it takes a similar number of keystrokes in my editor, and I even get to see highlight before it completes, so I’m not typing into a void. I think the thing is just that people who like to go deep end up learning vi, but it turns out if you go deep in basically any editor there are ways to do the same sorts of things with a similar number of keystrokes.

                      1. 14

                        There is not only the difference in the number of keystrokes but more importantly in ergonomics. In Vim I don’t need to hold 4 keys at once but I can achieve this by the usual flow of typing. Also things are coherent and mnemonic.

                        E.g. to change the text within the quotes I type ci”(change inner “) as the parent already explained. However this is only one tiny thing. You can do all the commands you use for “change(c)” with “delete(d)” or “yield(y)” and they behave the same way.

                        ci”: removes everything within the quotes and goes to insert mode di”: deletes everything within the quotes yi”: copies everything within the quotes

                        d3w, c3w, y3w would for example delete, replace or copy the next 3 words.

                        These are just the basics of Vim but they alone are so powerful that it’s absolutely worth to learn them.

                        1. 3

                          Just a small correction; I think you meant “yank(y)” instead of “yield(y)”.

                          1. 1

                            Haha yes thanks I really got confused :)

                          2. 2

                            And if you want to remove the delimiters too, you use ‘a’ instead of ‘i’ (I think the logic is that it’s a variation around ‘i’ like ‘a’ alone is).

                            Moreover, you are free to chose the pair of delimiters: “, ’, {}, (), [], and probably more. It even works when nested. And even with the nesting involves the same delimiter. foo(bar(“baz”)) and your cursor is on baz, then c2i) will let you change bar(“baz”) at once. You want visual mode stuff instead? Use v instead of c.

                            This goes on for a long time.

                          3. 6

                            One difference is that if you are doing the same edit in lots of places in your editor you have to do the cmd-shift-space backspace in every one, while in vi you can tap a period which means “do it again!” And the “it” that you are doing can be pretty fancy, like “move to the next EOL and replace string A with string B.”

                            1. 2

                              Sublime Text: ctrl+f search, ctrl+alt+enter select all results, then type your replacement.

                              1. 2

                                Yeah I just do CMD-D after selecting a line ending if I need to do something like that.

                            2. 3

                              I would remove the stuff in the quotes by doing cmd-shift-space backspace

                              What is a command-shift-space? Does it always select stuff between quotes? What if you wanted everything inside parentheses instead?

                              and then I say, it takes a similar number of keystrokes in my editor, and I even get to see highlight before it completes, so I’m not typing into a void

                              You can do it that way in vim too if you’re unsure about what you want, it’s only one keypress more (instead of ci" you do vi"c; after the " and before the c the stuff you’re about replace will be highlighted). You’re not forced to fly blind. Hell, if your computer is less than 30 years old you can probably just use the mouse to select some stuff and press the delete key and that will work too.

                              The point isn’t to avoid those modes and build strength through self-flagellation; the point is to enable a new mode of working where something like “replace this string’s contents” or “replace this function parameter” become part of your muscle memory and you perform them with such facility that you don’t need feedback on what you’re about to do because you’ve already done it and typed in the new value faster than you can register visual feedback. Instead of breaking it into steps, you get feedback on whether the final result is right, and if it isn’t, you just bonk u, which doesn’t even require a modifier key, and get back to the previous state.

                              1. 2

                                What if you wanted everything inside parentheses instead?

                                It is context sensitive and expands to the next context when you do it again.

                                Like I appreciate that vi works for other people but literally none of the examples I read ever make me think “I wish my editor did that”. It’s always “I know how I would do that in my editor. I’d just make a multiselection and then do X.” The really powerful stuff comes from using an LSP, which is orthogonal to the choice of editors.

                              2. 2

                                I do not disagree. For vim, as for your editor, the process is in both places somewhat complex.

                                Like you I feel I only want to learn one editor really well. So I choose the one which is installed by default on every system I touch.

                                For which I give up being able to preview what happens and some other niceties. Everything is a tradeoff in the end

                              3. 2

                                In a similar way, if you want to change the actual tag contents from “Stuff” to something else:

                                <tag style="tag stuff">Stuff</tag>
                                

                                you can use cit anywhere on the line (between the first < and the last >) to give you this (| is the cursor):

                                <tag style="tag stuff">|</tag>
                                

                                Or yit to copy (yank) the tag contents, dit to delete them etc.. You can also use the at motion instead of the it motion to include the rest of the tag: yat will yank the entire tag <tag style="tag stuff">Stuff</tag>.

                                Note that this only works in supported filetypes, html, xml etc., where vim knows to parse markup tags.

                              4. 2

                                I really like that I keep stumbling on tidbits like this one that continue to improve my workflow even further.

                              1. 2

                                Seems cool, but mp3? What year is it?

                                1. 4

                                  My guess is that it utilizes itch’s JS functionality to preview albums, like bandcamp. As a result, your browser needs to play the files, and probably FLAC isn’t as widely supported as mp3? Guessing, though…

                                  1. 3

                                    MP3 is indeed more widely supported in browsers but by a very small margin.

                                    https://caniuse.com/flac : 94% of marketshare support https://caniuse.com/mp3 : 96.6% of marketshare support

                                  2. 1

                                    what codecs do you prefer?

                                    1. 1

                                      FLAC if I’m paying money for music.

                                      Otherwise Opus

                                  1. 1

                                    It’s interesting that this says they’re going to enable compressed swap on all rpi4 devices. I’d think the larger memory systems don’t need it.

                                    1. 8

                                      Compression means spending CPU to reduce device IO. It makes sense if the storage device is slower (in relative terms) to the CPU. The amount of RAM isn’t really a factor - it decides how often the path executes, but not the optimal form of the path. I think what they are really saying is “SD cards are really slow, particularly for write.”

                                      1. 3

                                        SD cards also have endurance problems, making this an even better trade.

                                        1. 1

                                          Sure, personally I’ll just disable the swap completely on an rpi if my workload can fit comfortably in memory. Which in the case of 4/8GB is much more likely.

                                          1. 3

                                            It is almost always better to use swap to allow more memory to be used for bufcache.

                                            1. 2

                                              Isn’t the whole point of swap that it automatically uses your storage when your workload doesn’t fit in memory, and hardly costs anything when it does? I’m not sure you’re gaining anything by taking it into your own hands.

                                              1. 3

                                                I’d rather the OOM killer kill processes instead of adding extra wear on an SD card.

                                          2. 4

                                            This doesn’t write it to disk if possible. If compression frees up enough ram, it stays in ram. If not, it does write and read faster from storage…

                                            1. 3

                                              Enabling compressed swap doesn’t change how likely it is for something to be swapped. The logic is probably that if something is to be swapped, at least have it compressed first. Tuning swapiness would be a different topic.

                                              (and I think they could enable zram)

                                            1. 20

                                              Thermonuclear take: Why use a “TUI”? You’re crudely imitating a real GUI with the crippling limitations of a vt220, when you’re in an environment that can almost certainly handle a real GUI.

                                              1. 17

                                                The biggest reasons for me:

                                                • Low resource usage
                                                • I can run it on a different machine and SSH to it (e.g. IRC bouncer)

                                                (And for a combination of those two: I can run it on a low-powered machine like my home raspberry pi server…)

                                                1. 7

                                                  I’ve found that the richest TUIs are often very CPU heavy. Redrawing TUIs seems much more expensive than updating GUIs. It’s not very surprising since they’re not really meant for selective updates unlike current graphical technologies.

                                                2. 5

                                                  Terminals are an excellent example of a mixed bag. There’s a lot about terminals that is not great, perhaps most importantly the use of inband signalling for control sequences. That said, they’re also a testament to what we can achieve when we avoid constantly reinventing everything all the time.

                                                  There are absolutely limitations in the medium, but the limitations aren’t crippling or nobody would be getting anything done with terminal-based software. This is clearly just not true; people use a lot of terminal-based software to great effect all the time. Unlike most GUI frameworks, one even has a reasonable chance of building a piece of software that works the same way on lots of different platforms and over low-bandwidth or high-latency remote links.

                                                  1. 9

                                                    How are modern terminals not a case of reinventing? They’ve taken the old-school VT100 with its escape sequences and bolted on colors (several times), bold/italic/wide characters, mouse support, and so on. All of this in parallel with the development of GUIs, and mostly while running on top of an actual GUI.

                                                    I’m not denying there’s a benefit to having richer I/O in a CLI process where you’re in a terminal anyway, but a lot of the fad for TUI apps (Spotify? Really?) seems to me like hairshirt computing and retro fetishization.

                                                    If you’d asked 1986 me, sitting at my VT220 on a serial line to a VAX, whether I’d rather have a color terminal with a mouse or a Mac/Linux/Windows GUI desktop, I’d have thought you were crazy for even offering the first one.

                                                    1. 5

                                                      How are modern terminals not a case of reinventing? They’ve taken the old-school VT100 with its escape sequences and bolted on colors (several times), bold/italic/wide characters, mouse support, and so on. All of this in parallel with the development of GUIs, and mostly while running on top of an actual GUI.

                                                      I would not consider it reinventing because in many cases, at least when done well, you can still use these modern applications on an actual VT220. Obviously that hardware doesn’t provide mouse input, and is a monochrome-only device; but the control sequences for each successive new wave of colour support have generally been crafted to be correctly ignored by earlier or less capable terminals and emulators. Again, it’s not perfect, but it’s nonetheless an impressive display of stable designs: backwards compatibility and long-term incremental improvement with tangible results for users.

                                                      I’m not denying there’s a benefit to having richer I/O in a CLI process where you’re in a terminal anyway, but a lot of the fad for TUI apps (Spotify? Really?) seems to me like hairshirt computing and retro fetishization.

                                                      I’m not sure what you mean by “hairshirt” but it certainly sounds like intentionally loaded, pejorative language. I have been using the desktop Spotify application for a while, and it uses a lot of resources to be impressively sluggish and unreliable. I expect a terminal-based client would probably feel snappy and meet my needs. Certainly Weechat does a lot better for me than the graphical Slack or Element clients do.

                                                      I’m not going to make you use any of this software, but I would suggest that even if it is only a “fad”, who cares? If it makes people happy, and it hurts nobody, then people should probably just do it. Both graphical bitmap displays and character-cell terminals have been around for a long time; they both have pros and cons, and I don’t expect one size will ever fit all users or applications.

                                                  2. 4

                                                    That’s a very good question, honestly.

                                                    However, I haven’t seen any kind of graphical application (like, using the whole set of features gotten from full access to visual display) yet still being competely usable from keyboard only. Except Emacs, which is a very nice example, but I intentionally wanted to avoid any kind of text editors in this discussion.

                                                    After all, if I even stumble upon some sort of UI framework including full keyboard operation in REPL-style maner + shortcuts, showing various graphical data types (interactive tables, charts, data frames, scrollable windows, etc.) I’ll definitely test it thoroughly as long as it’s able to be shipped onto customers’ desktops (so yeah, Arcan is a suggestion, but not really fitting in current model of application deployment).

                                                    1. 6

                                                      Most GUI toolkits can be operated by keyboard? Windows was designed to be usable without a mouse, for instance.

                                                      I do note that GUI vs. CLI (and other things like mouse/keyboard dependency) isn’t a dichotomy. See: CLIM.

                                                      1. 1

                                                        A couple examples of the top of my head (though I’m not trying to make the case that all GUI apps can be driven this way, and there are tons of terrible GUI apps out there) that do offer full keyboard operation:

                                                        • IntelliJ IDEs
                                                        • Pan newsreader
                                                        • Nautilus file browser
                                                        • Evince PDF reader
                                                        • KeePassX

                                                        Those are just some apps I regularly use with no mouse usage at all.

                                                        1. 1

                                                          Most well-implemented Mac apps can be used keyboard-only, thanks to Apple’s accessibility features and lesser-known keyboard shortcuts like those for activating the menu bar.

                                                        2. 3

                                                          I think it’s for the same reason people write web GUI even if native GUI is generally superior.

                                                          1. 2

                                                            If I really mess up my Linux computer and I can’t get my window manager / X11 / Wayland to run, I can still get stuff done in TUIs while I attempt to fix it.

                                                            Also, while others point out low resource usage, I’ll specifically mention lack of GPU acceleration as a situation where I’d rather use a TUI. For example, software support for the GPU on my MNT Reform is spotty, which means some GUIs are painfully slow (e.g. Firefox crawls because WebRender doesn’t support the GPU), but there’s no noticeable difference in my terminal emulator.

                                                            1. 1

                                                              I currently do all my work sshed into my desktop in the office (combination of wfh and work’s security policies which mean I can’t access the code on my laptop). TUIs are great for that.

                                                              1. 1

                                                                Because TUI might be esoteric enough to avoid the attentions that might lead it into the same CADT that X11 got?

                                                                1. 4

                                                                  Unix retrofetishists love TUI stuff, so no.

                                                                  Besides, ncurses and the VT isn’t much better than X API-wise, anyways.

                                                                  1. 1

                                                                    Guess that’s true. Still want to shake my stick at them until they get off the grass.

                                                              1. 28

                                                                Why I appreciate the effort, I need to point out that SVG is basically a decompiled, XML-encoded version of the PDF graphic model (via PGML, a joint effort by Adobe, IBM and Netscape).

                                                                For the same of curbing the proliferation of new formats, I really would like to see a reduced PDF profile for icons (with an official compliance checker).

                                                                1. 31

                                                                  Yes. So many people hate on PDF without realizing that the core of it is (without exaggeration) the cleanest incarnation in widespread use of the following three things, layered on top of each other:

                                                                  1. an object graph, like JSON (but before JSON!)
                                                                  2. … that can be encoded in a compact/compressed binary form that allows seeking within the file, and
                                                                  3. a 2D vector graphics model with compositing

                                                                  There’s tons and tons of cruft on top of this: JavaScript, forms, 3D objects (!!), but the core of PDF is quite elegant. If you read the reference it was clearly started by people who knew what they were doing.

                                                                  SVG is an attempt to translate (3) into XML while throwing away (1) and (2).

                                                                  There is PDF/A, which is supposed to be restricted PDF for archival use, but I actually don’t know much about it. Adobe has tools which check for PDF/A compliance. A community tool to check for compliance to an even smaller PDF subset would be pretty cool and I think very feasible.

                                                                  1. 5

                                                                    Seems like there’s a lot of good reasons to “hate on” PDF, despite realizing that it’s quite clean at its core.

                                                                    1. 9

                                                                      It’s cat -v all over again. :-)

                                                                    2. 4

                                                                      an object graph, like JSON (but before JSON!)

                                                                      I don’t understand that part. JSON, the serialization format? It doesn’t even seem very fit for graphs to be honest.

                                                                      1. 3

                                                                        I’d have compared it to ASN.1 over JSON; I suspect @jyc was trying to go with something people were more familiar with.

                                                                    3. 5

                                                                      PDF is not gonna be the new popular vector graphics format. Maybe someone could make a separate “PDF for icons” standard (and make it an actual standard, not just proprietary Adobe garbage), but it would need a different name and a different extension. Image viewers must be able to tell the system, “I can open PDF icons but not PDFs”. The name “portable document format” is also just a misnomer for a vector graphics format.

                                                                      But honestly, I would prefer just a fresh format that’s not bogged down by all the crap that’s in PDF and that’s not affected by PDF’s or Adobe’s legacy. I’d take format proliferation any day over increased PDF proliferation.

                                                                      1. 14

                                                                        PDF is already an “actual standard”, ISO 32000. If you are implementing a tool that generates PDFs, you don’t need to use any “proprietary Adobe garbage.” If you are implementing a tool that renders PDFs, for 99% of PDFs you won’t need to use any “secret” proprietary extensions either (could you clarify what “proprietary Adobe garbage” you are referring to?)

                                                                        But honestly, I would prefer just a fresh format that’s not bogged down by all the crap that’s in PDF and that’s not affected by PDF’s or Adobe’s legacy.

                                                                        Why not just choose not to implement the parts of PDF that you don’t care about? You could even go off an existing PDF version and say “my renderer does not implement anything past PDF 1.2.” This is what all PDF viewers already do. It’s not dissimilar from a compiler saying “I support C++14 but not C++17.”

                                                                        Writing a new standard is all well and good until you realize you’ve just re-specced and re-implemented PDF 1.2 with new mistakes.

                                                                        1. 13

                                                                          could you clarify what “proprietary Adobe garbage” you are referring to?

                                                                          Sure:

                                                                          PDF 1.7, the sixth edition of the PDF specification that became ISO 32000-1, includes some proprietary technologies defined only by Adobe, such as Adobe XML Forms Architecture (XFA) and JavaScript extension for Acrobat, which are referenced by ISO 32000-1 as normative and indispensable for the full implementation of the ISO 32000-1 specification.[9] These proprietary technologies are not standardized and their specification is published only on Adobe’s website.[10][11][12][13] Many of them are also not supported by popular third-party implementations of PDF.

                                                                          (https://en.wikipedia.org/wiki/PDF#History)

                                                                          I don’t want anything to do with Adobe formats. Flash is dead, PDF ought to die too.

                                                                          Why not just choose not to implement the parts of PDF that you don’t care about? You could even go off an existing PDF version and say “my renderer does not implement anything past PDF 1.2.”

                                                                          The point of standards is that everyone implements them the same way. If every “PDF for icons” renderer implements a different subset of PDF, and every “PDF for icons” exporter uses a different subset of PDF, we don’t have a “PDF for icons” standard.

                                                                          Let’s just use a new, good vector format rather than try to bastardize an old, bloated, terrible, non-standard Adobe format.

                                                                          EDIT (since you edited your post):

                                                                          Writing a new standard is all well and good until you realize you’ve just re-specced and re-implemented PDF 1.2 with new mistakes.

                                                                          Sounds good to me. A reimplementation of the features of PDF 1.2, except with stuff like the PostScript legacy removed, assumptions about page sizes removed, support for multiple pages in a document removed, raster graphics support removed, with everything else that’s necessary for a document format but not an image format removed, with a radically simplified file format because we don’t have to consider extensibility, without the legacy of the PDF name to cause confusion? I love it.

                                                                          1. 18

                                                                            PDF 1.7, […] ISO 32000-1, includes some proprietary technologies defined only by Adobe, such as Adobe XML Forms Architecture (XFA) and JavaScript extension for Acrobat, which are referenced by ISO 32000-1 as normative and indispensable for the full implementation of the ISO 32000-1 specification.[9] These proprietary technologies are not standardized […]

                                                                            (https://en.wikipedia.org/wiki/PDF#History)

                                                                            If you scroll a bit further, you’ll see that PDF 1.7 was DOA.

                                                                            PDF 2.0, standardized as ISO 32000-2:2017

                                                                            eliminat[es] all proprietary elements, updating, enhancing and clarifying the documentation, and establish[es] tighter rules

                                                                            While we are talking about history, let’s talk about the fact that the full specification PDF 1.4 have been made available since 2001 by Adobe. https://www.adobe.com/content/dam/acom/en/devnet/pdf/pdfs/pdf_reference_archives/PDFReference.pdf

                                                                            And that the specifications for PDF/A (A = Archival subset) are available as ISO 19005-1:2005. Newer revisions of ISO 19005 (PDF/A-2, PDF/A-3) do reference PDF 1.7 and PDF 2.0 but specifically say that only their core functionalities are allowed in compliant documents. (From https://www.loc.gov/preservation/digital/formats/fdd/fdd000318.shtml: «The constraints for PDF/A-1, PDF/A-2, and PDF/A-3 include: * Audio and video content are forbidden. 3D artwork is also forbidden. * Javascript and executable file launches are prohibited. * All fonts must be embedded and also must be legally embeddable for unlimited, universal rendering. […]»

                                                                            I don’t want anything to do with Adobe formats. Flash is dead, PDF ought to die too.

                                                                            PDF 1.4 (as well as the core of PDF1.7 and PDF 2.0) is going to be supported until the end of the human civilization. It is incorporated in PDF/A and PDF/A is the standard archival format used and recommended by every national archive for long-term storage.

                                                                            1. 10

                                                                              From your own quote:

                                                                              Many of them are also not supported by popular third-party implementations of PDF.

                                                                              The point of standards is that everyone implements them the same way. If every “PDF for icons” renderer implements a different subset of PDF, and every “PDF for icons” exporter uses a different subset of PDF, we don’t have a “PDF for icons” standard.

                                                                              Your position is not different from “I don’t want to write C++14, because C++20 has some features I don’t like.” Or “I refuse to use cat, because these days Linux distros are using systemd.” The existence and utility of e.g. PDF 1.2 as a standard is not in any way affected by the existence of PDF 1.7, in the same way that the existence and utility of C++14 / cat is not in any way affected by the fact that newer “standards” built on top of those tools contain things you and I disagree with.

                                                                              I agree it’d be nice if Adobe stopped adding crap to PDF, but I can’t understand the conclusions you’re reaching from that premise.

                                                                              1. 3

                                                                                To my knowledge, there’s no PDF 1.2 standard. If there was, you could maybe have a point. But from what I can tell, 1.7 is the first version to get an associated standard. And even if there was a PDF 1.2 standard, you would still have to get people to adopt “PDF 1.2” as an image format unto itself, and to not just treat it as an old version of a document format. Image views would have to add PDF 1.2 support, browsers would have to add it to their IMG tags, OSes would need to learn to open PDF 1.2 documents in an image viewer rather than a PDF reader, etc… This is basically all the challenges met by a new image format, but complicated by the fact that there’s already a widely supported document format called PDF 1.2. How should Windows know whether a PDF 1.2 file is an image to be opened in an image viewer or a document to be opened in a PDF reader..?

                                                                                And this is all ignoring all the reasons why PDF 1.2 isn’t actually a good image format. I think I laid out some of those reasons in my comment.

                                                                                Repurposing PDF 1.2 to be the new widespread vector image format is just a terrible idea all around.

                                                                              2. 1

                                                                                You might want multiple pages to store dark mode and highlighted states.

                                                                        1. 4

                                                                          What’s painful is the following:

                                                                          % ps ax | grep -c firefox
                                                                          24
                                                                          

                                                                          No way to tell which ones actually use memory and whether I might want to reduce the number of processes created.

                                                                          My motivation for limiting memory use over other aspects is that I use a dedicated profile for work stuff that involves outlook web, teams web plus jira and confluence. I absolutely don’t care if something crash there but I care that these memory hogs are somehow constrained. They could even be twice slower if they used even 10% less memory. Right now with FF 95, I’m completely at loss regarding memory usage.

                                                                          1. 33

                                                                            Try any of

                                                                            • about:processes (to get an overview of the processes and what’s associated with which page)
                                                                            • about:performance (mostly about CPU usage)
                                                                            • about:unloads (to unload unused/busy tabs)
                                                                            • about:memory (the least user-friendly of them all, but quite insightful if you want to go deep down profiling)

                                                                            Unfortunately, we need this high amount of processes to mitigate Spectre vulnerabilities. See https://hacks.mozilla.org/2021/05/introducing-firefox-new-site-isolation-security-architecture/ for more

                                                                            1. 2

                                                                              Oh. I had already gone to about:processes but your comment made me spend more time in it and now I understand it better. TBH the UX could really be improved. At least the PID shouldn’t be only something at the end of the Name field because you might want to find by PID (if you’re looking at this because of something you’ve seen in another tool).

                                                                              What I’d like to know is what the current model is. It’s not one process per tab plus one process for each domain per frame for each tab. I have a single process for two of my tabs (same domain) and I have one process for each of for my three lobste.rs tabs.

                                                                              Also, is there a way to have more sharing or is that a thing of the past? In other words, is there any hope that my two awful outlook and teams tabs can share more?

                                                                              1. 1

                                                                                Good point. Would you be willing to file a bug?

                                                                                1. 2

                                                                                  Sure, will do.

                                                                              2. 2

                                                                                about:performance

                                                                                Interesting: I noticed Ghostery being quite active… even after having paused it. Not sure what it was doing, but I’m having none if it any more. NoScript and uBlock are probably enough anyway.

                                                                                1. 4

                                                                                  The more script and content blockers you have the more code will when a script is attempted to load. If performance is dear to you, set the Firefox builtin tracking protection to strict mode. It’s a bit more performant to do less JS/C++ context switches but it adds up.

                                                                                  When Firefox is dealing with the 70-90% of undesired scripts, you’re addons will be less busy

                                                                                  1. 2

                                                                                    I’ve always been under the impression that FF’s tracking protection ran after add-ons such as ublock because nothing is ever blocked on my machines while ublock blocks stuff.

                                                                                    1. 3

                                                                                      Normal mode doesn’t block at all. It loads stuff but with a separate, isolated cookie jar. This has shown as the best balance between privacy protection and web compatibility. It’s shown that most of our users would blame the browser if some important iframe doesn’t work or show up.

                                                                                      Now with a loading tab the user has something to interact with.

                                                                                      So gor power users the recommendation is to set it to “strict mode”, which doesn’t isolate but actually block.

                                                                                      1. 2

                                                                                        Good point, I’ve been wondering if FF blocks first, then addons, or vice-versa or even a mix of the two. Any thoughts?

                                                                                2. 6

                                                                                  Does ‘about:performance’ help in any way?

                                                                                1. 2

                                                                                  Finishing my own esp32 board: usable for prototyping but also usable in production (for personal projects).

                                                                                  It’s like a typical devboard but no on-board USB-TTL converter (they’re comparatively expensive, use board space with many additional components, and use precious power), some pins laid out several times in order to make connecting to SPI and I2C buses easier, and a battery connection but no on-board charge circuit.

                                                                                  1. 6

                                                                                    Impressive. When faced with a similar issue (laptop from 2015 with 4GB of RAM), I turned to zswap and then zram. Zram works incredibly well with 6:1 compression ratios. It seems web browsers data compresses particularly well (much better than it should).

                                                                                    1. 3

                                                                                      Hehe, kinda funny that in 2021 you can “download more RAM” for your computer!

                                                                                      1. 3

                                                                                        This idea is very old, e.g. in the 1990ies there was RAM Doubler for Macs:

                                                                                        https://tidbits.com/1996/10/28/ram-doubler-2/

                                                                                        Or Quarterdeck MagnaRAM for Windows 3.1:

                                                                                        https://en.wikipedia.org/wiki/QEMM#MagnaRAM

                                                                                        1. 1

                                                                                          It worked really well back then for day-to-day things like Office and Netscape… I remember a RAM Doubler/Speed Doubler combo making my PowerMac 7100 really noticeably snappier.

                                                                                          I really enjoyed this upgrade story, and laughed audibly at the conclusion:

                                                                                          I’ve now got an XPS13 with 16GB of memory.

                                                                                          But next time I think I’ll just buy the 16GB variant upfront.

                                                                                      2. 2

                                                                                        1:6 seems very reasonable given a lot of the content will be text which compresses very well. Then there will be a lot of UI memory which is also pretty repetitive. http://mattmahoney.net/dc/text.html

                                                                                      1. 35

                                                                                        I don’t really agree with a lot of the claims in the article (and I say this as someone who was very actively involved with XMPP when it was going through the IETF process and who wrote two clients and continued to use it actively until 2014 or so):

                                                                                        Truly Decentralized and Federated (meaning people from different servers can talk to each other while no central authority can have influence on another server unlike Matrix)

                                                                                        This is true. It also means that you need to do server reputation things if your server is public and you don’t want spam (well, it did for a while - now no one uses XMPP so no one bothers spamming the network). XMPP, unlike email, validates that a message really comes from the originating domain, but that doesn’t stop spammers from registering millions of domains and sending spam from any of them. Google turned off federation because of spam and the core problems remain unsolved.

                                                                                        End-To-End Encryption (unlike Telegram, unless you’re using secret chats)

                                                                                        This is completely untrue for the core protocol. End-to-end encryption is (as is typical in the XMPP world) multiple, incompatible, extensions to the core protocol and most clients don’t support any of them. Looking at the list of clients almost none of them support the end-to-end encryption XEP that the article recommends. I’d not looked at XEP-0384 before, but a few things spring to mind:

                                                                                        • It’s not encrypting any metadata (i.e. the stuff that the NSA thinks is the most valuable bit to intercept), this is visible to the operators of both party’s servers.
                                                                                        • You can’t encrypt presence stanzas (so anything in your status message is plaintext) without breaking the core protocol.
                                                                                        • Most info-query stanzas will need to be plain-text as well, so this only affects direct messages, but some client-to-client communication is via pub-sub. This is not necessarily encrypted and clients may or may not expose which things are and aren’t encrypted to the user.
                                                                                        • The bootstrapping thing involves asking people to trust new fingerprints that exist. This is a security-usability disaster: users will click ‘yes’. Signal does a good job of ensuring that fingerprints don’t change across devices and manages key exchange between clients so that all clients can decrypt a message encrypted with a key assigned to a stable identity. OMEMO requires a wrapped key for every client.
                                                                                        • The only protection against MITM attacks is the user noticing that a fingerprint has changed. If you don’t validate fingerprints out-of-band (again, Signal gives you a nice mechanism for doing this with a QR code that you can scan on the other person’s phone if you see them in person) then a malicious server can just advertise a new fingerprint once and now you will encrypt all messages with a key that it can decrypt.
                                                                                        • There’s no revocation story in the case of the above. If a malicious fingerprint is added, you can remove it from the advertised set, but there’s no guarantee that clients will stop sending things encrypted with it.
                                                                                        • The XEP says that forward secrecy is a requirement and then doesn’t mention it again at all.
                                                                                        • There’s no sequence counter or equivalent so a server can drop messages without your being aware (or can reorder them, or can send the same message twice - no protection against replay attacks, so if you can make someone send a ‘yes it’s fine’ message once then you can send it in response to a request to a different question).
                                                                                        • There’s no padding, so message length (which provides a lot of information) is available.

                                                                                        This is without digging into the protocol. I’d love to read @soatok’s take on it. From a quick skim, my view is that it’s probably fine if your threat model is bored teenagers.

                                                                                        They recommend looking for servers that support HTTP upload, but this means any file you transfer is stored in plain text on the server.

                                                                                        Cross-Platform Applications (Desktop, Web, and Mobile)

                                                                                        True, with the caveat that they have different feature sets. For example, I tried using XMPP again a couple of years ago and needed to have two clients installed on Android because one could send images to someone using a particular iOS client and the other supported persistent messaging. This may be better now.

                                                                                        Multi-Device Synchronization (available on some servers)

                                                                                        This, at least, is fairly mature. There are some interesting interactions between it and the security gurantees claimed by OMEMO.

                                                                                        Voice and Video Calling (available on most servers)

                                                                                        Servers are the easy part (mostly they do STUN or fall back to relaying if they need to). There are multiple incompatible standards for voice and video calling on top of XMPP. The most widely supported is Jingle which is, in truly fractal fashion, a family of incompatible standards for establishing streams between clients and negotiating a CODEC that both support. It sounds as if clients can now do encrypted Jingle sessions from their article. This didn’t work at all last time I tried, but maybe clients have improved since then.

                                                                                        1. 8

                                                                                          Strongly agree – claiming that XMPP is secure and/or private without mentioning all the caveats is surprising! There’s also this article from infosec-handbook.eu outlining some of the downsides: XMPP: Admin-in-the-middle

                                                                                          The state of XMPP security is a strong argument against decentralization in messengers, in my opinion.

                                                                                          1. 7

                                                                                            Spam in XMPP is largely a solved problem today. Operators of open relays, servers where anyone can create an account, police themselves and each other. Anyone running a server that originates spam without dealing with it gets booted off the open federation eventually.

                                                                                            Another part of the solution is ensuring smaller server operators don’t act as open relays, but instead use invites (like Lobste.rs itself). Snikket is a great example of that.

                                                                                            but that doesn’t stop spammers from registering millions of domains and sending spam from any of them.

                                                                                            Bold claim. Citation needed. Where do you register millions of domains cheaply enough for the economics of spam to work out?

                                                                                            Domains tend to be relatively expensive and are easy to block, just like the IP addresses running any such servers. All I hear from server operators is that spammers slowly register lots of normal accounts on public servers with open registration, which are then used once for spam campaigns. They tend to be deleted by proactive operators, if not before, at least after they are used for spam.

                                                                                            Google turned off federation because of spam and the core problems remain unsolved.

                                                                                            That’s what they claim. Does it really seem plausible that Google could not manage spam? It’s not like they have any experience from another federated communications network… Easier for me to believe that there wasn’t much in the way of promotion to be gained from doing anything more with GTalk, so they shut it down and blamed whatever they couldn’t be bothered dealing with at the time.

                                                                                            1. 3

                                                                                              Your reasonning about most clients not supporting OMEMO is invalid because noone cares about most clients: it’s all about the marketshare. Most XMPP clients probably don’t support images but that doesn’t matter.

                                                                                              For replays, this may be dealt with the double ratchet algorithm since the keys change fairly often. Your unknown replay would also have to make sense in an unknown conversation.

                                                                                              Forward secrecy could be done with the double ratchet algorithm too.

                                                                                              Overall OMEMO should be very similar to Signal’s protocol, which means that it’s quite likely the features and flaws of one are in the other.

                                                                                              Conversations on Android also offers showing and scanning QR codes for validation.

                                                                                              As for HTTP upload, that’s maybe another XEP but there’s encrypted upload with an AES key and a link using the aesgcm:// scheme (as you can guess: where to retrieve the file plus the key).

                                                                                              I concur that bootstrapping is often painful. I’m not sure it’s possible to do much better without a centralized system however.

                                                                                              Finally, self-hosting leads to leaking quite a lot of metadata because your network activity is not hidden in large amounts of network activity coming from others. I’m not sure that there’s really much more that is available by reading the XMPP metadata. Battery saving on mobile means the device needs to tell the server that it doesn’t care about status messages and presence from others but who cares if it’s unencrypted to the server (on the wire, there’s TLS) since a) it’s meant for the server, b) even if for clients instead, you could easily spot the change in network traffic frequency. I mean, I’m not sure there’s a lot more that is accessible that way (not even mentionning that if you’re privacy-minded, you avoid stuff like typing notifications and if you don’t, traffic patterns probably leak that anyway). And I’m fairly sure that’s the same with Signal for many of these.

                                                                                              1. 3

                                                                                                now no one uses XMPP so no one bothers spamming the network

                                                                                                I guess you’ve been away for awhile :) there is definitely spam, and we have several community groups working hard to combat it (and trying to avoid the mistakes of email, not doing server/ip rep and blocking and all that)

                                                                                                1. 3
                                                                                                  Cross-Platform Applications (Desktop, Web, and Mobile)
                                                                                                  

                                                                                                  True, with the caveat that they have different feature sets. For example, I tried using XMPP again a couple of years ago and needed to have two clients installed on Android because one could send images to someone using a particular iOS client and the other supported persistent messaging. This may be better now.

                                                                                                  Or they’ve also calcified (see: Pidgin). Last time I tried XMPP a few years ago, Conversations on Android was the only tolerable one, and Gajim was janky as hell normally, let alone on Windows.

                                                                                                  1. 3

                                                                                                    True, with the caveat that they have different feature sets. For example, I tried using XMPP again a couple of years ago and needed to have two clients installed on Android because one could send images to someone using a particular iOS client and the other supported persistent messaging. This may be better now.

                                                                                                    This was the reason I couldn’t get on with XMPP. When I tried it a few years ago, you really needed quite a lot of extensions to make a good replacement for something like WhatsApp, but all of the different servers and clients supported different subsets of the features.

                                                                                                    1. 3

                                                                                                      I don’t know enough about all the details of XMPP to pass technical judgement, but the main problems never were the technical decisions like XML or not.

                                                                                                      XMPP had a chance, 10-15 years ago, but either because of poor messaging (pun not intended) or not enough guided activism the XEP thing completely backfired and no two parties really had a proper interaction with all parts working. XMPP wanted to do too much and be too flexible. Even people who wanted it to succeed and run their own server and championed for use in the companies they worked for… it was simply a big mess. And then the mobile disaster with undelivered messages to several clients (originally a feature) and apps using up to much battery, etc.pp.

                                                                                                      Jitsi also came a few years too late, sadly, and wasn’t exactly user friendly either at the start. (Good people though, they really tried).

                                                                                                      1. 6

                                                                                                        I don’t know enough about all the details of XMPP to pass technical judgement, but the main problems never were the technical decisions like XML or not.

                                                                                                        XML was a problem early on because it made the protocol very verbose. Back when I started working on XMPP, I had a £10/month plan for my phone that came with 40 MB of data per month. A few extra bytes per message added up a lot. A plain text ‘hi’ in XMPP was well over a hundred bytes, with proprietary messengers it was closer to 10-20 bytes. That much protocol overhead is completely irrelevant now that phone plans measure their data allowances in GB and that folks send images in messages (though the requirement to base64-encode images if you’re using in-band bytestreams and not Jingle still matters) but back then it was incredibly important.

                                                                                                        XMPP was also difficult to integrate with push notifications. It was built on the assumption that you’d keep the connection open, whereas modern push notifications expect a single entity in the phone to poll a global notification source periodically and then prod other apps to make shorter-lived connections. XMPP requires a full roster sync on each connection, so will send a couple of megs of data if you’ve got a moderately large contact list (first download and sync the roster, then get a presence stanza back from everyone once you’re connected). The vcard-based avatar mechanism meant that every presence stanza contained the base64-encoded hash of the current avatar, even if the client didn’t care, which made this worse.

                                                                                                        A lot of these problems could have been solved by moving to a PubSub-based mechanism, but PubSub and Personal Eventing over PubSub (PEP) weren’t standardised for years and were incredibly complex (much more complex than the core spec) and so took even longer to get consistent implementations.

                                                                                                        The main lessons I learned from XMPP were:

                                                                                                        • Federation is not a goal. Avoiding having an untrusted admin being able to intercept / modify my messages is a goal, federation is potentially a technique to limit that.
                                                                                                        • The client and server must have a single reference implementation that supports anything that is even close to standards track, ideally two. If you want to propose a new extension then you must implement it at least once.
                                                                                                        • Most users don’t know the difference between a client, a protocol, and a service. They will conflate them, they don’t care about XMPP, they care about Psi or Pidgin - if the experience isn’t good with whatever client you recommend that’s the end.
                                                                                                        1. 2

                                                                                                          XMPP requires a full roster sync on each connection, so will send a couple of megs of data if you’ve got a moderately large contact list (first download and sync the roster, then get a presence stanza back from everyone once you’re connected).

                                                                                                          This is not accurate. Roster versioning, which means that only roster deltas, which are seldom, are transferred, is used widely and also specified in RFC 6121 (even though, not mandatory to implement, but given that it’s easy to implement, I am not aware of any mobile client that doesn’t use it)

                                                                                                          1. 1

                                                                                                            Also important to remember that with smacks people are rarely fully disconnected and doing a resync.

                                                                                                            Also, the roster itself is fully optional. I consider it one of the selling points and would not use it for IM without, but nothing prevents you.

                                                                                                            1. 1

                                                                                                              Correct.

                                                                                                              I want to add that, it may be a good idea to avoid using XMPP jargon to make the test more accessible to a wider audience. Here ‘smacks’ stands for XEP-198: Stream Management.

                                                                                                        2. 2

                                                                                                          XMPP had a chance, 10-15 years ago, but either because of poor messaging (pun not intended) or not enough guided activism the XEP thing completely backfired and no two parties really had a proper interaction with all parts working. XMPP wanted to do too much and be too flexible.

                                                                                                          I’d argue there is at least one other reason. XMPP on smartohones was really bad for a very long time, also due to limitations on those platforms. This only got better later. For this reason having proper messaging used to require spending money.

                                                                                                          Nowadays so you “only” need is too pay a fee to put stuff into the app store and in case of iOS development buy an overpriced piece of hardware to develop on. Oh and of course deal with a horrible experience there and be at the risk of your app being banned from the store, when they feel like. But I’m drifting off. In short: Doing what the Conversation does used to be harder/impossible on both Android and iOS until certain APIs were added.

                                                                                                          I think that gave it a pretty big downturn when it started to do okay on the desktop.

                                                                                                          I agree with the rest though.

                                                                                                        3. 2

                                                                                                          I saw a lot of those same issues in the article. Most people don’t realize (myself included until a few weeks ago) that when you stand up Matrix, it still uses matrix.org’s keyserver. I know a few admins who are considering standing up their own keyservers and what that would entail.

                                                                                                          And the encryption thing too. I remember OTR back in the day (which was terrible) and now we have OMEMO (which is ….. still terrible).

                                                                                                          This is a great reply. You really detailed a lot of problems with the article and also provided a lot of information about XMPP. Thanks for this.

                                                                                                          1. 2

                                                                                                            It’s not encrypting any metadata (i.e. the stuff that the NSA thinks is the most valuable bit to intercept), this is visible to the operators of both party’s servers. You can’t encrypt presence stanzas (so anything in your status message is plaintext) without breaking the core protocol.

                                                                                                            Do you know if this situation is any better on Matrix? Completely honest question (I use both and run servers for both). Naively it seems to me that at least some important metadata needs to be unencrypted in order to route messages, but maybe they’re doing something clever?

                                                                                                            1. 3

                                                                                                              I haven’t looked at Matrix but it’s typically a problem with any Federated system: you need at least an envelope that tells you the server that a message needs to be routed to to be public. Signal avoids this by not having federation and by using their sealed-sender mechanism to avoid the single centralised component from knowing who the sender of a message is.

                                                                                                              1. 1

                                                                                                                Thanks.

                                                                                                              2. 1

                                                                                                                There is a bit of metadata leaking in matrix, because of federation. But it’s something the team is working to improve.

                                                                                                              3. 2

                                                                                                                Fellow active XMPP developer here.

                                                                                                                I am sure you know that some of your points, like Metadata encryption, are a deliberate design tradeoff. Systems that provide full metadata encryption have other drawbacks. Other “issues” you mention to be generic and apply to most (all?) cryptographic systems. I am not sure why XEP-0384 needs to mention forward secrecy again, given that forward secrecy is provided by the building blocks the XEP uses and discussed there, i.e., https://www.signal.org/docs/specifications/x3dh/. Some points of yous are also outdated and no longer correct. For example, since the newest version of XEP-0384 uses XEP-0420, there is now padding to disguise the actual message length (XEP-0420 borrows this again from XEP-0373: OpenPGP for XMPP).

                                                                                                                From a quick skim, my view is that it’s probably fine if your threat model is bored teenagers.

                                                                                                                That makes it sound like your threat model shouldn’t be bored teenagers. But I believe that we should also raise the floor for encryption so that everyone is able to use a sufficiently secured connection. Of course, this does not mean that raising the ceiling shouldn’t be researched and tried also. But we, that is, the XMPP community of volunteers and unpaid spare time developers, don’t have the resources to accomplish everything in one strike. And, as I said before, if you need full metadata encryption, e.g., because you are a journalist in a suppressive regime, then the currently deployed encryption solutions in XMPP are probably not what you want to use. But for my friends, my family, and me, it’s perfectly fine.

                                                                                                                They recommend looking for servers that support HTTP upload, but this means any file you transfer is stored in plain text on the server.

                                                                                                                That depends on the server configuration, doesn’t it? I imagine at least some servers use disk or filesystem-level encryption for user-data storage.

                                                                                                                For example, I tried using XMPP again a couple of years ago and needed to have two clients installed on Android because one could send images to someone using a particular iOS client and the other supported persistent messaging. This may be better now.

                                                                                                                It got better. But yes, this is the price we pay for the modularity of XMPP due to its extensibility. I also believe it isn’t possible to have it any other way. Unlike other competitors, most XMPP developers are not “controlled” by a central entity, so they are free to implement what they believe is best for their project. But there is also a strong incentive to implement extensions that the leading Implementations support for compatibility. So there are some checks and balances in the system.

                                                                                                              1. 41

                                                                                                                Nice idea – would be great if there were multiple example languages to choose from (python, c++, something functional).

                                                                                                                And perhaps a “blind test” mode where you choose A or B so you are not biased by the fonts you know.

                                                                                                                1. 13

                                                                                                                  And perhaps a “blind test” mode where you choose A or B so you are not biased by the fonts you know.

                                                                                                                  Hear, hear.

                                                                                                                  I tried to get the window set up just right to avoid seeing the names, but it was tough.

                                                                                                                  1. 34

                                                                                                                    Your voice are heard! I added a new toggle button to hide the font names! Try refresh / hard refresh the page to find a toggle for “Blind Mode”

                                                                                                                    1. 10

                                                                                                                      I’d love to be able to see the full tournament bracket after doing a whole run in blind mode, so I can see what my second, third etc. choices were.

                                                                                                                      1. 4

                                                                                                                        Ah yeah that’s what I was saying as well. At least the 2nd-place runner-up, but yeah showing the full ladder would be great too!

                                                                                                                  2. 4

                                                                                                                    There is a “blind match” button at the bottom of the page.

                                                                                                                    1. 14

                                                                                                                      wow, you found it while I was developing the feature – i am being crappy not having a dev site and everything is done to the live site! I finished developing it, now it is moved from the bottom to a more prominent place on the page!

                                                                                                                      1. 2

                                                                                                                        That probably explains why it seemed that button wasn’t there the first time I loaded the page. ;-)

                                                                                                                        And well, you probably don’t need a QA server right now considering that you apparently didn’t (visibly) break the website while doing your changes!

                                                                                                                    2. 4

                                                                                                                      One thing I noticed was that there was only one pair of parentheses. A few of the fonts had braces that looked really like parentheses but it wasn’t obvious from just looking at the text. Something with nested brackets would make this much easier to spot. Similarly, most of these fonts made it easy to tell 0 and O apart, but I don’t know how many of them made it easy to distinguish 1 and I or I and l because the sample text didn’t have these characters nearby.

                                                                                                                      It would be a bit more robust if it showed things more than once. There were a couple of fonts in the list that were almost identical (different shape of 5, most other glyphs basically the same). A few of them happened to look really good or bad at 16pt with my monitor size and antialiasing mode (Ubuntu Mono, in particular, looked terrible) but might be very different at different sizes. Once you’ve made a selection against something though, it’s gone forever, so you don’t get to find a ranking of preferred fonts.

                                                                                                                      It would also be good if it didn’t tell me the name of the font until after I made my choice. My favourite according to this is Adobe’s Source Code Pro. Purely by coincidence, that’s the font that I have installed for all of my terminals to use. Or possibly seeing the name gave me a positive bias towards it that I wouldn’t have seen if I’d been comparing it without knowing the name.

                                                                                                                    1. 1

                                                                                                                      The benchmark here relies heavily on performance per watt… but Apple has never actually released the power usage of their chip, so the guess of 60 watts is just a guess. Intel, AMD, and nVidia are certainly not fully transparent in everything they do, but at least they release basic performance numbers on their chips and publish architecture documents.

                                                                                                                      1. 1

                                                                                                                        Indeed: a power meter is a must. Intel’s TDP is absolutely not reliable nowadays since their CPUs can roughly use twice as much power; AMD goes over too but not as much. And Apple probably doesn’t do better.

                                                                                                                        1. 1

                                                                                                                          Outside of microcontrollers (where they don’t do it anywhere as much), the manufacturers absolutely bullshit their public TDP numbers. It has been this way for at least a decade.

                                                                                                                          Only third party analysis of the actual power drawn in a variety of benchmark situations can be somewhat trusted.

                                                                                                                      1. 8

                                                                                                                        BTW, is bsdiff still the state of the art?

                                                                                                                        1. 7

                                                                                                                          Pretty much, yes. https://github.com/divvun/bidiff is a variation on the theme, but only in that it uses zstd instead of bzip2 for the final compression pass and consumes less memory.

                                                                                                                          1. 1

                                                                                                                            The memory consumption of bsdiff is something that I’ve long had trouble with. Well, now I have much more RAM but the usage is really high anyway. Do you have more detailled numbers for bidiff? The github page is light on details for memory usage.

                                                                                                                            1. 2

                                                                                                                              It uses 5 × old file size.

                                                                                                                        1. 3

                                                                                                                          Ubuntu 21.10 brings the all-new PHP 8 and GCC 11 including full support for static analysis

                                                                                                                          Why is PHP of all things suddenly the headliner?

                                                                                                                          1. 6

                                                                                                                            PHP 8 is much faster. That’s pretty good for something that’s basically old and boring tech nowadays.

                                                                                                                            1. 3

                                                                                                                              Going purely off memory here, but doesn’t Wikipedia run on Ubuntu and use PHP?

                                                                                                                              1. 2

                                                                                                                                PHP is still pretty massive.

                                                                                                                                1. 1

                                                                                                                                  brings the all-new PHP 8

                                                                                                                                  also, the next major release 8.1 is about to be released in about a month. I don’t think “all new” is a valid qualifier any more.

                                                                                                                                1. 25

                                                                                                                                  Fascinating read. Audio was the thing that made me switch from Linux to FreeBSD around 2003. A bit before then, audio was provided by OSS, which was upstream in the kernel and maintained by a company that sold drivers that plugged into the framework. This didn’t make me super happy because those drivers were really expensive. My sound card cost about £20 and the driver cost £15. My machine had an on-board thing as well, so I ended up using that when I was running Linux.

                                                                                                                                  A bit later, a new version of OSS came out, OSS 4, which was not released as open source. The Linux developers had a tantrum and decided to deprecate OSS and replace it with something completely new: ALSA. If your apps were rewritten to use ALSA they got new features, but if they used OSS (as everything did back then) they didn’t. There was only one feature that really mattered from a user perspective: audio mixing. I wanted two applications to be able both open the sound device and go ‘beep’. I think ALSA on Linux exposed hardware channels for mixing if your card supported it (my on-board one didn’t), OSS didn’t support it at all. I might be misremembering and ALSA supported software mixing, OSS only hardware mixing. Either way, only one OSS application could use the sound device at the time and very few things had been updated to use ASLA.

                                                                                                                                  GNOME and KDE both worked around this by providing userspace sound mixing. These weren’t great for latency (sound was written to a pipe, then at some point later the userspace sound daemon was scheduled and then did the mixing and wrote the output) but they were fine for going ‘bing’. There was just one problem: I wanted to use Evolution (GNOME) for mail and Psi (KDE) for chat. Only one out of the KDE and GNOME sound daemons could play sound at a time and they were incompatible. Oh, and XMMS didn’t support ALSA and so if I played music the neither of them could do audio notifications.

                                                                                                                                  Meanwhile, the FreeBSD team just forked the last BSD licensed OSS release and added support for OSS 4 and in-kernel low-latency sound mixing. On FreeBSD 4.x, device nodes were static so you had to configure the number of channels that it exposed but then you got /dev/dsp.0, /dev/dsp.1, and so on. I could configure XMMS and each of the GNOME and KDE sound daemons to use one of these, leaving the default /dev/dsp (a symlink to /dev/dsp.0, as I recall) for whatever ran in the foreground and wanted audio (typically BZFlag). When FreeBSD 5.0 rolled out, this manual configuration went away and you just opened /dev/dsp and got a new vchan. Nothing needed porting to use ALSA, GNOME’s sound daemon, KDE’s sound daemon, PulseAudio, or anything else: the OSS APIs just worked.

                                                                                                                                  It was several years before audio became reliable on Linux again and it was really only after everything was, once again, rewritten for PulseAudio. Now it’s being rewritten for PipeWire. PipeWire does have some advantages, but there’s no reason that it can’t be used as a back end for the virtual_oss thing mentioned in this article, so software written with OSS could automatically support it, rather than requiring the constant churn of the Linux ecosystem. Software written against OSS 3 20 years ago will still work unmodified on FreeBSD and will have worked every year since it was written.

                                                                                                                                  1. 8

                                                                                                                                    everything was, once again, rewritten for PulseAudio. Now it’s being rewritten for PipeWire

                                                                                                                                    Luckily there’s no need for such a rewrite because pipewire has a PulseAudio API.

                                                                                                                                    1. 1

                                                                                                                                      There was technically no need for a rewrite from ALSA to PulseAudio, either, because PulseAudio had an ALSA compat module.

                                                                                                                                      But most applications got a PulseAudio plug-in anyway because the best that could be said about the compat module is that it made your computer continue to go beep – otherwise, it made everything worse.

                                                                                                                                      I am slightly more hopeful for PipeWire, partly because (hopefully) some lessons have been drawn from PA’s disastrous roll-out, partly for reasons that I don’t quite know how to formulate without sounding like an ad-hominem attack (tl;dr some of the folks behind PipeWire really do know a thing or two about multimedia and let’s leave it at that). But bridging sound stacks is rarely a simple affair, and depending on how the two stacks are designed, some problems are simply not tractable.

                                                                                                                                      1. 2

                                                                                                                                        One could also say that a lot of groundwork was done by PulseAudio, revealing bugs etc so the landscape that PipeWire enters in 2021 is not the same that PulseAudio entered in 2008. For starters there’s no Arts, ESD etc. anymore, these are long dead and gone, the only thing that matters these days is the PulseAudio API and the JACK API.

                                                                                                                                        1. 3

                                                                                                                                          I may be misremembering the timeline but as far as I remember it, aRts, ESD & friends were long dead, gone and buried by 2008, as alsa had been supporting proper (eh…) software mixing for several years by then. aRts itself stopped being developed around 2004 or so. It was definitely no longer present in KDE 4, which was launched in 2008, and while it still shipped with KDE 3, it didn’t really see much use outside KDE applications anyway. I don’t recall how things were in Gnome land, I think ESD was dropped around 2009, but pretty much everything had been ported to canberra long before then.

                                                                                                                                          I, for one, don’t recall seeing either of them or using either of them after 2003, 2004 or so, but I did have some generic Intel on-board sound card, which was probably one of the first ones to get proper software mixing support on alsa, so perhaps my experience wasn’t representative.

                                                                                                                                          I don’t know how many bugs PulseAudio revealed but the words “PulseAudio” and “bugs” are enough to make me stop consider going back to Linux for at least six months :-D. The way bug reports, and contributors in general, technical and non-technical alike were treated, is one of the reasons why PulseAudio’s reception was not very warm to say the least, and IMHO it’s one of the projects that kickstarted a very hostile and irresponsible attitude that prevails in many Linux-related open-source projects to this day.

                                                                                                                                    2. 4

                                                                                                                                      I might be misremembering and ALSA supported software mixing, OSS only hardware mixing.

                                                                                                                                      That’s more like it on Linux. ALSA did software mixing, enabled by default, in a 2005 release. So it was a pain before then (you could enable it at least as early as 2004, but it didn’t start being easy until 1.0.9 in 2005)… but long before godawful PulseAudio was even minimally usable.

                                                                                                                                      BSD did the right thing though, no doubt about that. Linux never learns its lesson. Now Wayland lololol.

                                                                                                                                      1. 4

                                                                                                                                        GNOME and KDE both worked around this by providing userspace sound mixing. These weren’t great for latency (sound was written to a pipe, then at some point later the userspace sound daemon was scheduled and then did the mixing and wrote the output) but they were fine for going ‘bing’.

                                                                                                                                        Things got pretty hilarious when you inevitably mixed an OSS app (or maybe an ALSA app, by that time? It’s been a while for me, too…) and one that used, say, aRTs (KDE’s sound daemon).

                                                                                                                                        What would happen is that the non-aRTs app would grab the sound device and clung to it very, very tight. The sound daemon couldn’t play anything for a while, but it kept queuing sounds. Like, say, Gaim alerts (anyone remember Gaim? I think it was still gAIM at that point, this was long before it was renamed to Pidgin).

                                                                                                                                        Then you’d close the non-aRTs app, and the sound daemon would get access to the sound card again, and BAM! it would dump like five minutes of gAIM alerts and application error sounds onto it, and your computer would go bing, bing, bing, bang, bing until the queue was finally empty.

                                                                                                                                        1. 2

                                                                                                                                          I’d forgotten about that. I remember this happening when people logged out of computers: they’d quit BZFlag (yes, that’s basically what people used computers for in 2002) and log out, aRTs would get access to the sound device and write as many of the notification beeps as it could to the DSP device before it responded to the signal to quit.

                                                                                                                                          ICQ-inspired systems back then really liked notification beeps. Psi would make a noise both when you sent and when you received a message (we referred to IM as bing-bong because it would go ‘bing’ when you sent a message and ‘bong’ when you received one). If nothing was draining the queue, it could really fill up!

                                                                                                                                          1. 1

                                                                                                                                            Then you’d close the non-aRTs app, and the sound daemon would get access to the sound card again, and BAM! it would dump like five minutes of gAIM alerts and application error sounds onto it, and your computer would go bing, bing, bing, bang, bing until the queue was finally empty.

                                                                                                                                            This is exactly what happens with PulseAudio to me today, provided the applications trying to play the sounds come from different users.

                                                                                                                                            Back in 2006ish though, alsa apps would mix sound, but OSS ones would queue, waiting to grab the device. I actually liked this a lot because I’d use an oss play command line program and just type up the names of files I want to play. It was an ad-hoc playlist in the shell!

                                                                                                                                          2. 4

                                                                                                                                            This is just an example of what the BSDs get right in general. For example, there is no world in which FreeBSD would remove ifconfig and replace it with an all-new command just because the existing code doesn’t have support for a couple of cool features - it gets patched or rewritten instead.

                                                                                                                                            1. 1

                                                                                                                                              I’m not sure I’d say “get right” in a global sense, but definitely it’s a matter of differing priorities. Having a stable user experience really isn’t a goal for most Linux distros, so if avoiding user facing churn is a priority, BSDs are a good place to be.

                                                                                                                                              1. 1

                                                                                                                                                I don’t know; the older I get the more heavily I value minimizing churn and creating a system that can be intuitively “modeled” by the brain just from exposure, i.e. no surprises. If there are architectural reasons why something doesn’t work (e.g. the git command line), I can get behind fixing it. But stuff that just works?

                                                                                                                                            2. 4

                                                                                                                                              I guess we can’t blame Lennart for breaking audio on Linux if it was already broken….

                                                                                                                                              1. 7

                                                                                                                                                You must be new around here - we never let reality get in the way of blaming Lennart :-/

                                                                                                                                                1. 2

                                                                                                                                                  Same as with systemd, there were dozens of us where everything worked before. I mean, I mostly liked pulseaudio because it brought a few cool features, but I don’t remember sound simply stopping to work before. Sure, it was complicated to setup, but if you didn’t change anything, it simply worked.

                                                                                                                                                  I don’t see this as blaming. Just stating the fact that if it works for some people, it’s not broken.

                                                                                                                                                2. 3

                                                                                                                                                  Well, can’t blame him personally, but the distros who pushed that PulseAudio trash? Absolutely yes they can be blamed. ALSA was fixed long before PA was, and like the parent post says, they could have just fixed OSS too and been done with that before ALSA!

                                                                                                                                                  But nah better to force everyone to constantly churn toward the next shiny thing.

                                                                                                                                                  1. 4

                                                                                                                                                    ALSA was fixed long before PA was, and like the parent post says, they could have just fixed OSS too and been done with that before ALSA!

                                                                                                                                                    Huh? I just setup ALSA recently and you very much had to specifically configure dmix, if that’s what you’re referring to. Here’s the official docs on software mixing. It doesn’t do anything as sophisticated as PulseAudio does by default. Not to mention that on a given restart ALSA devices frequently change their device IDs. I have a little script on a Void Linux box that I used to run as a media PC which creates the asoundrc file based on outputs from lspci. I don’t have any such issue with PulseAudio at all.

                                                                                                                                                    1. 3

                                                                                                                                                      dmix has been enabled by default since 2005 in alsa upstream. If it wasn’t on your system, perhaps your distro changed things or something. The only alsa config I’ve ever had to do is change the default device from the hdmi to analog speakers.

                                                                                                                                                      And yeah, it isn’t sophisticated. But I don’t care, it actually works, which is more than I can say about PulseAudio, which even to this day, has random lag and updates break the multi-user setup (which very much did not just work). I didn’t want PA but Firefox kinda forced my hand and I hate it. I should have just ditched Firefox.

                                                                                                                                                      Everyone tells me the pipewire is better though, but I wish it could just go back to the default alsa setup again.

                                                                                                                                                      1. 6

                                                                                                                                                        Shrug, I guess in my experience PulseAudio has “just worked” for me since 2006 or so. I admit that the initial rollout was chaotic, but ever since it’s been fine. I’ve never had random lag and my multi-user setup has never had any problems. It’s been roughly 15 years, so almost half my life, since PulseAudio has given me issues, so at this point I largely consider it stable, boring software. I still find ALSA frustrating to configure to this day, and I’ve used ALSA for even longer. Going forward I don’t think I’ll ever try to use raw ALSA ever again.

                                                                                                                                                    2. 1

                                                                                                                                                      I’m pretty sure calvin is tongue in cheek referencing that Lennart created PulseAudio as well as systemd.

                                                                                                                                                  2. 3

                                                                                                                                                    I cannot up this comment more. The migration to ALSA was a mess, and the introductions of Gstreamer*, Pulse*, or *sound_daemon fractured the system more. Things in BSD land stayed much simpler.

                                                                                                                                                    1. 3

                                                                                                                                                      I was also ‘forced’ out of Linux ecosystem because of mess in sound subsystem.

                                                                                                                                                      After spending some years on FreeBSD land I got hardware that was not FreeBSD supported at that moment so I tried Ubuntu … what a tragedy it was. When I was using FreeBSD I got my system run for months and rebooted only to install security updates or to upgrade. Everything just worked. Including sound. In Ubuntu land I needed to do HARD RESET every 2-3 days because sound will went dead and I could not find a way to reload/restart anything that caused that ‘glitch’.

                                                                                                                                                      Details here:

                                                                                                                                                      https://vermaden.wordpress.com/2018/09/07/my-freebsd-story/

                                                                                                                                                      1. 1

                                                                                                                                                        From time to time I try to run my DAW (Bitwig Studio) in Linux. A nice thing about using DAWs from Mac OS X is that, they just find the audio and midi sources and you don’t have to do a lot of setup. There’s a MIDI router application you can use if you want to do something complex.

                                                                                                                                                        Using the DAW from Linux, if it connects via ALSA or PulseAudio, mostly just works, although it won’t find my audio interface from PulseAudio. But the recommended configuration is with JACK, and despite reading the manual a couple times and trying various recommended distributions, I just can’t seem to wrap my head around it.

                                                                                                                                                        I should try running Bitwig on FreeBSD via the Linux compatibility layer. It’s just a Java application after all.

                                                                                                                                                        1. 7

                                                                                                                                                          Try updating to Pipewire if your distribution supports it already. Then you get systemwide Jack compatibility with no extra configuration/effort and it doesn’t matter much which interface the app uses. Then you can route anything the way you like (audio and MIDI) with even fewer restrictions than MacOS.

                                                                                                                                                          1. 1

                                                                                                                                                            I’ll give that a try, thanks!

                                                                                                                                                      1. 8

                                                                                                                                                        I don’t think this is objective at all. With such a title, the article should definitely mention that releases are supported for only six months. See https://utcc.utoronto.ca/~cks/space/blog/unix/OpenBSDSupportPolicyResults .

                                                                                                                                                        I know more places with “outdated” (i.e. > 6 months) openbsd installs, than up-to-date ones. One of the biggest issue is that there is no salvation once you’re running an unsupported setup since you can’t skip any version and manual tweaks when upgrading.

                                                                                                                                                        1. 5

                                                                                                                                                          I was running an OpenBSD e-mail server for myself and ran into this. 6.3->6.4 had some MASSIVE changed to opensmtpd, resulting in needing an entirely different configuration file format. I just kept the old one running, but after a while, certbot stopped working (ACMEv1 support ended) and the new version of certbot wasn’t in the 6.3 ports tree. I tried to install it manually with pip, but it depended on cryptography, which now requires Rust, and the version of rust on that system was too old to build it. I then switched from certbot to dehydrate, a fully bash implementation of ACMEv2, but it spit out ECDSA certs which dovecot could read, but not opensmtpd.

                                                                                                                                                          I’m sure I could have just edited dehydrate, but at that point I finally started looking at 6.3->6.4 migration guides (there were none when it came out. There are a couple now. I’m currently writing one myself) and got updated to the latest opensmtpd .. now running in an Alpine container, on my big dedicated server. I then deleted my openbsd VM.

                                                                                                                                                          I liked OpenBSD, and still like the simplicity of their SMTP server, but I’ll run it on Linux for now.