1. 3

    The eigen- prefix is used to describe something which is “the same” before and after a transformation. e.g. the eigenvector of a matrix is a vector which still points in the same direction after multiplying by the matrix (scaled by its eigenvalue). What exactly is the transformation going on here?

    1. 6

      eigen comes from Germanic languages such as German and Dutch where it means own, particular to, etc.

      Mijn eigen auto -> My own car.

      Mijn eigenwaarde/eigendunk -> My self-esteem

      So, the prefix eigen- does not imply a transformation. Though, I agree that it is confusing in CS to use this prefix, since non-native speakers immediately associate it with eigenvalues/vectors.

      1. 2

        The term “singleton class” that the article briefly mentions is actually Ruby’s official term for this concept. Ruby’s standard library defines the Object#singleton_class method; you can use .singleton_class in place of .eigenclass in all the article’s examples. Perhaps this potential confusion is the reason the Ruby team chose “singleton class” over “eigenclass”.

        1. 2

          Thank you for your comment.

          Eigen means “own” in German. So “eigenclass” means “owned class” (as this anonymous class is owned by its defined class).

          1. 2

            Etymologically speaking, the “eigen-”-prefix was used by David Hilbert to describe that the value/vector is inherent to the thing. To confuse things a little, “eigen” is a pretty versatile word in german: https://www.dict.cc/?s=eigen

            Hilbert probably used in the sense of “possessive”, “inherent”, “in-built”. In that sense, which is not focused on the transformation, “eigenclass” is rather similar.

            (loosely taken and translated from: http://faql.de/etymologie.html#eigen)

          1. 3

            Great to hear and good for the rest of us ;). Lightweight user of sr.ht and loving it.

            1. 1

              I’m glad, please send feeback if you have any!

            1. 6

              Am I the only one who happily uses both Macs and GNU/Linux computers?

              1. 10

                Nope, one MUST use one or the other, never both, or even a third at the same time. Can only ever be either/or!

                /s obviously but most of these posts are starting to get tiring

                1. 2

                  It is not like there are no benefits to using a single platform. For example I use Linux, NixOS specifically, just so I can configure everything–from kernel to user packages to development projects–in declarative fashion. Good luck doing that with OSX (which I use solely for building iOS projects).

                  1. 1

                    I use nix on OS X, its not that different to be honest, not sure I agree with this overall sentiment even as a nixos and macos (amongst others) user.

                    1. 1

                      nix + nix-darwin?

                  2. 2

                    I live out of a suitcase, so I don’t want to carry two machines around with me. I also don’t want to run two operating systems, because disk space is at a premium already on my MacBook Air. I have to garbage collect my Nix store and rebuild my Docker build slave (I need to compile for Linux arch) every week.

                    I don’t need any MacOS GUI programs, but I do want the It Just Works thing when it comes to running the webcam (I don’t want to carry an external one), audio, battery life, etc.

                    I could switch away from Apple hardware but so far I still think it’s best in class, although this is less true now that they’ve done away with scissor switches and MagSafe.

                    1. 3

                      With Nix and Docker on OS X it’s kind of like you’re running a few operating systems already. How many copies of the GNU userland are on your drive right now? 😜

                      1. 1

                        That’s true, it’s a sort of hybrid but only because I’m not yet confident I can have the best hardware for me (basically a penultimate generation MacBook Air, maybe with more disk space and memory), with NixOS but with all the other conveniences working.

                      2. 2

                        I think I’m one of the most diehard pro-Linux and anti-Mac developers (just shy of the line between pragmatism and zealotry) I know but if I had to live with one machine and one machine only I guess I wouldn’t think twice about choosing a Macbook Pro. (Right now I’ve been on Windows for games all my life, and using Linux casually since 1998 and for work, fulltime, for over 10 years).

                        1. 3

                          I use MacOS personally (2013 MBP) and Windows at work. Linux in a VPS.

                          I’m seriously considering going Windows for personal stuff when I need to update my computer.

                          Why? Because while I love the Apple hardware, the latest iterations don’t look compelling to me. KB issues, no magsafe, no SD-reader (I do a lot of photography), lack of USB-A ports, and higher price. And I really don’t use any Apple-made software (apart from Terminal.app). It’s Chrome, Lightroom, VLC… all of which are basically cross-platform. I don’t feel I’ll miss much from OSX, and it will be much easier to game in Windows too.

                          1. 2

                            I tried switching to windows from mac at home on account of games, but my use case is really centered around audio. Turns out windows is still 3rd place for handling external audio devices (alongside more general web browsing sound) so I’m probably going to take the NUC and sell it or Linuxify it and get a mac mini. I use *nix side of the mac (and linux) constantly at work. Windows doesn’t have that side and the reliability although better in win 10 isn’t quite where I need it to be when I just want to hit record.

                      3. 1

                        As an anecdote, I use three computers, a mac, a windows and a Linux. Recently, I was going to give a talk at a meetup and I had two demos to show regarding p2p in the browser. Unfortunately, I did a poor job in testing them prior to the event day. I didn’t test my old demo, just the new one (my thought was, I didn’t change the old stuff, it should just work), well, it didn’t. In the end, one demo worked only on the mac and the other only on windows. I gave the talk with two computers on the stand and switched from one to the other… sometimes using two computers is ok :-)

                        1. 0

                          no

                        1. 14

                          My feelings are kind of mixed so far. The lightweight UI and responsive site are a breath of fresh air. What’s a little jarring is how much of the service is centered around email. I’ve never been part of a mailing list, and emailing code to other people sounds like something from 20 years ago, but maybe I’m just a young whippersnapper that doesn’t know what he’s talking about. Git is already a complicated tool, and adding email to the mix just increases the cognitive load. I’ll still learn how to use it because it sounds kind of interesting, but my preference would still be some kind of browser interface.

                          1. 19

                            I think you should give email a chance. Git has built-in tools for working with email and you can do your entire workflow effectively without ever leaving your terminal. Sending a patch along for feedback is just git send-email -1. (-1 meaning one commit, -2 meaning the last 2 commits, etc). Here’s a guide, which is admittedly terse and slated to be replaced with a more accessible tutorial:

                            https://man.sr.ht/git.sr.ht/send-email.md

                            That being said, web tools are planned to seamlessly integrate with this workflow from a browser.

                            1. 11

                              That being said, web tools are planned to seamlessly integrate with this workflow from a browser.

                              I would use that.

                              1. 4

                                I like the email workflow, but I also have to be realistic - it is unlikely that my colleagues or drive-by contributors would adopt it. So, in practice it will mean fewer contributions and less cooperation.

                                The GitHub-like workflow is something that is ingrained now and has a relatively low barrier to entry. So, if something is going to take over, it’s something that is very similar, such as GitLab or Gitea.

                                Of course, there will always be projects that cater to an audience that feels at home with an email workflow.

                                It’s good to hear that there will be web tools as well.

                                1. 4

                                  I like the email workflow, but I also have to be realistic - it is unlikely that my colleagues or drive-by contributors would adopt it.

                                  I think as this workflow proliferates, this will become less and less true. It’s remarkably easy to make a drive-by contribution with email if you already have git send-email working, easier even than drive-by GitHub pull requests.

                                  1. 4

                                    git send-email working

                                    that’s a big ask

                                    git send-email needs a bunch of perl packages, which often means you need to set up perl packaging.

                                    Depending on your distro/OS this can be tricky, especially because git send-email needs a bunch of network packages and they don’t always cleanly install and you have to figure out why (except you don’t know much about perl packaging, so you can’t).

                                    There have been multiple cases on different OSes (i think osx and some version of ubuntu) that i gave up after half an hour of various cpan commands trying to get things to work. I’m not even going to try setting that up on Windows.

                                    Furthermore, the UX of git-send-email is terrible. Sending followup patches is annoying, for one.

                                    All this has forced me to try and paste patches directly into an email client. But this is broken, too. GMail, for one, converts tabs to spaces in plaintext emails, breaking patches. I could use a local client, but setting up a client well is a lot of work and confusing (i could also rant for a while about why this is the case, but i won’t) and I don’t really want to switch my workflow over to using a client.

                                    Furthermore, half the patch mailing lists I’ve worked with have hard-to-figure-out moderation rules. They’ll outright reject some kinds of emails without telling you, and because many are human moderated it’s hard to know if your email setup worked especially if you’re using git-send-email (which you may not have invoked or set up correctly) because 90% of the time your patch won’t show up on the list and you have no idea which of the many possible reasons for that is the case.

                                    Despite all this I’ve submitted quite a few patches to a patch mailing list (hell, I’ve been involved enough in mailing-list-based project to have commit), either by lucking out on the perl setup for send-email, by temporarily setting up a client that doesn’t sync, or by sending patches through gmail with “ignore the whitespace please, that’s gmail’s fault, I’ll fix it when I commit”. It’s a chore each time.

                                    I’ve given email multiple chances. It doesn’t work. The activation energy of email for patch contributions is quite high.


                                    The web UI thing sounds like a good idea, especially if it can handle replies. It’s basically what I’ve been suggesting projects on mailing lists do for ages. Use email all you want, just give me a way to interact with the project that doesn’t involve setting up email.

                                    1. 2

                                      Almost no one has to package git’s perl dependencies themselves. Doesn’t your OS have a package for it already? And as someone who has packaged git before, it wasn’t really that bad.

                                      Also, the golden rule of emailing patches is never paste them into some other mail client.

                                      1. 2

                                        Also, the golden rule of emailing patches is never paste them into some other mail client.

                                        Paste not, but maybe attach? FreeBSD don’t like it, but it’s OK for Postgres.

                                        1. 2

                                          I generally prefer that people don’t attach patches, either. IMO the best way to send patches is send-email.

                                          1. 1

                                            “IMO” and “the best” is perfectly fine. But I was under impression that it was unconditionally the only way to submit patches, when I wanted to improve sr.ht’s PG DB schemas.

                                            1. 2

                                              Each project on sr.ht can have its own policies about how to accept patches. sr.ht itself does require you to submit them with send-email (i.e. for patches to the open source sr.ht repositories).

                                              1. 1

                                                Can you elaborate on what you dislike about sending patches with a normal MUA? It’s certainly a lot easier for someone who has spent the time to configure their MUA to be able to re-use the config they’ve already got rather than configuring a new tool they’ve never used before.

                                                1. 3

                                                  The main issue is that nearly all MUAs will mangle your email and break your patches, which is annoying for the people receiving them and will be more work for you in the long run. Also, most end-user MUAs encourage the use of HTML emails, which are strictly forbidden on sr.ht. Also, code review usually happens by quoting your patch, trimming the fat, and replying inline. This is more annoying if you attach your patch to the email.

                                                  Setting up git send-email is pretty easy and will work every time thereafter. It’s also extremely convenient and fits rather nicely into the git workflow in general.

                                                  1. 1

                                                    I see; so it has more to do with the fact that you can’t trust most popular MUAs not to screw up the patch rather than any inherent problem with that flow. For a well-behaved MUA it should be fine, but assuming a MUA is well-behaved (or even assuming that a user knows whether theirs is or not) isn’t a good bet.

                                                    Thanks.

                                        2. 1

                                          Almost no one has to package git’s perl dependencies themselves. Doesn’t your OS have a package for it already?

                                          No, i don’t mean you have to package them, but you have to install them and the installation isn’t always smooth. It’s been a while since I did this so I don’t remember the precise issues but i think it has a lot to do with the TLS part of the net stack. Which kinda makes sense, openssl packaging/linking has issues pretty much everywhere (especially on OSX).

                                          Also, again, Windows. A lot of devs use Windows. I got involved in open source on Windows, back when I didn’t have my own computer. I could use Git and Github, but I’m pretty sure I’d have been unable to set up git-send-email if I had to at the time. Probably can now, but I’m an experienced programmer now.

                                          Also, the golden rule of emailing patches is never paste them into some other mail client.

                                          I know, except:

                                          • now handling replies is annoying
                                          • now i need to set up git-send-email, which doesn’t always work
                                          1. 1

                                            Windows devs aren’t in the target audience. I heard from a macOS user that they were able to get send-email working without too much trouble recently, maybe the situation has improved.

                                            now handling replies is annoying

                                            Not really?

                                            now i need to set up git-send-email, which doesn’t always work

                                            git send-email will always work if your email provider supports SMTP, which pretty much all of them do.

                                            1. 1

                                              Windows devs aren’t in the target audience

                                              If you’re wishing for email to be the future, you’re going to have to think about windows devs at some point.

                                              (this choice is also even more hostile to new programmers, as if patch email workflows weren’t newbie-hostile enough already)

                                              Not really?

                                              You have to copy message ids and stuff to get lists to thread things properly

                                              git send-email will always work

                                              I just told you why it doesn’t always work :)

                                              1. 3

                                                I’m prepared to lose the Windows audience outright on sr.ht. Simple as that.

                                                (edit)

                                                Regarding message IDs, lists.sr.ht (and many other email archives) have a mailto: link that pre-populates the in-reply-to for you.

                                                1. 1

                                                  I’m prepared to lose the Windows audience outright on sr.ht. Simple as that.

                                                  oh, sure, for your own tool it’s fine.

                                                  what I’m saying is that if you’re expecting this workflow to proliferate you will have to deal with this too.

                                                  Do whatever you want with your own tool: I’m just explaining why send-email proliferating is a tall order, and windows is a major draw here.

                                                  Regarding message IDs, lists.sr.ht (and many other email archives) have a mailto: link that pre-populates the in-reply-to for you.

                                                  ah, that’s nice. I may not have encountered lists with this (or been interacting only by email and not using the archive)

                                        3. 1

                                          git send-email needs a bunch of perl packages, which often means you need to set up perl packaging.

                                          Personally I’ve never once seen a unix machine where the perl stack wasn’t already installed for unrelated system-level stuff.

                                          1. 1

                                            to be clear, perl is usually installed, it’s the relevant packages (specifically, the networking/TLS stuff) that usually aren’t

                                            this is particularly bad on OSX which has its own openssl issues, so the Perl SSL packages refuse to compile

                                        4. 4

                                          if you already have git send-email working

                                          Sadly, I think this is extremely uncommon :’(

                                          1. 3

                                            Hence:

                                            I think as this workflow proliferates

                                            1. 5

                                              I think I’d phrase this as if it proliferates, as if anything I think the number of people with sendmail (or equivalent) working on their computer is going down, not up. It’d be fun to see it rise again due to sr.ht, though I don’t know that I’m optimistic. But perhaps I’m just being overly pessimistic :)

                                              I do worry about more casual developers though, who may not even really know how to use the command-line. I think an increasing number of developers interact with version control solely through their IDE, and only touch their command-line if they have to copy-paste some commands. It’d be interesting to see if that’s something this workflow can still cater to. Some simple web-based tooling may go a long way there!

                                              1. 3

                                                You don’t need to set up sendmail, you just need a mail server with SMTP - which nearly all of them support.

                                                1. 3

                                                  Sorry, what I meant was more that you have to set up git for e-mail sending. I happen to have sendmail already set up, so all I needed was git config --global sendemail.smtpserver "/usr/bin/msmtp", but I think it’s very uncommon to already have it set up, or to even be comfortable following the instructions on https://man.sr.ht/git.sr.ht/send-email.md.

                                        5. 2

                                          I like the email workflow, but I also have to be realistic - it is unlikely that my colleagues or drive-by contributors would adopt it. So, in practice it will mean fewer contributions and less cooperation.

                                          The great thing about this is it’s not all-or-nothing.

                                          For https://fennel-lang.org we accept patches over the mailing list or from GitHub pull requests. Casual contributions tend to come from GitHub, while the core contributors send patches to the mailing list and discuss them there. Conveniently, casual contributions tend to require less back-and-forth review, (so GitHub’s poor UI for their review features is less frustrating) while the big meaty patches going to the mailing list benefit more from the nicer review flow.

                                        6. 3

                                          … that is if you’ve managed to set it up in the first place, probably without an opportunity to test it – that means that you have send your commit, not knowing what will come out, to test your setup, your configuration and the command you chose in the first place, which puts quite a lot of pressure, especially on people who have little experience with projects, let alone email-projects.

                                          That being said, web tools are planned to seamlessly integrate with this workflow from a browser.

                                          very nice.

                                          1. 2

                                            Nah, on sr.ht I have an open policy of “if you’re unsure about your setup, send the patch to me (sir@cmpwn.com) first and I’d be happy to make sure it’s correct”.

                                            1. 2

                                              I wonder if it would make sense to set up a “lint my prospective patch” email address you could send your patch to first which could point out common mistakes, assuming that kind of thing is easy to write code to detect.

                                              1. 2

                                                I plan on linting all incoming emails to lists.sr.ht to find common mistakes like this and reject the email with advice on how to fix it.

                                                1. 1

                                                  If you can get this running well and cheaply, you could potentially do an end run around people’s send email setup related issues by hosting a “well formed, signed, patches-only” open email relay, and local git config instructions.

                                              2. 1

                                                Is there a way or a plan to have a patch-upload form for example? That might be helpful for beginners.

                                                1. 3

                                                  Yes, I plan on having a web UI which acts as a frontend to git send-email.

                                          2. 4

                                            I like that it’s using e-mail so it’s “federated” and decentralized by default.

                                            The e-mail workflow has two problems though:

                                            • integrations: usually projects have a lot of checks that can be automated (“DCO present”, “builds correctly”), for e-mail workflow this kind of stuff needs to be built (check out how Postgres does it),
                                            • client configuration: to correctly use this workflow, one need to configure git send-email (setting up credentials for example), project configuration (correct sendemail.to and format.subjectprefix) and e-mail client to send plain text, 72-characters wrapped messages. Apparently not everyone does that.

                                            Mailing lists vs Github nicely summarizes benefits of ML over Github but also highlight the number of things maintainers need to setup to run their projects on ML that Github gives them “for free”.

                                            From my point of view sr.ht looks like a great way to validate the idea if it’s possible bring easy project collaboration from Github to MLs.

                                            1. 2

                                              usually projects have a lot of checks that can be automated

                                              This is planned on being addressed soon on sr.ht with dispatch.sr.ht, which is used today to allow GitHub users to run CI on builds.sr.ht. The same will be possible with patches that arrive on lists.sr.ht.

                                              client configuration

                                              There’s a guide for send-email:

                                              https://man.sr.ht/git.sr.ht/send-email.md

                                              As for other emails, I’m working on some more tools to detect incorrectly configured clients and reject emails with advice on how to fix it.

                                              Thanks for the feedback!

                                              1. 2

                                                I’m really interested in how far can one push this model.

                                                Would it build the patch and e-mail back build results? For example with a link to build results and a quick summary?

                                                Are you also planning for some aggregation of patches? (Similar to what Postgres has). For example Gerrit uses Change-Id to correlate new patches that replace old ones. Would you for example use Message-Id and In-Reply-To with [Patch v2] to present a list on a web interface of patches that are new / accepted / rejected? This interface could be operated from e-mail too I think, e.g. mailing LGTM would switch a flag (with DKIM validation so that the vote is not spoofed).

                                                By the way I really like how sr.ht is challenging status-quo of existing solutions that just want to mimic GitHub without thinking about basic principles.

                                                Good luck!

                                                1. 7

                                                  Would it build the patch and e-mail back build results? For example with a link to build results and a quick summary?

                                                  Yep, and a link to a full build log as well.

                                                  Would you for example use Message-Id and In-Reply-To with [Patch v2] to present a list on a web interface of patches that are new / accepted / rejected?

                                                  Yep!

                                                  This interface could be operated from e-mail too I think, e.g. mailing LGTM would switch a flag (with DKIM validation so that the vote is not spoofed).

                                                  Aye.

                                                  1. 3

                                                    Great!

                                                    By the way I admire you pro-active approach of not only explaining the problem but also building beautiful software that solves the problem! 👍

                                            2. 3

                                              emailing code to other people sounds like something from 20 years ago

                                              At least OpenBSD is still doing that on the regular on the tech@ mailing list. It definitely still works.

                                              1. 2

                                                And I love it. It’s so damn easy to just email a one-off diff and watch someone land it. No accounts, no registration, no forking repos and dealing with fancy weird web UIs…

                                              2. 3

                                                One day, the current generation of “Email is SO 5 minutes ago!” kids are going to wake up and realize that e-mail is an amazing tool.

                                                Or so I’d like to think :)

                                                1. 1

                                                  I could be convinced. What’s your argument in favor of email?

                                                  1. 3
                                                    • Inherently de-centralized
                                                    • Can be tuned for nearly real time end to end response of low bandwidth batch processing for where network is at a premium
                                                    • Vendor neutral
                                                    • As rich or as minimal as you want it to be
                                                    • Arbitrary context types - you can send everything from 7 bit ASCII to arbitrarily complex HTML/CSS and varying payload types
                                                    • Readable with everything from /bin/cat to a nice client like Thunderbird and everything in between
                                                    • Rich capabilities for conversation threading
                                                    • Rich search capability client and server side
                                                    • Myriad archival and backup options

                                                    The list goes on.

                                                    For a more end user/business-centric version of this see In Defense of Email

                                              1. 4

                                                I need to do this. I’ve set up a few NixOS systems and I get a little further each time but as soon as I need to get work done, I hop back over to my existing Ubuntu box.

                                                It’s just too much overhead to have look up how to configure everything. I need to get over the hump because once I get the system going, it won’t need to change much.

                                                The linked cheat sheet is nice for the day-to-day maintenance commands.

                                                1. 2

                                                  This, very much. I come back to Arch Linux every time (before it was Linux Mint) because the wiki is excellent. I’d like to try Gentoo, FreeBSD, and NixOS, but for those OSes it feels like much more of a struggle to set up a usable system.

                                                  I feel like NixOS’ most important contribution is the idea of reproducible builds. I’ve heard that the package manager can also be used on other operating system, but I’m not sure how different the experience would from using NixOS (I don’t know about package managers in general, but I’d like to learn more).

                                                  1. 9

                                                    I feel like NixOS’ most important contribution is the idea of reproducible builds. I’ve heard that the package manager can also be used on other operating system, but I’m not sure how different the experience would from using NixOS (I don’t know about package managers in general, but I’d like to learn more).

                                                    Nix offers so much more than reproducible builds. E.g.:

                                                    • Transactional upgrades with rollback.
                                                    • Environments (with a set of dependencies) that are only live in a particular shell or directory (with direenv).
                                                    • Management of your package set jointly with your system or home directory configuration. See NixOS, nix-darwin, and home-manager
                                                    • Nix allows users to use their own sets of packages, without global visibility.
                                                    • You can have different versions of different libraries at the same time.
                                                    • Since Nix is a functional language, it is easy to override existing derivations (change some attributes) for your own custom Boost, OpenCV, or whatever. All non-conflicting with other versions of the same programs/libraries.

                                                    John Wiegley has a very nice introduction to Nix. It is a bit much to take all in without any experience, but it gives a good impression of what is possible:

                                                    https://www.youtube.com/watch?v=G9yiJ7d5LeI Just to give an example of the power of Nix. It supports ZFS installs, but is not supported by the default ISO configuration. You add support for ZFS to the installation editing the system configuration of the live system booted from DVD/USB, then doing a nixos-rebuild switch in-place. Afterwards your live system

                                                    1. 1

                                                      Cool! I mostly meant ‘I think this would be the most convenient feature for me, if I would use NixOS’. But these are very nice as well. I ought to give it a try :)

                                                    2. 5

                                                      I feel like NixOS’ most important contribution is the idea of reproducible builds.

                                                      I would attribute that to Debian. They started the activity in 2013 and did most of the work to fix the builds on Linux. While NixOS is surprisingly old (2003) it became popular around 2015. The actual impulse seems have come from the Tor project.

                                                    3. 1

                                                      Responding to myself, I didn’t notice it mentioned in the article but to find the configuration options for things, you can look them up at https://nixos.org/nixos/options.html

                                                      1. 2

                                                        You might be aware of this already, but in case you’re not: there’s also the configuration.nix(5) manpage which contains every nixos configuration option.

                                                        1. 2

                                                          There’s also the nixos-option command line tool.

                                                      1. 2

                                                        Looks nice! The landing page would be a bit more readable if it didn’t have the animated gif on the side, it is very distracting.

                                                        Though I wonder if it is not a solution looking for a problem, why would someone want to read my notes as a presentation? I usually just want to convert my own notes into a presentation so that I can present them. But then I typically want a different granularity than presenting per sentence. I have good experiences with the following tools doing that:

                                                        • org-mode: does everything from simple notes to Jupyter-like notebooks with in-line code execution. Supports export to LaTeX beamer, S5, reveal.js, etc.
                                                        • pandoc: converts between various formats, including Markdown to LaTeX beamer, S5, reveal.js, etc.
                                                        • Deckset: native Mac app that allows you to present Markdown files, with PDF export. Has great support for source code listings, highlighting in steps, etc. I don’t like the formatting as much as LaTeX beamer, but it is particularly useful when I am in a hurry.

                                                        My favorite among these org-mode, since I can generate graphs and such directly from the document.

                                                        Of course, I might just not be in the target audience.

                                                        1. 3

                                                          Thank you for the feedback! Yes, the gif is distracting. I’ll see how to give the crux of the app without the gif.

                                                          This largely came out of my own itch. Just like you, I organise my thoughts into notes because that works better than slides, titles and content. Eventually I want to convert it into a rough presentation with absolutely zero effort, re-organising. And that’s how I ended up building one.

                                                          The tools you’ve mentioned look good. There’s no reason why Writango is better than those if you are looking for serious re-organization of notes into presentations. I guess Writango is for the not-so-serious, rough ones :)

                                                          1. 2

                                                            Sounds fair. It definitely has a less steep learning curve than org-mode or pandoc.

                                                            Keep up the good work!

                                                          2. 1

                                                            How do you generate graphs in org mode?

                                                              1. 1

                                                                My personal favorite is ggplot through R. I suspect for interactivity that Python/bokeh might be an option that works well.

                                                            1. 2

                                                              “Cosine Similarity tends to determine how similar two words or sentence are, It can be used for Sentiment Analysis, Text Comparison and being used by lot of popular packages out there like word2vec.”

                                                              Wouldn’t any distance metric do? As long as you choose the right vector space?

                                                              I was under the impression cosine similarity makes it easy to batch computation with highly tuned matrix multiplications

                                                              1. 4

                                                                Cosine similarity is not actually a metric and I think that is why people use it. Showing it is not a metric is easy because for metric spaces the only points that are zero distance away from another point are the points themselves. Cosine similarity in that sense fails to be a metric because for any given vector there are infinitely many vectors orthogonal to it and hence “zero” distance away. (But I just realized it’s even simpler than that because cosine similarity also gives negative values so it fails the positivity test as well.)

                                                                The relation to matrices that you mentioned are about positive definite bilinear forms and those give rise to dot products that are expressible as matrix multiplications and in vector spaces there is a way to define a metric based on dot products by defining the metric to be the dot product of the difference between two vectors with itself. Following through the logic the positive definite condition ends up being what is required to make this construction a metric.

                                                                1. 3

                                                                  This is not really the problem. People convert cosine similarly into a pseudo distance by taking 1 - cos(u,v). This solves the problems that you mention.

                                                                  The true problem is that the cosine is a non-linear function of the angle between two vectors, this violates the triangle inequality . Consider the vectors a and c with an angle of 90 degrees. Their cosine pseudo-distance is 1. Now add a vector b that has an angle of 45 degrees of both a and c. The cosine pseudo-distances between a and b, b and c are 0.29 rounded. So, the distance from a to c via b is shorter than the distance from a to c.

                                                                  1. 2

                                                                    Even with the pseudo-distance you still have the same problem with zero distances, as 1 - cos(u,v) is zero whenever the two points are tau radians apart.

                                                                    1. 2

                                                                      In most vector space models, these would be considered to be vectors with the same direction, so 0 would be the correct distance. Put differently, the maximum angle between two vectors is 180 degrees.

                                                                    2. 1

                                                                      Good point. I didn’t think about the triangle inequality failure.

                                                                    3. 1

                                                                      Thanks! Why would people want that (not actually being a metric)?

                                                                      1. 1

                                                                        The Wikipedia article has a good explanation I think. When working with tf-idf weights the vector coefficients are all positive so cosine similarity ends up being a good enough approximation of what you’d want out of an honest metric that is also easy to compute because it only involves dot products. But I’m no expert so take this with a grain of salt.

                                                                        So I take back what I said about using it because it’s not a metric. I was thinking it has something to do with clustering by “direction” and there is some of that but it’s also pretty close to being a metric so it seems like a good compromise when trying to do matching and clustering types of work where the points can be embedded into some vector space.

                                                                    4. 3

                                                                      I was under the impression cosine similarity makes it easy to batch computation with highly tuned matrix multiplications

                                                                      The same applies to Euclidean distance when computed with the law of cosines. Since the dot product is the cosine of the angle between two vectors, scaled by the vector magnitudes, the squared Euclidean distance between two points is:

                                                                      |u|^2 + |v|^2 - 2u·v

                                                                      (|u|^2 is the squared L2 norm of u, equations are a bit hard to do here)

                                                                      Computing the euclidean distance in this way especially pays off when you have to compute the euclidean distance between a vector and a matrix or two matrices. Then the third term is a matrix multiplication UV, which as you say, is very well-optimized in BLAS libraries. The first two terms are negligible (lower order of complexity).


                                                                      One of the reasons that cosine similarity is popular in information retrieval is that it is not sensitive to vector magnitudes. Vector magnitudes can vary quite a bit with most common metrics (term frequency, TF-IDF) because of varying document lengths.

                                                                      Consider e.g. two documents A and B, where document B is document A repeated ten times. We would consider these documents to have exactly the same topic. With a term frequency or TF-IDF vector, the vectors of A and B would have the same direction, but different lengths. Since cosine similarity measures just the angle between the vectors, it correctly tell us that the documents have the same topic, whereas a metric such as Euclidean distance would indicate a large difference due to the different vector magnitudes. Of course, Euclidean distance could be made to work by normalizing document (and query) vectors to unit vectors.

                                                                      I don’t want to bash this article too much, but it is not a very good or accurate description of the vector model of information retrieval. Interested readers are better served by e.g. the vector model chapter from Manning et al.:

                                                                      https://nlp.stanford.edu/IR-book/pdf/06vect.pdf

                                                                      1. 1

                                                                        I must have been confusing the two

                                                                    1. 3
                                                                      • Experiments with different word embeddings and how they influence the accuracy of bi-directional RNN taggers. This mostly as an example to encourage students to write their own hypotheses how different types of embeddings and hyperparameters influence their chosen task and test them. (Some folklore from the field surprisingly does not seem to hold up: e.g. shorter contexts and lower dimensionality, since this is a syntax-oriented task.)

                                                                      • Reading up some papers on vector quantization. I have started implementing some approaches in Rust, so that I can use them later in my crate to experiment with compressed embeddings.

                                                                      • Porting my approximate randomization tests package from Haskell to Rust. I haven’t done Haskell in years, and when trying to compile it with up-to-date dependencies usually requires a lot of work. So I am taking the lazy approach and redo it in Rust.

                                                                      • Finishing up a paper with some colleagues. Must be done before the holidays.

                                                                      • Baking Christmas cookies with our daughter. We made the dough together today and we are baking tomorrow. We also have to prepare a starter for their Christmas dinner at school.

                                                                      • Getting and wrapping the last presents for Xmas.

                                                                      1. 1

                                                                        I didn’t know what Wabi was, but found the Wikipedia article useful

                                                                        1. 1

                                                                          Win4Lin 9x was based on the somewhat similar SCO Merge and was a very fast way to run Windows 95/98 on Linux:

                                                                          https://en.m.wikipedia.org/wiki/Win4Lin

                                                                          I used it for several years on Linux. Typing ‘win’ in your shell and having Windows 98 on X11 in a few seconds was magical. It was also faster and better intergrated than the first versions of VMWare.

                                                                          Though I guess the merge kernel extension is a nightmare. Different times.

                                                                        1. 3

                                                                          So here’s the list of parts we need to build a working, useful Unix system:

                                                                          • init
                                                                          • a supervisor
                                                                          • a login program
                                                                          • a shell
                                                                          • a C library

                                                                          Isn’t really more needed when it comes to Linux? Using only that, you will have an empty /dev, as mknod isn’t part of the shell (not even talking about udev), and I think you get a kernel panic in such case.

                                                                          On a side note, I’ve been working on a self-bootstrapping distro for quite a while, and I was stunned by the amount of programs needed to reach this goal (think gcc, yacc, perl, python, etc…)

                                                                          1. 1

                                                                            defvs entries aren’t populated from the userspace for a while now, and udev isn’t a requirement unless you handle hotplug.

                                                                            1. 1

                                                                              defvs entries aren’t populated from the userspace for a while now

                                                                              I thought systemd/udev populated /dev on modern systems

                                                                              1. 3

                                                                                Indeed, devfs has been marked obsolete in 2003 in favor of udev:

                                                                                https://lwn.net/Articles/44731/

                                                                                Of course, you could just pre-create the device nodes, as was done before udev and devfs ;).

                                                                                1. 2

                                                                                  Slip of the pen, it’s devtmpfs.

                                                                                  https://lwn.net/Articles/330985/

                                                                                2. 2

                                                                                  See my sibling reply. Neither udev nor systemd are requirements for a usable system, Linux predates both by many years.

                                                                                3. 1

                                                                                  Who creates them then? The kernel itself? Take /dev/console for example. Wouldn’t it prevent the kernel to boot if it is not present? AFAIK, udev does more than hotplugging, as it replaces what devfs used to do.

                                                                                  1. 4

                                                                                    When a device driver registers new device, the kernel creates a node in devtmpfs. There is no need to manage it with udev in static configurations. Many embedded Linux systems never run udev. There’s been no practical hard requirement for mknod since 2.6.

                                                                                    1. 2

                                                                                      Take /dev/console for example. Wouldn’t it prevent the kernel to boot if it is not present?

                                                                                      The kernel does not write to the console (or any other device) by opening a device node in the filesystem. The device driver already lives inside the kernel, there’s no need to go via the filesystem (it writes to the console well before the root filesystem is mounted). So no, it wouldn’t prevent the kernel from booting if /dev/console wasn’t present. (It might create a problem for early userspace, though IIRC even init will have handles to the console set up directly by the kernel).

                                                                                      1. 1

                                                                                        This makes sense indeed, thanks for the reply!

                                                                                        I wonder to what extent a system would be usable with an empty /dev. Would a shell actually start without a TTY? I was playing with containers the other day, and running mksh inside it required that I create a few nodes in $chroot/dev/ before being able to start it. Maybe other shells can work around that?

                                                                                        1. 1

                                                                                          Containers share a kernel, which is already aware of the devices, on an isolated file system and process space where there is limited kernel introspection in this regard. Thus the rules are a little different.

                                                                                          1. 1

                                                                                            I don’t know the specific details of various shells, but in theory I think it should be possible to have a shell that doesn’t try to open any devices. (It’s a pretty onerous restriction though; there’s not much you’ll be able to do in a system with no device nodes).

                                                                                          2. 1

                                                                                            Though the fork children could always inherit standard output et al.

                                                                                            1. 1

                                                                                              That’s what I was implying trying to imply, yes.

                                                                                    1. 1

                                                                                      It says something that our version control systems are complicated enough that a bespoke, single user system makes sense (in niche cases), I’m just not sure what it says…

                                                                                      1. 10

                                                                                        But is e.g. git really that complicated as a single user system? You could work with a small subset of commands (git checkout, git branch, git commit -a, git reset --hard HEAD) and you would probably have something similar. Plus you can easily do more advanced stuff if you go beyond one person for a reason.

                                                                                        1. 1

                                                                                          you’re probably right but this raises two things in my mind.

                                                                                          What’s the small subset and is it like C++ that it’s a different small subset for different groups?

                                                                                          A nice feature would be the ability to transition from SRC to git, I’ll have to look/file an issue.

                                                                                      1. 6

                                                                                        I love and use emacs every day as a text editor. Tools like org mode and just general emacs customization is great!

                                                                                        However, outside of the text editing sphere, the emacs implementation of thing such as shell, email, and a window manager always seem “almost there” but unfortunately not useable. This saddens me because I would love to never leave emacs.

                                                                                        That being said, things like TRAMP completely shifted my ideas on how to manage remote files, so who knows. I am optimistic about the continued progress of the emacs ecosystem.

                                                                                        1. 8

                                                                                          Yes, I agree! For the shell environment, the brawback of emacs buffers becomes apparent. Most shell emulations (emacs has several) work fine as long as the executed programs do not produce much text, like cating a large file. When that happens, the shell becomes sluggish or freezes up, which in turn increases the cognitive burden, i. e. “May I execute this line or will this stop my workflow?” This is a major reason why I do not use the shell within Emacs. In general st feels much more resonsive than Emacs and that saddens me.

                                                                                          For mail, I simply do not have that much mail that I consider the elaborate mail configurations necessary. Mostly I just do IMAP searches to find a messge I’m looking for and that works well enough for me. But I still find the approach with offline mailboxes quite nice; but there are still some smaller corners.

                                                                                          As far as I understand it, when exwm is used, the window manager will freeze up, if emacs hangs and that is something that I do not want to experience, hence I’ve tried to make emacs play nicer with the window manager by mostly opening new Emacs frames instead of the internally managed windows. I’m satisfied with that setup.

                                                                                          TRAMP is almost there. I wish it had a mosh-like connection to the remote server, but I understand that this is actually quite hard to implement. But still ssh editing via tramp works quite nicely, especially once you configure ssh aliases and keys properly.

                                                                                          1. 4

                                                                                            As a heavy Emacs in terminal user I’m pretty happy with the ability to just bg emacs and run cat and less when needed. And having a terminal multiplexer helps too of course.

                                                                                            But I realize that if you’re in a windowing environment having everything in Emacs becomes more desirable.

                                                                                            As an aside, isn’t a “normal” terminal emulator like rxvt already much faster than Emacs? What does st bring to the table?

                                                                                            1. 3

                                                                                              bg

                                                                                              May I ask you how you put a emacs (in terminal mode, i.e. emacs -nw) in the background? I am running emacs with spacemacs + evil configuration (mostly for org-mode) and C-z messes the state completely up, the key-bindings don’t work as usual, but doesn’t put emacs in the background. Maybe it’s spacemacs’ fault. Just wondering.

                                                                                              1. 2

                                                                                                I use vanilla emacs, running under tmux. I just hit Ctrl-Z and it’s in the background, visible in the output of jobs. fg brings it back.

                                                                                                I think it’s your specific configuration in this case.

                                                                                                1. 1

                                                                                                  Thank you! Then indeed it’s probably the spacemacs configuration in the terminal mode. Will have to look there.

                                                                                                  1. 3

                                                                                                    Ctrl-z is the default toggle key for evil. You can set evil-toggle-key to some other shortcut:

                                                                                                    https://askubuntu.com/questions/99160/how-to-remap-emacs-evil-mode-toggle-key-from-ctrl-z

                                                                                                    1. 1

                                                                                                      Many thanks! It helped indeed and I learned something.

                                                                                                      I find it so strange, that Ctrl-Z has been chosen for this toggle mode, if this is the combination that is used in terminals to send programs to the background. Maybe there are not many people using emacs in the terminal with evil mode.

                                                                                                      1. 1

                                                                                                        The dude in the answers who modified the source to fix this really doesn’t understand the Emacs mindset ;)

                                                                                                2. 3

                                                                                                  Yeah, I prefer the window environment, especially for writing TeX documents and using pdf-tools to view it. Most of the time I have a terminal around somewhere, so I use both simultanously. For example, I have three windows open with the TeX code in one emacs frame, the pdf in another an then the terminal that runs latexmk -pvc.

                                                                                                  As an aside, isn’t a “normal” terminal emulator like rxvt already much faster than Emacs? What does st bring to the table?

                                                                                                  Yes, I used urxvt before but switched to st at some point. The differnces between those two are minor compared to a shell inside emacs. The blog post by Dan Luu showed that st performed quite well, and further highlights the point about throughput of the emacs shells. But yeah, the preference for st is mostly personal.

                                                                                                  1. 2

                                                                                                    Alright, that’s giving me LaTeX flashbacks from uni, I know just what you mean!

                                                                                                3. 1

                                                                                                  Most shell emulations (emacs has several) work fine as long as the executed programs do not produce much text, like cating a large file. When that happens, the shell becomes sluggish or freezes up, which in turn increases the cognitive burden, i. e. “May I execute this line or will this stop my workflow?” This is a major reason why I do not use the shell within Emacs. In general st feels much more resonsive than Emacs and that saddens me.

                                                                                                  I’ve found it’s long lines that cause Emacs to freeze. I tried working around this by having a comint filter insert newlines every 1000 characters, which worked but with really long lines that filter itself would slow down Emacs. One day I got fed up, and now I pipe the output of bash through a hacked version of GNU fold to do this newline insertion more efficiently. Unfortunately bash behaves differently when part of a pipe, so I use expect to trick it into thinking it’s not. Convoluted, but WorksForMe(TM)!

                                                                                                  (The code for this is in the fold.c and wrappedShell files at http://chriswarbo.net/git/warbo-utilities/git/branches/master ).

                                                                                                4. 2

                                                                                                  However, outside of the text editing sphere, the emacs implementation of thing such as shell, email, and a window manager always seem “almost there” but unfortunately not useable. This saddens me because I would love to never leave emacs.

                                                                                                  Shell depends, as @jnb mentions, for a lot of text it’s cumbersome, but especially with eshell, if you alias find-file and find-file-other-window (eg. ff and ffo) then you get something you can get very used to, very quickly.

                                                                                                  Maybe it’s not universal, but I’ve been using Gnus for a while now, and I just can’t change to anything else ^^. Integration to org-mode is great, the only thing that’s lacking imo is good integrated search with IMAP.

                                                                                                  Honestly, I can’t say anything about window managers. I use Mate, and it works.

                                                                                                  1. 1

                                                                                                    The search in Gnus and various other quirks (like locking up sometimes when getting new mail) caused me to finally switch to notmuch recently. I miss some of the splitting power, but notmuch gets enough of what I need to be content. The search in notmuch is really good, although it has a potentially serious hinderance, so I can’t recommend it without reservations.

                                                                                                    find-file from eshell is why I’ve been making a serious effort to try it out. I also implemented a /dev/log virtual target (M-x describe-variable <RET> eshell-virtual-targets) so I could redirect output to a new buffer easily.

                                                                                                  2. 2

                                                                                                    Regarding the shell. I also had shell issues but now use the shell exclusively in emacs. I work over ssh/tmux into a remote machine and only use the emacs term. I made a little ansi-term wrapper to allow the benefits of eshell (well the scrolling, yanking, etc) but it uses ansi-term still so it can use full screen stuff like htop. I’ve been using it for years now. Might help be worth checking out.

                                                                                                    plug: https://github.com/adamrt/sane-term

                                                                                                    1. 1

                                                                                                      Oh my God. Not only that is beautiful and perfectly suits what I was aiming to do, it also solves a couple of tangent problems I had with the section about loading the environment variables from .profile. Thank you so much!

                                                                                                      1. 1

                                                                                                        definitely will! I always run into issues with curses programs in emacs shell modes, which is the only thing that keeps me from using emacs shell exclusively,

                                                                                                    1. 8

                                                                                                      Working on learning C++ for a new job next month. (Junior dev, after 10 years of linux sysadmin / cloud / devops). This week chapter seven and up: https://www.learncpp.com/cpp-tutorial/71-function-parameters-and-arguments/ - last week finished a simple blackjack game (https://i.imgur.com/m2rm0u1.png). Also working through the c++ primer 5th edition.

                                                                                                      1. 6

                                                                                                        Effective C++ by Scott Meyers is a must-read after you are done with the basic introductions ;):

                                                                                                        http://shop.oreilly.com/product/0636920033707.do

                                                                                                        1. 2

                                                                                                          Or if you don’t feel like buying the books, g++ has a -Weffc++ flag that issues warnings for violations of the principles in either of the Meyers books. If you want those to be errors, also add the -Werror flag.

                                                                                                          1. 2

                                                                                                            Thanks! Will add that to the reading list. Any more tips?

                                                                                                            1. 4

                                                                                                              Also subscribe and listen to CppCast from Rob Irving!

                                                                                                              1. 4

                                                                                                                Thank you - I did not know about these! Apparently there is both a podcast and YouTube channel.

                                                                                                                1. 2

                                                                                                                  I believe the YT channel is no longer maintained.

                                                                                                          2. 1

                                                                                                            Hey too funny we’re both learning C++ at the same time! Though it seems you’re much farther along than I.

                                                                                                            I really didn’t like C++ primer and switched to Stroustrup’s Tour of C++ instead which I’m much happier with. I may steal the exercises from C++ primer just to get more practice though :)

                                                                                                            1. 2

                                                                                                              Your last sentence hits home. I need to do excersices and a lot off them to make sure I remember everything and can apply it. Good luck with learning

                                                                                                          1. 22

                                                                                                            To start, the ZFS filesystem combines the typical filesystem with a volume manager. It includes protection against corruption, snapshots and copy-on-write clones, as well as volume manager.

                                                                                                            It continues to baffle me how “mainstream” filesystems like ext4 forgo checksumming of the data they contain. You’d think that combatting bitrot would be a priority for a filesystem.

                                                                                                            Ever wondered where did vi come from? The TCP/IP stack? Your beloved macOS from Apple? All this is coming from the FreeBSD project.

                                                                                                            Technically, vi and the BSD implementations of the TCP/IP stack can be attributed to 4.xBSD at UCB; FreeBSD is not the origin of either.

                                                                                                            1. 10

                                                                                                              It continues to baffle me how “mainstream” filesystems like ext4 forgo checksumming of the data they contain. You’d think that combatting bitrot would be a priority for a filesystem.

                                                                                                              At least ext4 supports metadata checksums:

                                                                                                              https://wiki.archlinux.org/index.php/ext4#Enabling_metadata_checksums

                                                                                                              At any rate Ted T’so (the ext[34] maintainer) has said as far back as 2009 that ext4 was meant to be transitional technology:

                                                                                                              Despite the fact that Ext4 adds a number of compelling features to the filesystem, T’so doesn’t see it as a major step forward. He dismisses it as a rehash of outdated “1970s technology” and describes it as a conservative short-term solution. He believes that the way forward is Oracle’s open source Btrfs filesystem, which is designed to deliver significant improvements in scalability, reliability, and ease of management.

                                                                                                              https://arstechnica.com/information-technology/2009/04/linux-collaboration-summit-the-kernel-panel/

                                                                                                              Of course, the real failing here is not ext4, but that btrfs hasn’t been able to move to production use in more than ten years (at least according to some people).

                                                                                                              That said, ZFS works fine on Linux as well and some distributions (e.g. NixOS) support ZFS on root out-of-the-box.

                                                                                                              1. 3

                                                                                                                Of course, the real failing here is not ext4, but that btrfs hasn’t been able to move to production use in more than ten years (at least according to some people).

                                                                                                                I think it’s good to contrast “some people’s” opinion with the one from Facebook:

                                                                                                                it’s safe to say every request you make to Facebook.com is processed by 1 or more machines with a btrfs filesystem.

                                                                                                                Facebook’s open-source site:

                                                                                                                Btrfs has played a role in increasing efficiency and resource utilization in Facebook’s data centers in a number of different applications. Recently, Btrfs helped eliminate priority inversions caused by the journaling behavior of the previous filesystem, when used for I/O control with cgroup2 (described below). Btrfs is the only filesystem implementation that currently works with resource isolation, and it’s now deployed on millions of servers, driving significant efficiency gains.

                                                                                                                But Facebook employs btrfs project lead.

                                                                                                                There is also the fact that Google is now using BTRFS on Chromebooks with Crostini.

                                                                                                                As for opinions I’ve seen one that claims that “ZFS is more mature than btrfs ON SOLARIS. It is mostly ok on FreeBSD (with various caveats) and I wouldn’t recommend it on Linux.”.

                                                                                                                1. 2

                                                                                                                  I wouldn’t recommend it on Linux.

                                                                                                                  I’d still say that ZFS is more usable than lvm & linux-softraid. If only due to the more sane administration tooling :)

                                                                                                              2. 9

                                                                                                                Ext4, like most evolutions of existing filesystems, is strongly constrained by what the structure of on-disk data and the existing code allows it to do. Generally there is no space for on-disk checksums, especially for data; sometimes you can smuggle some metadata checksums into unused fields in things like inodes. Filesystems designed from the ground up for checksums build space for checksums into their on-disk data structures and also design their code’s data processing pipelines so there are natural central places to calculate and check checksums. The existing structure of the code matters too because when you’re evolving a filesystem, the last thing you want to do is to totally rewrite and restructure that existing battle-tested code with decade(s) of experience embedded into it; if you’re going to do that, you might as well start from scratch with an entirely new filesystem.

                                                                                                                In short: that ext4 doesn’t have checksums isn’t surprising; it’s a natural result of ext4 being a backwards compatible evolution of ext3, which was an evolution of ext2, and so on.

                                                                                                                1. 4

                                                                                                                  It continues to baffle me how “mainstream” filesystems like ext4 forgo checksumming of the data they contain. You’d think that combatting bitrot would be a priority for a filesystem.

                                                                                                                  Ext4 doesn’t aim to be that type of filesystem, for desktop use on the average user, this is fairly okay since actual bitrot in data the user cares about is rare (most bitrot occurs either in system files or empty space or in media files where the single corrupt frame barely matters).

                                                                                                                  If you want to check out a more modern alternative, there is bcachefs. I’ve been using it on my laptop for a while (until I stopped but now I’m back on it) and it’s been basically rock solid. The developer is also working on erasure coding and replication in a more solid way than btrfs currently has.

                                                                                                                1. 2

                                                                                                                  Reading this clarified some notions for me, and make me wonder: is it precisely Virtualization that we need ?

                                                                                                                  I got to the point that the large demand of virtual machine is more like a demand of a stable, known, dedicated environment whose resources can evolve than the fact to have hardware emulating hardware.

                                                                                                                  Instead of solving it by adding virtualization, maybe there are other means of managing hardware which would allow to allocate resources arbitrarily to some reproducible environment without having another instance of an operating system.

                                                                                                                  For instance, distributed file systems allow to give an “account” arbitrary amount of storage. It is more indirection than virtualization then.

                                                                                                                  Has anyone knowledge of some work toward alternative ways to manage hardware resources, and bonds with virtualization ?

                                                                                                                  1. 4

                                                                                                                    Containerisation is probably what you’re talking about.

                                                                                                                    Still one kernel, one set of device drivers, but individual environments for software to run.

                                                                                                                    LXC/LXD provides this as “System containers” - each container operates like a lightweight VM (including an init). Docker/etc provides this as “app containers” - each container is designed to run a single process/application.

                                                                                                                    1. 3

                                                                                                                      Also interesting in this context is gVisor, which further sandboxes containers by implementing Linux system calls in user space, acting as a guest kernel:

                                                                                                                      https://github.com/google/gvisor

                                                                                                                      1. 1

                                                                                                                        Somewhat related are rump kernels; kernels (or kernel components like drivers) running in user space.

                                                                                                                    2. 4

                                                                                                                      I believe Sun Microsystems’ Solaris zones were one of the first examples of this.

                                                                                                                      1. 1

                                                                                                                        Of course it was first invented by IBM as WPARs.

                                                                                                                        1. 3

                                                                                                                          Apparently WPARs were only introduced with AIX 6.1, or were you referring to something else? Solaris zones were, of course, extensively based upon FreeBSD jails.

                                                                                                                          1. 2

                                                                                                                            Hm, maybe I was thinking about LPARs, but they’re more like full VM. Thanks for the clarification.

                                                                                                                            1. 1

                                                                                                                              Yeah, LPARs have been around for a while (they predate AIX I believe) but, like you say, they’re not as lightweight as a zone/jail.

                                                                                                                      2. 4

                                                                                                                        It’s a good point. One advantage of VMs over containers that comes to mind is ability to run different OS. Also as seller of platform as a service (or whatever it’s called when we rent computers from Linode, Digital Ocean, AWS etc), VMs provide the level of sandboxing that is required but can’t be fulfilled by containers.

                                                                                                                        1. 3

                                                                                                                          One advantage of VMs over containers that comes to mind is ability to run different OS.

                                                                                                                          Unless you implement a loader for the binary format and system calls:

                                                                                                                          https://protocolsyntax.wordpress.com/2017/06/09/debian-7-wheezy-installation-in-freebsd-10-jail/

                                                                                                                          (Of course, whether you’d call this running a different OS depends on whether you define the OS to just be the kernel.)

                                                                                                                          1. 1

                                                                                                                            This is very interesting actually. Binary format loader might not be that complicated but implementing system calls would be a fairly big ask - second only to implementing the OS itself in terms of complexity. Imagine running Windows in user space of FreeBSD. That may also not be as performant or as true an implementation as VM.

                                                                                                                          2. 3

                                                                                                                            The term you’re looking for is Infrastructure-as-a-Service. You wouldn’t be selecting and maintaining an own operating system as part of PaaS. PaaS is about deploying an application; Heroku is one example of PaaS.

                                                                                                                            1. 1

                                                                                                                              I see, thanks for clarifying!

                                                                                                                            2. 2

                                                                                                                              A bit like switching between processes needs no particular support from each and every binary to be setup by the operating system.

                                                                                                                              Yes switching os. It’s enjoyable while discovering new systems.

                                                                                                                            3. 3

                                                                                                                              It’s not required to only run complete OS images. Unikernels for instance.

                                                                                                                              What virtualization gives you is another level of protection hierarchy. A VM is just a “process” except it can create memory protected processes within itself, unlike threads.

                                                                                                                              This is maybe possible at the kernel level, but it requires a system call for everything, which offers little flexibility. VMM allows the code it’s running to organize itself.

                                                                                                                              1. 1

                                                                                                                                I never noticed that there is no generic way to nest memory-protected execution environments besides VMs.

                                                                                                                              2. 1

                                                                                                                                Maybe paravirtualization is the first step of a process of merging the host OS with the guest OS.

                                                                                                                              1. 42

                                                                                                                                This is what running a VM in Microsoft Azure means. Microsoft controls the underlying datacenter hardware, host OS, hypervisors etc.

                                                                                                                                There is no defense against sustained physical access.

                                                                                                                                1. 10

                                                                                                                                  I think the difference raymii is making is between online and offline access - yes, they can unrack the servers and look at your disks with a magnifying glass, but online access where they can log in live to your running instance is a different threat model. If you rack hardware somewhere, sure, they have your hardware, but they most likely don’t have (an equivalent of) the root password. This story surprised me.

                                                                                                                                  1. 18

                                                                                                                                    But we’re talking about virtual machines here, right? So you don’t need to unrack anything; your magnifying glass is just /proc/$(pgrep qemu)/mem (or whatever the hyper-v equivalent is), to peruse at your leisure, online, from the host.

                                                                                                                                    (And even in the case of rented physical servers, there are still probably BMCs and such in scope that could achieve analogous things.)

                                                                                                                                    1. 2

                                                                                                                                      But that is still more work than just executing commands via an agent that’s already running. You still have to do something to get root access to a specific machine, instead of being able to script against some agent and accessing all machines.

                                                                                                                                      Leaving your door unlocked is one thing; setting it wide open with a sign “Enter here” is another.

                                                                                                                                      1. 2

                                                                                                                                        On the plus side, though it is “easy” it also appears to be logged and observable within the VM, which is the part most obviously unlike actual backdoors.

                                                                                                                                    2. 13

                                                                                                                                      There is absolutely nothing to be done from within a VM to prevent the host flipping a bit and backdooring it arbitrarily, or snapshotting it without shutting it down and doing the same. I’d be very surprised all the big names don’t have this functionality available internally – at least Google support live migration, which is the same tech.

                                                                                                                                      There are open toolkits for doing arbitrarily nasty poking and introspection to a running VM, e.g. volatility framework

                                                                                                                                      Hard to point fingers at Microsoft here

                                                                                                                                      1. 3

                                                                                                                                        Moreover, Live Migration of VM’s is a functionality available in widely deployed VMware ESXi software since 90’s. I suppose even longer than that on Big Iron.

                                                                                                                                      2. 2

                                                                                                                                        They can access the memory. That is equivalent to a root password. IMHO CPU supported memory encryption like Intel SGX is snake-oil at most, if you are targeted by the phisycal host of your VM.

                                                                                                                                        Hosting in the cloud is a matter of trust and threat analysis.

                                                                                                                                      3. 5

                                                                                                                                        I’m really suprised, it seems that everybody thinks it’s common knowledge and they seem to think it’s normal. I don’t like my hosting provider having this level of access to my data and machines. We are smart enough to find a solution to this, hosting infrastructure without giving up on all security…

                                                                                                                                        1. 26

                                                                                                                                          With managed virtualized infrastructure, “this level of access” is completely unavoidable. They run the virtualized hardware your “server” is running on; they have complete memory and CPU state access, and they can change anything they want.

                                                                                                                                          I guess it makes backdooring things marginally simpler to write a guest-side agent, but their actual capabilities are totally unchanged.

                                                                                                                                          This is something that ought to be common knowledge, but unfortunately doesn’t seem to be.

                                                                                                                                          1. 1

                                                                                                                                            The risk of your provider taking a snapshot of your disk and ram is always there with virtualization. But, you could encrypt the disk, which would make it harder for them (they have to scan ram for the key, then decrypt). But just an agent with root privileges… what bothers me the most I guess is that it is not made clear. A note in /etc/issue or motd with “we have full root acces in your vm, read http://kb.ms.com/kb77777 for more info” would make it clear right from the get-go.

                                                                                                                                            1. 10

                                                                                                                                              (they have to scan ram for the key, then decrypt)

                                                                                                                                              Not even that, just put a backdoor in the BIOS, boot loader, initramfs, or whatever code is used to unlock the encrypted disk to intercept key entry.

                                                                                                                                              1. 3

                                                                                                                                                Do you know of any isolated / trusted vm like solution? Where provider access is mitigated?

                                                                                                                                                1. 12

                                                                                                                                                  No. Even the various “gov clouds” are mainly about isolation from other customers and data center location.

                                                                                                                                                  The cloud providers are executing the cpu instructions that the VM image provided by you (or picked from the store) contains. There isn’t any escaping that access level.

                                                                                                                                                  The only option is to actually run your own physical hardware that you trust in an environment you consider good enough.

                                                                                                                                                  1. 4

                                                                                                                                                    In my comment about host and TLA resistance, I had a requirement for setups resistant to domestic TLA’s that might give orders for secrets to be turned over or use advanced attacks (which are getting cheaper/popular). It can be repurposed for an untrusted, host setup.

                                                                                                                                                    “If it has to be U.S. and it’s serious, use foreign operated anti-tamper setup. The idea is all sensitive computations are run on a computer stored in a tamper detecting container that can detect radiation, temperature changes, power surges, excessive microwaves, etc. Tamper detection = data wipe or thermite. The container will be an EMSEC safe and the sensors/PC’s will always be located in a different spot in it. The system is foreign built and operated with the user having no control of its operation except what software runs in deprivileged VM’s in it. Status is monitored remotely. It helps to modify code so that most sensitive stuff like keys are stored in certain spot in memory that will be erased almost instantly.”

                                                                                                                                                    The clouds aren’t built anything like this. They have total control like those in physical possession of hardware and software almost always have total control. They can do what they want. You won’t be able to see them do it most of the time without some clever detection mechanisms for security-relevant parts of the stack. That’s before we get to hardware risks.

                                                                                                                                                    Bottom line: external providers of computing services should always considered trusted with full access to your data and services. By default. Every time. It’s why I encourage self-hosting of secrets. I also encourage pen, paper, and people for most confidential stuff. Computers aren’t as trustworthy.

                                                                                                                                                    1. 3

                                                                                                                                                      What is your threat model?

                                                                                                                                                      There is something based on selinux for Xen https://wiki.xen.org/wiki/Xen_Security_Modules_:_XSM-FLASK which can by design prevent the privileged “dom0” from reading the memory of nonprivileged guest domains. But that assumes you trust your provider to actually implement this when they say they do.

                                                                                                                                                  2. 7

                                                                                                                                                    A note in /etc/issue or motd with “we have full root acces in your vm, read http://kb.ms.com/kb77777 for more info” would make it clear right from the get-go.

                                                                                                                                                    I think this is a combination of “common knowledge, so not worth mentioning specially” for users who already know this and “let sleeping dogs lie” for people who don’t already know. I mean, why press people with their noses on a fact that the competitor is equally mum about? Seems like a bad PR move; you’d get clueless people all alarmed and leaving your platform for reasons that are totally bogus, as any competitor has the same kind of access.

                                                                                                                                            1. 10

                                                                                                                                              My only real complaint about getting rid of the /usr split is that /usr still exists as something other than a symlink to /. The name /usr is basically meaningless. If you aren’t going to support /usr on a separate partition, why bother having /usr at all? Just shallow your hierarchy:

                                                                                                                                              • /bin
                                                                                                                                              • /dev
                                                                                                                                              • /local
                                                                                                                                              • /share
                                                                                                                                              • and so forth

                                                                                                                                              Keep /usr as a symlink to / for compatibility. I’ve suggested this to folks a few times over the years and haven’t gotten much response. By symlinking /bin and friends, you keep /usr as a naming convention, even though as Landley’s post points out, it is one that has lost its relevance.

                                                                                                                                              1. 2

                                                                                                                                                Having a separate /usr leads to a system of layered configuration. You have the distributor-originating artifacts in /usr, you have your configs in /etc, and local databases/pods/containers/caches/spools/etc. in /var.

                                                                                                                                                Also see systemd-tmpfiles(8).

                                                                                                                                                Coincidentally something I recall from Haiku.

                                                                                                                                                1. 3

                                                                                                                                                  From a 60ies/70ies UNIX perspective, I can understand this line of reasoning (limited disk space, more primitive file systems). But we can just put every application/configuration in its own flat filesystem namespace to avoid name clashes. It also makes it possible to have multiple versions of applications or configuration files available at the same time.

                                                                                                                                                  (E.g. nix and guix follow this approach and to a lesser extend macOS application bundles.)

                                                                                                                                                  1. 3

                                                                                                                                                    The central problem with just refactoring the file system structure to something that makes more intuitive sense from a high-level view is that it makes life a living hell for package maintainers. They have to grind through the process of not just making sure binaries and libraries get where they should be, they also have to make sure that the software will actually work correctly with things moved around. Some software (especially stuff using autoconf) can deal with this pretty okay, but other stuff is more nightmarish. The closer your system is to “legacy,” the more likely $random_third-party_app is likely to more or less work out of the box.

                                                                                                                                                2. 1

                                                                                                                                                  having circular symlinks can create some interesting tarballs, or make the linker recurse through it a few times.

                                                                                                                                                1. 11

                                                                                                                                                  What is Yggdrasil?, its an end to end encrypted networking protocol.

                                                                                                                                                  It’s also a sign that you’re Officially Old, if you’re wondering why you’d be installing a completely dead distro in another, living distro.

                                                                                                                                                  1. 4

                                                                                                                                                    The ‘find peers’ part of the title was a bit weird. But then I found it completely plausible that there is some community that specializes in running old, obscure Linux distributions. There were so many unique (but not always great) distributions: Yggdrasil, EasyLinux, DLD, Caldera OpenLinux, TurboLinux, Corel Linux, Storm Linux, Stampede.

                                                                                                                                                    1. 2

                                                                                                                                                      My first distro will always have a special place.

                                                                                                                                                      Coming from Atari ST and Amiga I spend days and days trying to get X to work (this was before internet being available to the common people and search engines). When I finally got it up it was like completing a game: now what?

                                                                                                                                                      Also FVWM (I think it was) looked horrible and unusable being used to the Atari ST and Amiga desktop environments.

                                                                                                                                                    1. 3

                                                                                                                                                      This is the one tool that is making me seriously consider flipping my workflow upside down and switching from vim + git cli to emacs.

                                                                                                                                                      That said, if anyone knows of something similar to magit that doesn’t require emacs, I’m interested in hearing about it! I’ve played with tig[0] a bit, but always go back to using git from the cli..

                                                                                                                                                      1. https://jonas.github.io/tig/
                                                                                                                                                      1. 3

                                                                                                                                                        I recently started using spacemacs in vim mode (which is its default) and it’s lovely. It’s been a much better experience getting a decent environment going than I’ve ever had with vim. Haven’t dug into Magit yet but I’m eager to. The docs for spacemacs are a bit scattered but this was the most useful for me to get going:

                                                                                                                                                        https://github.com/syl20bnr/spacemacs/blob/master/doc/BEGINNERS_TUTORIAL.org

                                                                                                                                                        1. 1

                                                                                                                                                          How long did you use vim workflows before switching? I’ve been a fulltime vim user for about 10 years, and I’m afraid that my productivity will take a serious spill as I try to unlearn muscle memory and build new ones. If you used vim for a while, I’m curious how that experience was for you.

                                                                                                                                                          1. 5

                                                                                                                                                            I switched mostly to emacs + evil (also used for Spacemacs) after a decade and a half or so on various vi implementations. Evil is very complete, evil often feels more like a complete vi implementation in Emacs Lisp rather than ‘vi emulation’. So far I didn’t have to unlearn any muscle memory.

                                                                                                                                                            I used Spacemacs for the initial switch, but I found it to be very buggy and slow. So, at some point I just switched to Emacs + hand-picked packages. I use use-package to load/install packages that I install with Nix and manage with home-manager. I use general to set up spacemaps-like key bindings (e.g. SPC-p-f to find a file in a project SPC-b-b to search buffers, etc.).

                                                                                                                                                            However, the Emacs ecosystem can be a bit overwhelming, so Spacemacs is a good entry point.

                                                                                                                                                            1. 1

                                                                                                                                                              I never spent longer than six months or so using vim full time, so I’m in a very different situation. You’ll be able to put much of your muscle memory to good use in spacemacs, using the same keystrokes to navigate and edit within a buffer. However you’ll need to re-learn many other common tasks.

                                                                                                                                                              You will absolutely take a big productivity hit at first, but if you stick with it then you’ll start feeling comfortable pretty quickly. Learning is its own reward! :)

                                                                                                                                                          2. 2

                                                                                                                                                            I have used vimagit and fugitive (both Vim/Neovim plugins) together for a while now. Vimagit is far from equal to magit in terms of power, but provides my Vim setup with the main features I missed from magit (visual range staging, easy amends, etc.). Fugitive is also useful on its own, but I currently mostly use it to asychronously push/pull from within Neovim (as vimagit does not yet provide these features itself).

                                                                                                                                                            1. 2
                                                                                                                                                              1. There’s nothing wrong with converting to the true faith ;)
                                                                                                                                                              2. There was a Go project called lazygit that was posted here a while back that a few people claimed was quite nice (haven’t tried it myself, so I can’t say), and reminded me of Magit – maybe that would be worthwhile?
                                                                                                                                                              1. 2

                                                                                                                                                                One thing I’m wondering about trying is a TUI-only Emacs configuration with a virtually completely empty .emacs file, flset up for just magit and nothing else. I’m wondering if the load time with a maximally stripped-down configuration would be short enough to make it feasible to use magit in a non-Emacs-oriented workflow. So, edit in something else, launch Emacs+magic in terminal, stage changes, close Emacs.

                                                                                                                                                                The Emacs daemon mode might be an option too but it adds a bit of complexity to the setup. :/

                                                                                                                                                                1. 1

                                                                                                                                                                  I use tig a lot for browsing commits, but for making commits, vimagit is a pretty cool magit-inspired vim-based git add -p type thing.

                                                                                                                                                                  (Though I keep using add -p anyway lol)

                                                                                                                                                                  1. 1

                                                                                                                                                                    Fugitive is amazing. I use it extensively at work.

                                                                                                                                                                    1. 1

                                                                                                                                                                      I’ve been using its most basic functionality (Gblame, and gutter highlighting) for at least 1 year now. Perhaps it’s time to invest more time in learning the ‘advanced’ features.

                                                                                                                                                                      1. 1

                                                                                                                                                                        Yeah! It’s got good diffing, committing, etc.

                                                                                                                                                                      2. 1

                                                                                                                                                                        If you like fugitive, I’d also recommend checking out gina.

                                                                                                                                                                        1. 1

                                                                                                                                                                          What does it do differently/better?