1. 16

    I’d call this “drift”, like linguists see in language usage, but in this case due to scope and purpose changing.

    It’s easier to adjust things as you need then to take the API’s and restructure them as a group all at once (inviting a mass outrage as well).

    We’re also not ready for the next major paradigm shift, and because we don’t know what that “is”, we live in the moment and “change as needed”. Because we know what have, and what we think we need.

    1. 1

      I agree with Doug McIlroy - specialized and on target, “. For the ls example, it’s about file attributes. Now … you can have filesystems with different attributes, so perhaps having a plug-in architecture that matches underlying filesystem to only those attributes might make sense to scope the utilities use of those few flags, likewise what the man page shows. Yes there are issues with modals, but having concise utilities has some powerful benefits, especially if I broadcast them to a million nodes of a cluster/cloud to achieve a time critical parallel effect and response.

      1. 8

        The reason for UNIX-like OS’s having low level abstractions like byte streams, filesystems of blocks/inodes/directories and hundreds of other like things, was to retain the efficiencies of the low level machine exported to time multiplexed processes on a single processor, where you could build upon them whatever complexity you’d like.

        Many OS types I knew from the 69’s-70’s never even saw a LISP Machine (LISPM) OS, nor had much contact with such systems that never had any interest in retaining low level abstractions, because those users were more interested in high level abstractions to be fleshed out rapidly so they could be used on the machine, and then as they were used, they would be refined to make them efficient enough.

        We’re in a very different universe of multicore, cluster, and cloud, where you can buy all the cycles you want in the world for the nanoseconds you might need them. Waiting to code/debug/process/parallelize high to low level to run on a low-level OS takes more time than the high level abstraction itself needs to function. So there is “anti-efficiency” many face.

        Why is this happening? Well … standards. Like POSIX. Now that we’ve finally made them work, they can’t be replaced, too much depends on them. And, unlike on LISPM’s, you can’t just rebuild the APIs on the fly to suite your need, because several billion others need to work with them at the same time, and they want to build there own differently for both good and bad reasons.

        There’s no escape through metastandards either, because that evasion will be countered in turn by efficiency arguments (which are mostly specious because people work off of fear and not measured, pragmatic study because that takes considerable talent/skill/work, which often isn’t valued as much as it should be).

        We are at a crossroads where we need something to break the logjam w/o being sucked back into to old standards, but since no one trusts anyone to do it, it never goes anywhere. Such is what happens when the free OS’s sucked all the air out of OS research decades ago because it was no longer necessary, because the truth of a common low level OS was all we would ever need.

        Well guess what, we have it now, and by golly its nowhere near what we need for the future.

        1. 1

          Well guess what, we have it now, and by golly its nowhere near what we need for the future.

          Its also nowhere near to what we need for now. I literally created (and still create) same abstractions over and over again just to get away from this.

        1. 1

          Nice article. It’s almost as if things like reuse only make sense if you have augmented development where the codebase is entirely within its scoped repositories. Then the object abstractions can be unified across this scope, with commonality dropping down back to the base objects, with the high layers becoming populated with uniquely used code hoisted into its lesser frequented objects.

          As if such metaphors are beyond the reasonable care of mere mortal programmers, but only automated ones with the diligence to constantly adapt the object model as the codebase changes.

          This might be seen as insulting to most programmers.

          1. 3

            Suggest that teaching programming is viewed as a applied skill by many, like teaching how to be an auto mechanic with four cylinder imports, so the choice of C++ is equally bland for wide applicability.

            Suggest a better approach is from the start always teaching how to write a good, understandable program. It’s no different from writing a good essay, math proof, or sonnet. What I find is that even with those who don’t really want to continue to learn/apply more, they seem to be versatile enough to “shift language gears” to another, because they recognize “something’s missing”, locate it, and restore the “form” because it was what was practiced from this “habit”.

            To this end Python has been of great help. It’s got some dumb stuff like for for/while loops with else, so not perfect. But it’s concise enough that coaxes a form of obvious style regardless, because beginners will “blow off” style guides (they really don’t see the point, just “make work”). And if one of my student’s crap’s out some Python, all I need to do is pick out another student’s adequate job (maybe four of them), then they get embarrassed and resubmit because it doesn’t take much to meet appearance.

            I’ve also taught scientists to forgo IDL (and Fortran) in favor of Python, fighting the institutional bias. Most scientific code is horribly written, which is too easy with such languages, to the point that they even forget why they coded something the way that they did, as the obtuse mounts up. It’s annoying for them at first, but the dividends pile up fast because in proving the science results clarity of form adds to the confirmation they are on the right track. The irony is that

            After this, “form” is followed by function - thinking algorithmically. A sense of the machine matters, as does the algorithm itself.

            C and C++ no longer do this well, because the machine’s complexity/variability has grown too high.

            Even worse a choice is the language GO. It’s a sloppy way of getting a lot of crap code running quickly/performantly, with a lesser trained staff - this is exactly what it was intended for. But like Fortran and scientific code, the swampy mess that results may take a while to get into a finished state. It’s as if they took the worst of C/C++ and built on top of that.

            1. 5

              These projects are way easier to deploy when you use Docker. It will hide 90% of the stupid stuff that is automated for you in the Dockerfile.

              1. 4

                Is this a bug or a feature? One could fail to explain the “stupid stuff”, and be tripped up by a part of it that really mattered.

                It’s not enough to be “open source”, it needs to be transparent and credible too, so that one can reasonably maintain it. Hiding things in a Dockerfile doesn’t pass this test.

                1. 8

                  I think it’s probably “enough” for a project to be whatever the maintainers want it to be. A Dockerfile is just an abstraction like a Makefile or even a shell script; the built artefact is effectively a tar file containing the assembled software, ready for deployment. I’m not a fan of the ergonomics of the docker CLI, but the idea that you’re “hiding” anything with Docker any more than you are with any other packaging and deployment infrastructure seems specious at best.

                  1. 0

                    Instead of focusing on a single word, try considering the other, opposing two - being credible and transparent. Clearly this isn’t.

                    For one thing, the reason you don’t do this is that it’s easy to be taken advantage of and place exploitative code in a big pile of things. For another it’s bad form to not communicate your work well, because maintainer’s struggling to deal with an issue don’t create more (and possibly even worse) versions they might claim “fix” something, and in the fog of code it might not be easy to tell which end is up.

                    I’m surprised you’d defend bad practice, since nearly everyone has had one of these waste a few hundred hours of their time. Your defense sounds even more specious than focusing on the wrong word and missing the point of the comment.

                    1. 2

                      I highlighted the word enough because your comment seems to have come from a place of entitlement and I was trying to call that out. The project doesn’t owe you anything.

                      Indeed, most of my comment was attempting to address your apparent suggestion that using a Dockerfile instead of some other packaging or deployment mechanism is somehow not transparent (or, I guess, credible?). I’m not really defending the use of Docker in any way – indeed, I don’t have any use for it myself – merely addressing what I think is specious criticism.

                      Regardless of what point you were trying to make, your comment comes across as an effectively baseless criticism of the project for not delivering their software in a way that meets your personal sense of aesthetics. Things are only hidden in a Dockerfile to the extent that they are conveniently arranged for consumption by consumers that do not necessarily need to understand exactly how they work. This isn’t any different to any other process of software assembly that abstracts some amount of the internal complexity of its operation from the consumer; e.g., I am not in the habit of reviewing the Linux kernel source that executes on my desktop.

                      If you want to know how the Dockerfile works, you could always look inside! It seems no more or less transparent or credible than a shell script or a markdown document that executes or describes a similar set of steps.

                      1. -1

                        I build them so I know whats inside. You’re looking for something to be outraged at, and find it in my words.

                        Perhaps you can defend those who write programs with meaningless variable names, and stale comments that no longer reflect the code they were next to.

                        Point your anger at somewhere else. Meanwhile who speaks up for something unintentionally vague or misleading. Or are you also going to defend syntax errors and typos next.

                        1. 1

                          I’m not angry – merely flabbergasted! How is a Dockerfile “vague or misleading”? By definition, it contains a concrete set of executable steps that set up the software in the built image.

                2. 1

                  I hate the docker setups that are just one piece of the setup and you are expected to spend a few days writing a docker compose file to piece together the whole thing

                  1. 1

                    Which problems do you encounter when writing docker-compose files? I’ve mostly had the experience that the upstream Dockerfile is horrible (for example Seafile is trying to pack everything into a single image - which causes feature-creep for the setup scripts) - but writing docker-compose.yaml always felt rather straight forward (besides Volumes, I’m still confused by Volume management on occasion).

                1. 6

                  This happens with a lot of bigger “open source” projects. Basically code dumps without any consideration for making it easy to deploy it by someone who’s not a developer and intimately familiar with the code base. I guess as good a definition of “devops” as any.

                  The easiest to improve this is to get it properly packaged for the most popular Linux distributions. This is a great QA tool as well. If your software (and its dependencies) are easy to package it means you have your dependencies under control, and you can build the software from source at any point.

                  Unfortunately nowadays you can be happy to get any of those projects to even work in a provider Docker container. Running them as a production service yourself is a completely different story, and practically impossible.

                  1. 21

                    To be fair, I don’t think every piece of open source is really meant to be run as-is by users so much as “here’s what we use to run this, you can build off it if you want.” It’s perfectly fair to release your internal tools for the purposes of knowledge sharing and reference without intending to support public consumption of them by non-developers.

                    Further, it looks like the author made minimal effort to actually use what is distributed as it was meant to be:

                    • The suggested route of using docker images wasn’t used. Wanting to understand something and be able to run it without a pre-built image is fine, but totally skipping the intended usage and trying to roll your own from the beginning is only likely to make the system harder to understand.
                    • The projects appear to provide release packages, yet he pulled the source directly from git, and at whatever state master happened to be in at the time, rather than a branch or tag. At least one of them looks to be failing CI in its current state, so it’s not even clear that what he had was a correctly functioning version to start with.
                    • He’s ignored the npm-shrinkwrap provided and automatically upgraded dependencies without any indication or testing to confirm that they will work. While it would be great to think that this wouldn’t be an issue, the state of npm is not such that this is a realistic expectation.
                    1. 3

                      Where is the purpose of knowledge sharing when you do not make things in order to be understandable? Knowledge sharing is not just take my stuff and understand. You have to make it understandable and be sure the person understand it well. That’s why you have wiki and documentation, to facilitate the understanding.

                      Where do you find the suggested route is Docker ? You know, when you try to deploy something, you begin by somewhere. I read the install part of one the repository, Firefox Accounts Server, as I do for a lot of application I install and I followed the process. The written process is git clone and npm install. After some research, I discovered there was not an unique repository needed but there are several links together. Where is it written? How can I suppose to know?

                      You can’t say I did minimal effort when I took so much time on it. I am used to deploy stuff and configure application. I configure by my own each microservice in order to make it works. The problem was after three days, I still have to guess things by my own, understand it, configure it properly and fix issues I got. I am sorry but it is too much. It is not my job to make the application easily understandable and easy to deploy. It is the job of maintainers and it is what I said in my blog post.

                      When I compare other applications I deployed, and some of them are bigger than this, FXA has a lot of work to do. The master branch is actually a development branch, there is no master branch and the documentation told you to pull from it to deploy in production. :o

                      Here we have just a big thing which is our stuff, deal with it and make it workable if you want. I made a try and failed. It is not suppose to be deploy by someone who is not working in this project full time. That’s all and it is questionable when it is a Mozilla Foundation and who is publicly saying that privacy matter.

                      1. 5

                        Knowledge sharing is not just take my stuff and understand. You have to make it understandable and be sure the person understand it well.

                        Your opinion is not universal.

                        Different cultures handle this problem differently. Some culture/language pairings are listener-responsible and some are writer-responsible.

                        1. 2

                          Clearly. I think the best way is to have writer and listener responsible but it is not the debate here I guess.

                          1. 0

                            I’m in agreement. Can’t understand how people can defend bad practice of the art. Why would anyone who cares about good work defend anything like this? It’s like they are working against themselves, karmic-ally setting themselves up for a later fall through someone else’s failing, … for no sensible gain.

                          2. 1

                            Shouldn’t they both be “responsible”? And please tell me which cultures? Are we talking professional vs unprofessional, or what? I’ve worked in hundreds of different cultures worldwide over many decades, and I’ve never seen a claim like this.

                            1. 2

                              Come to Asia, never cease being frustrated.

                              1. 1

                                Been there. Once I understood “hectoring”, learned pretty quick how to generate a larger, louder response.

                      2. 4

                        Here is some more concrete documentation that I found:

                        Run your own Firefox Accounts Server https://mozilla-services.readthedocs.io/en/latest/howtos/run-fxa.html

                        Self Hosting via Docker https://github.com/michielbdejong/fxa-self-hosting/blob/master/README.md

                        Installing Firefox Accounts https://webplatform.github.io/docs/WPD/Infrastructure/procedures/Installing_and_theme_Firefox_accounts/

                        The ‘code dump’ argument is a bit odd .. these projects are all super accessible and being developed in the open on Github. No project is perfect. If you think something specific, like self-hosting documentation, is missing, file a bug or make the investment and work on it together with the devs. Open source is not one way.

                      1. 2

                        Can someone ELI5 why Firefox is not to be trusted anymore?

                        1. 4

                          They’ve done some questionable things. They did this weird tie-in with Mr. Robot or some TV show, where they auto-installed a plugin(but disabled thankfully) to like everyone as part of an update. It wasn’t enabled by default if I remember right, but it got installed everywhere.

                          Their income stream, according to wikipedia: is funded by donations and “search royalties”. But really their entire revenue stream comes directly from Google. Also in 2012 they failed an IRS audit having to pay 1.5 million dollars. Hopefully they learned their lesson, time will tell.

                          They bought pocket and said it would be open sourced, but it’s been over a year now, and so far only the FF plugin is OSS.

                          1. 4

                            Some of this isn’t true.

                            1. Mr. Robot was like a promotion, but not a paid thing, like an ad. Someone thought this was a good idea and managed tto bypass code review. This won’t happen again.
                            2. Money comes from a variety of search providers, depending on locale. Money ggoes directly into the people, the engineers, the product. There are no stakeholders we need to make happy. No corporations we got to talk to. Search providers come to us to get our users.
                            3. Pocket. Still not everything, but much more than the add-on: https://github.com/Pocket?tab=repositories
                            1. 3
                              1. OK, fair enough, but I never used the word “ad”. Glad it won’t happen again.

                              2. When like 80 or 90% of their funding is directly from Google… It at the very least raises questions. So I wouldn’t say not true, perhaps I over-simplified, and fair enough.

                              3. YAY! Good to know. I hadn’t checked in a while, happy to be wrong here. Hopefully this will continue.

                              But overall thank you for elaborating. I was trying to keep it simple, but I don’t disagree with anything you said here. Also, I still use FF as my default browser. It’s the best of the options.

                            2. 4

                              But really their entire revenue stream comes directly from Google.

                              To put this part another way: the majority of their income comes from auctioning off being the default search bar target. That happens to be worth somewhere in the 100s of $millions to Google, but Microsoft also bid (as did other search engines in other parts of the world. IIRC the choice is localised) - Google just bid higher. There’s a meta-level criticism where Mozilla can’t afford to challenge /all/ the possible corporate bidders for that search placement, but they aren’t directly beholden to Google in the way the previous poster suggests.

                              1. 1

                                Agreed. Except it’s well over half of their income, I think it’s up in the 80% or 90% range of how much of their funding comes from Google.

                                1. 2

                                  And if they diversify and, say, sell out tiles on the new tab screen? Or integrate read-it-later services? That also doesn’t fly as recent history has shown.

                                  People ask from Mozilla to not sell ads, not take money for search engine integration, not partner with media properties and still keep up their investment into development of the platform.

                                  People don’t leave any explanation of how they can do that while also rejecting all their means of making money.

                                  1. 2

                                    Agreed. I assume this wasn’t an attack on me personally, and just as a comment of the sad state of FF’s diversification woes. They definitely need diversification. I don’t have any awesome suggestions here, except I think they need to diversify. Having all your income controlled by one source is almost always a terrible idea long-term.

                                    I don’t have problems, personally, with their selling of search integration, I have problems with Google essentially being their only income stream. I think it’s great they are trying to diversify, and I like that they do search integration by region/area, so at least it’s not 100% Google. I hope they continue testing the waters and finding new ways to diversify. I’m sure some will be mistakes, but hopefully with time, they can get Google(or anyone else) down around the 40-50% range.

                                  2. 1

                                    That’s what “majority of their income” means. Or at least that’s what I intended it to mean!

                              2. 2

                                You also have the fact they are based in the USA, that means following American laws. Regarding personal datas, they are not very protective about them and even less if you are not an American citizen.

                                Moreover, they are testing in nightly to use Cloudfare DNS as DNS resolver even if the operating system configure an other. A DNS knows all domaine name resolution you did, that means it know which websiste you visit. You should be able to disable it in about:config but in making this way and not in the Firefox preferences menu, it is clear indication to make it not easily done.

                                You can also add the fact it is not easy to self host datas stored in your browser. I am not sure they are not sold when there first financial support is Google which have based is revenue from datas?

                                1. 3

                                  Mozilla does not have your personal data. Whatever they have for sync is encrypted in such a way that it cannot be tied to an account or decrypted.

                                  1. 1

                                    They have my sync data, sync data is personal data so they have my personal data. How do they encrypt it? Do you have any link about how they manage it? In which country is it stored? What is the law about it?

                                    1. 4

                                      Mozilla has your encrypted sync data. They do not have the key to decrypt that data. Your key never leaves your computer. All data is encrypted and decrypted locally in Firefox with a key that only you have.

                                      Your data is encrypted with very strong crypto and the encryption key is derived from your password with a very strong key derivation algorithm. All locally.

                                      The encrypted data is copied to and from Mozilla’s servers. The servers are dumb and do not actually know or do crypto. They just store blobs. The servers are in the USA and on AWS.

                                      The worst that can happen is that Mozilla has to hand over data to some three letter organization, which can then run their supercomputer for a 1000 years to brute force the decryption of your data. Firefox Sync is designed with this scenario in mind.

                                      This of course assuming that your password is not ‘hunter2’.

                                      It is starting to sound like you went through this effort because you don’t trust Mozilla with your data. That is totally fair, but I think that if you had understood the architecture a bit better, you may actually not have decided to self host. This is all put together really well, and with privacy and data breaches in mind. IMO there is very little reason to self host.

                                      1. 1

                                        “The worst that can happen is that Mozilla has to hand over data to some three letter organization, which can then run their supercomputer for a 1000 years to brute force the decryption of your data. Firefox Sync is designed with this scenario in mind.”

                                        That’s not the worst by far. The Core Secrets leak indicated they were compeling via FBI suppliers to put in backdoors. So, they’d either pay/force a developer to insert a weakness that looks accidental, push malware in during an update, or (most likely) just use a browser sploit on the target.

                                        1. 1

                                          In all of those cases, it’s game over for your browser data regardless of whether you use Firefox Sync, Mozilla-hosted or otherwise.

                                          1. 1

                                            That’s true! Unless they rewrite it all in Rust with overflow checking on. And in a form that an info-flow analyzer can check for leaks. ;)

                                        2. 1

                                          As you said, it’s totally fair to not trust Mozilla with data. As part of that, it should always be possible/supported to “self-host”, as a means to keep that as an option. Enough said to that point.

                                          As to “understanding the architecture”, it also comes with appreciating the business practices, ethics, and means to work to the privacy laws of a given jurisdiction. This isn’t being conveyed well by any of the major players, so with the minor ones having to cater to those “big guys”, it’s no surprise that mistrust would be present here.

                                        3. 2

                                          How do they encrypt it?

                                          On the client, of course. (Even Chrome does this the same way.) Firefox is open source, you can find out yourself how exactly everything is done. I found this keys module, if you really care, you can find where the encrypt operation is invoked and what data is there, etc.

                                          1. 2

                                            You don’t have to give it to them. Firefox sync is totally optional, I for one don’t use it.

                                            Country: almost certainly the USA. Encryption: looks like this is what they use: https://wiki.mozilla.org/Labs/Weave/Developer/Crypto

                                        4. 2

                                          The move to Clouflare as dns over https is annoying enough to make me consider other browsers.

                                          You can also add the fact it is not easy to self host datas stored in your browser. I am not sure they are not sold when there first financial support is Google which have based is revenue from datas?

                                          Please, no FUD. :)

                                          1. 3

                                            move to Clouflare

                                            It’s an experiment, not a permanent “move”. Right now you can manually set your own resolver and enable-disable DoH in about:config.

                                      1. 3

                                        I love blog posts about failures. It usually prevents the audience from repeating a mistake.

                                        For this one, I’m not sure. Because I’m convinced there must be a solution to the problem. It’s (admittedly) difficult, but I don’t think giving up should be the end state

                                        Did you file bugs or reach out to devs on IRC? My assumption is that developers will consider this aa documentation bug that is worth resolving.

                                        1. 3

                                          You’re right, there’s little detail. But it’s a short blog post, and there are advantages in it not being tl;dr. More of a warning than a diagnostic. Think that was what was meant.

                                          I have noticed that sometimes when people do a long “tried this, didn’t work … tried this, didn’t work … tried this, didn’t work … “ that many pick the critique apart, even though playing “Twenty Questions” because nobody documented their work, thus the “guessing game”. You get criticized for not being a “good guesser”, or reading someones mind wrong.

                                          You shouldn’t have to guess or “mind read”. Consider it a part of writing clear code, that what it attaches to is likewise spoken for. I wish it was the case that all software that I’ve read to discern details of operation allowed me to “compensate” for lack of documentation, but that’s the rare case. Many often fix the code and don’t update even the comments.

                                          And I’ve had prior experience with Mozilla not unlike this, so not surprised.

                                        1. 1

                                          The article is flawed because the focus on eliminating loop-else needed an example that maintained the loop but didn’t have the loop-else. So it appears spurious.

                                          By resorting to the filter example, it effectively removed the loop thus no need for the loop-else.

                                          And yes loop-else is under justified and might not really “belong”. Just needs to be argued against better.

                                          BTW, it’s not a bad idea to remove spurious loops as well, as that can allow significant improvement in cases where, for example, the filter case allows for a more efficient/compact implementation (or parallelism).

                                          But then it’s not getting rid of the loop-else.

                                          1. 3

                                            Consider Python as a means to write good code. Perhaps it may be difficult to decide what “writing good code” means in say a Dropbox environment, verses other environments, and since one has obvious human limits it may be impossible to step back from that.

                                            More to the point with PEP 572, perhaps a threshold issue of how far to take a concept, and the manner of “selling” such to a much broader audience of Python programmers given the pattern suggested. Reminds of those I’ve seen who cross to another discipline, and then return later - they aren’t even aware of the perception changes that others notice, and the nuance others observe in scope of behavior.

                                            1. 5

                                              C matched the programming metaphors in UNIX because both occurred contemporaneously. That is why it was such a useful systems programming language, just as PL/1 did for MULTICS.

                                              If we evaluate systems programming languages against current need inside kernel/OS/utilities/libraries, won’t we come up short because the metaphors don’t match well anymore? Sure, we’ve stretched things with extensions to C, but that just makes C suffice for need, and no one will attempt to match metaphors to any new language because it’s shooting at TWO moving targets (the language and what it’s used for).

                                              Perhaps the success of C/UNIX derivatives forestalls their necessary concurrent replacement?

                                              1. 5

                                                After watching all the language wars play out, I started pushing the idea that a C replacement should keep its style and compatibility as much as possible. The same data types, calling conventions, automatic C FFI, extract to C for its compilers, whatever. Maybe close on the syntax like some competitors were. Then, on top of that, build something better without its shortcomings. Might be better module system, concurrency, macros, live coding a la Lisp/Smalltalk, and so on. It should be something that leverages the experience of those converting that plugs right into the ecosystem getting all its benefits. Plus, allows incremental rewrites of legacy codebases.

                                                Clay and Cyclone at least attempted something like this on safety side. I found another one that did on productivity side I’m submitting Sunday. I don’t see many attempts, though. Most take people far away from C model to then try to do something in a C model system.

                                                1. 4

                                                  I’m keeping an eye on zig.

                                                  1. 3

                                                    D.

                                                    1. 2

                                                      That’s a quite reasonable concern. Share it as well. Because minimalism has a fundamental advantage in and of itself in composition.

                                                      I think what replaces all of C (and all of UNIX) is in like manner such “composable minimalism”. But I’m not convinced that it will be “C like” or “UNIX like” at all, because the metaphors that are incompletely fit in this modern environment.

                                                      I greatly enjoyed working in Python for its clarity and focus on writing concise and understandable code, but what with the PEP 572 that is compelling van Rossum to step down as Python BDFL, once can see the limits of how far that can be taken. (He’s very emotional about what I regard as an “overreach”.)

                                                      I’ve been wrestling with Rust, attempting to rewrite my earlier kernel work in it in place of C, and it does have definite advantages. However, unlike C and Python, too much is “lost in translation” - code becomes obscure. Gets back to Ken Thompson’s comment in UNIX V5/6 “you are not expected to understand this” in doing his backwards coroutine context switch.

                                                      So again we are at a crossroads - we might need a new metaphor, but not have it in sight yet.

                                                      You want more low level hardware “involvement”, but wish to have the logic become more densely abstract to deal with complexity. You want greater “stop on a dime” debugging, but also “obviousness” in exposition to avoid much need for awkward comments. I’ve been thinking about AR/ML means to do augmented development as a means to bridging these, but we’ll see.

                                                      1. 1

                                                        Far as low-level Rust, you might find this work interesting given it’s about composing abstractions to deal with low-level stuff in embedded. The Tock people are also publishing interesting stuff. For now, I’m not sure if you were having problems due to the language itself, abstractions you were using, or some combination. Rust programmers are still in early exploration stage for that stuff.

                                                        There’s also the possibility, which I encourage, of using safe/proven C or assembly for unsafe stuff with safe Rust for the rest that can support better abstraction. By safe/proven, I’m talking types like Cyclone/Clay or automated solvers like Frama-C/SPARK. Even if manual, there’s probably only a small amount of code we’d need specialists for. If doing generic components with reuse, then even that might be reduced. To be clear, I’m just brainstorming here based on all the reuse I’m seeing in Why3-based, CompCert-based and Myreen et al-based work.

                                                        re AR/ML. I’ve been thinking of them for verification more than debugging. I just don’t have the depth required to know more past “attempt to apply the generic methods that work on everything to see what each accomplishes.” Monkey try thing, monkey see it work, monkey do more of that. That suggestion isn’t worth a Lobsters or IEEE submission, though. ;)

                                                  1. 2

                                                    It always amazes me the amount of blithe ignorance that allows pirate businesses to take off. SPAM started because early IETF/IAC didn’t care about policy, viruses started because uncritical MAC OS/Windows installation/isolation, and key leakage / weak passwords / uncritical AP&DSL NAT boxes allowed mass security attacks as well.

                                                    1. 3

                                                      Compiled languages definitely get an advantage out of strong typing and concrete API statement, because the planning of resource utilization allows many execution strategies, data layout, and even caching on code hoisting to be exploited. The more the desire to maximally “use” the hardware architecture, the more these grow to “fit” the hardware.

                                                      At the same time dynamic languages/JITs are getting better to fit the abstract expression of the programmer - functional programming can express compactly/accurately/clearly very elaborate programs, irrespective of the intermediate data types/APIs used in constructing them. The idea is to “fit” the nature of abstractions being manipulated rather than the nature of how they are executed.

                                                      I’m currently debugging a symbolic configuration mechanism that was prototyped in a week in a dynamic language, but is meant to function in an embedded OS with a very low-level language, as part of a bootstrap. It is taking months to finish, mostly due to adapting the code to work in such a programming environment - you alter the assemblage of primitives to build enough of a virtual machine to handle the semantics of necessary symbol processing. An oddball case, but it’s an example of the two (the virtue of this is that it allows enough “adaptability” at the low-level that you don’t need to drag along the entire dynamic programming environment to serve huge amounts of low-level code that otherwise fits the compiled model perfectly.

                                                      1. 2

                                                        Compiled/interpreted and strongly/weakly typed have little to do with each other. Ditto for low/high level: Swift compiles to machine code but good luck maintaining any cache locality with its collections.

                                                        1. 1

                                                          Depends on application. And yes we don’t have a good model for cache locality. How much can we get vs complexity to code/maintain?

                                                        2. 1

                                                          Can you elaborate on the strengths of the dynamic language that allowed you to prototype it so quickly? The difference in development time stated here is really striking.

                                                          1. 1

                                                            Sure. First about the problem - “how do you configure unordered modules while discovering the graph of how they are connected?”. The problem requires multilevel introspection of constructed objects with “temporary” graph assignments in multilevel discovery phase, then successive top down constructor phase with exception feedback.

                                                            The symbolic “middle layer” to support this was trivial to write in a language like Python using coroutines/iterators, and one could refactor the topological exception handling mechanism to deal with the corner cases quickly, by use of the annotation methods to handle the cases. So the problem didn’t “fight” the implementation.

                                                            While with the lower level compiled language, too much needed to be rewritten each time to deal with an artifact, so in effect the data types and internal API changed to compensate to fit the low-level model. Also, it was too easy to introduce boundary condition “new” errors each time, while the former’s more compact representation that didn’t thrash so much didn’t have this.

                                                            Sometimes with low level code, you almost need an expert system to maintain it.

                                                        1. -1

                                                          Repetitive, irrelevant, and … pointless.

                                                          Much of the man page corpus is just plain wrong. Many changed the code and never bothered to change the documentation. One can easily get misled.

                                                          UNIX/POSIX … is getting massively “bit-rotted” in its old age. Time for different metaphors, possibly maintained by ML to keep them effective and relevant?

                                                          1. 13

                                                            Do you have any examples of outdated manpages? Your comment is awfully vague.

                                                            1. 5

                                                              I run across examples semi-regularly, and try to report upstream when I find them (some upstreams are easier to report to than others). Mostly I’m pretty happy with manual pages, though.

                                                              Just recently, I noticed that pngcrush(1) on Debian is missing the newish -ow option. Upstream doesn’t ship a manpage at all, so Debian wrote one, but it doesn’t stay in sync automatically. Therefore I should probably report this to Debian. Upstream does ship a fairly complete pngcrush -v help text though, so I wonder if the manpage could be auto-synced with that.

                                                              I’m pretty sure I’ve reported a bunch of others in the past, but the only example that comes to mind at the moment is that privileges(5) on illumos used to be years out of date, missing some newer privileges, but it was fixed fairly quickly after I reported a bug.

                                                            2. 1

                                                              I really want to see documentation generated via machine learning systems. I wouldn’t want to use that documentation for anything, but I’d like to see it.

                                                            1. 1

                                                              This is an example of “low level” vs “high level” abstractions. One can climb around the path tree picking through files, or aggregate into directory path “bushel’s” , or poke away at a project’s components via an IDL.

                                                              These choices seem to be set by how the particular activity was set-up. Perhaps there are better ways to setup activities than optimizing default ways of working them?

                                                              Maybe the effort should be spent in asking / answering the question - is there a better way to do this? Rather than just “doing this”?

                                                              1. 4

                                                                This is going to get expensive for companies pretty quickly.

                                                                1. 7

                                                                  So that means buying more servers, with… Intel processors in them!

                                                                  1. 4

                                                                    Maybe. With the slowdown that KPTI incurs, it makes EPYC even more attractive.

                                                                    Now whether AMD can fab enough to keep up with demand is another question.

                                                                    1. 1

                                                                      Unfortunately AMD historically hasn’t had the management and the stockholder return to take on Fortress Intel. So Intel board hires weasel CEO’s to exploit the situation. Ironically, the tech is more than good enough.

                                                                  2. 2

                                                                    It already is. Across the board 30% hit is fairly common on cloud services. So the hit is worse than say, Apple and it’s battery/clock down issue, but clearly Intel weasels think they can outlast it - what are you going to do, not buy more Intel?

                                                                  1. 5

                                                                    Nicely written and turns the resolution of a bug into a victory of both discovery of the flaw as well as an deeper appreciation of the design of the language and its advantages of writing solid code with it.

                                                                    Having written/debugged much C, a very dangerous language for memory structures, examples like this reach to the heart of how to step beyond the theory of use of a language to the practice of use that illustrates both the risks and rewards of a language that improves upon C.

                                                                    1. 11

                                                                      Some people want easy access to the benefits of containerization such as: resource limits, network isolation, privsep, capabilities, etc. Docker is one system that makes that all relatively easy to configure, and utilize.

                                                                      1. 4

                                                                        Docker is one system that makes me wish Solaris Zones took off, which had all of that, but without the VM.

                                                                        1. 15

                                                                          What VM? Docker only requires a VM if you run it “non-natively”, on OS X or Windows.

                                                                          1. 1

                                                                            Docker isn’t running in a VM on Linux machines. It uses LXC.

                                                                            1. 10

                                                                              Docker hasn’t used LXC on Linux in a while. It uses its own libcontainer which sets up the Linux namespaces and cgroups.

                                                                          2. 1

                                                                            This is the correct answer. It’s a silly question. Docker has nothing to do with fat binaries. It’s all about creating containers for security purposes. That’s it. It’s about security. You can’t have security with a bunch of fat binaries unless you use a custom jail, and jails are complicated to configure. You have to do it manually for each one. Containers just work.

                                                                            1. 9

                                                                              security

                                                                              That is definitely not why I use it. I use it for managing many projects (go, python, php, rails, emberjs, etc) with many different dependencies. Docker makes managing all this in development very easy and organized.

                                                                              I don’t use it thinking I’m getting any added security.

                                                                              1. 3

                                                                                I don’t use it thinking I’m getting any added security.

                                                                                The question was “Why would anyone choose Docker over fat binaries?”

                                                                                You could use fat binaries of the AppImage variety to get the same, and probably better organization.

                                                                                Maybe if AppImages could be automatically restricted with firejail-type stuff they would be equivalent. I just haven’t seen many developers making their apps that way. Containers let you deal with apps that don’t create AppImages.

                                                                                1. 1

                                                                                  Interesting. So in effect you wish to “scope” portions for “protected” or “limited” use in a “fat binarie”. As opposed to the wide open scope implicit in static linking?

                                                                                  So we have symbol resolution by simply satisfying an external, resolution by explicit dynamic binding (dynload call), or chains of these connected together? These are all the cases, right?

                                                                                  We’d get the static cases handled via the linker, and the dynamic cases through either the dynamic loading functions or possibly wrapping the mmap calls they use.

                                                                                2. 1

                                                                                  That sounds genuine.

                                                                                  So I get that its one place, already working, to put all the parts in one place. I buy that.

                                                                                  So in this case, it’s not so much Docker as Docker, as it is a means to an end. This answers my question well, thank you. Any arguments to the contrary with this? Please?

                                                                                  1. 5

                                                                                    This answers my question well, thank you. Any arguments to the contrary with this? Please?

                                                                                    While I think @adamrt is genuine, I’m interested in seeing how it pans out over the long run. My, limited, experience with Docker has been:

                                                                                    • It’s always changing and hard to keep up, and sometimes changing in backwards breaking ways.
                                                                                    • Most container builds I come across are not reproducible, depending on HEAD of a bunch of deps, which makes a lot of things more challenging.
                                                                                    • Nobody really knows what’s in these containers or how they were built, so they are big black boxes. One can open it up and poke around but it’s really hard to tell what was put in it and why.

                                                                                    I suspect the last point is going to lead to many “we have this thing that runs but don’t know how to make it again so just don’t touch it and let’s invest in not touching” situations. People that are thoughtful and make conscious decisions will love containers. People inheriting someone’s lack of thoughtfulness are going to be miserable. But time will tell.

                                                                                    1. 1

                                                                                      Well these aren’t arguments to the contrary but accurate issues with Docker that I can confirm as well. Thank you for detailing them.

                                                                                3. 5

                                                                                  I think there’s something more to it than that. On Solaris and SmartOS, you can have security/isolation with either approach. Individual binaries have privileges, or you can use Zones (a container technology). Isolating a fat binary using ppriv is if anything less complicated to configure than Zones. Yet people still use Zones…

                                                                                  1. 4

                                                                                    I thought it was about better managing infrastructure. Docker itself runs on binary blobs of priveleged or kernel code IIRC (dont use it). When I pointed out its TCB, most people talking about it on HN told me they really used it for management and deployment benefits. There was also a slideshow a year or two ago showing security issues in lots of deployments.

                                                                                    What’s the current state in security versus VM’s on something like Xen or a separation kernel like LynxSecure or INTEGRITY-178B?

                                                                                    1. 5

                                                                                      Correct. It is unclear the compartmentalization aspect of containers to security specially.

                                                                                      I’ve implemented TSEC Orange Book Class B2/B3 systems with labelling, and worked with Class A hardware systems that had provable security at the memory cycle level. Even these had intrusion evaluations that didn’t close, but at least the models showed the bright line of where the actual value of security was delivered, as opposed to a loose, vague concept of security present as a defense here of security.

                                                                                      FWIW, what the actual objective that the framers of that security model was, was program verifiable object oriented programming model to limit information leakage in programming environments that let programs “leak” trusted information to trusted channels.

                                                                                      You can embed crypto objects inside an executable container and that would deliver a better security model w/o additional containers, because then you deal with issues involving key distribution w/o having the additional leakage of the intervening loss of the additional intracontainer references that are necessary for same.

                                                                                      So again I’m looking for where’s the beef instead of the existing marketing buzz that makes people feel good/scure because they use the stuff that’s cool of the moment. I’m all ears for a good argument for all this things, I really am, … but I’m not hearing it yet.

                                                                                      1. 1

                                                                                        Thanks to Lobsters, I already met people that worked in capability companies such as that behind KeyKOS and E. Then, heard from one from SecureWare who had eye opening information. Now, someone that worked on the MLS systems I’ve been studying a long time. I wonder if it was SCOMP/STOP, GEMSOS, or LOCK since your memory cycle statement is ambiguous. I’m thinking STOP at least once since you said B3. Do send me an email to address in my profile as I rarely meet folks knowledgeable about high-assurance security period much less that worked on systems I’ve studied for a long time at a distance. I stay overloaded but I’ll try to squeeze some time in my schedule for those discussions esp on old versus current.

                                                                                      2. 2

                                                                                        thought it was about better managing infrastructure.

                                                                                        I mean, yes, it does that as well, and you’re right, a lot of people use it just for that purpose.

                                                                                        However, you can also manage infrastructure quite well without containers by using something like Ansible to manage and deploy your services without overhead.

                                                                                        So what’s the benefit of Docker over that approach? Well… I think it’s security through isolation, and not much else.

                                                                                        Docker itself runs on binary blobs of priveleged or kernel code IIRC (dont use it).

                                                                                        Yes, but that’s where capabilities kicks in. In Docker you can run a process as root and still restrict its abilities.

                                                                                        Edit: if you’re referring to the dockerd daemon which runs as root, well, yes, that is a concern, and some people, like Jessie Frazelle, hack together stuff to get “rootless container” setups.

                                                                                        When I pointed out its TCB, most people talking about it on HN told me they really used it for management and deployment benefits. There was also a slideshow a year or two ago showing security issues in lots of deployments.

                                                                                        Like any security tool, there’s ways of misusing it / doing it wrong, I’m sure.

                                                                                      3. 4

                                                                                        According to Jessie Frazelle, Linux containers are not designed to be secure: https://blog.jessfraz.com/post/containers-zones-jails-vms/

                                                                                        Secure container solutions existed long before Linux containers, such as Solaris Zones and FreeBSD Jails yet there wasn’t a container revolution.

                                                                                        If you believe @bcantrill, he claims that the container revolution is driven by developers being faster, not necessarily more secure.

                                                                                        1. 2

                                                                                          According to Jessie Frazelle, Linux containers are not designed to be secure:

                                                                                          Out of context it sounds to me like you’re saying “containers are not secure”, which is not what Jessie was saying.

                                                                                          In context, to someone who read the entire post, it was more like, “Linux containers are not all-in-one solutions like FreeBSD jails, and because they consist of components that must be properly put together, it is possible that they can be put together incorrectly in an insecure manner.”

                                                                                          Oh sure, I agree with that.

                                                                                          Secure container solutions existed long before Linux containers, such as Solaris Zones and FreeBSD Jails yet there wasn’t a container revolution.

                                                                                          That has exactly nothing (?) to do with the conversation? Ask FreeBSD why people aren’t using it as much as linux, but leave that convo for a different thread.

                                                                                          1. 1

                                                                                            That has exactly nothing (?) to do with the conversation?

                                                                                            I’m not sure how the secure part has nothing to do with the conversation since the comment this is responding to is you saying that security is the reason people use containers/Docker on Linux. I understood that as you implying that was the game change. My experience is that it has nothing to do with security, it’s about developer experience. I pointed to FreeBSD and Solaris as examples of technologies that had secure containers long ago, but they did not have a great developer story. So I think your believe that security is the driver for adoption is incorrect.

                                                                                            1. -1

                                                                                              Yes. Agree not to discuss more on this thread, … but … jails both too powerful and not enough at the same time.

                                                                                            2. 2

                                                                                              Generally when you add complexity to any system, you decrease its scope of security, because you’ve increased the footprint that can be attacked.

                                                                                        1. 12

                                                                                          I’m a big fan of having data in files, where you can look at their contents or replace or alter them in a pinch to solve a problem. Where the operating system can be at least somewhat aware of the structure of the data, so that we can observe and instrument an application with OS-level tools without needing to trust whatever debugging facilities are able to run inside the process itself. There are lots of occasions where I want to make use of facilities provided by another process alongside, e.g., cron or rsyslogd. Once you have at least two thoroughly different programs in the mix, the whole “fat binary” approach doesn’t really help anyway.

                                                                                          I really don’t buy the “everything must be in the one binary!” approach at all, regardless of how trendy it is in Go and Rust at the moment. If it works for you, I suppose that’s great – but some of us will continue along the less windswept and interesting path of incrementally improving the systems that already work very well for us and countless others.

                                                                                          1. 2

                                                                                            I could build into a “fat binary” a FUSE-like filesystem and you could integrate it like same, so I’d like to understand your objection better. Is it convenience, taste, … or operational need.

                                                                                            A long time ago I had to move a comprehensive system from OS/360 under MVS, and a variant running under Multics … to an early UNIX system. There were tons of dependencies on different OS features/capabilities not then present on UNIX. Eventually found that all of them were distractions, some quite costly, that I remedied with a “fat binary”, because that was the only thing possible at the time.

                                                                                            The experience left me wary of arbitrary OS abstractions that in the end did not pass muster. I intentionally left out shared libraries from an OS I did, because it did not benefit enough for the added complexity that it added.

                                                                                            1. 11

                                                                                              Why would I want to reinvent the file system that I already have, which works just fine, inside a program?

                                                                                              I understand that shared libraries are a minefield of nuance and are difficult to get right, which is why they often get left out of new projects (e.g., Go) in the early years. Even Go seems to be getting some sort of nascent shared library support, though: see plugins.

                                                                                              On an established system where we’re already able to use them to great effect, I really see no reason to stop. As with the file system: we’ve built it already, and it works, and we’ll keep improving it. I’m really not at all worried about some hypothetical future in which we’re all suddenly forced to throw out the baby, the bath water, and the bath itself.

                                                                                              1. 2

                                                                                                Not to mention the security implications. If there’s a security problem in a library, you can update that library. For Rust/Go apps, you need to update the dependency and recompile and redistribute that application.

                                                                                                There was a researcher at Ruxcon years ago who was studying embedded libraries in C/C++ projects. An excellent example is Firefox, which doesn’t link to a lot of system libraries, but has its own embedded JPEG and PNG decoders. FF is kinda insane because it’s pretty much it’s own little operating system in a lot of ways.

                                                                                                1. 2

                                                                                                  It’s a tough balance to strike sometimes. If you’re trying to ship software that will run on lots of systems, but you need to depend on things which don’t really promise a stable interface, sometimes you have no choice but to bundle a private copy.

                                                                                                2. 1

                                                                                                  You misunderstand. No reinvention is required, one can redirect the kernel to perform the filesystem within an embedded object that is part of an existing container, namely the executable. And this was a “for example” to deal with your need for “data in files” call out. Please don’t obsess on the leaves instead of the forest being discussed.

                                                                                                  The direction being argued here is why do we have some much crap in OS/kernel in the first place. Understand that many just make use of what happens to be there, that’s fine, we all need to get shit done.

                                                                                                  But shared objects create contention for multi-threaded, multi-core systems - they add complexity and reduce the benefits of parallelism and fault-tolerance. So if one aspires to 100x cores/threads/parallelism … we don’t want to spend resource uselessly for abstractions that subtract for little/no gain.

                                                                                                  So back to what I asked you - what are the objections to “everything in the existing container” other than “that’s not what I do right now”? I don’t find all the overhead for additional containers justified by that.

                                                                                                  1. 10

                                                                                                    I don’t “misunderstand”, I just think you’re talking out of your hat.

                                                                                                    What do you mean shared objects create contention? Once the initial relocation is performed, subsequent calls to functions are generally just calls. It’s not like there’s a mutex in the path of every call to a shared library.

                                                                                                    Also, the idea that you can just “redirect the kernel” to store all your files inside an ELF object seems like a stretch at best. What if you need to append to or modify one of those files? Why take on the complexity of something like FUSE to paper over the deficiency of requiring everything to be jammed into the executable, when you could have otherwise just called fopen()?

                                                                                                    It’s true that operating systems are larger and more complicated than they used to be, but that’s because they used to be bloody awful. They used to panic instead of failing a system call. They used to need to operate on machines that had a single processor and I/O bus, and a flat memory architecture. Machines now are hugely more complicated themselves, and modern operating systems reflect our best efforts to provide a substrate for a wide variety of applications.

                                                                                                    I’d rather continue to improve on all of the work done already, where we already have some amazing tools and a lot of accumulated experience, rather than pretend that reinventing everything is a good engineering decision.

                                                                                                    1. 2

                                                                                                      At work, I wrote a program in using Lua. To simplify the installation of the program, I embedded all the Lua modules (both written in C and Lua) into the executable. All I had to do then was extend the normal Lua method of loading modules to include looking inside the executable for them (not hard to do—Lua has hooks for doing just that). That way, we only have one file to install in production, instead of a dozen or so. I don’t need the ability to modify those files, so in this case, it works.

                                                                                                      You got me about the shared libraries though.

                                                                                                      1. 3

                                                                                                        Yup, do this with statically linked modules and an interpreter (not Lua but similar). Works great, for all the same reasons.

                                                                                                      2. 1

                                                                                                        Do you understand the term “shared”? That means multiple processes/threads “share” it. As opposed to a “shared nothing” environment, which is entirely impermeable.

                                                                                                        In a true shared library system, the same library file is mmap’ed content in all the MMU’s of all using processes/threads. Sure the data is copy on modify, but all the support for the shared abstraction involves hardware/software to maintain this abstraction, which isn’t for free. And yes you can have re-entrent sitations with libraries, like I/O and event handing, where you do need to have the code able to anticipate these issues.

                                                                                                        Many of the early multicore systems had tons of esoteric bugs of this sort, which is why we had global locks to “safe” the programming environment.

                                                                                                        The kernel has virtual filesystem interfaces that express the semantics of a filesystem implementation, where you can redirect the functionality elsewhere (other hosts via RPC, user processes via library reflection/exception). With it one can embed the content of a filesystem inside an executable container in various ways. And I didn’t say ELF either. (Note that one can shortcut content transfer various ways so its even faster than going through the kernel.)

                                                                                                        Your argument is presumptive that something is being “papered over” - it’s actually more optimal because there is less code for the common case of the subsystem being within the address space for the references needed by the microservice it is implementing. In net far simpler than what is being done today.

                                                                                                        Operating systems have a much larger scope of what they have to contend with, that’s why they all tend to “bit rot”, because everyone wants to keep their bit of crap in it, human nature.

                                                                                                        I didn’t ask what you liked, I asked to defend the need for something. You act as if I’m stealing your dog or something. I doubt you have a clue to anything I’m talking about, and just want to keep obfuscating the discussion so as to hide you lack of understanding of why you use containers, because … you just use them.

                                                                                                        FIne. Use them. But you don’t know why, and you don’t want to do anything but distract from this fact by focusing on a red herring that was served up as a means to start a discussion. Die on that hill if you must, but you still aren’t responsive to my inquiry.

                                                                                                        1. 15

                                                                                                          Hello person relatively new to lobste.rs, just a comment on your how you communicate:

                                                                                                          In this thread you seem to make a vague and poorly specified claim, like put a FUSE filesystem in your binary or same thing about how the named shared objects means they are shared so things have to be more complicated and error prone. I believe that your vague comments do not help the discussion. While I do not think @jclulow is doing himself a service by responding to you, I to assumed you meant an ELF binary with some crazy FUSE setup.

                                                                                                          It’s really hard for me to tell if you have a good idea or are just being contrarian given that you aren’t being more specific in your proposal. We’re left making assumptions about what you mean (which is actually our fault) but rather than clarify how our assumptions are incorrect you are seeming to use it as a way to be smug. In particular:

                                                                                                          Do you understand the term “shared”? That means multiple processes/threads “share” it. As opposed to a “shared nothing” environment, which is entirely impermeable.

                                                                                                          and

                                                                                                          Please don’t obsess on the leaves instead of the forest being discussed.

                                                                                                          I think this discussion would be much more productive if you could specify what you are proposing. I’m quite interested in it finding out.

                                                                                                          Feel free to completely disregard this comment if you think it’s full of shit.

                                                                                                          1. 3

                                                                                                            Even if there is less code running using this method, that does not make the code less potentially buggy. Any OS you’ll likely deploy on will have more tested and proven code than what you or your small team can produce.

                                                                                                            Less depedance on third parties can be good, but up to a certain to a certain point.

                                                                                                            I believe, by the way, that Docker at least shares the filesystem layers between similar containers. Not sure how well that works, and if then binaries are also still shared in memory.

                                                                                                            1. -1

                                                                                                              But then we have dependencies between containers? Don’t we want containers to be idempotent?

                                                                                                      3. 0

                                                                                                        Why would I want to reinvent the file system that I already have, which works just fine, inside a program?

                                                                                                        Why don’t we put sqlite into the kernel?

                                                                                                        Invention is not the issue. You would use well known libraries in your program instead of developing stuff yourself.

                                                                                                        I picked sqlite as an example, because it considers itself a competitor to the open syscall and thus related the file systems. Similarly compression and encryption is file system related. Why can’t my kernel treat zip files as directories? Instead some GUIs reinvented that while bash/zsh cannot do that.

                                                                                                        1. 2

                                                                                                          Could you elaborate on what your response means? The proposed solution is to build a fat binary that includes a file system in it and to use FUSE to interact with this fat binary. What does that have to do with sqlite in the kernel? To me, the FUSE suggestion seems like a huge hack to get around using kernel primitives for no particularly good reason other than one can, as far as I can tell. I’m not even really sure what it would mean to start editing the binary given one’s usual expectations on binaries.

                                                                                                          1. 0

                                                                                                            It is a balancing act, what should be put where. Sometimes it makes sense to put the file system into the binary. Sometimes to put the file system into the kernel. Sometimes to put the file system into a daemon (microkernels). For example I have heard that some databases run directly on block devices because using the file system is slower.

                                                                                                            Jclulow is “a big fan of having data in files” because the default tools of the operating system can then be used to inspect and change the data. To pursue that way means we should extend the capabilities of the OS, for example by putting sqlite into the kernel. Then default tools of the operating system can then be used to inspect and change the data. I now think that zip file support is actually the better example. You could use ls, grep, find, and awk on zip file contents. It seems to be available via FUSE. It does not seem to be a popular option though. Why? Genuine question but I guess there are good reasons.

                                                                                                            I do not consider the Unix philosophy that great and it seems to be underlying this discussion (Just use simple files!). Unix likes to compose via files, pipes, and processes (leading to containers). Others prefer to compose via data types, functions, and libraries (leading to fat binaries). I do not see inherent advantages in either of the approaches.

                                                                                                            1. 3

                                                                                                              I don’t see the connection to the Unix philosophy here. In Windows it’s common for executables and their configurations to be separate as well. I’m trying to understand exactly what is being proposed but struggling. Are you saying that when I build nginx, the nginx executable has all of the resources it will use including config, and any code for dynamic content it will generate?

                                                                                                              1. 1

                                                                                                                Please describe what you mean by “UNIX philosphy”?