Thanks for posting this @wesleyac. I’ve been experimenting with this a lot lately and I’m starting to feel that a lot of complexity in software is down to trying to be reusable instead of reditable or recombinable. Eg I mentioned in another comment that last year I replaced anki with my own spaced repetition app and the code ended up smaller than my anki configuration. If anki had instead been written as a collection of loosely-linked functions that you glue together to make your own main function I believe it would be a much simpler codebase and it would be much easier to change it’s behavior, because:
they wouldn’t have to try to anticipate all the changes that any user might want to make
they wouldn’t have to reason about all the combinations of all possible configurations and how they interact
writing code as code is much easier than writing code as config files and templates.
Trying to think of examples of editable software:
The bitsquid game engine shipped as a collection of libraries and a ~300loc lua script that glued them together. So rather than having a configuration mechanism you just edited the glue script. I can’t find where they wrote about that decision, but ‘managing coupling’ has some related thinking.
Xmonad is similar - it’s a library and you have to write your own main function to use it as a window manager.
Emacs has some examples of this - while a lot of packages provide explicit extension points, it’s also really easy to just copy and edit a single function in your own init without having to fork the original. I often use that to fix minor bugs or to change behavior that isn’t configurable.
Excel/airtable/etc have a culture of providing templates which you edit to customize, rather than trying to build customization points into an otherwise immutable ‘library’ spreadsheet.
Suppose that I’d made Panel_Layout’s data entirely private and didn’t support any way of querying anything in it. In that case, you would be forced to either use or not use Panel_Layout for everything in a panel. So if you’d done all your buttons with push_button() and the modified toggle-boolean push_button(), then got to a button that couldn’t use one of those, you’d literally be forced to rewrite the entire panel using the low-level UI calls!
It is always important to avoid granularity discontinuities, but fortunately, it is really easy to do if you just remember a simple rule of thumb: never supply a higher-level function that can’t be trivially replaced by a few lower-level functions that do the same thing.
I think the sweet spot here might be shipping simple granular libraries that cover the 80% case along with a default example of how to glue them together, rather than shipping a monolithic app with a configuration file that inevitably tends towards Turing-complete.
When libraries are written in a granular way, it’s easy to inline functions into your own project and edit them without having to maintain a large diff over the original library. But we don’t seem to have much cultural experience with writing libraries in this way so I’m constantly stumbling over my own broken habits when I try.
One good example is microui which I’ve found very easy to alter and extend without having to fork the original code, because it offers layered access to all of it’s internals. Most libraries want to enforce hiding of their internal implementation details which means you can’t add new functionality without forking them.
Have you found that this approach leads to a higher learning curve? I can’t say that I’ve tried any of the programs in question, but I know if the first thing I had to do to use a window manager was write a glue script, it’d make it harder to get started
I know if the first thing I had to do to use a window manager was write a glue script, it’d make it harder to get started
I don’t use XMonad but I would guess this is a feature not a bug insofar as I expect that they take a suckless-esque stance of mild hostility towards less technical users.
It’s possible to configure a lot of XMonad without knowing Haskell. It’s not particularly fun though. In my case I took some else’s config and modified it, which gave me 90% - but I failed to implement a simple layout that didn’t exist. Was a thing of 10 minutes for someone who knows Haskell.
Something that I’ve been thinking about a lot is that the way that most software is distributed is really hostile to modifications - this post touches on that, but doesn’t go very deep into it. A question I’ve been asking recently is - what would a package manger that’s designed with end-users patching software as a first-class concern look like? And a somewhat more challenging version for some systems - what would it look like to allow end-users to patch their kernels as a first-class concern?
NixOS has the start to an answer for this (fork the nixpkgs repo, make whatever changes you like), but it still doesn’t seem ideal. I guess maybe gentoo is a sort of answer to this as well, but it doesn’t seem like gentoo answers the question of “how do you keep track of the changes that you’ve made”, which is I think a really important part of systems like this (and the thing that’s missing in most current package managers support for patching)
Note you don’t need to fork nixpkgs to edit individual packages, you can just add them to an overlay with an override eg I was trying to debug sway recently so I have:
One of the guix devs gave a talk called Practical Software Freedom (slides) where the core point was that it’s not enough to have foss licensing if editing code is so much of a pain that noone does it. It looks like guix has a pretty nice workflow for editing installed packages, and since the tooling is all scriptable I bet you could streamline it even more - eg add a command for “download the source, fork it to my personal repo, add the fork to my packages list”.
I only very briefly used guix, but its kernel configuration system is also just scheme. It’s so much nicer than using any of the kernel configurators that come with the kernel, and I actually played around with different configurations rather than opting to play it safe which is what I normally do since it’s such a faff if I accidentally compile out support for something important.
How often do you find your overlays breaking something in strange ways? My impression is that most packages don’t come with tests or other guardrails to give early warning if I break something. Is that accurate?
I haven’t had any problems so far. I guess if you upgrade a package and it changes the api then you might be in trouble, but for the most part it seems to just work.
We aren’t just reordering bits here. There’s a new constraint that has no counterpart in current programming practice — remove a feature, and everything before it should build and pass its tests
If we then built software such that reordering commits was less likely to cause merge conflicts, you could imagine users having a much easier time checking out a simpler version of the system and adding their functionality to that if the final version of the system was too complex to modify.
I’ve gotten close with rpm and COPRs. I can fairly quickly download a source RPM, use rpmbuild -bp to get a prepped source tree where I can build a patch, track it, and add it back to the spec, then push it to a COPR which will build it for me and give me a yum repository I can add to any machines I want. Those pick up my changed version of the package instead of the upstream with minimal config fiddling.
It’s not quite “end users patching as a first class concern” but it is really nice and in that ballpark.
Interesting point! And I guess that’s kinda the approach that Go took from the start, with imports directly from GitHub. Except, ideally, you’d have a mutable layer on top. So something like IPFS, where you have the mutable IPNS namespace that points to the immutable IPFS content-addressed distributed filesystem.
Still, unlike direct links to someone elses GitHub repo, you would want to be able to pin versions. So you would want to point to your own namespace, and then you could choose how and when to sync your namespace with another person’s.
This is how https://www.unisonweb.org/ works. Functions are content-addressed ie defined by the hash of their contents, with the name as useful metadata for the programmer. The compiler has a bunch of builtin refactoring tools to help you splice changes into an existing graph of functions.
Just watched the 2019 strangeloop talk. Absolutely brilliant. Only drawback I see is that it doesn’t help with code written in other languages. So dependency hell is still a thing. But at least you’re not adding levels to that hell as you write more code (unless you add more outside dependencies).
what would a package manger that’s designed with end-users patching software as a first-class concern look like?
If you just want to patch your instance of the software and run it locally, it is very straightforward in Debian and related distributions, using apt-src. I do it often, for minor things and pet peeves. Never tried to package these changes and share them with others, though.
All the stuff package managers do doesn’t seem to help (and mostly introduces new obstacles for) the #1 issue in modifying software: “where are the sources?” It should be dead easy to figure out where the sources are for any program on my system, then to edit them, then to run their tests, then to install them. Maybe src as the analogue to which?
I appreciate npm and OpenBSD to a lesser extent for this. Does Gentoo also have a standard place for sources?
what would a package manger that’s designed with end-users patching software as a first-class concern look like?
A DVCS. (It’s Christmas, “Away in a package manger, no .dpkg for a bed…”)
We drop in vendor branches of 3rd party libraries into our code as needed.
We hack and slash’em as needed.
We upstream patches that we believe upstream might want.
We upgrade when we want, merging upstream into our branch, and the DVCS (in our case mercurial) tells us what changed up stream vs what changed locally. Some of the stuff that changed upstream is our patches of improvements on them, so 99% of the time we take the upstream changes, and 0.9% of time take our changes and 0.1% of the time have to think hard. (ps: 99% of stats including this one are made up (from gut feel) on the spot. What makes these stats special is I admit it.)
A related topic is the plague of configuration knobs.
Every time any programming asks, “What should these configuration value be? He says, dunno, make it an item (at best) in the .conf file or at worst, in the UI.”
The ‘net effect is the probability of the exact configuration ever having been tested by anyone else on the planet is very very very low and a UI that you’d either need a Phd to drive (or more likely is full of wild arsed guesses)
A good habit is, unless there is a screaming need, by default put config items in a hidden (to the user) file and that is not part of the user documentation.
If anybody starts howling that they need to alter that knob, you might let them into the secret for that knob, and consider moving it into a more public place.
This may be a little off-topic, but I think it is relevant. User modifications in computer games are a lot more common than other software types and are even possible in some big budget proprietary software. I think this can offer some insight into how this can work for other software.
The most basic version is to move all content/assets (3d models, textures, sound files, etc.) into its own subfolder tree and give it clear human readable names. Users can then simply swap out files.
The next level is to abstract how the metadata for high level objects that use these assets is stored out to a text based format like json or xml so that the assets can be reused and new high level objects defined.
This can be extended to more and more engine content, including defining behaviours and simple functions in json/xml
Another useful step is to publish the source (this does not mean libre, see e.g. Space Engineers) so that users can read and understand how things work under the hood. The cherry on top is then to add a system that parses in user assembly libs and overwrites/extends internal types with anything found inside (see e.g. Rimworld). This has major security drawbacks but at least in single player games (more generally non-networked desktop applications) security is not that important.
Games that implement all or most of these concepts tend to have a much longer life, some still have dedicated and active user bases decades after release, and they continue to get more content and evolve long after the original developers have moved on.
Not all of these concepts are directly translatable to general software but it is still a model worth looking at for inspiration. I even see a lot of young children releasing basic mods for some games which not only means that the modding is easy, but also has the added benefit of teaching a new generation some basic software and programming concepts.
This is not at all off-topic and I think you’ve got a very good point here. Games with mods are one of the places where we see the largest amount of end-user modification of software, and associated problems&benefits of different ways of dealing with that. I think it’s a really good idea to study what happens when people try to make these things work in games, especially as the ecosystem gets bigger, projects get more ambitious and people start running into problems with mutually incompatible mods, mods having compatibility issues with newer versions of the base game and mods that try to build on top of others.
For a start, from what I’ve seen:
almost always a mod package manager evolves (unless the base game shipped with a good-enough one)
many games have (more) problems with uninstalling mods than with installing them. Games that have clean “overlay” style file loading (like how in quake and quake derivatives, resources are searched for in multiple pak0.pak files) seem to suffer less with this
it seems to be really common for people to end up creating bundles of hundreds of very small mods that do similar things. e.g. bundling 100 mods that each fix a visual misalignment in one asset into one “Alice’s visual misalignment fixes pack” mod
there are some really interesting politics around heavily-modded shared servers where it takes a very long time to update the server to the newest version of the base game because every mod has to be updated for compatibility or (contentiously) dropped
I think this might be underestimating the fact that copying software is zero effort and so a lot of it can be free. That makes the cost for modification relatively bigger.
And for something more complicated then chairs it’s a lot easier to modify for the original creator. A better comparison would be modern cars with all their extras and options. And for that there are similar models in software like SAP where you usually pay someone to customize it for you.
That’s just really expensive compared to putting in all features and letting the user decide.
I think the correct comparison is the cost of copying and configuring software though. I often find that software that tries to be very general via configuration ends up being very complicated. Eg last year I replaced anki with my own spaced repetition app and the code ended up smaller than my anki configuration.
I feel very cynical/fatalistic about Emacs/Lisp/Smalltalk lately. Yes, the core design choices exhibit a high trust for users. But then it seems to be inevitable that the layers on top start chipping away at this trust. Vim’s package managers introduce hurdles for anyone who wants to modify sources. (See my recent war story.) How does the Emacs eco-system compare? Is it really common for people to modify packages? Racket’s raco feels the same, additional complexity between me and the libraries they want me to ‘use’.
Emacs is great for this, especially if you use straight. Straight makes a local clone of each package repo. If you want to edit it, edit it and commit (or don’t), and your emacs will use the local copy.
There is a built in facility called advice that let you intercept functions and change how they behave.
Finally there’s a very neat macro called el-patch that lets you make modifications to the source of other functions and will tell you if they’ve changed out from under your mods, if you wanna get dirty.
I typically don’t modify the original source directly, but copy-paste the function into my own init and edit it there. I find it easier to have all my changes in one file with one version control history rather than spread out across many repos.
But the answer to your general complaint about finding the source code is easy in emacs - xref-find-definitions works across packages.
That seems like a key point. A lot of the reasoning around locking down code and having pre-defined, well-controlled configuration points is to prevent breakage. But if you think of your users as responsible adults then maybe you can just say “here is the sharp edge, you know the risks”?
This seems similar to access control in languages like python and julia where there are conventions around private interfaces but they aren’t enforced by the compiler. So you can reach in and call some private function if you need to do it, but you’re made aware that you’re taking on a higher risk that future versions of the library will break your code. There’s kind of a social aspect too - if you rely on a public interface and it breaks you can blame the library developer but if you rely on a private interface and it breaks then it’s clear that you took on that responsibility yourself.
I think your reflection here picked out the most important point, which I didn’t really key on when I wrote the comment.
On that vein, I want to gesture at the communities which produce different kinds of software, along with the overall broader erosion of trust in society, and suggest that perhaps the trust levels of software design is inherent in the community and time which forms it.
This post made me instantly think of st. I think they have swung too far the other direction; I am not a fan of needing to edit the source to something in order to, say, change the font it uses to display text when one of its primary jobs is displaying text.
A lot of software is already like this through plugins/extensions. For example Firefox, Wordpress, Vim, etc.
The problem with locally modifying software is that it’ll become rather hard to update. I run a modified version of dwm, and even though it’s pretty simple and doesn’t get many updates, updating it is time-consuming as I have to merge my local changes with upstream. I’m not entirely sure how practical truly editable software will be.
I’ve been thinking about writing my own WM which just handles the basics, and everything else is an external program. You don’t need the WM to do tiling or to move the windows, something like xdotool can do that just as well. I’m not sure how practical it would be, or when exactly you’ll run in to limitations about this though, but it would be an interesting experiment.
The author contrasts plugins/extensions against editing software. Extensions require the original developer to try to anticipate all the places that a user might want to change the behavior and provide a fixed api that can accommodate everyone’s needs. As opposed to encouraging the user to just download the source and make the changes directly.
Obviously that isn’t going to work with firefox today because you need some kind of supercomputer just to compile it from scratch. But maybe we can figure out how to design software to make that kind of direct editing easier.
Have you thought of using a version control system? What I tend to do is download the source to a package, and if it’s via a tarball, before I do anything (./configure, etc.) I’ll create a git repo out of it, then immediately create a branch and check it out. Any changes I make are to that local branch. A new version comes out? I checkout master, update, then switch back to the local branch and merge.
The truly tragic thing IMO: all the software that tries to be simple and hackable has no tests. All the software that has tests gets complex and baroque. As a result tests end up seeming just a convenience for maintainers, when they could be so much more.
I think that for a lot of simple and hackable software, the need for tests is much smaller, whereas for complex and baroque software it’s pretty much required.
It’s not that simple software couldn’t benefit from tests, it’s just that the ROI on it is much smaller. After all, if you can understand all of the program then you don’t need to unit test isolated parts.
The same applies to some other “best practices” btw. The smaller and simpler your program, the more acceptable it becomes to just use global variables for example.
Totally agreed on global variables and unit vs integration tests. But I disagree on total tests. I refuse to believe that dwm’s sources wouldn’t benefit from tests. Yes, the author may not need them. But they seem super useful to a user wanting to hack on them. Particularly if you just encountered a merge conflict. 2kLoC isn’t insignificant. And just look at the number of concepts needed to deal with X windows!
I refuse to believe that dwm’s sources wouldn’t benefit from tests
Yeah sure, it would certainly benefit; it’s just that the benefit is smaller. How many bugs would be prevented by testing? Probably not that many. Like I said, it’s all about ROI; you can probably chart project size vs. benefit from testing on a chart quite nicely.
Your argument makes sense, but leaves one question open: benefit for whom? If you mean benefit for the main repo, you’re absolutely right. But I’m addressing your original point at the top of this thread:
The problem with locally modifying software is that it’ll become rather hard to update. I run a modified version of dwm, and even though it’s pretty simple and doesn’t get many updates, updating it is time-consuming as I have to merge my local changes with upstream. I’m not entirely sure how practical truly editable software will be.
It would be less time-consuming for you if dwm had tests. And it would make editable software seem more practical.
Of course, this isn’t in your control. I was thinking of that when I used the word ‘tragic’. But your subsequent defense of lack of tests strikes me as Stockholm Syndrome :)
I posted about something like this on the LV2 plugin devel mailing list a couple of days ago, about both the flexibility to relabel input and output ports and a preset (or similar mechanism) to save and recall these changes.
Thanks for posting this @wesleyac. I’ve been experimenting with this a lot lately and I’m starting to feel that a lot of complexity in software is down to trying to be reusable instead of reditable or recombinable. Eg I mentioned in another comment that last year I replaced anki with my own spaced repetition app and the code ended up smaller than my anki configuration. If anki had instead been written as a collection of loosely-linked functions that you glue together to make your own main function I believe it would be a much simpler codebase and it would be much easier to change it’s behavior, because:
Trying to think of examples of editable software:
Casey Muratori’s post about complexity and granularity also seems very relevant here. Money-quote:
I think the sweet spot here might be shipping simple granular libraries that cover the 80% case along with a default example of how to glue them together, rather than shipping a monolithic app with a configuration file that inevitably tends towards Turing-complete.
When libraries are written in a granular way, it’s easy to inline functions into your own project and edit them without having to maintain a large diff over the original library. But we don’t seem to have much cultural experience with writing libraries in this way so I’m constantly stumbling over my own broken habits when I try.
One good example is microui which I’ve found very easy to alter and extend without having to fork the original code, because it offers layered access to all of it’s internals. Most libraries want to enforce hiding of their internal implementation details which means you can’t add new functionality without forking them.
Have you found that this approach leads to a higher learning curve? I can’t say that I’ve tried any of the programs in question, but I know if the first thing I had to do to use a window manager was write a glue script, it’d make it harder to get started
I think all of the projects above ship with an example that has reasonable settings and then you tweak it from there.
It does require that you figure out how to compile the thing though. Nix/guix make that a lot easier which is why I’m so interested in them.
I don’t use XMonad but I would guess this is a feature not a bug insofar as I expect that they take a suckless-esque stance of mild hostility towards less technical users.
It’s possible to configure a lot of XMonad without knowing Haskell. It’s not particularly fun though. In my case I took some else’s config and modified it, which gave me 90% - but I failed to implement a simple layout that didn’t exist. Was a thing of 10 minutes for someone who knows Haskell.
Something that I’ve been thinking about a lot is that the way that most software is distributed is really hostile to modifications - this post touches on that, but doesn’t go very deep into it. A question I’ve been asking recently is - what would a package manger that’s designed with end-users patching software as a first-class concern look like? And a somewhat more challenging version for some systems - what would it look like to allow end-users to patch their kernels as a first-class concern?
NixOS has the start to an answer for this (fork the nixpkgs repo, make whatever changes you like), but it still doesn’t seem ideal. I guess maybe gentoo is a sort of answer to this as well, but it doesn’t seem like gentoo answers the question of “how do you keep track of the changes that you’ve made”, which is I think a really important part of systems like this (and the thing that’s missing in most current package managers support for patching)
Note you don’t need to fork nixpkgs to edit individual packages, you can just add them to an overlay with an override eg I was trying to debug sway recently so I have:
One of the guix devs gave a talk called Practical Software Freedom (slides) where the core point was that it’s not enough to have foss licensing if editing code is so much of a pain that noone does it. It looks like guix has a pretty nice workflow for editing installed packages, and since the tooling is all scriptable I bet you could streamline it even more - eg add a command for “download the source, fork it to my personal repo, add the fork to my packages list”.
I only very briefly used guix, but its kernel configuration system is also just scheme. It’s so much nicer than using any of the kernel configurators that come with the kernel, and I actually played around with different configurations rather than opting to play it safe which is what I normally do since it’s such a faff if I accidentally compile out support for something important.
How often do you find your overlays breaking something in strange ways? My impression is that most packages don’t come with tests or other guardrails to give early warning if I break something. Is that accurate?
I haven’t had any problems so far. I guess if you upgrade a package and it changes the api then you might be in trouble, but for the most part it seems to just work.
One thing this brought to mind was @akkartik’s “layers” script: http://akkartik.name/post/wart-layers
The post is short, but the money quote is here:
If we then built software such that reordering commits was less likely to cause merge conflicts, you could imagine users having a much easier time checking out a simpler version of the system and adding their functionality to that if the final version of the system was too complex to modify.
I’ve gotten close with rpm and COPRs. I can fairly quickly download a source RPM, use
rpmbuild -bpto get a prepped source tree where I can build a patch, track it, and add it back to the spec, then push it to a COPR which will build it for me and give me a yum repository I can add to any machines I want. Those pick up my changed version of the package instead of the upstream with minimal config fiddling.It’s not quite “end users patching as a first class concern” but it is really nice and in that ballpark.
At that point the package system would just be a distributed version control system, right?
Interesting point! And I guess that’s kinda the approach that Go took from the start, with imports directly from GitHub. Except, ideally, you’d have a mutable layer on top. So something like IPFS, where you have the mutable IPNS namespace that points to the immutable IPFS content-addressed distributed filesystem.
Still, unlike direct links to someone elses GitHub repo, you would want to be able to pin versions. So you would want to point to your own namespace, and then you could choose how and when to sync your namespace with another person’s.
This is how https://www.unisonweb.org/ works. Functions are content-addressed ie defined by the hash of their contents, with the name as useful metadata for the programmer. The compiler has a bunch of builtin refactoring tools to help you splice changes into an existing graph of functions.
Just watched the 2019 strangeloop talk. Absolutely brilliant. Only drawback I see is that it doesn’t help with code written in other languages. So dependency hell is still a thing. But at least you’re not adding levels to that hell as you write more code (unless you add more outside dependencies).
If you just want to patch your instance of the software and run it locally, it is very straightforward in Debian and related distributions, using apt-src. I do it often, for minor things and pet peeves. Never tried to package these changes and share them with others, though.
All the stuff package managers do doesn’t seem to help (and mostly introduces new obstacles for) the #1 issue in modifying software: “where are the sources?” It should be dead easy to figure out where the sources are for any program on my system, then to edit them, then to run their tests, then to install them. Maybe
srcas the analogue towhich?I appreciate
npmand OpenBSD to a lesser extent for this. Does Gentoo also have a standard place for sources?I believe Debian is trying to streamline this with Salsa. That’s the first place I look when I’m looking for source code of any package on my system.
A DVCS. (It’s Christmas, “Away in a package manger, no .dpkg for a bed…”)
We drop in vendor branches of 3rd party libraries into our code as needed.
We hack and slash’em as needed.
We upstream patches that we believe upstream might want.
We upgrade when we want, merging upstream into our branch, and the DVCS (in our case mercurial) tells us what changed up stream vs what changed locally. Some of the stuff that changed upstream is our patches of improvements on them, so 99% of the time we take the upstream changes, and 0.9% of time take our changes and 0.1% of the time have to think hard. (ps: 99% of stats including this one are made up (from gut feel) on the spot. What makes these stats special is I admit it.)
A related topic is the plague of configuration knobs.
Every time any programming asks, “What should these configuration value be? He says, dunno, make it an item (at best) in the .conf file or at worst, in the UI.”
The ‘net effect is the probability of the exact configuration ever having been tested by anyone else on the planet is very very very low and a UI that you’d either need a Phd to drive (or more likely is full of wild arsed guesses)
A good habit is, unless there is a screaming need, by default put config items in a hidden (to the user) file and that is not part of the user documentation.
If anybody starts howling that they need to alter that knob, you might let them into the secret for that knob, and consider moving it into a more public place.
This may be a little off-topic, but I think it is relevant. User modifications in computer games are a lot more common than other software types and are even possible in some big budget proprietary software. I think this can offer some insight into how this can work for other software.
The most basic version is to move all content/assets (3d models, textures, sound files, etc.) into its own subfolder tree and give it clear human readable names. Users can then simply swap out files.
The next level is to abstract how the metadata for high level objects that use these assets is stored out to a text based format like json or xml so that the assets can be reused and new high level objects defined.
This can be extended to more and more engine content, including defining behaviours and simple functions in json/xml
Another useful step is to publish the source (this does not mean libre, see e.g. Space Engineers) so that users can read and understand how things work under the hood. The cherry on top is then to add a system that parses in user assembly libs and overwrites/extends internal types with anything found inside (see e.g. Rimworld). This has major security drawbacks but at least in single player games (more generally non-networked desktop applications) security is not that important.
Games that implement all or most of these concepts tend to have a much longer life, some still have dedicated and active user bases decades after release, and they continue to get more content and evolve long after the original developers have moved on.
Not all of these concepts are directly translatable to general software but it is still a model worth looking at for inspiration. I even see a lot of young children releasing basic mods for some games which not only means that the modding is easy, but also has the added benefit of teaching a new generation some basic software and programming concepts.
This is not at all off-topic and I think you’ve got a very good point here. Games with mods are one of the places where we see the largest amount of end-user modification of software, and associated problems&benefits of different ways of dealing with that. I think it’s a really good idea to study what happens when people try to make these things work in games, especially as the ecosystem gets bigger, projects get more ambitious and people start running into problems with mutually incompatible mods, mods having compatibility issues with newer versions of the base game and mods that try to build on top of others.
For a start, from what I’ve seen:
pak0.pakfiles) seem to suffer less with thisI think this might be underestimating the fact that copying software is zero effort and so a lot of it can be free. That makes the cost for modification relatively bigger.
And for something more complicated then chairs it’s a lot easier to modify for the original creator. A better comparison would be modern cars with all their extras and options. And for that there are similar models in software like SAP where you usually pay someone to customize it for you.
That’s just really expensive compared to putting in all features and letting the user decide.
I think the correct comparison is the cost of copying and configuring software though. I often find that software that tries to be very general via configuration ends up being very complicated. Eg last year I replaced anki with my own spaced repetition app and the code ended up smaller than my anki configuration.
I think this is where emacs shines. The whole Lisp/Smalltalk world is sort of ideologically like this. High trust of users.
I feel very cynical/fatalistic about Emacs/Lisp/Smalltalk lately. Yes, the core design choices exhibit a high trust for users. But then it seems to be inevitable that the layers on top start chipping away at this trust. Vim’s package managers introduce hurdles for anyone who wants to modify sources. (See my recent war story.) How does the Emacs eco-system compare? Is it really common for people to modify packages? Racket’s
racofeels the same, additional complexity between me and the libraries they want me to ‘use’.Emacs is great for this, especially if you use straight. Straight makes a local clone of each package repo. If you want to edit it, edit it and commit (or don’t), and your emacs will use the local copy.
There is a built in facility called advice that let you intercept functions and change how they behave.
Finally there’s a very neat macro called el-patch that lets you make modifications to the source of other functions and will tell you if they’ve changed out from under your mods, if you wanna get dirty.
Thank you,
straight.ellooks wonderful. This whole thread has also had me (finally!) reading up on Guix.I typically don’t modify the original source directly, but copy-paste the function into my own init and edit it there. I find it easier to have all my changes in one file with one version control history rather than spread out across many repos.
But the answer to your general complaint about finding the source code is easy in emacs - xref-find-definitions works across packages.
That seems like a key point. A lot of the reasoning around locking down code and having pre-defined, well-controlled configuration points is to prevent breakage. But if you think of your users as responsible adults then maybe you can just say “here is the sharp edge, you know the risks”?
This seems similar to access control in languages like python and julia where there are conventions around private interfaces but they aren’t enforced by the compiler. So you can reach in and call some private function if you need to do it, but you’re made aware that you’re taking on a higher risk that future versions of the library will break your code. There’s kind of a social aspect too - if you rely on a public interface and it breaks you can blame the library developer but if you rely on a private interface and it breaks then it’s clear that you took on that responsibility yourself.
I think your reflection here picked out the most important point, which I didn’t really key on when I wrote the comment.
On that vein, I want to gesture at the communities which produce different kinds of software, along with the overall broader erosion of trust in society, and suggest that perhaps the trust levels of software design is inherent in the community and time which forms it.
This post made me instantly think of st. I think they have swung too far the other direction; I am not a fan of needing to edit the source to something in order to, say, change the font it uses to display text when one of its primary jobs is displaying text.
But doing that has really made their software easy for users to modify to their liking. As evidenced by the pile of available patches that various users have made.
A lot of software is already like this through plugins/extensions. For example Firefox, Wordpress, Vim, etc.
The problem with locally modifying software is that it’ll become rather hard to update. I run a modified version of
dwm, and even though it’s pretty simple and doesn’t get many updates, updating it is time-consuming as I have to merge my local changes with upstream. I’m not entirely sure how practical truly editable software will be.I’ve been thinking about writing my own WM which just handles the basics, and everything else is an external program. You don’t need the WM to do tiling or to move the windows, something like
xdotoolcan do that just as well. I’m not sure how practical it would be, or when exactly you’ll run in to limitations about this though, but it would be an interesting experiment.The author contrasts plugins/extensions against editing software. Extensions require the original developer to try to anticipate all the places that a user might want to change the behavior and provide a fixed api that can accommodate everyone’s needs. As opposed to encouraging the user to just download the source and make the changes directly.
Obviously that isn’t going to work with firefox today because you need some kind of supercomputer just to compile it from scratch. But maybe we can figure out how to design software to make that kind of direct editing easier.
Have you thought of using a version control system? What I tend to do is download the source to a package, and if it’s via a tarball, before I do anything (./configure, etc.) I’ll create a git repo out of it, then immediately create a branch and check it out. Any changes I make are to that local branch. A new version comes out? I checkout master, update, then switch back to the local branch and merge.
Yeah, dwm is in git, but it’s still manual work fixing merge conflicts, which only gets worse the more modifications you make.
The truly tragic thing IMO: all the software that tries to be simple and hackable has no tests. All the software that has tests gets complex and baroque. As a result tests end up seeming just a convenience for maintainers, when they could be so much more.
I think that for a lot of simple and hackable software, the need for tests is much smaller, whereas for complex and baroque software it’s pretty much required.
It’s not that simple software couldn’t benefit from tests, it’s just that the ROI on it is much smaller. After all, if you can understand all of the program then you don’t need to unit test isolated parts.
The same applies to some other “best practices” btw. The smaller and simpler your program, the more acceptable it becomes to just use global variables for example.
Totally agreed on global variables and unit vs integration tests. But I disagree on total tests. I refuse to believe that
dwm’s sources wouldn’t benefit from tests. Yes, the author may not need them. But they seem super useful to a user wanting to hack on them. Particularly if you just encountered a merge conflict. 2kLoC isn’t insignificant. And just look at the number of concepts needed to deal with X windows!Yeah sure, it would certainly benefit; it’s just that the benefit is smaller. How many bugs would be prevented by testing? Probably not that many. Like I said, it’s all about ROI; you can probably chart project size vs. benefit from testing on a chart quite nicely.
Your argument makes sense, but leaves one question open: benefit for whom? If you mean benefit for the main repo, you’re absolutely right. But I’m addressing your original point at the top of this thread:
It would be less time-consuming for you if
dwmhad tests. And it would make editable software seem more practical.Of course, this isn’t in your control. I was thinking of that when I used the word ‘tragic’. But your subsequent defense of lack of tests strikes me as Stockholm Syndrome :)
OP and everybody commenting on this thread feels like a kindred spirit. <3
I posted about something like this on the LV2 plugin devel mailing list a couple of days ago, about both the flexibility to relabel input and output ports and a preset (or similar mechanism) to save and recall these changes.