A straightforward way to run binaries without making a derivation or any patchelf business
That’s what broke the camel’s back for me. One too many not-packaged binary where writing a derivation was cumbersome, and the patchelf business too much complexity.
Though this was in 2018, things might be better now.
I’ve found that buildFHSUserEnv works for me in some cases. It took me literally years too understand how to use that, due to the usual docs problem. Basically, I have this in my config
This adds a fhs utility to my PATH, which drops me into a “normal linux”. It’s up to me to add all the required libraries here (and that huge block of packages is 99% copy-pasted from somewhere), but it gets the thing going:
$ cd downloads
$ aunpack sublime_text_build_4143_x64.tar.xz
$ ./sublime_text/sublime_text
exec: Failed to execute process '/home/matklad/downloads/sublime_text/sublime_text': The file exists and is executable. Check the interpreter or linker?
$ fhs
(fhs) $ ./sublime_text/sublime_text
# gui fires up and seems to work
Not saying that this is an acceptable solution, just sharing something which was useful for me after I’ve finally grokked how to hold it
Looks like too much magic under the hood. Like languages.python.enable = true;, that’s all well and good, but how nasty does it get when I need a specific version of Python? Or when I need to start generating devenv.yaml because it’s so complex?
Photos and LightRoom both fail on the “let me put thousands of photos on a remote share and let any number of laptops/phones/tablets poke around, tagging, organizing, making smart albums, etc.”. I don’t care about RAW, I don’t care about anything more than basic editing, and would like to keep all the face recognition/etc. that the basic Photos app can do. If I could have it generate a “feed” or something I could point a smart TV or smart picture frame to and have it cycle through a smart album or something that’d be amazing.
This seems insane that Photos can’t do this. I tried something from Adobe too a while ago (can’t remember the name now) but it wasn’t up to the task either.
Did you ever look into digiKam? I use it a lot, from what you’re describing, it looks like it matches a lot of your “requirements”.
Some things that might fit your bill:
it can do face recognition for you (and learn and help you automate this stuff)
it doesn’t “import” your pics anywhere, it just works on wherever they are.
it has its database, but:
you can write most of the changes in the sidecar file, so that other programs can also use it.
the db is sqlite (by default) so you can even do something with tools like datasette e.g. query what’s the most frequent time of day or location your photos are taken, or - if you manage RAWs there as well - what focal lengths do you have the most or similar.
it can do some basic editing for you(for proper edits I still use external software)
you should be able to point it to a smart TV, I think, and even if not, there are many cool plugins for such stuff (as well as for e.g. backups to box, google, posts to twitter, facebook, regular html galleries and a lot of others.
The part that I like the most about it is that it is focused on the library functions: searching, tagging, filtering etc. (I also combine it with rapid photo downloader to get pics off my cameras, and darktable and/or gimp to do actual editing but that’s just me).
After using digiKam for a few years I only use it to import images from the camera. Persistent issues:
The DB is corrupted once in a while, and the application crashes on the regular.
The UI is 1990s. Why is the settings button not on the home screen? Why is there a splash screen (by default, you can turn this off but dammit, who actually wants that?)? Why is there no way to go from “WTF does this do?” to the actual documentation? Like, unless I spend every waking second editing photos, why would I know which “quality” setting to choose?
Darktable uses sidecar files, and although they refuse to believe the filesystem is important (so I have to keep re-importing the entire catalogue and run a service to clean out missing files nightly), at least it isn’t actually corrupted. And it’s much faster than digiKam, making it barely usable on a reasonably modern desktop.
Wow, we really have different experiences there. I don’t think I’ve ever seen the database get corrupt, for example, and with darktable I had that happen.
And the UI, some of it might seem “1990s”, but for me it’s just fine, and far, far better then Darktable for managing a library (not editing, I still use dt for that).
For importing, did you ever consider rapid photo downloader? For me this does noticeably better job then the other two.
I think Photoprism should be able to handle thousands of photos. However I’ve tried to import around 100 000 photos and it was almost impossible to manage that much in it.
Literally just set up my own PhotoPrism instance a week ago, and it currently has some 55k images and videos in it.
There are a few things that could be improved, e.g. allowing renaming of persons tagged through face detection, easier labeling of multiple images and better face detection for kids. All of these improvements already have issues on GitHub BTW.
One thing it doesn’t support is timeshifting of images with bad EXIF tags. For some reason I had a whole lot of photos that had GPS data but a wrong EXIF create date. Luckily exiftool was able to fix all bad timestamps. Here’s a oneliner that can fix the issue that I had:
exiftool -progress -r \
-if '$createdate eq "2002:12:08 12:00:00"' \
-if '$gpsdatetime' '-createdate<$gpsdatetime' \
-overwrite_original_in_place . |&
tee -a log.txt
Yea, I have a similar dream. For me, it would be an open source project with a single-binary self-hosting approach, which embeds a high performance web server for an interface and API similar to Google Photos. The storage is “just files” so the binary + photos can be backed up easily. Metadata stored in SQLite alongside the photos on the filesystem. Cool tricks like face recognition and autosizing, autocropping, etc. outsourced to other binaries with a scheme for storing “derived” images non-destructively, similar to what Picasa used to do. And then sync tooling that puts the same thing in a self-hosted cloud, with a VM + cache for 30 days of photos, with rest backed by Amazon S3, Google GCS, or Backblaze B2. Then maybe some tooling that can auto install this on a VM and secure it via Google Login, while supporting private photo sharing at stable URLs similar to Google Photos.
This would be a big project because, to be good, there’d also have to be great open source mobile apps on iOS and Android.
Some friends of mine prototyped the start of this project using redbean (Lua single binary web server) and got as far as the basics of photo serving from disk and having access to SQLite schema for metadata. It’s totally doable.
For the time being, I have been using Android’s support for SD card readers to sync all my photos from phones, DSLR, and mirrorless cams into Google Photos, and then keeping RAW storage separate on my Mac Mini and backed up using standard 3-2-1 backup approach. But, it’s cumbersome (requires manual processes), lossy (I lose backup of edits in GPhotos), proprietary, and I’m afraid for Google’s track record with longevity of services. It also saddens me that Google Photos doesn’t give me any way to post public photo albums at my domain (https://amontalenti.com) with stable URLs that I know I can make last forever, regardless of photo serving tech.
My theory for why something like this never got built is that Apple Photos and Google Photos are just so darn convenient, and non-phone photography relatively rare among consumers these days, so it just never fell into the indieweb and f/oss sweet spot of itch worth scratching. But I still have the itch and have for a long time.
It also seems like a lot of the backend functionality could be done by somehow modularizing the GNOME Shotwell codebase. For example to handle image metadata, color correction, cropping, and so forth.
At #PreviousJob, we used BuildKite, and it was fantastic - you could deploy the agent pretty much anywhere you wanted, and I found the configuration much easier than any other system I’ve used (such as GitHub Actions).
I recently realized that what I really want out of a CI system is running locally. Ideally, with decoupled backends, so, it can run tasks in containers, but also just processes, for simpler projects.
Most CI configuration languages don’t really let you do that, so you need to duplicate the commands for building, linting, testing, etc.
There’s that terra something tool, but it requires containers always, I think.
I had a couple (very) rough ideas on a CI system. One was to make the actual task configuration a Lua script with a bunch of predefined functions to make a declarative pipeline possible, but to also allow the user to drop into more imperative steps. Using Lua lets you more effectively sandbox the tasks than using something like the JVM, the runner could be much leaner, and users could possibly get real autocompletion for their steps instead of only relying on docs for some made up yaml DSL. I also really want to integrate it more deeply with metrics, so you can see annotations in Grafana when you deployed and have automatic rollback when something goes wrong.
I want a calling convention not stuck decades in the past and lugging C’s type system forever forward. I want a network stack, free of Berkeley sockets; capabilities for content and compute. I want the same programs on my phone, tablet, laptop, desktop, and Other People’s Computers; and I mean the same— browsers and the web are so much closer than “native apps” and yet so fucking far.
I want this every day I make another REST API or gRPC service or Kafka topic. I want this every day I shepherd my team building a product written for Android and iOS. I want this every day I play games on my Switch, knowing the word “buy” on screen is a lie, I’ve only rented this experience. I want this every day the music on my headphones playing from Bandcamp won’t seamlessly switch to my living room’s stereo. I want this every day my girlfriend wonders why she can’t stream Ted Lasso to her Chromecast. I want this every day my colleagues block out “Monday” on their calendars and I’m in a timezone on the other side of the world. I want this every single day I wait for funds to move from my bank account in one developed economy to pay taxes in another developed economy.
Once upon a time, we invented regular expressions as a comfortable midpoint between expressive power and efficiency on 1970s computers. We invented tools to search for regexes (grep), tools to automatically edit streams based on regexes (sed), programming languages designed to run code when a regex was matched in the input stream (awk) and even interactive editors with regexes as first-class UI primitives (vi).
Here in the 21st century, text processing has advanced a long way. In particular, parsing context-free grammars no longer requires days of swearing and wrestling with shift/reduce errors; you can write grammars nearly as succinctly as regexes, and more readably. Using packrat parsing or Earley parsing or some other modern algorithm, I’d love to have a suite of text-processing tools like grep, sed, awk and vi based on grammars rather than regexes.
In most cases, I end up doing text processing using (neo)vim macros rather than regexp. It can run surprisingly fast even on large datasets (in headless mode or with lazyredraw).
Feels like a very modern/ergonomic/incremental/less abstract approach compared to regular expressions.
I do like the premise of clearly defined grammars, however! Could compound nicely with macros, too.
Now that I think of it, treesitter is essentially a database of grammars that can be used for data annotating and processing. 🙃 I guess the next step is to have a more on-the-fly way to describe and use these things.
I want to be able to slap a QR UUID sticker on a box. Then added the contents of the box to a db with
Image(s) of object
Name
UPC of item if possible
Purchase infomation if available
Lastly add all of the images to a to image recognition system so I can look up what an item is and where it goes.
Also being able to query from the other direction (e.g. where is tiny tiny machine square).
It ultimately would not be helpful for keeping organized, and potentially counter productive, but it’s nice to dream that this would magically fix my poor organization skills.
The image recognition system isn’t present, but snipe-it can do the inventory management with UPC and purchase info + QR code generation parts of this idea. I self host it to track computer parts and systems through the process of repair, it’s been very useful in staying organized so far.
I’m actually working on an app like this. You can take a photo, scan a barcode, and add a description. There’s also a barcode generation tool. Any feedback is welcome.
Containerized desktop manager. Apps, or groups of apps, or maybe each virtual desktop, run in isolated containers. You can pause, stop, start, etc, new groups, and access to hardware and shared data (i.e., $HOME) are granted per group. You should be able to move apps between isolated environments.
The main use cases would be keeping a more strict separation between work/personal stuff, and being able to save to disk the whole state of a virtual desktop, during pauses in work in specific projects or tasks.
Have you actually measured this? With memory deduplication, VMs will share as much memory for shared libraries as containers (more than containers that aren’t sharing a layer). A minimal paravirtualised Linux kernel build will add a fairly small amount of memory consumption per process in comparison to a typical desktop application.
VMs have an hypervisor, containers don’t. So you might get close with memory, but I doubt that the gap can be bridged as much on CPU.
And even then, I’m giving the idea a wide margin because there’s a ton of different VM technologies, so there really might be some setups that are competitive. The run of the mill virtualbox, though, doesn’t stand a chance.
VMs have an hypervisor, containers don’t. So you might get close with memory, but I doubt that the gap can be bridged as much on CPU.
There’s a lot to unpack here. When you say ‘containers’, I presume you mean ‘shared-kernel virtualisation’, since containers don’t necessarily imply a specific isolation technology (and, often, are deployed on VMs for isolation). With that assumption:
The overheads of shared-kernel virtualisation are not uniform. Docker on Linux uses namespaces, cgroups, and seccomp-bpf. The BPF filtering can have a fairly high overhead on system calls and can be over 10% of total CPU usage in the worst case. Something like containerd has a bunch of different options for filesystems, but the layered model for containers is built on a snapshot abstraction. With ZFS, that’s relatively cheap, but the default on Linux is to use some form of union filesystem, which means that filesystem namespace operations can be linear complexity in terms of the depth (you need to sequentially look through each layer to find a match). This, again, can incur a large performance penalty.
In contrast, the overheads from VMs come from two places. First, you need nested paging. You may be paying that anyway (XNU and Windows both enable it by default for various security features) and you may well have it enabled to be able to run VMs, unless your host uses shared first and second-level page tables (this is increasingly difficult as modern architectures provide different layouts of the two for their different uses). The overhead for something like SEV-SNP can be double-digit percentages, but the overhead for things with good locality is very low. It basically makes TLB fills more expensive. The second overhead from VMs is for any I/O, which generally goes via a paravirtualised interface for performance. For a Qubes-like use case, with Linux-on-Linux, I’d assume that they’re using FUSE-over-VirtIO. This is an efficient asynchronous (batched) protocol and it allows the guest to share read-only mappings of buffer-cache pages, so a single page (e.g. for a shared library) can be mapped read-only into many guests with CoW mappings.
On top of that, there are some second-order effects. For example, the last couple of FreeBSD releases have included VNET, which lets you have a separate instance of the network stack for each jail. This consumes more memory but can significantly increase performance because there’s no contention between jails, anywhere between the system call and ethernet driver layers. VMs get this benefit for free: each one can have a separate network stack, filesystem stack, process and thread namespace, and so on. In contrast, a VM has its own scheduler, which runs on top of a VCPU abstraction, which can cause some other performance artefacts.
TL;DR: Full VMs and shared-kernel virtualisation both have overheads and neither is a clear winner for all workloads. There have been numerous papers published on this topic.
Seems like I’ve been overestimating the advantages of containers over VMs on the general case, thanks for the thorough explanation.
Maybe I should give qubes a go. Back when I first heard of it my hardware was very limited, but I have much more capacity now, so, could be interesting.
The main use cases would be keeping a more strict separation between work/personal stuff, and being able to save to disk the whole state of a virtual desktop, during pauses in work in specific projects or tasks.
Do you need containers for this, or just use different user accounts?
In theory, malicious code could go looking for the other user account, but the same is perhaps true of containers (I’m not up to date - but I understand you shouldn’t use them as a security perimeter).
For security, yes. But the main reason I came up with the idea was to be able to save a group of apps and resume the state later, which requires some level of virtualization.
A persistent-object-store / filesystem-replacement, somewhat along the lines of the NewtonOS “soup”, or of the various ideas for tag-based file management that have floated around here.
A really good object-capability-based RPC library. Cap’n Proto is decent but the C++ API is awkward and it doesn’t yet support persisting capabilities or passing them around between clients, which is where it gets really interesting.
A persistent-object-store / filesystem-replacement, somewhat along the lines of the NewtonOS “soup”, or of the various ideas for tag-based file management that have floated around here.
This is what we were trying to build with CoreObject, but updated for modern usage, including:
Persistent undo. With non-destructive editing, keeping this even for tasks like video editing is not a large overhead.
Named versions. Save is not a UI abstraction that should exist. Data should always be saved but sometimes you want to add some tags to a specific version of a thing to make it easy to jump back to later.
Branching undo, so if I undo and then edit then I can always revisit a previous branch in the undo history. This also gives you templates for free: a template is just a named version that you then edit and give a new name to some derivatives.
Model-specific diffing and merging for arbitrary document types, so you could do offline editing on different devices and merge the result easily.
Live collaborative editing built on top of the diffing and merging support.
I haven’t worked on it for about 10 years. It became clear that GNUstep and Objective-C were not adequate for building the kind of system I wanted, so I started working on hardware isolation that would let us safely combine C code with languages with no unsafe mode, so that we could build things like this with vast numbers of isolated components.
A task manager & time tracker that works together with the file-system, git and other tools. E.g. if I start working on project x, task y, any downloads and screenshots automatically go to ~/my/projects/project-x/task-y. It would also create a feature branch in the corresponding git repo, start dev server, open editor, etc.
Emacs, but not trapped in the ‘70s. A system – a holistic system, top to bottom, UI to drivers – that is live, inspectable, plastic, and fun. Let me explore my computer. I don’t care about security beyond the most basic, “don’t let unknown folks do things” extent. I don’t care about reliving the multi-user systems of my youth – I want my device to be mine. I don’t care about how easily it interacts with the rest of creation.
I want a UI at least as polished and sophisticated as the classic Mac OS, but not based on the world of scare storage and slow machines.
Adjustment layers been the longest WIP. At least the prerequisite GEGL is here now. I adjusted my photography workflow so that I can do everything with darktable (and Hugin) to get around the non-destructive situation with GIMP—luckily you’ve been able to retouch skin via wavelet decompose inside a darktable module.
Not quite. It’s goals are to be more like an Instagram clone which isn’t suitable—you don’t need ‘stories’. Digital art doesn’t need to be limited to a square, or sRGB, or have its metadata (including licensing) stripped and compressed to hell to save a few bytes because the kind of content is different. Flickr also keeps an uncompressed original as a backup which can be used in an emergency and the groups features was a big part of what used to make it so popular. It’s also still a good place to search for Creative Commons licensed content.
MediaGoblin is much closer in that sense. Butif you want to hack on either project, you’re looking at PHP or Python on Docker and that’s not very interesting to me.
We should shake up the UNIX monoculture a bit. I’d hope for a secure microkernel OS (seL4?) written in Rust (or Ada). Ideas from e.g. Genode, EROS, QNX, Plan9. For starters it would just need a microkernel + a way to act as a hypervisor. You’d be able to run a Linux / Windows guest to work and continue improving the host OS.
We need more security, privacy and reliability from the ground up. But we can’t forget about usability, or nobody is ever going to use the worlds most secure paper weight.
A good recipe app designed for power users. I want to be able to plan party menus and see flavor pairings and take notes on new recipes. All the recipe apps I tried are little more than recipe storage + shopping list.
The funny thing is that I know so many people, and I’ve seen so many people [on github], that tried to solve this problem with their own bunch of scripts or apps. It always break down later either because of unexpected feature creep that’s too hard to add or just by lack of motivation.
(I think I found something to try to model in alloy xD)
The funniest thing to me is that “a place for all your recipes!” was a big selling point of computers in the 1980s and we still don’t have a good recipe app
Ha! Just like computers were supposed to get rid of all paper documents… We’re getting there, but meanwhile computers made the quantity of paper we use simply explode :shrug:
What I really want is hardware – a box, about the size of a phone, with a few dozen terabytes of storage, that all my various screens can use as backing state. The phone, the desktop, I don’t know, the TV, the watch, &c. All of them should use my own pocket cloud. I want centralized storage of my data without having to give control over to someone else.
An accurate emulator for a modern CPU (say, skylake). It would allow for much greater visibility into performance, answering questions like ‘why isn’t this shit scaling?’ or ‘to what extent is btb aliasing actually a problem for things like bounds/overflow checks and inline caches?’ with much greater precision than would otherwise be possible. Some things that are cool but aren’t it:
performance counters: not enough information
valgrind: too primitive, and decades out of date
uica: not enough reach (basically just simulates scheduling)
Surprisingly, a lot of the information you’d need for something like this is publicly available, but not, obviously, in executable form. This would also be immensely valuable as a platform for doing further research into modern CPUs; if you observe behaviour X, and can’t explain it, but think it might be caused by Y, then you can implement Y in the emulator and see if it causes behaviour X.
I actually have started working on something along these lines, but it is far down my priority list and I despair of ever finishing.
(Plus, obviously, a broader computing ecosystem that isn’t complete garbage, but I have to pick and choose my battles, and have chosen to forego that one.)
This sound very cool. Where do you find the publicly available details on microarchitecture of contemporary desktop CPUs? Would it allow you to get reasonably close to a cycle accurate description of a processor?
While a lot of information comes from third-party researchers, a surprising amount comes from intel’s own optimisation manual (I won’t bother linking it because they have a habit of breaking links—even worse than microsoft in that respect, given that they actively remove pages containing useful and interesting information, rather than just moving them around—… /rant).
This makes sense when you consider their market position: they effectively have a monopoly on the high-performance market, both internet/web and scientific. Due to their scale, their users are strongly incentivised to optimise for microarchitectural details of their target CPUs. Intel, in turn, is incentivised to make information available about their CPUs, in order to enable the same, and in order to ensure software runs better on their own CPUs than their competitors’ (kind of a positive version of the ‘cripple amd’ feature of their compiler). More subtly, they are incentivised to double down on microarchitectural compromises, making only monotonic improvements rather than rebalancing, because else, in exactly the same fashion, they’d end up competing against their previous hardware revisions. Intel has been refining the same basic uarch for upwards of a decade now, meaning knowledge and information about it have aged reasonably well.
Aside from third-party research, there are also patents, from which you can often glean some information, but that’s obviously prone to be fraught…
Getting to cycle-accuracy will require quite some work. There are plenty of holes or dim spots. I don’t think anyone knows the exact operation of any branch predictors, for instance, because the main question you need to ask from a performance perspective is ‘is this branch predictable (and hot enough to be predicted)?’ If the answer is yes, then you probably don’t care exactly how long it takes to warm up. Similar applies to e.g. the speculative prefetcher. There’s also the fact that ‘cycle-accuracy’ is not so clearly defined as it is for something like the NES. Timings on real hardware will depend on real-world factors like the weather, or the quality of the wiring connecting you to your RAM.
Anyway, here are some links. I’m surely forgetting some things:
Agner has published information on all intel/amd/via x86 processors from the last decade or so. Not super detailed (aside from the instruction timings), compared with some of the other resources, but it is nicely comprehensive along the other two axes.
Henry wong’s blog contains a lot of useful information and measurement methodology.
John mccalpin’s blog is nice, and in particular is (seemingly) one of the few people that actually talks about cache coherency protocols.
Brandon falk is mostly focused on security, but has done some interesting work in this area (some of which, iirc, was used by andreas abel).
Andreas abel did a lot of work on intel processors as part of his phd, including detailed instruction timings and a pipeline model, and cache replacement algorithms.
Maynard handley has done a lot of detailed research into recent revisions of apple’s cpus, and some of the discussion of the memory subsystem likely applies to other cpus as well.
Dougall johnson has also done some nice measurement work on apple’s cpus
Travis downs’s uarch-bench contains some information I haven’t seen elsewhere, and there’s some on his blog as well.
I recently started to look for a replacement for my self-hosted Drone CI, trying to find something I like, which is fully open source and self-hostable, and came up empty handed. Since then, I’ve been dreaming about a CI system that’s somewhere between Drone and GitHub Actions. I even started researching and designing my own CI, but… I really don’t have the time - nor the expertise, really - to build it. Not anytime soon.
I’ve definitely enjoyed working on buildomat at Oxide. It began out of necessity (GitHub Actions is never going to support illumos really, and the relatively inflexible hosted runner stuff is all .NET which… is certainly a choice one could make!) but has become a way to make functionality choices for us which are pretty different to those made by Actions, etc.
GitHub Actions is never going to support illumos really, and the relatively inflexible hosted runner stuff is all .NET which… is certainly a choice one could make
Historical nit: Azure Pipelines, not Azure proper. The rebrand to be an Azure product happened pretty late, and it went by various Visual Studio brands previously.
When the “Build vNext” system (what became Azure Pipelines) was being designed, cross-platform for the agent was a key requirement. The agent was initially written in NodeJS. I joined the team after the agent rewrite in C# was completed, so I don’t have first-hand knowledge about why it changed platforms.
Go wasn’t nearly as prevalent or obvious a choice as it would be today. There was a pretty vocal contingent which wanted to rewrite again in Go, but the business value wasn’t quite there yet. And then the GitHub acquisition happened, further complicating things.
(I was a product manager on Azure Pipelines from 2017-2020, for a time the sole product manager.)
cross-platform for the agent was a key requirement
It’s a shame that they chose one of the least cross-platform languages available to implement it if that’s the case. Getting it working on even something very Linux-like, such as FreeBSD, is a pain. Getting it working on any architecture that isn’t x86 or Arm is impossible (and not always possible even if that is the architecture). The code is written in an incredibly non-portable style (it has no platform abstraction layer and does things like query the platform to determine whether it’s Linux to see if the filesystem is case sensitive, so every new platform needs to grep the code for macOS, Linux, and Windows to see what things have been conflated).
There is a third-party implementation in Go, which we’ve used very successfully (we do snmalloc CI for FreeBSD using it and we also use it in our Morello cluster, neither of which is possible with the C# agent). I would love to see us ditch the C# version, hire ChristopherHX, and make his implementation the official one.
That said, I was astonished that a CI system created in the last ten year didn’t separate the command executer portion from the control plane. With this separation, it’s easy to support running CI jobs on anything that supports SSH or a serial console. Without it, you’ll never be able to support embedded targets.
I have been dreaming about software designed to make self-hosting easier for over 10 years now. And I have spent a lot of time on coding it myself, although my results haven’t been very good so far.
I’ve thought about it a lot, tried different approaches and projects, and I think at this point im looking at creating a “lite” infrastructure as a service platform thats self-hostable and has similarities to Fediverse / Mastodon.
distinction between “admins” and “users”, where “users” get full access to the system, the only extra capability admins get is control over who gets a user account and rules that users have to follow.
can get multiple servers in different localities to work together (in this case for replication/failover instead of social media sharing / discovery)
A sort of Ruby scripting platform for Android that lets you write Ruby scripts and run them once or periodically, with bindings to Android libraries to let you send notifications and send and respond to intents and all that, and prebundled with libraries to parse HTML, JSON, an HTTP client, anything that might be useful really.
Other nice things to have:
A repository to share libraries and scripts with other users
Bindings to a GUI toolkit (maybe even some immediate mode thing) and OpenGL so you can write small apps and games/visualizations
An IDE optimized for phones without a physical keyboard to write scripts
I was not aware of that but I don’t think so, my idea was mostly about scripting API stuff and running cronjobs while that’s definitely more game-oriented.
Ah, my apologies – the itch.io version is. The main site’s version is much less focused on game development as it is focused on “quick deploy” and “write once run anywhere”
Ah I see, I think you could probably use that as the runtime, but that’s only one part of it. I probably didn’t explain what I meant very well; the main concept is an app I can use to write quick scripts on a phone and then run them on that same phone, without having to go through a normal computer or a VM or something like Termux. The language doesn’t necessarily have to be Ruby, it’s just the scripting language I like the most.
I know you said “without having to go through termux”, however, you can easily install ruby in termux, and there’s a widgets application so you can run them directly from the homescreen
A self-hosted, open-source replacement for Discord. The pieces are all there: VOIP is a solved problem thanks to Mumble, text chat is an XMPP client, etc. All it really needs is some duct tape, and the barest of UI polish.
Debugger GUI that’s better than inserting prints everywhere. Maybe just a modern ddd would do.
VCS that doesn’t break code by automerging and doesn’t require manual merges for obvious situations. Think Pijul, but with semantic diffing and merging.
ABI description language. Think C headers, but with added annotations for possibly shared features (slices, (un)validated UTF{8,16,32} strings, etc.) and without C’s warts.
I’ve found both Intellij and VSCode debugging UIs to be fairly ok. Problem is they’re tied to the editor, which might be a problem if you don’t like the editors.
Sometime between today and 5 billion years from now, the sun will expand to such an extent that the earth will become uninhabitable. Between now and then, the web as we know it will come to an end.
Eventually, the cost of supporting all of the web’s the bad design decisions will obviously outweigh the benefits of maintaining it. In short, we will eventually need to “break the web”—something hardly anyone has any appetite to do today—and come up with something new. I’m not talking about “Web 2.0” or “Web3”. That’s just business jargon for VCs.
I want to know, what would a genuinely new web look like? Here’s what I’d like to see:
Things to keep
URLs. People have been trying to kill the address bar for years. But addresses are the universal way of sharing information between increasingly walled gardens. There may be some way to improve the UX of the address bar, perhaps by exposing some new UI/flow based on some standardized form of the HATEOS idea in REST.
HTTP over TCP/IP. This is a language every tech stack already speaks. It works well and is well understood. Yes, there are some security flaws in HTTP/2, but reinventing that wheel is not going to yield much better.
WASM. An intermediate language that already has a lot of momentum behind it. Hopefully it will get garbage collection soon and a huge raft of garbage collected languages will jump in.
SVG. Not sure about this one. Love it right up to the point where it allows embedded content in other languages. Can we define a subset of SVG we can support?
Things to throw away
CSS. No scoping = unwanted global collisions. Non-deterministic behavior caused by cascading selectors. CSS is the shittiest language I’ve ever encountered, and I think I get to say that because I’ve been shoveling that shit professionally for over 20 years.
The DOM. See HTML/CSS replacement for a native, component-based architecture.
Things to replace
HTML & CSS. It’s folly to completely separate content from presentation. The medium is the message. Dynamic front-end framework patterns have settled over the years into a clear need for component-based front-ends with component-level scoping for both content and presentation. These components will need to have explicitly defined parameters and explicitly designated abilities (e.g. static or dynamic, the ability to render content outside its container, access to specified user data and devices).
JavaScript. This is a tricky decision. It’s tempting to adopt/adapt a language like Elm, PureScript, or TypeScript. Another option is to focus on WASM support and tooling instead of having a favored scripting language.
Things to add
A blend of responsive components and grids. Start with the UX/DX for individuals and teams using tools to create content in this new web and work backwards to design the ideal format for tool makers to use. A designer should be able to use a GUI to create a basic, static component and edit the visual design of a dynamic, scripted component without any assistance from a developer.
Proper accessibility and multilingual support from the start. Needs research into the varieties of disabilities and languages and the existing tools people use for them. Writing assistive tools for the new web should be as straightforward as reasonably possible. No more band-aid solutions.
A custom LWML for content authors. Markdown is close, but not good enough. Basic formatting should be allowed in text content, but not too much. Should also support right-to-left languages and math and scientific formatting. Could possibly also support other, existing LWMLs as alternatives.
Built in authentication. This subject is out of my depth, but the overall goal is reduce security complexity for users and developers by baking authentication into the standard. No more pasting passwords in from managers or logging in with credentials from a dying social media platform. Should include MFA, but how do we make this better?
Some kind of config file format, tool(s) or ecosystem with migrations and backwards compat built-in. To fight bit-rot or revisiting codegen’d, copy-pasta’d boilerplate (or not) files with semver transitions, deprecation. To make wow config files move through time better.
I would like a fast binary diff with insertion/deletion detection that works on multiple-GB files. There are some tools that can do that (ECMerge and Beyond Compare), but they are not open-source and have a GUI.
A Lua variant that knows how to suspend itself, combined with a good debugger (thing “pdb”) that lets you look at suspensions, and powerful metrics on that.
The idea? Being able to write workflows across multiple days as imperative code, where you can easily observe the state and write things like “sleep for 5 days” and not have to worry. That and being able to have CI/CD that isn’t declarative, but doesn’t require writing a bunch of webhook nonsense to get anything done. let job_result = run_job_on_other_machine(configuration).wait(10.days) and having the tools that can handle in progress stuff nicely.
There are so many worms here but Salesforce Dot Com is proof that people are fine with tinkering with their stuff (SFDC is the Smalltalk lover’s dream).
A “brute force” total evaluation executor for Python
Be able to write a test like property = True | False; some_function(property) and have that smartly calculate all possible results from that.
To combinatorial explosion types: the first thing would be just “blow up if there are too many universes”, the second thing would be letting people write code branches for “underspecified mode” so that you can avoid this. I think this sort of tool would work with almost all of the code that people write for enterprise software. It would be 100% fine for this to not work on systems software.
(might have time for this but) Typescript but for Python
Take Python code, transform it into Typescript code that is “semantically equivalent” (as closely as possible), so that I can use Typescript’s type system in Python. Mypy is good, but Typescript just is so powerful that it’s hard to look at any other type system the same for enterprise software
Good forum software that isn’t Discourse
Just like… a Django thing a la wagtail that makes it easy to set up a forum, customize for your community, and just look like a webpage
A fast and reliable scanner for my Pokemon cards so that I could have an electronic database.
The official app has a terrible browser for cards; you have to know which release the card was a part of. You can’t say “show me all of the psychic type pokemon”.
TCGPlayer has a scanner, but it’s never worked for me beyond once as an accident.
I have a webcam and a powerful desktop. Surely there should be a way I could hold the card in front of the camera, have it recognize the card, and add it to a sqlite database for later consumption by … something else.
I’ve always found it very difficult to get anything useful done with opencv for computer vision. It feels like these things should be achievable these days, but it’s unclear how to do them.
Vendor based (might be wrong word) package managers
I want to do something like npm vendor component-library which would add component-library to ./vendor & i can go in there and modify only ./vendor/Input.svelte & somehow git would only track my changes to Input.svelte
So many times i want to make 1 small edit to a library & i end up doing some major hacks because im too lazy to create & manage a forked repo.
There’s so many packages that acquire so much bloat due to trying to let each user have their own config, vs easily modifiable packages with tools to manage the diffs.
I believe this also would make library code easier to read if its created with the intention to be modified & application devs would understand the frameworks better.
This is cool! I might use this for my component library at work because I’m already making so many global stylistic changes to suit our project.
The description makes it seem like its only for bug fixes / inconvenience, but I’d want a whole new way of thinking of packages. Technically I think its not far off, but my dream involves packages designed to be modified.
Imagine how much simpler APIs would be if you were expected to go into /vendor/package-name/variables.ts to set the global options.
Imagine how much more readable library code would be expected to be if it was expected to be modified by the average user.
I imagine working with packages that are tailor suited towards 1 thing & my needs at work slightly deviate so I change it to fit my need.
In the world we live in packages are built to suit every use-case under the sun because the only ways of providing value to most users is providing a flexible API, rather than a solid base that is easily modifiable.
Clean API & deep understanding of your system at the cost of having to deal with manual upgrades for the components you modify. I’m not sure if the pay off is always worth it, but sometimes it definitely is.
A multiplayer real time tactical space game with a technology level similar to The Expanse. Stealth and orbital trickery are the keys to survival. Friends and I can crew up a ship and divide the various roles of navigation, fire control, and management of ship systems among us. We can play against other real people, but not as an MMO. More like an invite-only scheduled event.
Super realistic physics and ideally beautiful visuals (the part I’d struggle the most with).
I guess since the last time I now can formulate an additional DVCS dream (a very different one)
Basically, be better rsync. Or better Unison? I already have a mess of large files synchronised across multiple devices with rsync, or something behaving more or less the same. I want no new copies, and I want the directory structure to stay and I some copies are on USB drives etc. and should be directly usable but first-class copies. So, no annex stuff or whatever. But I want to version how I move files around, and how I modify them. Modification is just hash-before/hash-after, the idea is that the tool can distinguish which file is older even if timestamps were messed up (see also: USB drives), and also alert me if two versions are actually incomparable. Oh, and as usual I want SVN-style narrow copies, too.
(hm, I also have other data-organisation projects that I kind of do code but very slowly, do they count?)
De/centralization-agnostic social media protocol + test implementation.
When AliceClient is started it looks into its local db of Alice’s friends, and where they store their content.
AliceClient then queries AliceServer, BobServer, CharlieServer, for top-level posts (Bob and Charlie are friends)
BobPost001 is returned
AliceClient then queries AliceServer, BobServer, CharlieServer, for replies to BobPost001
…etc. and so on…
Pull-type data transfer + auth would make it resistant to spam from sources you don’t subscribe to. Open protocol + the network graph being stored locally by the client and fetching data from possibly many servers means that the server provider is not a gatekeeper and cannot hold users hostage due to network effects, should they wish to leave.
A server could be anything, a cupboard SBC, a VPS instance, or something analogous to a web hotel tenancy.
By now I’ve been away from social media for so long that I can’t really see myself coming back, even to my own implementation, and if I can’t convince even myself it doesn’t really matter if I had the time in the first place.
simple, small, non-systemd init + service management system that keeps
declarative unit configs, but disposes of the journal & the rest of
systemd
Have you looked at dinit?
The service configuration isn’t systemd’s unit config, but it is
declarative. Essentially, a service is defined in a file containing
lines of key=value pairs. For example, here’s the definition of sshd
from one of my systems:
type = process
smooth-recovery = true
command = /usr/sbin/sshd -D
depends-on = ssh-keygen
depends-on = networking
I’ve been using dinit for well over a year now. If you dig systemd’s
declarative config and don’t care for its mission creep, it is worth a
look.
I believe honeycomb.io is at least trying to build a version that you can use outside of Facebook, even if it isn’t entirely open source. But they are leveraging open telemetry for ingestion which itself is open!
Clickhouse comes close, doesn’t it? Supports streaming ingestion via Kafka, has a new experimental WATCH statement for monitoring output for “live” queries via what they call a Live View. I think this would get you most of the way there.
In the past, I combined Kafka, Amazon S3, Storm, Cassandra & Elasticsearch to get the same value proposition of Scuba, but that required quite a lot of (closed source) engineering and code to make it work well. Described a bit here.
Clickhouse comes close if you have a mostly static schema. If it’s highly dynamic, there’s a point after which too many columns and/or querying json blob doesn’t work.
I think Arcan might be evolving to be what I would end up wanting if I actually tried to implement this, but I’d really like to try to build a modern version of NeWS / Display PostScript, using technologies that I could run both within a web browser and in a native application:
Image / video / audio objects sent to the display server where they can be queued directly.
WebAssembly for display-server-side parts.
PostScript-style drawing model exposed to WASM for 2D, OpenGL for 3D.
On top of this, I’d like a widget set and GUI that runs things like a rich text system (based on SILE for preference) for rendering and interactivity near the user, so if I type something in a text field then it’s displayed immediately and then a message is sent to the application with the update, so that there’s the kind of latency hiding that NeWS offered.
You can build this kind of thing on the web already, but I want a narrower interface (close to WASM + Canvas) so that you can also run it without all of the baggage of a full web stack.
One thing I’ve wanted to build for ages is a web-based Hotline client that works on old computers, I guess? I haven’t really thought it through and won’t know what would work the best until I started using it.
The other idea I think about frequently is a multi-room music player server like Mopidy+Snapcast but also with a whole UI for basically ripping media from wherever (it recently occurred to me the entire project could solely be this UI, living alongside Mopidy). Paste a [YTMusic|Bandcamp|Soundcloud|Spotify|…] link and it parses the artist and songname, rips the audio and permanently stores that in your local library. Eventually you amass a colossal archive of everything you and your family/visitors have added to your Library. Kind of like a high-quality frontend for yt-dlp, but more fully-featured (the artist/song-parsing particularly, and ripping other sources like Spotify). I mean, the whole “legal issues” aspect is one huge blocker to this one, unsurprisingly.
a truly cross platform (web, iOS, Android) note taking app that supports handwriting but also markdown text and is either local-first or has easy export to portable formats
A truly plug and play personal web server, along the lines of this post. Something that is easy enough for my mom to just plug in and start using.
I’ve been making games lately – Semantle was a big success, Surfwords … wasn’t (it’s cool, but not making any money). I’ve got two more in the works.
What I really want to make is a game along the lines of Myst or The Witness: a graphical puzzle game. I have been collecting individual puzzle concepts for years, and I have almost as many as Myst had (depending on how you count). I also have an obviously doable idea for a flavor of puzzle that’s straightforward to make a fair number of of (like The Witness, but very different in style), enjoyable to solve, and flexible.
What I need is art. Well, art and a story, but if I had a genie that would make art, I would probably be able to put together a story. The problem is that a game of that scale has a lot of art. Myst had 2,500 images. It’s just way more than I can afford to hire someone to do, and I don’t have enough of a track record yet to find a collaborator.
I thought about using Stable Diffusion, but it’s not quite there yet – even if I ignore the very weak animation support, it’s just bad at making physically plausible objects. People (excluding hands) are fine, because there’s so much input data. But things, especially things that don’t exist, are too often too surreal. Also, image generation is slow enough to be a bottleneck.
I think the modern version of this sort of thing is “escape the room”, which, since they take place in a small space, reduce the amount of art needed. But probably they don’t reduce it to a feasible scale, and anyway I have never seen one that had the level of atmosphere that Myst had. And it’s not just the quality of graphics: Monkey Island had worse graphics, but it had a sense of space.
Nix for normies:
recandwithbuiltinandlibNixOS for normies:
nix buildvsnix-build, for examplenixpkgs for normies:
I would add to NixOS for normies:
That’s what broke the camel’s back for me. One too many not-packaged binary where writing a derivation was cumbersome, and the patchelf business too much complexity.
Though this was in 2018, things might be better now.
I’ve found that
buildFHSUserEnvworks for me in some cases. It took me literally years too understand how to use that, due to the usual docs problem. Basically, I have this in my confighttps://github.com/matklad/config/blob/49ace0891e4566a3199dba08ba49e909ed3ad9ad/hosts/default.nix#L19-L36
This adds a
fhsutility to my PATH, which drops me into a “normal linux”. It’s up to me to add all the required libraries here (and that huge block of packages is 99% copy-pasted from somewhere), but it gets the thing going:Not saying that this is an acceptable solution, just sharing something which was useful for me after I’ve finally grokked how to hold it
There’s so much in nixpkgs, I think I’ve only done this once. Of course, YMMV.
I never tried it, but there’s nix-ld.
The readme also mentions
nix-autobahnandnix-alien, which seem to try to solve the same problem.Or Nix but localized so I stop getting errors like
Isn’t https://devenv.sh the Nix for normies, or at least a very good first step?
Looks like too much magic under the hood. Like
languages.python.enable = true;, that’s all well and good, but how nasty does it get when I need a specific version of Python? Or when I need to start generating devenv.yaml because it’s so complex?Decent photo organizing software.
Photos and LightRoom both fail on the “let me put thousands of photos on a remote share and let any number of laptops/phones/tablets poke around, tagging, organizing, making smart albums, etc.”. I don’t care about RAW, I don’t care about anything more than basic editing, and would like to keep all the face recognition/etc. that the basic Photos app can do. If I could have it generate a “feed” or something I could point a smart TV or smart picture frame to and have it cycle through a smart album or something that’d be amazing.
This seems insane that Photos can’t do this. I tried something from Adobe too a while ago (can’t remember the name now) but it wasn’t up to the task either.
Did you ever look into digiKam? I use it a lot, from what you’re describing, it looks like it matches a lot of your “requirements”.
Some things that might fit your bill:
The part that I like the most about it is that it is focused on the library functions: searching, tagging, filtering etc. (I also combine it with rapid photo downloader to get pics off my cameras, and darktable and/or gimp to do actual editing but that’s just me).
After using digiKam for a few years I only use it to import images from the camera. Persistent issues:
Darktable uses sidecar files, and although they refuse to believe the filesystem is important (so I have to keep re-importing the entire catalogue and run a service to clean out missing files nightly), at least it isn’t actually corrupted. And it’s much faster than digiKam, making it barely usable on a reasonably modern desktop.
Wow, we really have different experiences there. I don’t think I’ve ever seen the database get corrupt, for example, and with darktable I had that happen.
And the UI, some of it might seem “1990s”, but for me it’s just fine, and far, far better then Darktable for managing a library (not editing, I still use dt for that).
For importing, did you ever consider rapid photo downloader? For me this does noticeably better job then the other two.
I think Photoprism should be able to handle thousands of photos. However I’ve tried to import around 100 000 photos and it was almost impossible to manage that much in it.
So far I’ve settled with plex for building playlist + digikam for organizing collection + my own photoframe software: https://github.com/carbolymer/plex-photo-frame
oh wow this looks very promising… thank you for the recommendation!
Literally just set up my own PhotoPrism instance a week ago, and it currently has some 55k images and videos in it.
There are a few things that could be improved, e.g. allowing renaming of persons tagged through face detection, easier labeling of multiple images and better face detection for kids. All of these improvements already have issues on GitHub BTW.
One thing it doesn’t support is timeshifting of images with bad EXIF tags. For some reason I had a whole lot of photos that had GPS data but a wrong EXIF create date. Luckily
exiftoolwas able to fix all bad timestamps. Here’s a oneliner that can fix the issue that I had:All in all I’m pretty satisfied with PhotoPrism!
Yea, I have a similar dream. For me, it would be an open source project with a single-binary self-hosting approach, which embeds a high performance web server for an interface and API similar to Google Photos. The storage is “just files” so the binary + photos can be backed up easily. Metadata stored in SQLite alongside the photos on the filesystem. Cool tricks like face recognition and autosizing, autocropping, etc. outsourced to other binaries with a scheme for storing “derived” images non-destructively, similar to what Picasa used to do. And then sync tooling that puts the same thing in a self-hosted cloud, with a VM + cache for 30 days of photos, with rest backed by Amazon S3, Google GCS, or Backblaze B2. Then maybe some tooling that can auto install this on a VM and secure it via Google Login, while supporting private photo sharing at stable URLs similar to Google Photos.
This would be a big project because, to be good, there’d also have to be great open source mobile apps on iOS and Android.
Some friends of mine prototyped the start of this project using redbean (Lua single binary web server) and got as far as the basics of photo serving from disk and having access to SQLite schema for metadata. It’s totally doable.
For the time being, I have been using Android’s support for SD card readers to sync all my photos from phones, DSLR, and mirrorless cams into Google Photos, and then keeping RAW storage separate on my Mac Mini and backed up using standard 3-2-1 backup approach. But, it’s cumbersome (requires manual processes), lossy (I lose backup of edits in GPhotos), proprietary, and I’m afraid for Google’s track record with longevity of services. It also saddens me that Google Photos doesn’t give me any way to post public photo albums at my domain (https://amontalenti.com) with stable URLs that I know I can make last forever, regardless of photo serving tech.
My theory for why something like this never got built is that Apple Photos and Google Photos are just so darn convenient, and non-phone photography relatively rare among consumers these days, so it just never fell into the indieweb and f/oss sweet spot of itch worth scratching. But I still have the itch and have for a long time.
It also seems like a lot of the backend functionality could be done by somehow modularizing the GNOME Shotwell codebase. For example to handle image metadata, color correction, cropping, and so forth.
I know it’s not software, but in case you didn’t know about this, if you have an Apple TV you can show slideshows on your TV.
Drone for your first one? Bob?
I should clarify that by “self-hosted” I mean “by organizations employing teams of engineers” and not “by individuals”.
While easy to use software is possible, I reckon making it is quite hard.
Netdata?
At #PreviousJob, we used BuildKite, and it was fantastic - you could deploy the agent pretty much anywhere you wanted, and I found the configuration much easier than any other system I’ve used (such as GitHub Actions).
I recently realized that what I really want out of a CI system is running locally. Ideally, with decoupled backends, so, it can run tasks in containers, but also just processes, for simpler projects.
Most CI configuration languages don’t really let you do that, so you need to duplicate the commands for building, linting, testing, etc.
There’s that terra something tool, but it requires containers always, I think.
I had a couple (very) rough ideas on a CI system. One was to make the actual task configuration a Lua script with a bunch of predefined functions to make a declarative pipeline possible, but to also allow the user to drop into more imperative steps. Using Lua lets you more effectively sandbox the tasks than using something like the JVM, the runner could be much leaner, and users could possibly get real autocompletion for their steps instead of only relying on docs for some made up yaml DSL. I also really want to integrate it more deeply with metrics, so you can see annotations in Grafana when you deployed and have automatic rollback when something goes wrong.
(nods) Though something like this remains, to me, the most ideal architecture.
I want a calling convention not stuck decades in the past and lugging C’s type system forever forward. I want a network stack, free of Berkeley sockets; capabilities for content and compute. I want the same programs on my phone, tablet, laptop, desktop, and Other People’s Computers; and I mean the same— browsers and the web are so much closer than “native apps” and yet so fucking far.
I want this every day I make another REST API or gRPC service or Kafka topic. I want this every day I shepherd my team building a product written for Android and iOS. I want this every day I play games on my Switch, knowing the word “buy” on screen is a lie, I’ve only rented this experience. I want this every day the music on my headphones playing from Bandcamp won’t seamlessly switch to my living room’s stereo. I want this every day my girlfriend wonders why she can’t stream Ted Lasso to her Chromecast. I want this every day my colleagues block out “Monday” on their calendars and I’m in a timezone on the other side of the world. I want this every single day I wait for funds to move from my bank account in one developed economy to pay taxes in another developed economy.
I don’t dream, I live in a nightmare.
But it pays the bills. I eat well!
Well, now I’m sad.
Once upon a time, we invented regular expressions as a comfortable midpoint between expressive power and efficiency on 1970s computers. We invented tools to search for regexes (
grep), tools to automatically edit streams based on regexes (sed), programming languages designed to run code when a regex was matched in the input stream (awk) and even interactive editors with regexes as first-class UI primitives (vi).Here in the 21st century, text processing has advanced a long way. In particular, parsing context-free grammars no longer requires days of swearing and wrestling with shift/reduce errors; you can write grammars nearly as succinctly as regexes, and more readably. Using packrat parsing or Earley parsing or some other modern algorithm, I’d love to have a suite of text-processing tools like
grep,sed,awkandvibased on grammars rather than regexes.Do you have any links or examples? I struggle at this!
In most cases, I end up doing text processing using (neo)vim macros rather than regexp. It can run surprisingly fast even on large datasets (in headless mode or with
lazyredraw).Feels like a very modern/ergonomic/incremental/less abstract approach compared to regular expressions.
I do like the premise of clearly defined grammars, however! Could compound nicely with macros, too.
Made me think of this recent Structural Search & Replace plugin that uses treesitter (basically grammars under the hood).
Now that I think of it, treesitter is essentially a database of grammars that can be used for data annotating and processing. 🙃 I guess the next step is to have a more on-the-fly way to describe and use these things.
I kind of got that feeling back when I played with turtle:
You can see some examples in the comment here: https://github.com/Gabriella439/turtle/blob/main/src/Turtle/Pattern.hs
hackage seems down right now, but there is a tutorial there: https://hackage.haskell.org/package/turtle/docs/Turtle-Tutorial.html
semgrep?
That’s interesting. I think you’d be looking at matching/querying an AST-like structure?
For matching elements of specific kinds of tree-like structure, we have
jqand xpath.Is that the kind of thing you mean (but perhaps for general grammars?)
If not, how do these differ from your thoughts/vision?
Personal inventory management.
I want to be able to slap a QR UUID sticker on a box. Then added the contents of the box to a db with
Lastly add all of the images to a to image recognition system so I can look up what an item is and where it goes. Also being able to query from the other direction (e.g. where is tiny tiny machine square).
It ultimately would not be helpful for keeping organized, and potentially counter productive, but it’s nice to dream that this would magically fix my poor organization skills.
The image recognition system isn’t present, but snipe-it can do the inventory management with UPC and purchase info + QR code generation parts of this idea. I self host it to track computer parts and systems through the process of repair, it’s been very useful in staying organized so far.
I’m actually working on an app like this. You can take a photo, scan a barcode, and add a description. There’s also a barcode generation tool. Any feedback is welcome.
https://stuffo.app
What would you need to inventory? like your closet of clothes? Or stuff in your storage or garage? Or for your small shopify business?
Containerized desktop manager. Apps, or groups of apps, or maybe each virtual desktop, run in isolated containers. You can pause, stop, start, etc, new groups, and access to hardware and shared data (i.e., $HOME) are granted per group. You should be able to move apps between isolated environments.
The main use cases would be keeping a more strict separation between work/personal stuff, and being able to save to disk the whole state of a virtual desktop, during pauses in work in specific projects or tasks.
Isn’t this 90% of QubesOS?
Yeah. Not sure about the pausing/hibernating part, though. And containers have a much smaller overhead.
Have you actually measured this? With memory deduplication, VMs will share as much memory for shared libraries as containers (more than containers that aren’t sharing a layer). A minimal paravirtualised Linux kernel build will add a fairly small amount of memory consumption per process in comparison to a typical desktop application.
VMs have an hypervisor, containers don’t. So you might get close with memory, but I doubt that the gap can be bridged as much on CPU.
And even then, I’m giving the idea a wide margin because there’s a ton of different VM technologies, so there really might be some setups that are competitive. The run of the mill virtualbox, though, doesn’t stand a chance.
There’s a lot to unpack here. When you say ‘containers’, I presume you mean ‘shared-kernel virtualisation’, since containers don’t necessarily imply a specific isolation technology (and, often, are deployed on VMs for isolation). With that assumption:
The overheads of shared-kernel virtualisation are not uniform. Docker on Linux uses namespaces, cgroups, and seccomp-bpf. The BPF filtering can have a fairly high overhead on system calls and can be over 10% of total CPU usage in the worst case. Something like
containerdhas a bunch of different options for filesystems, but the layered model for containers is built on a snapshot abstraction. With ZFS, that’s relatively cheap, but the default on Linux is to use some form of union filesystem, which means that filesystem namespace operations can be linear complexity in terms of the depth (you need to sequentially look through each layer to find a match). This, again, can incur a large performance penalty.In contrast, the overheads from VMs come from two places. First, you need nested paging. You may be paying that anyway (XNU and Windows both enable it by default for various security features) and you may well have it enabled to be able to run VMs, unless your host uses shared first and second-level page tables (this is increasingly difficult as modern architectures provide different layouts of the two for their different uses). The overhead for something like SEV-SNP can be double-digit percentages, but the overhead for things with good locality is very low. It basically makes TLB fills more expensive. The second overhead from VMs is for any I/O, which generally goes via a paravirtualised interface for performance. For a Qubes-like use case, with Linux-on-Linux, I’d assume that they’re using FUSE-over-VirtIO. This is an efficient asynchronous (batched) protocol and it allows the guest to share read-only mappings of buffer-cache pages, so a single page (e.g. for a shared library) can be mapped read-only into many guests with CoW mappings.
On top of that, there are some second-order effects. For example, the last couple of FreeBSD releases have included VNET, which lets you have a separate instance of the network stack for each jail. This consumes more memory but can significantly increase performance because there’s no contention between jails, anywhere between the system call and ethernet driver layers. VMs get this benefit for free: each one can have a separate network stack, filesystem stack, process and thread namespace, and so on. In contrast, a VM has its own scheduler, which runs on top of a VCPU abstraction, which can cause some other performance artefacts.
TL;DR: Full VMs and shared-kernel virtualisation both have overheads and neither is a clear winner for all workloads. There have been numerous papers published on this topic.
Now, that’s what I call a reply XD
Seems like I’ve been overestimating the advantages of containers over VMs on the general case, thanks for the thorough explanation.
Maybe I should give qubes a go. Back when I first heard of it my hardware was very limited, but I have much more capacity now, so, could be interesting.
Also would let you run a glibc desktop app on a musl host.
Do you need containers for this, or just use different user accounts?
In theory, malicious code could go looking for the other user account, but the same is perhaps true of containers (I’m not up to date - but I understand you shouldn’t use them as a security perimeter).
For security, yes. But the main reason I came up with the idea was to be able to save a group of apps and resume the state later, which requires some level of virtualization.
I think if you want checkpoint/restart, you can get somewhere applying CRIU (but it doesn’t currently support GUI apps) but VMs are a much better bet…
A persistent-object-store / filesystem-replacement, somewhat along the lines of the NewtonOS “soup”, or of the various ideas for tag-based file management that have floated around here.
A really good object-capability-based RPC library. Cap’n Proto is decent but the C++ API is awkward and it doesn’t yet support persisting capabilities or passing them around between clients, which is where it gets really interesting.
This is what we were trying to build with CoreObject, but updated for modern usage, including:
This looks very cool! But past-tense? Has it been abandoned?
I haven’t worked on it for about 10 years. It became clear that GNUstep and Objective-C were not adequate for building the kind of system I wanted, so I started working on hardware isolation that would let us safely combine C code with languages with no unsafe mode, so that we could build things like this with vast numbers of isolated components.
Have you seen https://perkeep.org/
A task manager & time tracker that works together with the file-system, git and other tools. E.g. if I start working on project x, task y, any downloads and screenshots automatically go to ~/my/projects/project-x/task-y. It would also create a feature branch in the corresponding git repo, start dev server, open editor, etc.
Get into the demoscene. But I never take the time…
Emacs, but not trapped in the ‘70s. A system – a holistic system, top to bottom, UI to drivers – that is live, inspectable, plastic, and fun. Let me explore my computer. I don’t care about security beyond the most basic, “don’t let unknown folks do things” extent. I don’t care about reliving the multi-user systems of my youth – I want my device to be mine. I don’t care about how easily it interacts with the rest of creation.
I want a UI at least as polished and sophisticated as the classic Mac OS, but not based on the world of scare storage and slow machines.
I am afraid that if you want sanity in drivers, what you dream of is not just reasonable software, but also hardware not actively hostile to reason.
Have you tried Smalltalk?
It sure sounds like SmallTalk, doesn’t it? I’ll have to give it a shot.
You might be interested in following https://graphite.rs/ development. I’m definitely looking forward to it.
Adjustment layers been the longest WIP. At least the prerequisite GEGL is here now. I adjusted my photography workflow so that I can do everything with darktable (and Hugin) to get around the non-destructive situation with GIMP—luckily you’ve been able to retouch skin via wavelet decompose inside a darktable module.
More PureScript compile targets for 0.15+ (like Lua, Zig, etc.)
AsciiDoc in Pandoc
FOSS, federated AGPL Flickr
FP-oriented game engine (not tied to .NET)
Would PixelFed fulfill the FOSS federated AGPL Flickr? Or not close enough?
Not quite. It’s goals are to be more like an Instagram clone which isn’t suitable—you don’t need ‘stories’. Digital art doesn’t need to be limited to a square, or sRGB, or have its metadata (including licensing) stripped and compressed to hell to save a few bytes because the kind of content is different. Flickr also keeps an uncompressed original as a backup which can be used in an emergency and the groups features was a big part of what used to make it so popular. It’s also still a good place to search for Creative Commons licensed content.
MediaGoblin is much closer in that sense. Butif you want to hack on either project, you’re looking at PHP or Python on Docker and that’s not very interesting to me.
Its* goals. English is hard.
We should shake up the UNIX monoculture a bit. I’d hope for a secure microkernel OS (seL4?) written in Rust (or Ada). Ideas from e.g. Genode, EROS, QNX, Plan9. For starters it would just need a microkernel + a way to act as a hypervisor. You’d be able to run a Linux / Windows guest to work and continue improving the host OS.
We need more security, privacy and reliability from the ground up. But we can’t forget about usability, or nobody is ever going to use the worlds most secure paper weight.
Take a look at Fuschia which is a microkernel that has parts written in Rust.
A good recipe app designed for power users. I want to be able to plan party menus and see flavor pairings and take notes on new recipes. All the recipe apps I tried are little more than recipe storage + shopping list.
This 1000x.
The funny thing is that I know so many people, and I’ve seen so many people [on github], that tried to solve this problem with their own bunch of scripts or apps. It always break down later either because of unexpected feature creep that’s too hard to add or just by lack of motivation.
(I think I found something to try to model in alloy xD)
The funniest thing to me is that “a place for all your recipes!” was a big selling point of computers in the 1980s and we still don’t have a good recipe app
Ha! Just like computers were supposed to get rid of all paper documents… We’re getting there, but meanwhile computers made the quantity of paper we use simply explode :shrug:
Oooh… is this the “Alloy” you’re referring to? https://en.wikipedia.org/wiki/Alloy_(specification_language)
Yes, and the personI was replying to knows a thing or two about specifications. https://www.hillelwayne.com/tags/alloy/
I gotta say, Alloy is pretty cool because it’s somewhat simple and it gives you nice visualization
What I really want is hardware – a box, about the size of a phone, with a few dozen terabytes of storage, that all my various screens can use as backing state. The phone, the desktop, I don’t know, the TV, the watch, &c. All of them should use my own pocket cloud. I want centralized storage of my data without having to give control over to someone else.
Don’t forget the backup one!
The best possible code cad system.
Curv 3d?
Yeah within that realm. Curv has its own issues that need resolution, but IMO its approach is currently the best one.
BRL-CAD is worth looking at, even if it is ancient - everything can be described in code.
An accurate emulator for a modern CPU (say, skylake). It would allow for much greater visibility into performance, answering questions like ‘why isn’t this shit scaling?’ or ‘to what extent is btb aliasing actually a problem for things like bounds/overflow checks and inline caches?’ with much greater precision than would otherwise be possible. Some things that are cool but aren’t it:
performance counters: not enough information
valgrind: too primitive, and decades out of date
uica: not enough reach (basically just simulates scheduling)
Surprisingly, a lot of the information you’d need for something like this is publicly available, but not, obviously, in executable form. This would also be immensely valuable as a platform for doing further research into modern CPUs; if you observe behaviour X, and can’t explain it, but think it might be caused by Y, then you can implement Y in the emulator and see if it causes behaviour X.
I actually have started working on something along these lines, but it is far down my priority list and I despair of ever finishing.
(Plus, obviously, a broader computing ecosystem that isn’t complete garbage, but I have to pick and choose my battles, and have chosen to forego that one.)
This sound very cool. Where do you find the publicly available details on microarchitecture of contemporary desktop CPUs? Would it allow you to get reasonably close to a cycle accurate description of a processor?
While a lot of information comes from third-party researchers, a surprising amount comes from intel’s own optimisation manual (I won’t bother linking it because they have a habit of breaking links—even worse than microsoft in that respect, given that they actively remove pages containing useful and interesting information, rather than just moving them around—… /rant).
This makes sense when you consider their market position: they effectively have a monopoly on the high-performance market, both internet/web and scientific. Due to their scale, their users are strongly incentivised to optimise for microarchitectural details of their target CPUs. Intel, in turn, is incentivised to make information available about their CPUs, in order to enable the same, and in order to ensure software runs better on their own CPUs than their competitors’ (kind of a positive version of the ‘cripple amd’ feature of their compiler). More subtly, they are incentivised to double down on microarchitectural compromises, making only monotonic improvements rather than rebalancing, because else, in exactly the same fashion, they’d end up competing against their previous hardware revisions. Intel has been refining the same basic uarch for upwards of a decade now, meaning knowledge and information about it have aged reasonably well.
Aside from third-party research, there are also patents, from which you can often glean some information, but that’s obviously prone to be fraught…
Getting to cycle-accuracy will require quite some work. There are plenty of holes or dim spots. I don’t think anyone knows the exact operation of any branch predictors, for instance, because the main question you need to ask from a performance perspective is ‘is this branch predictable (and hot enough to be predicted)?’ If the answer is yes, then you probably don’t care exactly how long it takes to warm up. Similar applies to e.g. the speculative prefetcher. There’s also the fact that ‘cycle-accuracy’ is not so clearly defined as it is for something like the NES. Timings on real hardware will depend on real-world factors like the weather, or the quality of the wiring connecting you to your RAM.
Anyway, here are some links. I’m surely forgetting some things:
Agner has published information on all intel/amd/via x86 processors from the last decade or so. Not super detailed (aside from the instruction timings), compared with some of the other resources, but it is nicely comprehensive along the other two axes.
Henry wong’s blog contains a lot of useful information and measurement methodology.
John mccalpin’s blog is nice, and in particular is (seemingly) one of the few people that actually talks about cache coherency protocols.
Brandon falk is mostly focused on security, but has done some interesting work in this area (some of which, iirc, was used by andreas abel).
Andreas abel did a lot of work on intel processors as part of his phd, including detailed instruction timings and a pipeline model, and cache replacement algorithms.
Maynard handley has done a lot of detailed research into recent revisions of apple’s cpus, and some of the discussion of the memory subsystem likely applies to other cpus as well.
Dougall johnson has also done some nice measurement work on apple’s cpus
Travis downs’s uarch-bench contains some information I haven’t seen elsewhere, and there’s some on his blog as well.
Thanks a lot for the links, this is a treasure trove :)
A useful AI assistant
Fully open-source, self-hosted Keybase.
Have you looked at https://keyoxide.org ?
Have you seen https://keys.pub/?
I recently started to look for a replacement for my self-hosted Drone CI, trying to find something I like, which is fully open source and self-hostable, and came up empty handed. Since then, I’ve been dreaming about a CI system that’s somewhere between Drone and GitHub Actions. I even started researching and designing my own CI, but… I really don’t have the time - nor the expertise, really - to build it. Not anytime soon.
So I’ll keep dreaming.
https://bob-cd.github.io/ keeps popping in my brain
For some odd reason, I have not heard of this before. Thanks for pointing me there!
I’ve definitely enjoyed working on buildomat at Oxide. It began out of necessity (GitHub Actions is never going to support illumos really, and the relatively inflexible hosted runner stuff is all .NET which… is certainly a choice one could make!) but has become a way to make functionality choices for us which are pretty different to those made by Actions, etc.
.NET? I thought it was all Go.
Behold: https://github.com/actions/runner and https://github.com/actions/runner-images
AIUI GHA was born from Azure.
Historical nit: Azure Pipelines, not Azure proper. The rebrand to be an Azure product happened pretty late, and it went by various Visual Studio brands previously.
When the “Build vNext” system (what became Azure Pipelines) was being designed, cross-platform for the agent was a key requirement. The agent was initially written in NodeJS. I joined the team after the agent rewrite in C# was completed, so I don’t have first-hand knowledge about why it changed platforms.
Go wasn’t nearly as prevalent or obvious a choice as it would be today. There was a pretty vocal contingent which wanted to rewrite again in Go, but the business value wasn’t quite there yet. And then the GitHub acquisition happened, further complicating things.
(I was a product manager on Azure Pipelines from 2017-2020, for a time the sole product manager.)
It’s a shame that they chose one of the least cross-platform languages available to implement it if that’s the case. Getting it working on even something very Linux-like, such as FreeBSD, is a pain. Getting it working on any architecture that isn’t x86 or Arm is impossible (and not always possible even if that is the architecture). The code is written in an incredibly non-portable style (it has no platform abstraction layer and does things like query the platform to determine whether it’s Linux to see if the filesystem is case sensitive, so every new platform needs to grep the code for macOS, Linux, and Windows to see what things have been conflated).
There is a third-party implementation in Go, which we’ve used very successfully (we do snmalloc CI for FreeBSD using it and we also use it in our Morello cluster, neither of which is possible with the C# agent). I would love to see us ditch the C# version, hire ChristopherHX, and make his implementation the official one.
That said, I was astonished that a CI system created in the last ten year didn’t separate the command executer portion from the control plane. With this separation, it’s easy to support running CI jobs on anything that supports SSH or a serial console. Without it, you’ll never be able to support embedded targets.
Wow, hats off to you! 🙇🏾♂️
Do you still live and breath build/CI/CD? If so, is there anywhere you write your thoughts in the public? I’d love to learn / pick your mind.
We use woodpecker which is a foss fork of drone. It’s not the best but it does work. We trigger it via push/merge hooks on our gitea instance.
Did drone go non foss?
In 2018, from the looks of it.
Is SourceHut’s CI easy to host?
From what I can tell, no.
NixOS mainline support for ppc64le
A liburing based embedded database, something like SQLite or better like RockDB but with a simpler API.
I have been dreaming about software designed to make self-hosting easier for over 10 years now. And I have spent a lot of time on coding it myself, although my results haven’t been very good so far.
I’ve thought about it a lot, tried different approaches and projects, and I think at this point im looking at creating a “lite” infrastructure as a service platform thats self-hostable and has similarities to Fediverse / Mastodon.
A sort of Ruby scripting platform for Android that lets you write Ruby scripts and run them once or periodically, with bindings to Android libraries to let you send notifications and send and respond to intents and all that, and prebundled with libraries to parse HTML, JSON, an HTTP client, anything that might be useful really.
Other nice things to have:
Isn’t this just some variation of Dragon Ruby?
I was not aware of that but I don’t think so, my idea was mostly about scripting API stuff and running cronjobs while that’s definitely more game-oriented.
Ah, my apologies – the itch.io version is. The main site’s version is much less focused on game development as it is focused on “quick deploy” and “write once run anywhere”
See uhh, https://dragonruby.org/ and http://www.rubymotion.com/
Ah I see, I think you could probably use that as the runtime, but that’s only one part of it. I probably didn’t explain what I meant very well; the main concept is an app I can use to write quick scripts on a phone and then run them on that same phone, without having to go through a normal computer or a VM or something like Termux. The language doesn’t necessarily have to be Ruby, it’s just the scripting language I like the most.
I know you said “without having to go through termux”, however, you can easily install ruby in termux, and there’s a widgets application so you can run them directly from the homescreen
A self-hosted, open-source replacement for Discord. The pieces are all there: VOIP is a solved problem thanks to Mumble, text chat is an XMPP client, etc. All it really needs is some duct tape, and the barest of UI polish.
How about https://revolt.chat
matrix / element is getting there
Debugger GUI that’s better than inserting prints everywhere. Maybe just a modern ddd would do.
VCS that doesn’t break code by automerging and doesn’t require manual merges for obvious situations. Think Pijul, but with semantic diffing and merging.
ABI description language. Think C headers, but with added annotations for possibly shared features (slices, (un)validated UTF{8,16,32} strings, etc.) and without C’s warts.
Fast PDF viewer.
I’ve found both Intellij and VSCode debugging UIs to be fairly ok. Problem is they’re tied to the editor, which might be a problem if you don’t like the editors.
Sometime between today and 5 billion years from now, the sun will expand to such an extent that the earth will become uninhabitable. Between now and then, the web as we know it will come to an end.
Eventually, the cost of supporting all of the web’s the bad design decisions will obviously outweigh the benefits of maintaining it. In short, we will eventually need to “break the web”—something hardly anyone has any appetite to do today—and come up with something new. I’m not talking about “Web 2.0” or “Web3”. That’s just business jargon for VCs.
I want to know, what would a genuinely new web look like? Here’s what I’d like to see:
Things to keep
Things to throw away
Things to replace
Things to add
Counterpoint: our cellular biology’s hacks, carried forever…
If humans are still writing CSS on alien planets 5 billion years from now, something has gone seriously wrong.
Some kind of config file format, tool(s) or ecosystem with migrations and backwards compat built-in. To fight bit-rot or revisiting codegen’d, copy-pasta’d boilerplate (or not) files with semver transitions, deprecation. To make wow config files move through time better.
I would like a fast binary diff with insertion/deletion detection that works on multiple-GB files. There are some tools that can do that (ECMerge and Beyond Compare), but they are not open-source and have a GUI.
A port of the cwm window manager to Wayland.
Tweet deletion tool based on retweets and likes.
Let me clear out all tweets filtered by engagement then show me what’s left and let me pick out the rest.
Ideally would also let me filter by if it’s a reply to someone else or is my own tweet (or thread).
The idea? Being able to write workflows across multiple days as imperative code, where you can easily observe the state and write things like “sleep for 5 days” and not have to worry. That and being able to have CI/CD that isn’t declarative, but doesn’t require writing a bunch of webhook nonsense to get anything done.
let job_result = run_job_on_other_machine(configuration).wait(10.days)and having the tools that can handle in progress stuff nicely.There are so many worms here but Salesforce Dot Com is proof that people are fine with tinkering with their stuff (SFDC is the Smalltalk lover’s dream).
Be able to write a test like
property = True | False; some_function(property)and have that smartly calculate all possible results from that.To combinatorial explosion types: the first thing would be just “blow up if there are too many universes”, the second thing would be letting people write code branches for “underspecified mode” so that you can avoid this. I think this sort of tool would work with almost all of the code that people write for enterprise software. It would be 100% fine for this to not work on systems software.
Take Python code, transform it into Typescript code that is “semantically equivalent” (as closely as possible), so that I can use Typescript’s type system in Python. Mypy is good, but Typescript just is so powerful that it’s hard to look at any other type system the same for enterprise software
Just like… a Django thing a la wagtail that makes it easy to set up a forum, customize for your community, and just look like a webpage
General Purpose:
Reddit Moderation:
Realistic physics engine for real-time games. Something where the game-play mechanics only derive from the physics of the world.
A fast and reliable scanner for my Pokemon cards so that I could have an electronic database.
The official app has a terrible browser for cards; you have to know which release the card was a part of. You can’t say “show me all of the psychic type pokemon”.
TCGPlayer has a scanner, but it’s never worked for me beyond once as an accident.
I have a webcam and a powerful desktop. Surely there should be a way I could hold the card in front of the camera, have it recognize the card, and add it to a sqlite database for later consumption by … something else.
I’ve always found it very difficult to get anything useful done with opencv for computer vision. It feels like these things should be achievable these days, but it’s unclear how to do them.
An auto encoder that could transform miles Davis songs into Jimi Hendrix performances by learning the styles
I blog a bit about these:
An ai assistant that could help my elderly father and uncle deal with all the tech in the world they are being pushed into.
Vendor based (might be wrong word) package managers
I want to do something like
npm vendor component-librarywhich would add component-library to ./vendor & i can go in there and modify only ./vendor/Input.svelte & somehow git would only track my changes to Input.svelteSo many times i want to make 1 small edit to a library & i end up doing some major hacks because im too lazy to create & manage a forked repo.
There’s so many packages that acquire so much bloat due to trying to let each user have their own config, vs easily modifiable packages with tools to manage the diffs.
I believe this also would make library code easier to read if its created with the intention to be modified & application devs would understand the frameworks better.
You might be looking for https://www.npmjs.com/package/patch-package
This is cool! I might use this for my component library at work because I’m already making so many global stylistic changes to suit our project.
The description makes it seem like its only for bug fixes / inconvenience, but I’d want a whole new way of thinking of packages. Technically I think its not far off, but my dream involves packages designed to be modified.
Imagine how much simpler APIs would be if you were expected to go into /vendor/package-name/variables.ts to set the global options.
Imagine how much more readable library code would be expected to be if it was expected to be modified by the average user.
I imagine working with packages that are tailor suited towards 1 thing & my needs at work slightly deviate so I change it to fit my need.
In the world we live in packages are built to suit every use-case under the sun because the only ways of providing value to most users is providing a flexible API, rather than a solid base that is easily modifiable.
Clean API & deep understanding of your system at the cost of having to deal with manual upgrades for the components you modify. I’m not sure if the pay off is always worth it, but sometimes it definitely is.
A multiplayer real time tactical space game with a technology level similar to The Expanse. Stealth and orbital trickery are the keys to survival. Friends and I can crew up a ship and divide the various roles of navigation, fire control, and management of ship systems among us. We can play against other real people, but not as an MMO. More like an invite-only scheduled event.
Super realistic physics and ideally beautiful visuals (the part I’d struggle the most with).
I guess since the last time I now can formulate an additional DVCS dream (a very different one)
Basically, be better rsync. Or better Unison? I already have a mess of large files synchronised across multiple devices with rsync, or something behaving more or less the same. I want no new copies, and I want the directory structure to stay and I some copies are on USB drives etc. and should be directly usable but first-class copies. So, no annex stuff or whatever. But I want to version how I move files around, and how I modify them. Modification is just hash-before/hash-after, the idea is that the tool can distinguish which file is older even if timestamps were messed up (see also: USB drives), and also alert me if two versions are actually incomparable. Oh, and as usual I want SVN-style narrow copies, too.
(hm, I also have other data-organisation projects that I kind of do code but very slowly, do they count?)
De/centralization-agnostic social media protocol + test implementation.
…etc. and so on…
Pull-type data transfer + auth would make it resistant to spam from sources you don’t subscribe to. Open protocol + the network graph being stored locally by the client and fetching data from possibly many servers means that the server provider is not a gatekeeper and cannot hold users hostage due to network effects, should they wish to leave.
A server could be anything, a cupboard SBC, a VPS instance, or something analogous to a web hotel tenancy.
By now I’ve been away from social media for so long that I can’t really see myself coming back, even to my own implementation, and if I can’t convince even myself it doesn’t really matter if I had the time in the first place.
Have you looked at dinit? The service configuration isn’t systemd’s unit config, but it is declarative. Essentially, a service is defined in a file containing lines of key=value pairs. For example, here’s the definition of sshd from one of my systems:
I’ve been using dinit for well over a year now. If you dig systemd’s declarative config and don’t care for its mission creep, it is worth a look.
Uselessd, is that project still around?
Open source Scuba
https://research.facebook.com/publications/scuba-diving-into-data-at-facebook/
I believe honeycomb.io is at least trying to build a version that you can use outside of Facebook, even if it isn’t entirely open source. But they are leveraging open telemetry for ingestion which itself is open!
Honeycomb is a fantastic product, but there’s very little of it that is open source, and that’s pretty understandable.
On day though, i’ll find the time to make it happen, at least that’s the dream.
Clickhouse comes close, doesn’t it? Supports streaming ingestion via Kafka, has a new experimental WATCH statement for monitoring output for “live” queries via what they call a Live View. I think this would get you most of the way there.
In the past, I combined Kafka, Amazon S3, Storm, Cassandra & Elasticsearch to get the same value proposition of Scuba, but that required quite a lot of (closed source) engineering and code to make it work well. Described a bit here.
Clickhouse comes close if you have a mostly static schema. If it’s highly dynamic, there’s a point after which too many columns and/or querying json blob doesn’t work.
But yes, CH is an amazing piece of software!
I think Arcan might be evolving to be what I would end up wanting if I actually tried to implement this, but I’d really like to try to build a modern version of NeWS / Display PostScript, using technologies that I could run both within a web browser and in a native application:
On top of this, I’d like a widget set and GUI that runs things like a rich text system (based on SILE for preference) for rendering and interactivity near the user, so if I type something in a text field then it’s displayed immediately and then a message is sent to the application with the update, so that there’s the kind of latency hiding that NeWS offered.
You can build this kind of thing on the web already, but I want a narrower interface (close to WASM + Canvas) so that you can also run it without all of the baggage of a full web stack.
One thing I’ve wanted to build for ages is a web-based Hotline client that works on old computers, I guess? I haven’t really thought it through and won’t know what would work the best until I started using it.
The other idea I think about frequently is a multi-room music player server like Mopidy+Snapcast but also with a whole UI for basically ripping media from wherever (it recently occurred to me the entire project could solely be this UI, living alongside Mopidy). Paste a [YTMusic|Bandcamp|Soundcloud|Spotify|…] link and it parses the artist and songname, rips the audio and permanently stores that in your local library. Eventually you amass a colossal archive of everything you and your family/visitors have added to your Library. Kind of like a high-quality frontend for yt-dlp, but more fully-featured (the artist/song-parsing particularly, and ripping other sources like Spotify). I mean, the whole “legal issues” aspect is one huge blocker to this one, unsurprisingly.
Lately two things:
I’ve been making games lately – Semantle was a big success, Surfwords … wasn’t (it’s cool, but not making any money). I’ve got two more in the works.
What I really want to make is a game along the lines of Myst or The Witness: a graphical puzzle game. I have been collecting individual puzzle concepts for years, and I have almost as many as Myst had (depending on how you count). I also have an obviously doable idea for a flavor of puzzle that’s straightforward to make a fair number of of (like The Witness, but very different in style), enjoyable to solve, and flexible.
What I need is art. Well, art and a story, but if I had a genie that would make art, I would probably be able to put together a story. The problem is that a game of that scale has a lot of art. Myst had 2,500 images. It’s just way more than I can afford to hire someone to do, and I don’t have enough of a track record yet to find a collaborator.
I thought about using Stable Diffusion, but it’s not quite there yet – even if I ignore the very weak animation support, it’s just bad at making physically plausible objects. People (excluding hands) are fine, because there’s so much input data. But things, especially things that don’t exist, are too often too surreal. Also, image generation is slow enough to be a bottleneck.
I think the modern version of this sort of thing is “escape the room”, which, since they take place in a small space, reduce the amount of art needed. But probably they don’t reduce it to a feasible scale, and anyway I have never seen one that had the level of atmosphere that Myst had. And it’s not just the quality of graphics: Monkey Island had worse graphics, but it had a sense of space.