Crustaceans,
When you are looking for a piece of software or a tool for your needs, what is your technique? I find it very hard to discover hobbyist projects.
Let me develop, a month ago I was looking for an append-only key-value store. I remember it having some kind of flower name “Kamelia” or something like this. It took me a lot of duckduckgoing/googling and searching on github: “append only store”, “key value store no deletion”. After multiple days of giving up and trying searching again, I found it “Parkeep (née Camlistore)”. Since the name changed and it doesn’t mention anywhere it is append-only, I dismissed it two or three times during my tries.
Another day, I was looking at alternative for gcc and clang. I clicked around github, sourcehut and gitlab READMEs to find scc, chibicc, cproc or lacc. Most of them look like high quality hobbyist projects, which are my favorite kind of projects.
Sometimes I find insanely high quality hobbyist projects hidden in comments on Lobste.rs, the orange website or Reddit.
When looking for these types of high quality hobbyist projects, github is not always the solution. Some are hosted on sourcehut, gitlab or codeberg. But sometimes, you can also find some isolated gitlab or gitea instances. It is very hard to search these platforms.
For this usecases, Duckduckgo and Google are literally useless. I’m looking for an alternative to Ansible right now, the results are filled with Search-Engine-Optimized articles that list the same thing “Puppet”, “CFEngine”, “Chef”. I’m not looking for this, I’m looking for some person in their garage who decided: “I think all of this sucks, I will do my own!”, kinda like ziglang started.
Do you have a secret technique or website of which I’m not aware? Or does it suck as well for you, and you just spend hours clicking around and bookmarking websites?
I start really wondering how these Lobste.rs/Reddit comments find these projects…
Maybe not the answer you’re looking for, but I generally don’t go actively looking for new tools.
If I’m at a job I’ll learn what they use, and sometimes that’s new tech. If I’m on a project and something falls across my desk while researching I might follow up on it. Perhaps most frequently, if people I’m chatting with bring up “oh have you used so-and-so” I might investigate it if it looks promising.
If I’m really, really stuck on something and I can clearly articulate what my problem is and it isn’t worth it to solve it myself, then I might go looking for a new tool–in which case, polling coworkers, Googling, and so forth is a good idea.
In general, though, I dislike actively looking for tools because:
Maybe the biggest thing for me is that, at the end of the day, I get enjoyment from actually solving problems and building things, and it is hard to do that if you keep finding strangers in your toolbox.
There’s no greater friction than going against the grain of what your organization is built around.
This really deserves to be repeated more. People are always harping about “using the best tool for the job”, and while that may be true, the best tool might only look like the best tool from a distance. When you get to know the tool better you’ll almost certainly find inadequacies that are (deliberately or not) being hidden. Alternatively, it might be the best tool but the learning curve could be so high that you will only be able to use it effectively after falling on your face a few times, and your current project that needs it might not be the right place or time to learn. With the tools you already know, you will know how to avoid the sharp edges and make the most effective use of them.
Of course, it’s important that you don’t stagnate, as technology does progress, and you don’t want to be that one dude that keeps clinging to his ancient practices (like for example in the early 2000s when people would be using shared FTP and manual backups instead of VCSes). Although I do find that more often than not, it seems like we as an industry are taking two steps forward, one step backward every single time.
I’m in this camp. I often find that my hunt for a new tool comes out of laziness. I know full well that I have what I need in front of me, but I can’t summon the ambition to make it work. That’s usually when I’ll take a step back and restate my assumptions about what it is I’m working on.
This is a fantastic point (and also a wonderful turn of phrase). Sure, there may be a “better” tool to use but oftentimes the tool you know will help get to a solution more successfully (faster, less bugs etc) than fumbling around with something unfamiliar.
This is me as well. Maybe it’s because I don’t really bother with software as a hobby (my hobbies are time consuming and not computer related), but I pretty much only focus on the current tool chain in place at my work. That has meant learning new tools (web frameworks, database admin tools, new languages), but I tend to focus on mastering what is in place instead of chasing the next shiny thing (that might never be used at my place of work anyway.)
I used to worry that this would cause my skills to atrophy, but I’ve mostly found the opposite. Like you say, a lot of what is out there is shovels. I also mostly focus on backend work, so I get more value out of my time understanding design patterns in my language than in searching for yet more frameworks.
You’ve pretty much described my process, too. The only other thing I sometimes do is look at the “Awesome” lists, like this one for React or this one for Typescript. I literally just search for “awesome [technology]”. The lists are a mixed bag, sometimes they’re carefully curated, othertimes they seem to just list every project under the sun that’s vaguely related. But even in the latter case they’re frequently a good pointer to what I should be searching for.
I prefer to “have them all” and filter myself, instead of “not getting it in my results”. Most of the time, using Google and Duckduckgo, I just “don’t see it”. That’s the sad part…
But you’re right, Awesome lists are also a good medium.
I think a lot of these lists are starred as “read it later”. But I feel like people don’t. If you want to get a star, make it so long that people think it’s useful.
I used to have a script that would scrape a list of libraries but also detect the star. Then I’d make a markdown table out of it. I was trying to use stars as “in use” even though this isn’t a great metric. The awesome style lists aren’t curated enough (I know this is non-trivial). It’d be cool to have an algorithm and stuff around libraries. Youtube does it for video and they can’t read the “source”.
There’s a handy search option for such lists here: https://awesomelists.top/
I search lobsters for that thing I saw once a few months ago that I thought was really cool and kept open in a tab on one of my phones for a while until I ended up closing it thinking “I’ll remember how to get that.”
Given how completely your description fits my own browsing behavior, it might be worth sharing that I’ve been largely successful at sticking to a low-friction system without relapsing to this previous approach.
I liberally use the “save” feature here, the “star” feature on GitHub, and the “favorite” feature on the orange site. The latter two are publicly viewable.
I make no categorization or organizational attempt upon saving items, nor do I pretend that there is any specific plan to return to those items. These lists merely serve as a smaller subset of items I am able to manually look over when I want to recall something I once found of interest. It has been highly effective for me, even given how low my bar is for adding something to those lists; they’re not curated, merely a log of my gut reaction that something might be of interest.
By the way, if you’re looking for GCC/Clang alternatives, tcc by Fabrice Bellard is very fast. I use it where possible. However, it’s x86-only and inline assembly is not supported.
It’s been a while since I looked at tcc and what do my sore eyes see in the recent commit history but something adding ARM support at least for BSD. Looking at the tinycc source tree, it appears to be expanding out, with x86, x86_64, ARM, ARM64, and RISCV64 support.
I’m glad I’m not the only one with this issue!! I also invariably lose track of the tools I have installed if they’re not being used every day, and have a devil of a time finding them again on my machine once I do need them.
For tools that I’ve installed and that I don’t use too often, I usually add aliases for them. For example:
It has the advantages of only having to remember the name of the functionality, having the name of the tool recorded somewhere in case I really need it, and not having to remember another program name if I end up using another one for the same functionality.
That reminds me that I still need to find that one program that records gif from a part of my screen, and that just works…
I used to use GitHub, but there the downside is that it’s really easy to get to hyped software, that ended up on HackerNews and alike. There’s projects with thousands of stars that effectively don’t exist yet or never will exist. So that’s not a good measure. What works a bit better is going over to users profile pages. Take someone that does a lot of GIS for example. It’s likely that they have other GIS projects, or contributed or starred them. Sometimes one finds something in READMEs.
But of course not everything is on GitHub. I also lock at competitors.
While a commercial(?) website (that I am completely unaffiliated with), alternative.to often works, when I know I want something that is similar enough to something else it actually does a pretty decent job.
Sometimes Wikipedia helps, but that’s now largely replaced with the Arch Linux Application List on their Wiki. That one is really nice for things like “I want a TUI version of X” or “I want X, but not depending on al of of GNOME/KDE/…”.
DuckDuckGo and Google seems to be getting worse and worse for that kind of research. I cannot pinpoint it exactly. Google sometimes has the edge when it knows your “interests”, but it can be really hard to exclude stuff and use it more like full text search. That’s something that used to be easier a decade or more ago. I assume fighting spam and the whole SEO thing could have stopped that from working properly. That assumption is based on there not being other search engine doing that kind of thing.
I also save posts on lobste.rs, if I think software (or software lists/comparisons) could be interesting at a later time. Search through it is not working so well, but I don’t think it’s that the search is bad, more that I would have to know what’s on the pages linked to. One line summaries are usually not enough and in general, summaries tend to be fairly bad for non-trivial tools. I am sure most of you came across situations where something different was understood based on them.
On top of that it’s getting worse, cause even very technical software is being marketed. Buzzwords and generic phrases “be more efficient”, “automate X”, etc. and sometimes blatant misuse of words can make it quite hard to find out what one is looking at.
I make profuse use of GitHub/GitLab/whatever stars when I see stuff linked from Lobsters or reddit or anywhere else. If there’s something I really want to play with, I make a ticket in the appropriate system. E.g., this ticket for adding
espansoto my default setup or a ticket in my work system for playing with vega.These kinds of referrals can pretty much only come from social networks (not necessarily social media networks, I mean the abstract concept) or from someone else who’s watching. Back in the day, I’d watch Freshmeat but I don’t know what’s out there that’s basically just a firehose of release announcements with changelogs. I guess tools like dependabot address the security/feature needs to watch that firehose but there’s still the “ooh shiny” part of me that misses that experience.
I miss Freshmeat very much. It was nice to have a system which didn’t assume anything about where the code was hosted or how it was distributed, had a large enough network to contain almost everything, and reviews to infer trust. I’ll never understand why it closed, and it seems almost impossible to replicate, because a new system with an empty catalog won’t see many visitors or submissions.
Before freshmeat, it seemed like there were a lot of curated software collections on CD-ROM, BBS, etc. These days there are a few still floating around that are basically neglected (remember tucows?) Even though we have more software than ever, there’s something strange in having so few catalogs.
Properly cataloguing things is an expensive and time-consuming process, and nobody pays for information any more, meaning the only way to keep the lights on is to pimp out space in your catalogue to the highest bidder, meaning nobody trusts catalogues…. insert your choice of lionking.gif and/or yodawg.gif
As an (ahem) heavy shopper:
For browsing? Awesome lists, blog posts on the topic, Arch wiki.
For search, mostly github. Which is… OK. But obviously and unfortunately incomplete. …Freshmeat.
Around the time Microsoft bought github and some people started I seriously considered rebuilding Freshmeat.
Not as maintainer-maintained, but as a meta search with the https://libraries.io/ (and other) data. With a more useful search than github.
But, where would the users come from?
Seemed like a lot of work to go from software packages that no one could find to a software search engine nobody could find.
I’m pretty sure it doesn’t exist currently. I want it. Anyone else?
I find new software quite often when I upgrade or install new homebrew (macOS) tools. Homebrew lists new formulas, so I copy them and then invoke
brew home FORMULA_NAMEto review each one. You can do multiple as well -brew home perkeep exa emacs tig. Alternatively https://formulae.brew.sh/ has search, but isn’t great.When I stumble upon something interesting via brew or on the internet I’ve got a little Python script that edits a Markdown file of URLs that I find interesting, which is then synced between all my devices. That file has the page’s title and a url to it and more often than not some note or just keywords for me to find it later, if I need to. Since it is a Markdown file I can edit it myself and add whatever I want via any text editor, but I have to keep the syntax, so my “parser” doesn’t break. The file holds not only software, but also articles and so on. It is exactly like old-days Bookmarks, but using a plain text file. The script also makes a backup every new day upon adding a new entry.
Another source of new software is my GitHub’s activity page (the one you get when you open github when logged in) - it lists the activity of all the people you follow - mostly starred projects. I do use the GitHub’s star button, but I find it quite often I want to add a few words to help me find what I’m starring easier, so I invoke my Python script to add the url and page title to my md file.
The awesome lists people mentioned are also… awesome, but they evolve or I lose what I once knew was there, so I end up adding to my URLStore, ensuring things stay put.
I’ve also signed up to this repo’s notifications for the languages I’m interested in:
There are also people who do curated lists of new stuff, not only software, but tech news related as well, which I’m subscribed to via email. I can dig some up if you want me to share.
This is really hard to solve and it’s my pain point. I have a great article about using a state machine when parsing HTML which I read on mobile and nowadays searching for state machine are all kind of articles that aren’t as good as that one anymore.
I rely on hacker news, a few news letter to curated that for me. I grow to the point I made my own news letter[1] and dedicated a [Tool] sections for it. I tried to have very good description with keyword in title/description so later on i can just
grepto find it. They look like this https://github.com/yeo/betterdev.link/blob/master/content/issues/184.yml#L73[1] https://betterdev.link
It sounds like you are doing the right things when it comes to looking for software and tools. Reading from a number of quality technical sources, combined with diligent googling, is likely the best way to find what you’re looking for.
It is worth realizing there is a “good-enough” solution for every problem out there. It might be helpful to give yourself a time limit for finding a tool. In other words, think about this using the Exploit-Explore Dilemma mental model. When do you want to stop exploring and start exploiting the (less than perfect) tools that you found?
I search on GitHub. Not the best technique but better than Google search in terms of cutting down all the listicles that make it harder to find the actual tool or library.
I used to seek out this kind of stuff quite a bit. I’m not saying I’m evolved or anything. I miss the energy. I am seeking less software these days. I recently firmware updated my coffee mug (srs).
Some ideas:
Hope this is useful or not redundant. I agree with others’ general sentiment of “I don’t do this”. I think it’s an interesting situation: lack of authority for recommendations (no heroes), abstract nature of software (no measurement) and a distributed web (no discovery).
This is a very heavily laden question!
First thing is, “why are you looking for new tools, why do you need a replacement for gcc?”. I am not personally interested in the answer, but am asking you to ask yourself the question. Sometimes things are hard because they are and we haven’t optimized those paths.
Second thing, it sounds like you need more skill in using a search engine. Duckduckgo is very shallow and not very smart. The above the fold Google has been gamed by google and the rest of the net. I could suggest looking into search operators that both provide, limiting searches to domains, file types etc. Look at the old Fravia articles.
I’m always interesting in sharpening my search engine skills. But I do feel that Google results are roten by Search-Engine-Optimized marketing articles, and I don’t see how to get around this.
I think you chose the wrong example by taking only the
gccpart. Sometimes I am genuinely looking for a tool that fits my need, and for which I don’t have a solution except “write it myself”.I’ve noticed a distinct decline in search results for upstream projects and source documents. Content farms and low-effort tech blogs seem to dominate the first page of every search I do lately.
Probably the first thing to consider is your motivation and your values.
Then you look for platforms (programming languages, operating systems, organizations) whose values are comlatible with your own.
Then you look in their forums, websites, chatgroups and so on for whatever it is you need.
If it doesn’t exist you look for a way to create it.
To do that you should first consider why it doesn’t already exist.
Then consider if those reasons also apply to your situation.
If they do then you must consider what kind of situation you would need to be in to be free of these restraints.
Then you either yak-shave or compromise.
I tend to use GitHub and GitLab searches, often filtering by language. GitHub topics can also be useful. If it’s a fairly mainstream tool AlternativeTo can also be useful.
If I see something potentially interesting/useful in passing, I bookmark it with the tag “warelogging” and maybe some relevant notes. I often refer back to this when I think “oh yeah, I saw something for this” later on.
If I have no idea where to look and a generic web search isn’t helping, I:
Have you found a good alternative to Ansible?
If yes, I’d be interested in what you found, and otherwise you can check out cdist.
Maybe mgmt but it competes more with Puppet’s approach than Ansible. I saw a demo of it at FOSDEM 2017 and was quite impressed, mainly because of its speed. Turns out they also gave a presentation at last year’s FOSDEM.
Unfortunately, I haven’t. I’ve looked into
cdist, unfortunately that’s not what I was looking for. I don’t want to write shell scripts manually.I’m border line thinking about doing my own thing :/ .
May I recommend having a look at pyinfra, a lightweight alternative to Ansible.
I feel you, that’s also my current approach for (FreeBSD) jail-aware stuff…