This really resonated with me. I’ve been watching the explainable AI scene for a while and also came away from it feeling that this is a very hard and very unsolved problem. I’ve been really enjoying working with language models because of how approachable and easy to understand that they are. I’ve seen it said multiple times and I think it bears repeating: a language model is like a really enthusiastic intern that you have to guide a little and reel back in when it goes too far. The ability to iteratively improve the outputs of an AI language model is something that’s hugely valuable and really, really hard with existing AI models.
The ability to iteratively improve the outputs of an AI language model is something that’s hugely valuable and really, really hard with existing AI models.
What do you mean by that? What does “existing AI models” refer to?
What I was thinking was: if you create a simple model that makes a decision, after it makes it decision, you don’t really have any ability to refine that decision. If a sentiment analysis model predicts that some text is positive, but on review, you know that the text is negative, there’s nothing built into the model to nudge it in a different direction other than changing the training dataset or by weighting inputs to the model differently. I think these each require a deeper understanding of how the model works internally than an LLM for steering the output of that model.
This is easily one of my favorite online tools that I use when trying to figure something out and explore something I’m not familiar with. For example, I’ve used it for figuring out ratios for making soda cake in a mini rice cooker (similarly, banana nut muffins). I’ve also used it for estimating dataset sizes (I’ve no examples on me, but stuff like “at 512x512 resolution, how many images fit in 1 GB of RAM? what if I shift the number of bits per color?”).
The one thing I’ve been bit with in the past is that copy-pasting sections can sometimes mess things up. For the game Overwatch, I wanted to figure out the math on how a change in the patch notes affects a characters strength, so I wanted to compare a before and after. I wrote up the before, copied it, pasted it back, and then tweaked it for the after, but I was getting weird results. I ended up having to recreate the after from scratch to get it to work correctly.
My one “ask” would be to have a way to tweak the raw data without the formatting. From looking into it, I recall it using some non-printable character to represent “insert the n-th previous value into this equation” (or… something), so I don’t know how that would interact with a normal text box. Also a feature like this would ideally never need to be used, so eh 🤷
My one “ask” would be to have a way to tweak the raw data without the formatting.
I’m actually thinking of removing the “reference the line” feature, in the flavor of variables with autocomplete. This references are very difficult to implement and giving me a lot of troubles =)))
I currently replaced the click on the answer to just copy the result to clipboard, instead of inserting the reference.
Oh! That explains why it wasn’t doing what I expected when I showed my friend earlier!
A feature I only just learned is that variables can be re-declared repeatedly, and even assigned new values based on their old values. That actually solves my other pain point of having a hard time finding variable names for everything, hence why I used the positional references instead.
Updating one of my previous links to use that feature instead yields this. The one big difference is that the number isn’t shown in-line with the text which was something I quite liked, since it made it easier to write a readable takeaway/conclusion. Beggars and choosers and all that aside, I will switch to this style.
I’m also planing to make variables easier to use. For example: highlighting all variables on mouse over, and maybe using different colors for each variable.
This is pretty cool! I just wish it included some photos of it in operation. It’s a shame it has to rely on ngrok but at the same time I understand the trade-off.
I don’t know if this has changed in the last 20 minutes, but if you go to the new beta UI ( https://beta.dwitter.net/ ) then there is a toggle to switch between the compressed and uncompressed versions.
This was absolutely fascinating to me, from beginning to end. It’s also very neat to see that the author is the one who developed the Newer SMB game(s) that I played with my family years ago.
I think it’s interesting to see a successful game company actually structures their code and how much of that code gets reused for later games. Perhaps the most surprising is that, while the naming and functional conventions appear misguided or archaic (why still use _c as a suffix for all classes? Why have the d prefix at all?), it clearly didn’t get in the developer’s way very much.
Based on all the details in this post, I wonder if it’d be possible or useful to create a standalone game engine like the one in NSMB.
To me, one of conventions I wouldn’t have thought of is the separate lists/queues for things that need to be deleted, created, executed, or drawn; and moreover, that this list might need to keep around things that still need to be acted on. I would have thought that if something did get added to the “to delete” list, then it would just be immediately deleted then removed from the list, rather than asking it to delete itself and postponing deletion until the next tick if necessary. I wonder how much of that decision is based on (I’m assuming) a technical decision to not have multithreading but instead use a form of cooperative multitasking (which is common for games that need to keep their core game loop within a certain number of milliseconds to be ready for outputting the next frame).
These are useful examples, i.e. 2 failed attempts and the right one.
I have found that quoting and evaluation is sort of a “missing topic” in programming education. I think I got exposed to it through Lisp but then it took awhile for my brain to transfer that knowledge to strings, Python, shell and C. It’s very important for security, i.e. understanding SQL injection, HTML injection (XSS), and shell injection.
For example this post has all sorts of quoting/evaluation errors, like manipulating shell code with sed and then piping directly to sh:
But it doesn’t work if you have a newline. It sometimes emits the $'\n' strings, but doesn’t understand them. These are the most general type of string (e.g. POSIX single quoted strings can’t contain single quotes).
So the fact that bash doesn’t do this correctly is more evidence that even the authors of languages are confused about quoting and evaluation.
I think that some people wondered why Oil even has QSN at all! It is so you can quote and unquote correctly 100% of the time. You don’t have to worry about data-dependent bugs, like when you strings contain spaces, newlines, single quotes, double quotes, or backslashes.
It’s just Rust string literals, which are a cleaned up version of C string literals. Most people understand 'foo\n' but not necessarily
x='foo
'
(the way to write a newline in POSIX shell)
That is, there is a trick to concatenate \' in POSIX shell, but it doesn’t have the property of fitting on a single line.
I think what would be useful is to have a post on the relationship between “quoting and evaluation” and say JSON serialization and deserialization. They are kind of the same thing, except the former is for code, and the latter is for data. It’s not an accident that JSON was derived from the syntax of JavaScript, etc.
So the problem here is to serialize an argv array as a string. And SSH does it naively by concatenating and separating with a space! This leads to the problem where you arguments with special characters are mangled!
Indeed, I’ve also noted this design flaw of Ssh (and that it’s concatenating the wrong way) in my how to do things safely in Bash guide. So not entirely unrecognised, for what it’s worth.
After reading your xargs post, I think I’m of the exact opposite opinion: I only use find -exec and haven’t touched xargs in years. I understand xargs enough to know that it needs to be treated carefully depending on what data is being passed around, and it doesn’t work well with my idea of iteratively building up commands. Consider going from find . -type f where it simply prints out the names, then moving on to using xargs requires changes to both the find command and to xargs: find . -type f -0 | xargs -0 rm. Of course, in this trivial example, it would just be better as find . -type f -exec rm {} + (for symmetry with \; I usually write it as \+).
Instead, I’ve taken to using a strategy where I go straight from find back into the shell. The pattern is kind of obtuse, admittedly, but there’s never a case where the filenames get passed around through a pipe and where delimiters have to be considered. The simple example above would be:
It’s a bit of a mouthful, but it then lets you use any shell features inside of the bash -c command, which I prefer because I already think in terms of shell expansions and commands. I use this a lot when I need to rename files with a weird convention. For example, I’ve used this before to convert a folder full of world.2017-01-01.converted.bin files that should be converted to world/converted/2017-01-01.bin could be written as:
The '<bash -c>' or '<bash -c rename>' argument is needed because it sets Bash’s argv[0] which is shown in process listings. Without it, the first argument gets lost.
At this level of effort, I think it could be better to just use shell completely and make use of shopt -s globstar. I think it would look like this:
$ shopt -s globstar
$ for src in **/world.*.converted.bin; do dst=$src; dst=${dst%.converted.bin}.bin; dst=${dst#world.}; dst=world/converted/$dst; mv "$src" "$dst"; done
But then you lose both: all of the extra features within find and being able to more easily build up the command iteratively. Plus, my brain prefers to go straight to find when I need to recursively go through directories, and globstar is more of an afterthought.
Side note: I realized that it could be somewhat straightforward to write a “find to bash” (or “find to posix sh”) converter to remove any find dependency altogether, something like the following:
$ find2bash . -type f -name 'world.*.converted.bin' -mmin 10 -exec echo Removing {} now... \;
#!/usr/bin/env bash
tempdir=$(mktemp -d)
printf -v escaped 'rm %q' "${tempdir:?}"
trap "${escaped:?}" EXIT
# find -mmin 10
# I think this has to be GNU's touch
touch --date="-10 minutes" "${tempdir:?}/-mmin 10"
shopt -s globstar
for arg in ./**; do # find .
# find -type f
[ -f "${arg:?}" ] || continue
# find -name world.*.converted.bin
case "${arg:?}" in
(world.*.converted.bin);;
(*) continue;;
esac
# find -mmin 10
[ "${arg:?}" -nt "${tempdir:?}/-mmin 10" ] || continue
# find -exec echo Removing {} now... ;
echo Removing "$arg" now...
done
That’s definitely a valid way of doing it and I will concede that find -exec \+ doesn’t have the gotcha of newlines in filenames, which xargs -d $'\n' does.
It’s nice to preview and separate the two issues: what to iterate on, and what to do. It’s basically like Ruby / Rust iteration vs. Python.
xargs -P is huge; can’t do this with find
find is its own language which I think is annoying. It has globs, regexes, and printf. I’d rather just use shell globs, regex, and printf.
Although your -exec bash idiom is very similar to the $0 dispatch pattern I mention. I use xargs to “shell back in”, and you are using find to “shell back in”.
To add another bit to it: if you use ®exp<Enter> (or &!regexp<Enter>) to filter things, another &<Enter> (or equivalently &!<Enter>) then it will clear out the filter.
The only problem I’ve found with this (that I haven’t been able to resolve yet) is that I can get less locked up in a really large file when filtering. The usual Ctrl-C or qqqqqqqs don’t seem to do anything. So that’s the only caveat and where grep is still useful: large files.
Not the OP and not 2 lines, but a shell solution might look like this (with lots of UUoC):
#!/usr/bin/env bash
set -euxo pipefail
temp=$(mktemp)
trap "rm -f $temp" EXIT
cat >"$temp"
${EDITOR:-vi}" "$temp"; then
exec cat <"$temp"
If you really wanted to squeeze it down to 2 lines, then you could do some hacky vim/bash stuff (and accept that the file will exist after the process runs, ‘cause I interpret “2 lines” as “2 subprocesses”, and I’m already bending the rule with the shebang).
$WORK: Managed to get a lot done on the paper during the week and then almost none during the weekend. I think I needed a break from working on it because I kept finding myself getting distracted on things that aren’t important (e.g. I tried to write my own mapping library to work around problems I found in Leaflet, but now I’m going to throw that code out because it’s probably got more problems of its own).
I start my new internship today and have a lot of on-boarding stuff to do (training, paperwork, etc). I’m hoping that I can start working on some of the work I’ll be doing, if I can get onto all of their systems I’ll need. So this week I’m just adjusting to all the new things and trying not to look too bad.
$HOME: Re-made that salad dressing from last week and it was even better. Sometime this week, we’ll make it another time but writing down measurements we use. I think it’s also possible we’ll finish watching Bones and can switch to another series. We’re missing the final/most recent seasons of 4 shows now. Some friends and I just finished our playthrough of Divinity: Original Sin 2 and we loved it. We’re looking to find another game like it to play.
I’ve spent a long time finalizing my home directory. I use a variation of the FHS for all my local files, though it took a while to really appreciate and make the most out of it.
Every time I reinstall my OS, I go through the same process of removing all the Documents, Pictures, Downloads, etc directories and create the ones I prefer. Most of them stay gone, but Desktop and Downloads often come back. The biggest perpetrators are Gnome and Chrome, respectively, but this is solved with a few settings that I always have to look up.
~/bin/ - for any scripts I write and use, and especially for wrapper scripts to fix things I don't like
~/mnt/ - for `sudo mount /dev/sdb ~/mnt/`
~/opt/ - for software and tools that might need compiled (like ~/opt/dwm, ~/opt/Write, ~/opt/yEd)
~/share/ - for writings or drawings that I create (like ~/share/documents/*.svg, produced by Write)
~/src/ - for any project I work on (every directory here is a git repo)
~/src/*_paper/ - for research papers I'm working on
~/src/tmp/ - for any projects that I don't intend to commit to git (like re-clones of my work projects for trying new things)
~/tmp/ - for any files I download
~/tmp/*/ - for any completely throwaway projects (e.g. ffmpeg scripts for making hour long sleep videos)
~/etc/ - for any configuration files for tools I use
I want to use the $XDG_CONFIG_HOME and $XDG_DATA_HOME directories but I never spend enough time making sure it all works, so I don’t really trust them to work right. I also just don’t like the idea of hiding config files in a hidden directory.
One thing I used to do that I no longer do is to keep a very organized ~/src/ directory. I had a script to clone every project into an exact path and then update symlinks to be easier to use. The trouble I ran into is that it was a little too regimented and didn’t make it easy to write throwaway code because I always had to create a GitHub repository when I wanted to work on a project.
~/src/github.com/$owner/$repo - for any GitHub projects cloned
~/src/bitbucket.org/$owner/$repo - for any BitBucket projects cloned
~/src/$repo -> ~/src/github.com/$owner/$repo - symlinks for easier access to the full path
I also try to have a small home directory as most of my work is on limited drives, so I need a solution to managing large datasets. The trick I use there is to make heavy use of symlinks to a data drive. For example, a project ~/src/foo would have ~/src/foo/data pointing to /mnt/mylargedrive/foo/data and ~/src/foo/gen would point to /mnt/mylargedrive/foo/gen. Then I can document the data source and formats in a /mnt/mylargedrive/foo/README and everything stays nice and clean.
$WORK: I got a ton of work done this past weekend on the code for a paper we’re writing. I got to learn and practice a lot of OpenGL stuff and made a ton of progress.
I’m hoping to get a couple more features done and continue the writing process so it can be done before I start my internship next week.
I’m also very excited about the direction our work is going and think that it’s going to be a great foundation for my thesis. For the past few years, I haven’t really known what I wanted to do for my thesis, so this is the first time I’ve really felt like I get it and that has been nice.
$HOME: I made more of those coffee bean style cookies and this batch turned out more like I was wanting. They’re a little dryer and crunchier which is what I was trying for. My wife and I also made some Asian salad dressing like we used to get from one of our favorite restaurants. We had been trying to make it for a while, and this batch turned out the best so far, so I’d like to write the recipe down. I didn’t start the #100DaysToOffload writing like I wanted, but that’s okay, maybe this week.
$WORK: Finished the paper! I’m very pleased with how it turned out, especially that it marks the end of an 18 month dry spell of writing and working. A lot of my moodiness with being unmotivated was resolved by having consistent work to do.
Now onto the next one, and then taking a writing break to focus on code by working with something entirely different. Today is about collecting some baseline results to compare against and writing a short form of the paper for a class. The rest of the week, I’ll be continuing to get enough data together to write the full paper.
$HOME: Now that that paper is done, I’m feeling really motivated to start Kev’s #100DaysToOffload challenge. I think I’ll be writing some about software and cooking mostly. This weekend, I made little coffee bean shaped cookies but I had to tweak the recipe a lot, so I want to consolidate my changes and also start making large batches to share with family/friends.
$WORK: It’s paper deadline time again! We’re submitting one paper on Thursday, so this week is all about wrapping that up. Then the next day, I have a final project presentation for a class before doing the last sprint on another paper due early May. After that, I start my virtual summer internship.
Quarantine has been getting to me more lately, so I’ve been resolving to try to have more of a schedule and get more time outside in the sun. I’m finding it pretty hard to stay motivated sometimes, but having this work schedule with the papers helps a lot, surprisingly.
$HOME: I had a lovely anniversary weekend with my wife, and I really enjoyed having time completely away from work, so I think I’ll keep that up. That said, I’ve been reading a professor’s blog and am feeling very motivated to do more writing on my blog and I’d love to start that up again. I felt very accomplished after finishing the last post I wrote up.
Other than that, it’s just going to be more crime drama bingeathons and relaxing.
I try to write a few blog posts here and there at https://mediocreatbest.xyz . I tend to follow the style of “let the code do the talking” without much exposition. For example, I just finished on the creation of compute-heavy scientific microservices which summarizes a lot of the things I’ve learned about embedding C code into Python servers, mostly in terms of how the code is written. My old blog includes a few more posts in that style.
My understanding is that PHONY rules are like any other rule, it just skips the check for whether the file exists. You can already depend on a file that doesn’t and will never exist, for example:
clean:
rm -rf ./output
Now make clean will run the rule like you might expect. The catch is that someone could create a file called “clean” and now your script won’t run. This is what PHONY solves: even if a file “clean” exists, it’ll pretend like it doesn’t.
From there, you can also depend on a rule that depends on a file that will never exist. For example, a clean-all rule could reuse the clean rule as follows:
This is all that .PHONY: phony rule is doing. It almost acts as if it’s inheriting the phony status, but that’s just a consequence of how Make handles transitive rules (if a sub-dependency doesn’t exist, it’ll re-run the whole chain of rules after that one).
The part I find interesting is that they say it slows down larger Makefiles, which I wouldn’t expect to be the case, at least not significantly.
Cheers for the != thing! I hadn’t seen that one before but it seems very useful.
$WORK: I’ve got a lot of stuff going on this week. I have a small presentation on containers and their history on Thursday which I have to throw together. In 3 weeks, I have my conference presentation, so I hope to start on that today and get it moving along so that I have enough time to practice and prepare. I also need to work on open sourcing the code that the conference paper/presentation is about, which I haven’t really started on yet.
I think I’m going to need to punt on a project I’m excited about for now, so that I’ll have enough time to get everything else done. Because everything is happening all at once this month, I’m going to have to get really good about managing my time. I’ve started journaling via jrnl.sh to keep track of that time better and I’m hoping to do some nightly reviews of the journal to see where I’ve lost time. I tend to treat myself poorly during busy months, which I’d like to improve on by making sure to give myself time to decompress and relax. After last year’s busy March month, I needed several months just to decompress, during which I didn’t really do anything, and I’m trying not to let that happen again.
$HOME: I’ve been playing with my 3D printer again. I finally have a copy of SolidWorks so I can start 3D modeling stuff. So far, I haven’t printed many useful things, but it’s been really nice to be able to do this again. There’s something really relaxing about taking measurements of things and recreating them in SolidWorks; it’s very systematic and the end result is always nice. Right now, I’m designing a little faux sink for my cat to play in so we don’t have to feel bad about wasting water (it’ll get reused and re-pumped through the system).
I want to get my personal server in use again. I didn’t get into blogging again like I’d hoped, and I think that’s because I made the system a little too DIY. I’d like to throw in someone else’s server that supports MicroPub that way I can blog from my phone, instead of having to get out my laptop. At the end of the day, it probably still won’t get much use, but it’s still fun to play around with.
I bought a laptop a week or two ago, a ThinkPad T420. I used to own one of those but ended up selling it to a friend, and I’ve been missing it. Unfortunately, the one I have doesn’t have all the nice specs the old one had (an i7, an SSD, maxed out RAM), but it works well enough once it gets going. It’s my main SolidWorks machine, from above. It also has a DVD drive, so I’m thinking about ripping some of the DVDs I own and setting up a plex server, perhaps on my gaming PC attached to the TV.
Lastly, a good friend is leaving for Australia for a few weeks, so we’re hanging out tonight before he goes. We’ve been playing through the game Divinity, which has a map/campaign editor that we’re hoping to play with and maybe port some simple One-Off D&D campaigns to, which should be fun.
$WORK: I finished revising my paper and submitted it and it’s completely done! I now have about 6 or 7 weeks to make and prepare a presentation for the conference, and also polish up the code for release.
I’m getting some more work done on my other paper about graph rendering. For this week, my focus is to get an E2E MVP built and start evaluating it.
$FUN: My website is passably working now: personal page and blog, all with appropriate tags to each other that can be automatically parsed.
I spent most of the weekend playing around with IndieWeb concepts: MicroPub and MicroSub, specifically. I’m excited for them because I’m hoping to build a “personal indexer” of sorts. I read lots of articles online and I always forget what I’ve read and where. It’d be nice if my browser automatically saved the contents of pages I visit, saving them in either a plain text or microformat way. Further, if someone is interested in the things I read about (or more likely, if I’m interested in what others wrote), they could subscribe to my personal archive and query it too. I suspect that it’d be possible to replace a lot of my Google searches with this, without sending ever more data to Big Google all the time.
$WORK: Over the weekend, I got some work done on the paper like I wanted, both coding and writing. With the initial draft out of the way, we should be able to start iterating on it and getting some work done. I also finished handing off that other project, so that’s off my plate.
One of my other papers got conditionally accepted with some minor revisions, so that’s my main focus this week: fixing the things our reviewers asked for and polishing for publication, plus starting on the presentation part of it so that I’m prepared for March. I will also need to get the code running again and make sure nothing has bitrotted.
$FUN: I didn’t get around to adding IndieWeb stuff to my website, but maybe I’ll find a few minutes here or there this week to at least get IndieAuth working. I solved my problem of “hard to access my self-hosted apps if they’re running from my network” the easy way: just run it on another network.
For my website/blog posts, I’m currently trying to decide between using lots of HTML things like microformats or just using plain text files. On the one hand: if I marked everything up, I could make good use of things like the recipe microformat and keep my recipes machine readable, and also have nice semantic links to other blogs or posts. However, then I have to give up the super easy “just throw text files on the internet” approach that I’ve been liking. The approach I’m leaning towards is to have things like recipes on their own standalone HTML pages and just link to them from my posts.
$WORK: Finally handing over a project I’ve been working on for a while so that I can work on something else. The last step is getting AWS instances to create their own Docker Swarm and connect to one another. After that’s done, I get to work on drafting a couple of papers and maybe start writing some code for them.
$FUN: I’m hoping to work on my website and add some IndieWeb/microformats stuff to it, plus finding a publishing workflow I like. I have something that’s mostly working so I can write some blog posts already, but I’d like it to be more streamlined.
I’m also looking into self hosting some services at home. One problem I’ve run into is: if you’re hosting your website on your local network, you can’t view your website at its domain name (easily) because it pulls up the router’s admin page. I’ve tried to fix this with a local DNS entry on my pi-hole, but I wish there was something a little easier, especially since I expect to host many services on the same device, all with different domain names. Right now I have to add an entry manually for each one.
This really resonated with me. I’ve been watching the explainable AI scene for a while and also came away from it feeling that this is a very hard and very unsolved problem. I’ve been really enjoying working with language models because of how approachable and easy to understand that they are. I’ve seen it said multiple times and I think it bears repeating: a language model is like a really enthusiastic intern that you have to guide a little and reel back in when it goes too far. The ability to iteratively improve the outputs of an AI language model is something that’s hugely valuable and really, really hard with existing AI models.
That last bit…
What do you mean by that? What does “existing AI models” refer to?
What I was thinking was: if you create a simple model that makes a decision, after it makes it decision, you don’t really have any ability to refine that decision. If a sentiment analysis model predicts that some text is positive, but on review, you know that the text is negative, there’s nothing built into the model to nudge it in a different direction other than changing the training dataset or by weighting inputs to the model differently. I think these each require a deeper understanding of how the model works internally than an LLM for steering the output of that model.
This is easily one of my favorite online tools that I use when trying to figure something out and explore something I’m not familiar with. For example, I’ve used it for figuring out ratios for making soda cake in a mini rice cooker (similarly, banana nut muffins). I’ve also used it for estimating dataset sizes (I’ve no examples on me, but stuff like “at 512x512 resolution, how many images fit in 1 GB of RAM? what if I shift the number of bits per color?”).
The one thing I’ve been bit with in the past is that copy-pasting sections can sometimes mess things up. For the game Overwatch, I wanted to figure out the math on how a change in the patch notes affects a characters strength, so I wanted to compare a before and after. I wrote up the before, copied it, pasted it back, and then tweaked it for the after, but I was getting weird results. I ended up having to recreate the after from scratch to get it to work correctly.
My one “ask” would be to have a way to tweak the raw data without the formatting. From looking into it, I recall it using some non-printable character to represent “insert the n-th previous value into this equation” (or… something), so I don’t know how that would interact with a normal text box. Also a feature like this would ideally never need to be used, so eh 🤷
Edit: Found a few more examples of how useful this tool is. I compared the cost of buying a domain for 1, 2, or 3 years. I found the average cost of a robot vacuum and self-cleaning litterbox. I estimated the amount of spendable “fun” money while still saving money.
Cool! Glad it is useful for someone else =)
I’m actually thinking of removing the “reference the line” feature, in the flavor of variables with autocomplete. This references are very difficult to implement and giving me a lot of troubles =)))
I currently replaced the click on the answer to just copy the result to clipboard, instead of inserting the reference.
Oh! That explains why it wasn’t doing what I expected when I showed my friend earlier!
A feature I only just learned is that variables can be re-declared repeatedly, and even assigned new values based on their old values. That actually solves my other pain point of having a hard time finding variable names for everything, hence why I used the positional references instead.
Updating one of my previous links to use that feature instead yields this. The one big difference is that the number isn’t shown in-line with the text which was something I quite liked, since it made it easier to write a readable takeaway/conclusion. Beggars and choosers and all that aside, I will switch to this style.
Thank you for making this tool! :)
I’m also planing to make variables easier to use. For example: highlighting all variables on mouse over, and maybe using different colors for each variable.
Got a lot of email asking to return references back)
So will be bringing them back, but with Alt+Enter now.
Also, I’ve reimplemented references and now they do not require extra spaces around them. So now old docs have a little bit extra white space in them)
Documentation?
Several links deep, I found a couple of pages that could serve as documentation.
The manual in
.texiformat: https://git.ageinghacker.net/git/cgit.cgi/jitter/tree/doc/jitter.texiThe work-in-progress tutorial as a normal web page: https://ageinghacker.net/projects/jitter-tutorial/
One tutorial is https://www.gnu.org/ghm/2022/#talk-jitter-vm
This is pretty cool! I just wish it included some photos of it in operation. It’s a shame it has to rely on ngrok but at the same time I understand the trade-off.
There’s a video in the Tweet link.
For completeness (since I didn’t see a direct link to the tweet in the article): https://twitter.com/aschmelyun/status/1506960015063625733
Some of the examples are compressed and end up with something like this shell
in order to keep the post under 140 charaters
It would be nice if as a user if one was presented with uncompressed code and the compressed code
Apart from that very interesting, similar to shadertoy
I don’t know if this has changed in the last 20 minutes, but if you go to the new beta UI ( https://beta.dwitter.net/ ) then there is a toggle to switch between the compressed and uncompressed versions.
Thanks, you are right it is in the beta I only checked the default site
This was absolutely fascinating to me, from beginning to end. It’s also very neat to see that the author is the one who developed the Newer SMB game(s) that I played with my family years ago.
I think it’s interesting to see a successful game company actually structures their code and how much of that code gets reused for later games. Perhaps the most surprising is that, while the naming and functional conventions appear misguided or archaic (why still use
_cas a suffix for all classes? Why have thedprefix at all?), it clearly didn’t get in the developer’s way very much.Based on all the details in this post, I wonder if it’d be possible or useful to create a standalone game engine like the one in NSMB.
To me, one of conventions I wouldn’t have thought of is the separate lists/queues for things that need to be deleted, created, executed, or drawn; and moreover, that this list might need to keep around things that still need to be acted on. I would have thought that if something did get added to the “to delete” list, then it would just be immediately deleted then removed from the list, rather than asking it to delete itself and postponing deletion until the next tick if necessary. I wonder how much of that decision is based on (I’m assuming) a technical decision to not have multithreading but instead use a form of cooperative multitasking (which is common for games that need to keep their core game loop within a certain number of milliseconds to be ready for outputting the next frame).
These are useful examples, i.e. 2 failed attempts and the right one.
I have found that quoting and evaluation is sort of a “missing topic” in programming education. I think I got exposed to it through Lisp but then it took awhile for my brain to transfer that knowledge to strings, Python, shell and C. It’s very important for security, i.e. understanding SQL injection, HTML injection (XSS), and shell injection.
For example this post has all sorts of quoting/evaluation errors, like manipulating shell code with
sedand then piping directly tosh:https://codefaster.substack.com/p/xargs-considered-harmful
I have used that pattern in the past, but I’ve moved away from it in favor of xargs, and I never put it in a shell script.
I responded to it here: http://www.oilshell.org/blog/2021/08/xargs.html
Fun fact: bash ALMOST does its quoting correctly with
printf %qor${x@Q}and the “not quite inverse”printf %bto unquote.https://github.com/oilshell/oil/wiki/Shell-Almost-Has-a-JSON-Analogue
But it doesn’t work if you have a newline. It sometimes emits the
$'\n'strings, but doesn’t understand them. These are the most general type of string (e.g. POSIX single quoted strings can’t contain single quotes).So the fact that bash doesn’t do this correctly is more evidence that even the authors of languages are confused about quoting and evaluation.
Oil has QSN instead: https://www.oilshell.org/release/latest/doc/qsn.html
I think that some people wondered why Oil even has QSN at all! It is so you can quote and unquote correctly 100% of the time. You don’t have to worry about data-dependent bugs, like when you strings contain spaces, newlines, single quotes, double quotes, or backslashes.
It’s just Rust string literals, which are a cleaned up version of C string literals. Most people understand
'foo\n'but not necessarily(the way to write a newline in POSIX shell)
That is, there is a trick to concatenate
\'in POSIX shell, but it doesn’t have the property of fitting on a single line.I think what would be useful is to have a post on the relationship between “quoting and evaluation” and say JSON serialization and deserialization. They are kind of the same thing, except the former is for code, and the latter is for data. It’s not an accident that JSON was derived from the syntax of JavaScript, etc.
Actually a really practical example of where this comes up is SSH quoting:
https://lobste.rs/s/8tki7j/ssh_quoting
https://www.chiark.greenend.org.uk/~cjwatson/blog/ssh-quoting.html
So the problem here is to serialize an argv array as a string. And SSH does it naively by concatenating and separating with a space! This leads to the problem where you arguments with special characters are mangled!
https://news.ycombinator.com/item?id=27483077
Top comment:
Indeed, I’ve also noted this design flaw of Ssh (and that it’s concatenating the wrong way) in my how to do things safely in Bash guide. So not entirely unrecognised, for what it’s worth.
After reading your xargs post, I think I’m of the exact opposite opinion: I only use
find -execand haven’t touchedxargsin years. I understand xargs enough to know that it needs to be treated carefully depending on what data is being passed around, and it doesn’t work well with my idea of iteratively building up commands. Consider going fromfind . -type fwhere it simply prints out the names, then moving on to using xargs requires changes to both the find command and to xargs:find . -type f -0 | xargs -0 rm. Of course, in this trivial example, it would just be better asfind . -type f -exec rm {} +(for symmetry with\;I usually write it as\+).Instead, I’ve taken to using a strategy where I go straight from
findback into the shell. The pattern is kind of obtuse, admittedly, but there’s never a case where the filenames get passed around through a pipe and where delimiters have to be considered. The simple example above would be:It’s a bit of a mouthful, but it then lets you use any shell features inside of the
bash -ccommand, which I prefer because I already think in terms of shell expansions and commands. I use this a lot when I need to rename files with a weird convention. For example, I’ve used this before to convert a folder full ofworld.2017-01-01.converted.binfiles that should be converted toworld/converted/2017-01-01.bincould be written as:The
'<bash -c>'or'<bash -c rename>'argument is needed because it sets Bash’sargv[0]which is shown in process listings. Without it, the first argument gets lost.At this level of effort, I think it could be better to just use shell completely and make use of
shopt -s globstar. I think it would look like this:But then you lose both: all of the extra features within
findand being able to more easily build up the command iteratively. Plus, my brain prefers to go straight tofindwhen I need to recursively go through directories, andglobstaris more of an afterthought.Side note: I realized that it could be somewhat straightforward to write a “find to bash” (or “find to posix sh”) converter to remove any
finddependency altogether, something like the following:</ramble></ramble></ramble>That’s definitely a valid way of doing it and I will concede that
find -exec \+doesn’t have the gotcha of newlines in filenames, whichxargs -d $'\n'does.However my responses:
xargs -d $'\n'composes with other tools like grep and shuf: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-composes-with-other-toolsxargs -Pis huge; can’t do this withfindAlthough your
-exec bashidiom is very similar to the $0 dispatch pattern I mention. I use xargs to “shell back in”, and you are using find to “shell back in”.Discussed here btw: https://lobste.rs/s/xestey/opinionated_guide_xargs
https://news.ycombinator.com/item?id=28258189
Small world! I just learned this trick as well.
To add another bit to it: if you use
®exp<Enter>(or&!regexp<Enter>) to filter things, another&<Enter>(or equivalently&!<Enter>) then it will clear out the filter.The only problem I’ve found with this (that I haven’t been able to resolve yet) is that I can get
lesslocked up in a really large file when filtering. The usualCtrl-Corqqqqqqqs don’t seem to do anything. So that’s the only caveat and wheregrepis still useful: large files.vipe is pretty useful. Can do things like:
To manipulate intermediate results with your $EDITOR. I’ve re-implemented this tool in Haskell:
https://hackage.haskell.org/package/editpipe
That’s funny, i thought it was based upon the moreutils author. I guess vipe predates Joey’s desire to write everything in Haskell?
You gonna tell us how? ;-)
Not the OP and not 2 lines, but a shell solution might look like this (with lots of UUoC):
If you really wanted to squeeze it down to 2 lines, then you could do some hacky vim/bash stuff (and accept that the file will exist after the process runs, ‘cause I interpret “2 lines” as “2 subprocesses”, and I’m already bending the rule with the shebang).
This doesn’t open a tty, so under some conditions it won’t actually open the editor UI.
Neat! Thanks!
Left as an exercise for the reader.
What are the two lines
Have a go, it’s a bit trickier than you think. I guarantee you won’t implement it correctly first try.
(last week: https://lobste.rs/s/qbjfe7/what_are_you_doing_this_week#c_277whb)
$WORK: Managed to get a lot done on the paper during the week and then almost none during the weekend. I think I needed a break from working on it because I kept finding myself getting distracted on things that aren’t important (e.g. I tried to write my own mapping library to work around problems I found in Leaflet, but now I’m going to throw that code out because it’s probably got more problems of its own).
I start my new internship today and have a lot of on-boarding stuff to do (training, paperwork, etc). I’m hoping that I can start working on some of the work I’ll be doing, if I can get onto all of their systems I’ll need. So this week I’m just adjusting to all the new things and trying not to look too bad.
$HOME: Re-made that salad dressing from last week and it was even better. Sometime this week, we’ll make it another time but writing down measurements we use. I think it’s also possible we’ll finish watching Bones and can switch to another series. We’re missing the final/most recent seasons of 4 shows now. Some friends and I just finished our playthrough of Divinity: Original Sin 2 and we loved it. We’re looking to find another game like it to play.
I’ve spent a long time finalizing my home directory. I use a variation of the FHS for all my local files, though it took a while to really appreciate and make the most out of it.
Every time I reinstall my OS, I go through the same process of removing all the Documents, Pictures, Downloads, etc directories and create the ones I prefer. Most of them stay gone, but Desktop and Downloads often come back. The biggest perpetrators are Gnome and Chrome, respectively, but this is solved with a few settings that I always have to look up.
I want to use the
$XDG_CONFIG_HOMEand$XDG_DATA_HOMEdirectories but I never spend enough time making sure it all works, so I don’t really trust them to work right. I also just don’t like the idea of hiding config files in a hidden directory.One thing I used to do that I no longer do is to keep a very organized
~/src/directory. I had a script to clone every project into an exact path and then update symlinks to be easier to use. The trouble I ran into is that it was a little too regimented and didn’t make it easy to write throwaway code because I always had to create a GitHub repository when I wanted to work on a project.I also try to have a small home directory as most of my work is on limited drives, so I need a solution to managing large datasets. The trick I use there is to make heavy use of symlinks to a data drive. For example, a project
~/src/foowould have~/src/foo/datapointing to/mnt/mylargedrive/foo/dataand~/src/foo/genwould point to/mnt/mylargedrive/foo/gen. Then I can document the data source and formats in a/mnt/mylargedrive/foo/READMEand everything stays nice and clean.(last week: https://lobste.rs/s/obwsyn/what_are_you_doing_this_week#c_vjp4di)
$WORK: I got a ton of work done this past weekend on the code for a paper we’re writing. I got to learn and practice a lot of OpenGL stuff and made a ton of progress.
I’m hoping to get a couple more features done and continue the writing process so it can be done before I start my internship next week.
I’m also very excited about the direction our work is going and think that it’s going to be a great foundation for my thesis. For the past few years, I haven’t really known what I wanted to do for my thesis, so this is the first time I’ve really felt like I get it and that has been nice.
$HOME: I made more of those coffee bean style cookies and this batch turned out more like I was wanting. They’re a little dryer and crunchier which is what I was trying for. My wife and I also made some Asian salad dressing like we used to get from one of our favorite restaurants. We had been trying to make it for a while, and this batch turned out the best so far, so I’d like to write the recipe down. I didn’t start the #100DaysToOffload writing like I wanted, but that’s okay, maybe this week.
(last week: https://lobste.rs/s/vcx5vu/what_are_you_working_on_this_week#c_ilxscv)
$WORK: Finished the paper! I’m very pleased with how it turned out, especially that it marks the end of an 18 month dry spell of writing and working. A lot of my moodiness with being unmotivated was resolved by having consistent work to do.
Now onto the next one, and then taking a writing break to focus on code by working with something entirely different. Today is about collecting some baseline results to compare against and writing a short form of the paper for a class. The rest of the week, I’ll be continuing to get enough data together to write the full paper.
$HOME: Now that that paper is done, I’m feeling really motivated to start Kev’s #100DaysToOffload challenge. I think I’ll be writing some about software and cooking mostly. This weekend, I made little coffee bean shaped cookies but I had to tweak the recipe a lot, so I want to consolidate my changes and also start making large batches to share with family/friends.
(8 weeks ago… a lot longer than I remembered https://lobste.rs/s/epx9km/what_are_you_working_on_this_week#c_bnj6ad)
$WORK: It’s paper deadline time again! We’re submitting one paper on Thursday, so this week is all about wrapping that up. Then the next day, I have a final project presentation for a class before doing the last sprint on another paper due early May. After that, I start my virtual summer internship.
Quarantine has been getting to me more lately, so I’ve been resolving to try to have more of a schedule and get more time outside in the sun. I’m finding it pretty hard to stay motivated sometimes, but having this work schedule with the papers helps a lot, surprisingly.
$HOME: I had a lovely anniversary weekend with my wife, and I really enjoyed having time completely away from work, so I think I’ll keep that up. That said, I’ve been reading a professor’s blog and am feeling very motivated to do more writing on my blog and I’d love to start that up again. I felt very accomplished after finishing the last post I wrote up.
Other than that, it’s just going to be more crime drama bingeathons and relaxing.
I try to write a few blog posts here and there at https://mediocreatbest.xyz . I tend to follow the style of “let the code do the talking” without much exposition. For example, I just finished on the creation of compute-heavy scientific microservices which summarizes a lot of the things I’ve learned about embedding C code into Python servers, mostly in terms of how the code is written. My old blog includes a few more posts in that style.
I didn’t know that phony tags are inherited, or how does this work?
Also, if you’re already using GNU extensions, you might like to replace
with
My understanding is that PHONY rules are like any other rule, it just skips the check for whether the file exists. You can already depend on a file that doesn’t and will never exist, for example:
Now
make cleanwill run the rule like you might expect. The catch is that someone could create a file called “clean” and now your script won’t run. This is what PHONY solves: even if a file “clean” exists, it’ll pretend like it doesn’t.From there, you can also depend on a rule that depends on a file that will never exist. For example, a clean-all rule could reuse the clean rule as follows:
This is all that
.PHONY: phonyrule is doing. It almost acts as if it’s inheriting the phony status, but that’s just a consequence of how Make handles transitive rules (if a sub-dependency doesn’t exist, it’ll re-run the whole chain of rules after that one).The part I find interesting is that they say it slows down larger Makefiles, which I wouldn’t expect to be the case, at least not significantly.
Cheers for the
!=thing! I hadn’t seen that one before but it seems very useful.(3 weeks ago: https://lobste.rs/s/2qzmtr/what_are_you_doing_this_week#c_cijuow)
$WORK: I’ve got a lot of stuff going on this week. I have a small presentation on containers and their history on Thursday which I have to throw together. In 3 weeks, I have my conference presentation, so I hope to start on that today and get it moving along so that I have enough time to practice and prepare. I also need to work on open sourcing the code that the conference paper/presentation is about, which I haven’t really started on yet.
I think I’m going to need to punt on a project I’m excited about for now, so that I’ll have enough time to get everything else done. Because everything is happening all at once this month, I’m going to have to get really good about managing my time. I’ve started journaling via jrnl.sh to keep track of that time better and I’m hoping to do some nightly reviews of the journal to see where I’ve lost time. I tend to treat myself poorly during busy months, which I’d like to improve on by making sure to give myself time to decompress and relax. After last year’s busy March month, I needed several months just to decompress, during which I didn’t really do anything, and I’m trying not to let that happen again.
$HOME: I’ve been playing with my 3D printer again. I finally have a copy of SolidWorks so I can start 3D modeling stuff. So far, I haven’t printed many useful things, but it’s been really nice to be able to do this again. There’s something really relaxing about taking measurements of things and recreating them in SolidWorks; it’s very systematic and the end result is always nice. Right now, I’m designing a little faux sink for my cat to play in so we don’t have to feel bad about wasting water (it’ll get reused and re-pumped through the system).
I want to get my personal server in use again. I didn’t get into blogging again like I’d hoped, and I think that’s because I made the system a little too DIY. I’d like to throw in someone else’s server that supports MicroPub that way I can blog from my phone, instead of having to get out my laptop. At the end of the day, it probably still won’t get much use, but it’s still fun to play around with.
I bought a laptop a week or two ago, a ThinkPad T420. I used to own one of those but ended up selling it to a friend, and I’ve been missing it. Unfortunately, the one I have doesn’t have all the nice specs the old one had (an i7, an SSD, maxed out RAM), but it works well enough once it gets going. It’s my main SolidWorks machine, from above. It also has a DVD drive, so I’m thinking about ripping some of the DVDs I own and setting up a plex server, perhaps on my gaming PC attached to the TV.
Lastly, a good friend is leaving for Australia for a few weeks, so we’re hanging out tonight before he goes. We’ve been playing through the game Divinity, which has a map/campaign editor that we’re hoping to play with and maybe port some simple One-Off D&D campaigns to, which should be fun.
(2 weeks ago: https://lobste.rs/s/gmbx5t/what_are_you_doing_this_week#c_s0pk5o)
$WORK: I finished revising my paper and submitted it and it’s completely done! I now have about 6 or 7 weeks to make and prepare a presentation for the conference, and also polish up the code for release.
I’m getting some more work done on my other paper about graph rendering. For this week, my focus is to get an E2E MVP built and start evaluating it.
$FUN: My website is passably working now: personal page and blog, all with appropriate tags to each other that can be automatically parsed.
I spent most of the weekend playing around with IndieWeb concepts: MicroPub and MicroSub, specifically. I’m excited for them because I’m hoping to build a “personal indexer” of sorts. I read lots of articles online and I always forget what I’ve read and where. It’d be nice if my browser automatically saved the contents of pages I visit, saving them in either a plain text or microformat way. Further, if someone is interested in the things I read about (or more likely, if I’m interested in what others wrote), they could subscribe to my personal archive and query it too. I suspect that it’d be possible to replace a lot of my Google searches with this, without sending ever more data to Big Google all the time.
(last week: https://lobste.rs/s/8uitwq/what_are_you_working_on_this_week#c_kez6sx)
$WORK: Over the weekend, I got some work done on the paper like I wanted, both coding and writing. With the initial draft out of the way, we should be able to start iterating on it and getting some work done. I also finished handing off that other project, so that’s off my plate.
One of my other papers got conditionally accepted with some minor revisions, so that’s my main focus this week: fixing the things our reviewers asked for and polishing for publication, plus starting on the presentation part of it so that I’m prepared for March. I will also need to get the code running again and make sure nothing has bitrotted.
$FUN: I didn’t get around to adding IndieWeb stuff to my website, but maybe I’ll find a few minutes here or there this week to at least get IndieAuth working. I solved my problem of “hard to access my self-hosted apps if they’re running from my network” the easy way: just run it on another network.
For my website/blog posts, I’m currently trying to decide between using lots of HTML things like microformats or just using plain text files. On the one hand: if I marked everything up, I could make good use of things like the recipe microformat and keep my recipes machine readable, and also have nice semantic links to other blogs or posts. However, then I have to give up the super easy “just throw text files on the internet” approach that I’ve been liking. The approach I’m leaning towards is to have things like recipes on their own standalone HTML pages and just link to them from my posts.
$WORK: Finally handing over a project I’ve been working on for a while so that I can work on something else. The last step is getting AWS instances to create their own Docker Swarm and connect to one another. After that’s done, I get to work on drafting a couple of papers and maybe start writing some code for them.
$FUN: I’m hoping to work on my website and add some IndieWeb/microformats stuff to it, plus finding a publishing workflow I like. I have something that’s mostly working so I can write some blog posts already, but I’d like it to be more streamlined.
I’m also looking into self hosting some services at home. One problem I’ve run into is: if you’re hosting your website on your local network, you can’t view your website at its domain name (easily) because it pulls up the router’s admin page. I’ve tried to fix this with a local DNS entry on my pi-hole, but I wish there was something a little easier, especially since I expect to host many services on the same device, all with different domain names. Right now I have to add an entry manually for each one.