Great article. Really enjoyed reading this, and it’s an issue I’ve run up against countless times by now. Glad to see that you covered both exponential backoff and jitter. Something else we use is token buckets to control flow and burstiness.
I’m going to check out more of your blog, but I noticed a lack of RSS. Is this something you would consider adding?
I use hledger to keep track of my spending, and the fact that it uses a special text format for transactions isn’t a huge concern for me. What is a big concern is that every financial institution I use makes it difficult to programmatically obtain my transaction records, and also to figure out what records I failed to grab properly if my calculated spending doesn’t match up with what the website says. This aspect is by far the biggest problem I have managing my finances and it’s not one that any program I run myself can help with, because the problem is entirely contained within the proprietary software running on the websites of financial institutions subject to their own business and regulatory priorities.
If there was a bank that made it possible for me to make an authenticated REST API request against a website to get my personal transaction data I would seriously consider switching to that bank. But I’m not aware of any such banks and there may be reasons why no bank offers this service.
In the UK, we have Monzo and Starling that both have an authenticated REST API for accessing transactios (and more). The latter is particularly good, but both work great.
That part is so absurd, I don’t understand which part I don’t understand about it. They already have the data ready and kind of expose it. Why not add the extra endpoint that returns it in an actually correct format. (Actually correct because I’m remembering CommBank using minutes instead of the month in the date and ING listing the amount as 1234.34.56 for international payments of 1234.56; I gave up trying ofx since everyone gets it wrong and the unique IDs are not unique).
This might be a hot take, but isn’t the addition of more configurability around LSP indication that, eventually, Helix is going to become more like any other editor, i.e. have lots of configuraiton knobs?
I don’t think so. Helix does have tons of configuration knobs. The difference always was that in Helix the out-of-the-box experience if very polished and featureful. Everything just works like one would expect. All the LSPs, languages, features. You add custom configs only … to customize. With *Vims the default experience is good enough only for the most basic editing of some system configuration file.
I like to customize. My Helix config is 60 lines. My Vim config in the past was … 1000? And Helix still wins with polish and features.
While I ultimately didn’t stick with Helix, I am forever grateful for stumbling upon the project as it gave the final push I needed to move away from VS Code! After trying Helix for a little while I came upon various YouTube videos about NeoVim and how to easily configure it with pre-packaged “distributions” like NvChad. I decided it makes more sense for me to learn Vim motions than Helix’s as a lot of my day-to-day does involve SSH-ing into machines, whether it is to edit something or not, and I felt I wanted to be better equipped to efficiently navigate within files there as well as my local machine.
I am in a similar position. I absolutely love Helix, and keep an eye on the project. Every time I’ve tried it out, it’s been a real dream to use and I think the mostly plug ‘n’ play nature of it is fantastic.
I’m a big fan of modal editing and minimalist terminal based software, but I begrudge endless tinkering with config files (I’m just not that much of a power user).
Helix ticks so many boxes for me. But ultimately, I reverted to using NVim with AstroNVIM as my “configuration framework” simply for the fact that the VI keys are hammered into me already, and using both causes me to do thing incorrectly on both.
Has NeoVim fixed their crashy LSP implementation? My major incentive for moving to Helix was that NeoVim would constantly crash when I was editing TS or Prisma files, and Helix wouldn’t. Once I moved over I really enjoyed the syntactical differences between Helix and Vim-like editors, especially the idea of “select first, action later”…because it maps a lot closer to how I think about text editing. But that’s not to say that I’ve lost my muscle memory for Vim, in fact I still do have NeoVim installed for vimdiff usage. I also have the Vim keybinding mode in ZSH so that I can use some Vim commands for line editing in the shell. This has kept me “in practice”, I suppose, but either way I’ve felt relatively comfortable moving between Vim keys and Helix keys, because they’re just not that different. Only thing that makes me “think twice” is the use of x and d…in Helix x means something very different :D
If you have control over your shell, I highly suggest using the ZSH Vim keybindings because it makes line editing on a phone or similar device without a regular keyboard a lot easier.
I’ve virtually never seen the neovim lsp client crash; the usual problem is that it didn’t gracefully handle cases where the server did (and that code doesn’t come from neovim and is shared between the IDEs). But yes, that’s been fixed. I have no problems with the typescript lang server but I don’t know anything about prisma and if that has a different LSP server, etc. Do note that there are multiple TS servers of varying quality; I’m using typescript-language-server on my neovim config: https://github.com/mqudsi/nvim-config/blob/00f48c84da8d46971eb989740279581269459ea5/init.vim#L486
My combination of Common Lisp, Ruby, and Perl programming combined with Emacs for nearly everything works just fine on a second-hand, years old, ThinkPad T470. I just upgraded from an X250 and appreciate the slightly larger screen and faster CPU.
I tend to buy refurbished ex-corporate laptops for myself and the kids, and we can get 1 x T470s and 3 or 4 x X250s for the price of an assembled base model Ryzen Framework.
From around 2015 - 2019 I was consulting - initially in Ruby and JavaScript, later moved into leadership & management.
At the time, the consultancy offered a bring your own device (BYOD) policy; you could either use one of their supplied Macbooks, or receive a stipend and you were on your own. I chose the latter, and ran an X220 with FreeBSD:
These days I use a client-issued MacBook Pro for their work, but still run FreeBSD on a T470s for other things. I don’t code for a living any more, but still maintain a Ruby codebase for a pre-consulting cofounded-startup-days client, and picked up a short term Common Lisp gig back in around ~2020 to try it out in a commercial setting. Perl I use purely for my own hacking when I want really broad platform support.
I have to ask: Why FreeBSD?
I am always curious to hear people’s motivations for using, what I consider to be, esoteric operating systems.
(Note that I absolutely appreciate the history and philosophy of FreeBSD. I’m just curious).
Several reasons; some accidental, some deliberate.
The initial accident was that I happened to have a colleague who ran FreeBSD on his laptop, and I was intrigued.
Once I’d gotten it running, and dug more into the design and culture of the OS, the more I liked it. Systemd was busy eating the GNU/Linux world at the time, and FreeBSD offered an alternative that was more in line with the distros I’d grown up with as a high-schooler. (Thanks to tireless support from my friends, I was a Slackware user back in, IIRC, 1995 … I still have a set of Walnut Creek CD-ROMs in my workshop somewhere).
I still run a lot of GNU/Linux, mind you. My kids’ laptops and my “gaming” laptop (X250s and W540) are all running Linux Mint. And my brace of Raspberry Pis are all running Raspbian. But my daily driver is FreeBSD, as is my “new” (second hand Xeon motherboard) server that I’m building to replace my ancient Ubuntu-running ProLiant. (Hallway cabinet photo here.).
But my medium-term plan is to switch everything over to FreeBSD, perhaps replacing the individual RPis with virtualised servers running under bhyve. My daily-driver laptop setup is almost completely scripted.
Re. the culture … that manifests in a bunch of subtle ways. The base / ports split, the philosophy of ports, the handbook and other (excellent) documentation. It’s just … more of a happy place for me. Hard to quantify.
Absolutely fascinating. That totally makes sense why you would have stuck to FreeBSD. As with most with things, we end up being a product of our environment.
Oops, too late to edit the original reply, so adding: except the systems for gaming. FreeBSD gaming on proprietary titles like Minecraft and Kerbal Space Program has never been a good (or even possible, in most cases) experience.
I have to say, since sharing the article here, I have had some time to think about it and really don’t agree with the author on many of their points. The article feels like a mixture of the author’s actual frustrations with some light-hearted japes, but it’s hard to differentiate them.
A lot of these ‘complexities’ the author complains about are features of the language that allow it to be so safe. What little time you spend making use of Option, Result or puzzling out lifetimes is potentially an order of magnitude less time than would be spent debugging errors down the line.
See that? No return statement. It’s like playing hide-and-seek with semicolons. The value of the last expression is returned automatically, whether you like it or not.
This statement suggests the author doesn’t fully understand the language features either, else he would have known that the value is only returned when omitting a semicolon at the end.
Having said that, I did find the article thought provoking. It has ultimately served to only increase my appreciation of Rust though.
I agree with you that most (but not all) of the complexity the author is complaining about is necessary/inherent complexity, but that doesn’t make it any more fun to learn. The article is more venting than making a technical point.
Much of https://www.oilshell.org/ is thinking as well. I just spent 3 weeks writing 5 posts, and that greatly clarified the language design. Now so many things are unblocked and can be implemented.
I don’t think there’s a shortcut – you may think you have an idea, but when you try to explain it to somebody, you will realize you don’t actually have the idea YET. Writing is thinking
Once you publish something, the convention is that whatever you wrote was what you thought before you wrote it. These were your ideas, and now you’ve expressed them. But you know this isn’t true. You know that putting your ideas into words changed them.
That is a lot of questions. I blog at softwaremaxims.com/.
Usually i have no problems finding what to write about. I do my usual stuff, and things that would benefit from being visible by others naturally occurs. I have dozens of these in a backlog. I rarely write them because I have limited time.
Sometimes something riles me up enough that i immediately set a few hours aside to write them fully and release them. These are usually the one that get a lot of readers.
What lead me into digging? Need. I have a really limited tolerance for painful or overly complicated ways to do things. So if I encounter one, i go dig if there are solutions.
I usually write about what i stumble upon. Usually about what I would like others to do.
Confidence: nothing give it to me. I don’t need it. If people disagree, that is not my problem. And they can simply not read it. That said i do not write hot takes without backing. I write about domains i know in depth and I have researched for years. I come with receipt and citations. I consider that if someone spend their precious time and brainpower to read me, they deserve something that is at least reasonably supported by scientific evidence or something close to it.
Why do i blog. Multiple reasons. Mostly so i can link it to others instead of constantly repeating myself. In other cases, to have something solid i can share with others in order to convey my ideas and get results. Sometimes, it is simply cathartic, like my supplier post. I needed to yell in anger.
How to start: make lists. Publish stuff. Noone will read it anyway so there is no risk.
Your penultimate point is one I hadn’t considered actually: Efficiency. I think that is a good reason to write a blog post, and more so if it’s a subject you have touched upon more than once across whatever platforms you subscribe to.
[PyPy] is slower to start than CPython, but if it runs for long enough, it will become faster and faster.
This is a common pair of myths. In VM Warmup Blows Hot and Cold, it is shown that JITs don’t necessarily start slow, and also don’t necessarily get monotonically faster over time. A more useful way to conceptualize PyPy is as giving an overall speedup to generic Python code; right now, that speedup is about 4.7x, as measured here.
[Psyco]’s now dead…
Psyco’s author went on to produce PyPy. Similarly, PyPy’s authors produced cffi. In both cases, the goal is to make it easier to write fast Python code and not write CPython-style extension modules.
Pypy is taking 0.44s to start on my machine, while python is taking 0.17, on a cold start. It’s has certainly be my experience that most small scripts are slower with pypy.
Things have nuances, but the article is not about getting into the very deep details of JIT, rather give a general understanding. I think it’s fair to have this approximate idea in mind to understand the concept.
The article mentions pypy’s inpiration from stackless, so certainly it can mention psyco’s author went to create it. I will add it.
Good write-up. I’m currently in the process of learning Elixir myself, as I’ve been wanting to get a functional language under my belt. Elixir stood out because of it’s expressiveness and combination of interpreter and compiler. It feels very much like a utility language that would work in many scenarios, and using it as scripting language isn’t one I’d considered.
I’m also very interested in the Phoenix framework.
Important bit about composability and extensibility is that composability is extensibility on the next level. kakoune is a plug-in for unix shell/terminal, and the terminal is your extensible world.
So, you don’t “prefer composability”, you rather select an extensible layer: is your editor extensible? Your terminal? Your browser? Your kernel?
And the problem I have here is that shell/terminal combo is an engineering abomination. It gets the thing done, but 99.9 explanation for that is pure inertia, rather than a particularly good design. It makes no sense for my software to emulate hardware which was obsolete before I was born for basic programming task. Especially if parts of the emulation live in the kernel.
Curiously, this is an area which sees very little experimentation from the community: everyone is writing their terminal editor, or terminal emulator, or a shell language, building on top of XX century APIs.
Of the top of my head, I can name only two projects which think along the lines of “what if there’s something better than a terminal” and try to build platforms to build composable software on top: Emacs (which isn’t exactly XXI century either) and Arcan (for which I am still waiting when I can nix run nixpkgs#cat9https://github.com/letoram/cat9).
That’s an interesting take.
Could you expand the part about emulating obsolete hardware? What hardware is being emulated by the shell?
My understanding of the shell/terminal, is that it offers a simple and efficient interface for interacting with a computer system. It provides a text-based command line interface that allows users to execute commands, automate tasks, and manipulate data in a straightforward manner. The idea of it emulating hardware is foreign to me, so I’d like to learn more about that.
To try and offer a different explanation / mental model:
Take the ASCII table (and thus Unicode as ASCII is a subset), https://www.asciitable.com/, the predominant encoding scheme for what is often, and dangerously, referred to as “just text”. It is actually about Information Interchange – the abbreviation expanded even says as much.
Notice how a fair portion of the first half of bitpatterns do not, in fact, encode symbols to carry the written word but some rather peculiar choices, like DEL – why would a text encoding scheme have processing instructions for stripping away its own data?!.
The common use for these entries is as a variable length instruction set intended for a machine to plow through. That is what is nowadays ‘emulated’, but this exact machine hasn’t existed for a grand long time, if ever, as it is a compromise. If you would follow the instructions strictly and try to run a modern program (“text file”), the results would likely be very unpleasant unless it happens to be “just text”.
The table itself is far from complete enough to reflect the capabilities of the hardware (terminals) that were out there yet people wanted their “just text” programs to do more. Some terminals only had a humble dot matrix printer as output, others glorious vector graphics CRTs. Do you see any instructions in the table for moving a turtle around a grid, drawing lines?
So through creative use of sequences of the less relevant bitpatterns from this table, you could invent such instructions. People did. A lot of them. A few got “Standardised” as ANSI, but far from all of them and it hasn’t exactly been kept in synch with modern times. Noadays you have (several) full display server protocols worth of them. There is clipboard, program execution, synchronisation, authentication, window management, scrolling, cursor control, history control and so on. Many of them use the ESC entry as a prefix, hence the colloquial ’ESCape sequences”. All this inside humble “just text”. Here is a rough state machine just for >decoding< some of the instructions out there https://vt100.net/emu/dec_ansi_parser .
That’s just emulating the instruction part. Then we another dimension of attributes to how these instructions should be interpreted to mimic variability in hardware not captures by the instructions; not all keyboards carried luxuries like a delete key, some had funky states like “num lock”, “print screen” and other bizarre. Not all hardware could communicate with the same speed. See the manpage for ‘stty’ for how inputs should affect the generated code.
Then we have data stores with bizarre access permissions (environment variables) that are also used to pick and chose between overlays from an external capability database (that is TERMCAP/TERMINFO, why things behave differently when you set TERM=xterm vs. TERM=linux vs. TERM=doesnotexist). To this day, the often used abstraction library (n)curses has an estimator that uses your capability description to infer an instruction set, combine that with the pending drawing operations and bin against the baudrate of your virtual device to reduce the chance of partial display updates.
Then we have Interrupts and Interrupt Service Routines(!) often overlooked just like stty- as you don’t see them in STDIN/STDOUT - i.e. signals and signal handlers. SIGWINCH (Window Changed) tells you that the display magically changed properties. I am just going to leave this here: http://www.rkoucha.fr/tech_corner/sigwinch.html
This is a baseline for terminal emulation. We still haven’t gotten to the interactive shells, multiplexers or network translation (i.e. ssh).
TLDR; This is absolutely not simple. It is absolutely not efficient.
All these features were originally provided by hardware terminals, which are big boxes with keyboards and screens, without computers:
hitting Enter – i.e. you enter a line on a hardware input device, AND it’s buffered there while you type. When you hit enter, it’s sent to the actual computer !
this also predates screens – teletypes were basically “interactive printers” as far as I understand. Like you got physical paper for everything you type !!!
control codes like Ctrl-C and Ctrl-Z - send messages to the computer from the hardware terminal
ANSI color (e.g. ls --color)
ANSI sequences to go back and forth (left arrow, right arrow)
clear the screen
Also my understanding is that curses was actually extracted from Vi. These kind of apps also ran on hardware:
On the web you send HTML from the server to the browser, which formats and displays the page. Different browsers will format things slightly differently. You can choose your font.
In mainframe computing, you send ANSI escape codes to the terminal device, which formats and displays the screen. Different terminals will format things slightly differently. You can choose your font.
There is a program/database called termcap that papers over the historical differences between all the terminals manufactured by different vendors.
A big difference is that hardware terminals weren’t programmable, and software terminals today (xterm) aren’t programmable either. That is, there’s no equivalent of JavaScript for the terminal.
Also web browsers started out as software, but something like a ChromeBook makes it a little more like hardware. The Chromebook is intended to be stateless and even non-programmable, with all your apps on the server. Just like a terminal is stateless.
Most of this is the background for what you call a terminal in combination with a shell. Which is also where the original color or symbol limits stem from.
“Command Symbols” are basically just an answer to the question how you get your data together with a display-command in one stream. When all the system can do is shove bytes around and display them as they come in. From the origin of having many non-smart interfaces to one mainframe / system and multiple users and a serial connection. For example the VT100 is such an old piece of original hardware.
Then you built a ton of defacto standards which all new programs need to implement, to be actually compatible with what real programs expect, and you end up with what we have. (See also bash/sh) For example problems detecting theterminal capabilities.
Terminal emulator, the kernel, the shell, and the program running in a shell all pretend that there’s a “VT100 with more features”. Thats why we have terminal emulator / shell split, and the emulator part at all.
Had I not been so stubborn about separating durden/pipeworld/console (wm models), core-arcan (desktop server), arcan-tui (the ncurses replacement), lash (shared shell building base) and cat9 (PoC shell) as interchangable layers but rather as an integrated vertical build you could have done so by now – with the usual nixpicking when it comes to input device and gpu access. I do not nix myself though there are a handful on the discord that do (including ehrm, fully statically linked, software rendering only on m1 mac..) and could perhaps be persuaded ..
On the context of the article and Kakounes integration ambition - one arcan dev/user uses zig/arcan-tui for that, see https://github.com/cipharius/kakoune-arcan . I do not think he has completed popups / windows as discrete objects yet though.
Yeah, after writing this comment, I realized that Arcan state in nixland is better these days. I was able to install&run arcan. I couldn’t get cat9 from nixpkgs work, but cloning the repo and following “not for the faint of heart” instructions worked. However, when launching cat9 I noticed that I can’t type ~ symbol, and the key repeat was super slow (like 1 key press/second), at which point I ran out of my tinkering time quant.
Key oscillation control is up to the WM side (durden needs to toggle it on / off contextually, console does not). It is set quite low so that a jumpy keyboard that performs their own repeat events and doesn’t tag them as such (several users had that problem) still can reach the runtime config.
There is a recent patch for the keymap script that other WMs that allows offline configuration of it.
arcan_db add_appl_kv console keyrepeat n ; (n=25Hz ticks between each rising/falling event edge)
arcan_db add_appl_kv console keydelay n (n=25Hz ticks before repetition is enabled)
Of the top of my head, I can name only two projects which think along the lines of “what if there’s something better than a terminal” and try to build platforms to build composable software on top
The main reason it’s not more prominent is because I’m not “driving it” … but if you want it to happen, you should help us with it :)
I will try to publish a demo – there is actually working code on both sides, and it was recently translated to C++ (by Melvin), so I’d say the last remaining barrier is removed. (In the past, you could say Oils isn’t “production quality” because it was in Python, which was indeed slow for our use case. But now it’s pure native code.)
As mentioned in the post, Oils is different than Arcan as it doesn’t require boiling an ocean, only a large lake :) That is, ls --color, Cargo, Rust compiler, and a bazillion other utilities still work in this model.
The shell itself should not require a terminal, but most tools use some kind of terminal support, so they still need terminals.
Another comparison is the Warp terminal. The difference there is that they inject shell code to your bash and zsh, and then parse the terminal codes in order to get “blocks”. I think this probably works fine for most cases, but I’m not sure it’s correct in the presence of concurrency.
In contrast, Oils does this with terminal FD passing over Unix domain sockets, in a simple and principled way.
So I’d say that Oils is the ONLY shell that can be easily divorced from the terminal. (It will also be the only POSIX-compatible shell that also has structured JSON-like data types; Melvin is also working on translating this now, i.e. divorcing it from Python.)
The slogan is that the shell UI should HAVE a terminal, but it shouldn’t BE a terminal.
So it would be cool if someone (who knows how to program terminals and Unix domain sockets, or wants to learn) can help revive and test it against the fast native implementation of the protocol
The idea is that the GUI creates a terminal and passes through the shell to say Cargo or rustc. The shell itself does not use the terminal – it can send all its error messages and prompts and completion on a different channel, over the socket.
This gets you the “blocks” feature of Warp terminal very easily and naturally, along with shell history / completion / error messages OUTSIDE the terminal, in a GUI.
So in this way the shell is decoupled from the terminal, but the child processes can still use the terminal. Again, unlike in Arcan FE.
The same binary (afsrv_terminal) that sets up lash can be set to:
be only a terminal emulator
be only a lash runtime
wrap single commands (e.g. a legacy shell), presents a raw or vt-interpreted rendered view as well as its respective stdio with arbitrary sets of inherited other descriptors for a shell to output through.
the inherited descriptor set from 3 can includes the socket channel that arcan shmif unfolds over, bchunkhint events there comes with the same dynamic descriptor passing mechanism as in your linked.
spawn new instances of itself in any of the above configurations that hands over to the outer WM or embeds into itself.
It does look pretty awesome … The “jobs” idea looks like the Warp “blocks”, which a FANOS-based GUI would also support naturally
Personally I’m spending most of my time on the new shell language, now called YSH (And that naturally goes into self-hosted / multi-cloud distributed git-like data and polyglot computation)
And I would like some kind of GUI for that, but don’t have time to build it myself (or the expertise, really)
But Oils/YSH is now pure C++ and requires only a shell to compile, not even Make
So it should be pretty trivial to drop into any environment
Partly a backpressure problem - too many features to present already that there was no real point in going even further (about 1/4th of readers drop for each ‘section’ with a tail of ‘reads it all’ judging by the video downloads).
The non-uniform grid bits are also not developed enough to really show off, while border drawing now works as cell attributes rasterised outside the grid - there are more fancy text rendering nuances that were present when the shell process rasterised the text that had to be temporarily disabled when moving to server-side text rendering.
The interchange format (TPACK) allows for varying line properties (shaped, RTL) and other annotations. WM deferred popups, scrollbars etc. also works but better shown off in an article explicitly about the APIs rather than an example shell.
Of the top of my head, I can name only two projects which think along the lines of “what if there’s something better than a terminal” and try to build platforms to build composable software on top:
That address bar search thing really took me a moment to notice on the video, but it’s really cool! Doesn’t work for me at all though. And I hope there is a key to toggle it between URL and search term…
It is now possible to override a JavaScript file in the debugger
Here’s a screenshot showing the feature. I have searched ‘test one two three’ using the URL bar on DuckDuckGo. You’ll notice the search terms are there in the URL bar.
A brief (but not totally inclusive) summary of changes:
Search terms will be displayed in the URL bar when using it for web search.
Enhanced Picture-in-Picture.
Rewind / Fast-forward controls.
Display video duration.
Private windows now block third-party cookies and storage of content trackers.
Passwords automatically generated by Firefox now include special characters.
improved speed, responsiveness, and stability when used with:
Screen readers, as well as certain other accessibility software;
East Asian input methods;
Enterprise single sign-on software; and
Other applications which use accessibility frameworks to access information.
Favicons will now be imported when importing bookmarks from Safari and Chrome-based browsers.
Support for AV1 Image Format files containing animations (AVIS).
Enhanced security for Windows GPU sandbox.
Support for files being drag-and-dropped directly from Microsoft Outlook.
Users on macOS can now access the Services sub-menu directly from Firefox context menus.
On Windows, the elastic overscroll effect has been enabled by default.
Firefox is now available in the Tajik (tg) language.
mozRTCPeerConnection, mozRTCIceCandidate, and mozRTCSessionDescription WebRTC interfaces have been removed.
New CSS functionality, including improved support for the color (level 4) specification (such as the lab(), lch(), oklab(), oklch(), and color() functions) and the scripting media query.
Support for a number of WebRTC features for improved interoperability: RTCMediaSourceStats, RTCPeerConnectionState, RTCPeerConnectionStats (“peer-connection” RTCStatsType), RTCRtpSender.setStreams(), and RTCSctpTransport.
I’m so glad you’ve posted this. I stumbled across it earlier and wanted to share it, but was unable to as I’m a new user.
I think this is a really novel way to introduce newer and intermediate programmers to the concept of building an application by consuming APIs. I’ve spent a bit of time poking around the project and it looks to be quite expansive and robust, and there is an active community surrounding it. I think using this as the basis of a programming course in a class-based setting would be very interesting.
Doesn’t look simple to me but I get the point. I guess this will only be necessary when moving a future to another task so pretty rare in my case.
A couple of years ago I was rather excited about Rust but recently every time I read about async Rust I am just left puzzled by the amount of complexity. As a language user maybe I should just stop worrying…
Supporting asynchronicity under Rust’s standards of “memory safety without garbage collection, concurrency without data races, stability without stagnation” is acknowledged as complex, but, if one doesn’t need asynchronicity, one can choose libraries such as ureq that reject async and its complexity.
Love this, absolutely something that could fit in my workflow.
One issue I’ve found on an Android Phone (One Plus 7 Pro), is that in notification settings there is nothing for default notifications. I only have Max, High, Low and Minimum. This result in no proper notification (i.e no pop up or vibration) for default notifications. I’m not sure if this is an issue with my phone or the app. I downloaded it from F-Droid.
Edit:
Using the Google Play version has solved the issue for me. I know there can be some funk with notifications and apps installed outside of Google Play, so not totally unexpected for the F-Droid version to be a little funky. I’ll stick to this version now.
That’s not something I have heard before, and there is no difference between F-Droid and Play in this regard. Odd. Feel free to file an issue if it’s a big issue.
I was under the impression that for proper push notifications, like your service is offering, that there was a dependency on Google Play Services? I’ve not developed for Android, so apologies if that is not correct. I seem to remember there being issues with Proton Mail on F-Droid that seemed to cite that reason before they fixed their notification issues.
Either way, since using the Google Play version it has all worked flawlessly. I will try the F-Droid version again at some point and just make sure it wasn’t something I’d done to stop it working properly.
I’m not certain I really agree with the article, but I do think it could prompt an interesting discussion here. Personally, I do not care about the “status” of software developers. I care about doing something I enjoy and earning a decent living. Even so, I actually don’t think our status has been eroded. On the contrary, I feel like it’s never been a more respected profession. The field is extremely accessible though, so you do end up with a very wide spectrum of developers and I think there is more of a divide between seasoned developers who learned in more traditional ways and career changers emerging in the field through bootcamps.
For those without and not wanting an account, here’s a version with that requirement removed.
Personally, I do not care about the “status” of software developers
I think ‘status’ is the wrong term, but respect for expertise is important. You could find a lot of opinions about the status of plumbing as an occupation but I don’t think you’d find much disagreement that the person planning and installing the system that removes sewage from your house has very important and valuable skills and the world would (locally, at least) be a much worse place if someone without those expertise tried to do the job.
The problem with a lot of the ‘eveyone can code’ movement is that it doesn’t differentiate between ‘everyone should be able to remove the u-bend under their sink and clean / replace it’ and ‘everyone should be able to plan a water and sewage system for a neighbourhood’ levels of skills. It’s pretty obvious to most people that the former is a useful life skill (you might still want a plumber to do it for you if the opportunity cost of you doing it is too high) whereas the latter is a specialised task that you shouldn’t expect someone to do just because they are able to do just because they fixed a leaky tap in the office kitchen that one time.
This lack of understanding and respect for expertise leads to a load of folks moving into management with the ability to do small programming tasks but without any understanding of software engineering.
Nice article. I’ve read many like this; typically featured on one of the major VPS providers; but I like that yours is rather concise whilst still explaining some of the less obvious aspects.
I also didn’t know about network.target not being a good way of determining that network interfaces are ready, so very glad I also came to the comments. Certainly a point worth updating in your article.
As a software developer, I adore having access to GPT3/4. Lately, I’ve been jumping around from language to language, and it’s a god-send when I know how to do something in one language and want to know how to do it in another. It’s not terribly difficult or time-consuming to do with a search engine, but with GPT I can ask follow-up questions to get an idea of idiomatic ways of doing things or about libraries that might exist in the new language to aid me.
How do you know it isn’t confabulating answers about idiomatic ways of doing things? Unlike the existence of libraries, it doesn’t seem like an easy thing to verify.
After using it for long enough you end up developing a pretty accurate intuition as to when to trust it and when to double-check.
Writing idiomatic code is something it’s actually really good at, because it’s seen a LOT of code in different languages and “idiomatic” basically means “in the most average way possible”. LLMs are really good at coming up with the most average way possible to express something!
That part just comes with experience I think. I don’t just blindly trust what it spits out. Often I’ll run similar prompts through it to check for consistency, and if I’m totally unfamiliar with a language I spend a bit of time on a forum/subreddit/channel for it initially just to get a feel.
I’d day it’s close to the mark more often that not though. To clarify, I’m not having it write code for me, but just having it give examples of how to get something done. Creating tests in an unfamiliar testing framework is something it really excels at in my experience.
I had a similar experience asking an LLM to generate some JavaScript code for me. The code was completely wrong, but it used all of the right APIs (I know enough of JavaScript-the-language to get things done, I haven’t looked at modern web APIs for ages) and so I could jump straight to the docs and write something correct in a few minutes.
Yes, this totally mirrors my experience. It getting the code wrong doesn’t matter, as we shouldn’t (yet) be relying on it for that. It’s a great tool for experienced developers to accelerate our workflows.
Great article. Really enjoyed reading this, and it’s an issue I’ve run up against countless times by now. Glad to see that you covered both exponential backoff and jitter. Something else we use is token buckets to control flow and burstiness.
I’m going to check out more of your blog, but I noticed a lack of RSS. Is this something you would consider adding?
Thank you!
https://samwho.dev/rss.xml I really need to add an obvious link to this somewhere. Thanks for reminding me. 🙈
Honestly, I looked everywhere for this. Really disappointed in myself for not simply trying /rss.xml… Cheers!
I use hledger to keep track of my spending, and the fact that it uses a special text format for transactions isn’t a huge concern for me. What is a big concern is that every financial institution I use makes it difficult to programmatically obtain my transaction records, and also to figure out what records I failed to grab properly if my calculated spending doesn’t match up with what the website says. This aspect is by far the biggest problem I have managing my finances and it’s not one that any program I run myself can help with, because the problem is entirely contained within the proprietary software running on the websites of financial institutions subject to their own business and regulatory priorities.
If there was a bank that made it possible for me to make an authenticated REST API request against a website to get my personal transaction data I would seriously consider switching to that bank. But I’m not aware of any such banks and there may be reasons why no bank offers this service.
In the UK, we have Monzo and Starling that both have an authenticated REST API for accessing transactios (and more). The latter is particularly good, but both work great.
That part is so absurd, I don’t understand which part I don’t understand about it. They already have the data ready and kind of expose it. Why not add the extra endpoint that returns it in an actually correct format. (Actually correct because I’m remembering CommBank using minutes instead of the month in the date and ING listing the amount as 1234.34.56 for international payments of 1234.56; I gave up trying ofx since everyone gets it wrong and the unique IDs are not unique).
This might be a hot take, but isn’t the addition of more configurability around LSP indication that, eventually, Helix is going to become more like any other editor, i.e. have lots of configuraiton knobs?
I don’t think so. Helix does have tons of configuration knobs. The difference always was that in Helix the out-of-the-box experience if very polished and featureful. Everything just works like one would expect. All the LSPs, languages, features. You add custom configs only … to customize. With *Vims the default experience is good enough only for the most basic editing of some system configuration file.
I like to customize. My Helix config is 60 lines. My Vim config in the past was … 1000? And Helix still wins with polish and features.
Very similar experience. It was odd to switch but now I loathe to revert to vim/vi. d and x get me every time.
Maybe. But there’s something to be liked about having some amount of additional configuration but coming mostly pre-baked with sensible defaults.
Maybe a bit of a warmish take, but also an important question worth thinking about.
While I ultimately didn’t stick with Helix, I am forever grateful for stumbling upon the project as it gave the final push I needed to move away from VS Code! After trying Helix for a little while I came upon various YouTube videos about NeoVim and how to easily configure it with pre-packaged “distributions” like NvChad. I decided it makes more sense for me to learn Vim motions than Helix’s as a lot of my day-to-day does involve SSH-ing into machines, whether it is to edit something or not, and I felt I wanted to be better equipped to efficiently navigate within files there as well as my local machine.
I am in a similar position. I absolutely love Helix, and keep an eye on the project. Every time I’ve tried it out, it’s been a real dream to use and I think the mostly plug ‘n’ play nature of it is fantastic.
I’m a big fan of modal editing and minimalist terminal based software, but I begrudge endless tinkering with config files (I’m just not that much of a power user).
Helix ticks so many boxes for me. But ultimately, I reverted to using NVim with AstroNVIM as my “configuration framework” simply for the fact that the VI keys are hammered into me already, and using both causes me to do thing incorrectly on both.
Took me 3 weeks to get used to Helix. Now I can’t go back to vim since everything I do in vim is wrong now.
Oh well. :-)
Has NeoVim fixed their crashy LSP implementation? My major incentive for moving to Helix was that NeoVim would constantly crash when I was editing TS or Prisma files, and Helix wouldn’t. Once I moved over I really enjoyed the syntactical differences between Helix and Vim-like editors, especially the idea of “select first, action later”…because it maps a lot closer to how I think about text editing. But that’s not to say that I’ve lost my muscle memory for Vim, in fact I still do have NeoVim installed for
vimdiffusage. I also have the Vim keybinding mode in ZSH so that I can use some Vim commands for line editing in the shell. This has kept me “in practice”, I suppose, but either way I’ve felt relatively comfortable moving between Vim keys and Helix keys, because they’re just not that different. Only thing that makes me “think twice” is the use ofxandd…in Helixxmeans something very different :DIf you have control over your shell, I highly suggest using the ZSH Vim keybindings because it makes line editing on a phone or similar device without a regular keyboard a lot easier.
I’ve virtually never seen the neovim lsp client crash; the usual problem is that it didn’t gracefully handle cases where the server did (and that code doesn’t come from neovim and is shared between the IDEs). But yes, that’s been fixed. I have no problems with the typescript lang server but I don’t know anything about prisma and if that has a different LSP server, etc. Do note that there are multiple TS servers of varying quality; I’m using typescript-language-server on my neovim config: https://github.com/mqudsi/nvim-config/blob/00f48c84da8d46971eb989740279581269459ea5/init.vim#L486
One thought … what are you using it for?
My combination of Common Lisp, Ruby, and Perl programming combined with Emacs for nearly everything works just fine on a second-hand, years old, ThinkPad T470. I just upgraded from an X250 and appreciate the slightly larger screen and faster CPU.
I tend to buy refurbished ex-corporate laptops for myself and the kids, and we can get 1 x T470s and 3 or 4 x X250s for the price of an assembled base model Ryzen Framework.
Interesting combination there. Is that for professional work? If so, what’s your job if I may ask?
It used to be :)
From around 2015 - 2019 I was consulting - initially in Ruby and JavaScript, later moved into leadership & management.
At the time, the consultancy offered a bring your own device (BYOD) policy; you could either use one of their supplied Macbooks, or receive a stipend and you were on your own. I chose the latter, and ran an X220 with FreeBSD:
My X220 setup
These days I use a client-issued MacBook Pro for their work, but still run FreeBSD on a T470s for other things. I don’t code for a living any more, but still maintain a Ruby codebase for a pre-consulting cofounded-startup-days client, and picked up a short term Common Lisp gig back in around ~2020 to try it out in a commercial setting. Perl I use purely for my own hacking when I want really broad platform support.
I have to ask: Why FreeBSD? I am always curious to hear people’s motivations for using, what I consider to be, esoteric operating systems. (Note that I absolutely appreciate the history and philosophy of FreeBSD. I’m just curious).
Several reasons; some accidental, some deliberate.
The initial accident was that I happened to have a colleague who ran FreeBSD on his laptop, and I was intrigued.
Once I’d gotten it running, and dug more into the design and culture of the OS, the more I liked it. Systemd was busy eating the GNU/Linux world at the time, and FreeBSD offered an alternative that was more in line with the distros I’d grown up with as a high-schooler. (Thanks to tireless support from my friends, I was a Slackware user back in, IIRC, 1995 … I still have a set of Walnut Creek CD-ROMs in my workshop somewhere).
I still run a lot of GNU/Linux, mind you. My kids’ laptops and my “gaming” laptop (X250s and W540) are all running Linux Mint. And my brace of Raspberry Pis are all running Raspbian. But my daily driver is FreeBSD, as is my “new” (second hand Xeon motherboard) server that I’m building to replace my ancient Ubuntu-running ProLiant. (Hallway cabinet photo here.).
But my medium-term plan is to switch everything over to FreeBSD, perhaps replacing the individual RPis with virtualised servers running under bhyve. My daily-driver laptop setup is almost completely scripted.
Re. the culture … that manifests in a bunch of subtle ways. The base / ports split, the philosophy of ports, the handbook and other (excellent) documentation. It’s just … more of a happy place for me. Hard to quantify.
Absolutely fascinating. That totally makes sense why you would have stuck to FreeBSD. As with most with things, we end up being a product of our environment.
Oops, too late to edit the original reply, so adding: except the systems for gaming. FreeBSD gaming on proprietary titles like Minecraft and Kerbal Space Program has never been a good (or even possible, in most cases) experience.
I have to say, since sharing the article here, I have had some time to think about it and really don’t agree with the author on many of their points. The article feels like a mixture of the author’s actual frustrations with some light-hearted japes, but it’s hard to differentiate them.
A lot of these ‘complexities’ the author complains about are features of the language that allow it to be so safe. What little time you spend making use of Option, Result or puzzling out lifetimes is potentially an order of magnitude less time than would be spent debugging errors down the line.
This statement suggests the author doesn’t fully understand the language features either, else he would have known that the value is only returned when omitting a semicolon at the end.
Having said that, I did find the article thought provoking. It has ultimately served to only increase my appreciation of Rust though.
I agree with you that most (but not all) of the complexity the author is complaining about is necessary/inherent complexity, but that doesn’t make it any more fun to learn. The article is more venting than making a technical point.
I think the page on Aaron Swartz’s website about why he wrote a blog is worth a read.
It amazes me how often his stuff comes up and once again it’s something I’ve not read but absolutely found interesting. Thanks for sharing that.
Yes absolutely, writing is thinking … I also like this post - https://gregorygundersen.com/blog/2020/01/12/why-research-blog/
Much of https://www.oilshell.org/ is thinking as well. I just spent 3 weeks writing 5 posts, and that greatly clarified the language design. Now so many things are unblocked and can be implemented.
I don’t think there’s a shortcut – you may think you have an idea, but when you try to explain it to somebody, you will realize you don’t actually have the idea YET. Writing is thinking
Similar: http://www.paulgraham.com/words.html
That is a lot of questions. I blog at softwaremaxims.com/.
Usually i have no problems finding what to write about. I do my usual stuff, and things that would benefit from being visible by others naturally occurs. I have dozens of these in a backlog. I rarely write them because I have limited time.
Sometimes something riles me up enough that i immediately set a few hours aside to write them fully and release them. These are usually the one that get a lot of readers.
What lead me into digging? Need. I have a really limited tolerance for painful or overly complicated ways to do things. So if I encounter one, i go dig if there are solutions.
I usually write about what i stumble upon. Usually about what I would like others to do.
Confidence: nothing give it to me. I don’t need it. If people disagree, that is not my problem. And they can simply not read it. That said i do not write hot takes without backing. I write about domains i know in depth and I have researched for years. I come with receipt and citations. I consider that if someone spend their precious time and brainpower to read me, they deserve something that is at least reasonably supported by scientific evidence or something close to it.
Why do i blog. Multiple reasons. Mostly so i can link it to others instead of constantly repeating myself. In other cases, to have something solid i can share with others in order to convey my ideas and get results. Sometimes, it is simply cathartic, like my supplier post. I needed to yell in anger.
How to start: make lists. Publish stuff. Noone will read it anyway so there is no risk.
Your penultimate point is one I hadn’t considered actually: Efficiency. I think that is a good reason to write a blog post, and more so if it’s a subject you have touched upon more than once across whatever platforms you subscribe to.
This is a common pair of myths. In VM Warmup Blows Hot and Cold, it is shown that JITs don’t necessarily start slow, and also don’t necessarily get monotonically faster over time. A more useful way to conceptualize PyPy is as giving an overall speedup to generic Python code; right now, that speedup is about 4.7x, as measured here.
Psyco’s author went on to produce PyPy. Similarly, PyPy’s authors produced cffi. In both cases, the goal is to make it easier to write fast Python code and not write CPython-style extension modules.
Pypy is taking 0.44s to start on my machine, while python is taking 0.17, on a cold start. It’s has certainly be my experience that most small scripts are slower with pypy.
Things have nuances, but the article is not about getting into the very deep details of JIT, rather give a general understanding. I think it’s fair to have this approximate idea in mind to understand the concept.
The article mentions pypy’s inpiration from stackless, so certainly it can mention psyco’s author went to create it. I will add it.
I had no idea the author of Psyco was also behind PyPy. That’s certainly something the article failed to mention; thanks.
Good write-up. I’m currently in the process of learning Elixir myself, as I’ve been wanting to get a functional language under my belt. Elixir stood out because of it’s expressiveness and combination of interpreter and compiler. It feels very much like a utility language that would work in many scenarios, and using it as scripting language isn’t one I’d considered.
I’m also very interested in the Phoenix framework.
Important bit about composability and extensibility is that composability is extensibility on the next level. kakoune is a plug-in for unix shell/terminal, and the terminal is your extensible world.
So, you don’t “prefer composability”, you rather select an extensible layer: is your editor extensible? Your terminal? Your browser? Your kernel?
And the problem I have here is that shell/terminal combo is an engineering abomination. It gets the thing done, but 99.9 explanation for that is pure inertia, rather than a particularly good design. It makes no sense for my software to emulate hardware which was obsolete before I was born for basic programming task. Especially if parts of the emulation live in the kernel.
Curiously, this is an area which sees very little experimentation from the community: everyone is writing their terminal editor, or terminal emulator, or a shell language, building on top of XX century APIs.
Of the top of my head, I can name only two projects which think along the lines of “what if there’s something better than a terminal” and try to build platforms to build composable software on top: Emacs (which isn’t exactly XXI century either) and Arcan (for which I am still waiting when I can
nix run nixpkgs#cat9https://github.com/letoram/cat9).That’s an interesting take. Could you expand the part about emulating obsolete hardware? What hardware is being emulated by the shell? My understanding of the shell/terminal, is that it offers a simple and efficient interface for interacting with a computer system. It provides a text-based command line interface that allows users to execute commands, automate tasks, and manipulate data in a straightforward manner. The idea of it emulating hardware is foreign to me, so I’d like to learn more about that.
To try and offer a different explanation / mental model:
Take the ASCII table (and thus Unicode as ASCII is a subset), https://www.asciitable.com/, the predominant encoding scheme for what is often, and dangerously, referred to as “just text”. It is actually about Information Interchange – the abbreviation expanded even says as much.
Notice how a fair portion of the first half of bitpatterns do not, in fact, encode symbols to carry the written word but some rather peculiar choices, like DEL – why would a text encoding scheme have processing instructions for stripping away its own data?!.
The common use for these entries is as a variable length instruction set intended for a machine to plow through. That is what is nowadays ‘emulated’, but this exact machine hasn’t existed for a grand long time, if ever, as it is a compromise. If you would follow the instructions strictly and try to run a modern program (“text file”), the results would likely be very unpleasant unless it happens to be “just text”.
The table itself is far from complete enough to reflect the capabilities of the hardware (terminals) that were out there yet people wanted their “just text” programs to do more. Some terminals only had a humble dot matrix printer as output, others glorious vector graphics CRTs. Do you see any instructions in the table for moving a turtle around a grid, drawing lines?
So through creative use of sequences of the less relevant bitpatterns from this table, you could invent such instructions. People did. A lot of them. A few got “Standardised” as ANSI, but far from all of them and it hasn’t exactly been kept in synch with modern times. Noadays you have (several) full display server protocols worth of them. There is clipboard, program execution, synchronisation, authentication, window management, scrolling, cursor control, history control and so on. Many of them use the ESC entry as a prefix, hence the colloquial ’ESCape sequences”. All this inside humble “just text”. Here is a rough state machine just for >decoding< some of the instructions out there https://vt100.net/emu/dec_ansi_parser .
That’s just emulating the instruction part. Then we another dimension of attributes to how these instructions should be interpreted to mimic variability in hardware not captures by the instructions; not all keyboards carried luxuries like a delete key, some had funky states like “num lock”, “print screen” and other bizarre. Not all hardware could communicate with the same speed. See the manpage for ‘stty’ for how inputs should affect the generated code.
Then we have data stores with bizarre access permissions (environment variables) that are also used to pick and chose between overlays from an external capability database (that is TERMCAP/TERMINFO, why things behave differently when you set TERM=xterm vs. TERM=linux vs. TERM=doesnotexist). To this day, the often used abstraction library (n)curses has an estimator that uses your capability description to infer an instruction set, combine that with the pending drawing operations and bin against the baudrate of your virtual device to reduce the chance of partial display updates.
Then we have Interrupts and Interrupt Service Routines(!) often overlooked just like stty- as you don’t see them in STDIN/STDOUT - i.e. signals and signal handlers. SIGWINCH (Window Changed) tells you that the display magically changed properties. I am just going to leave this here: http://www.rkoucha.fr/tech_corner/sigwinch.html
This is a baseline for terminal emulation. We still haven’t gotten to the interactive shells, multiplexers or network translation (i.e. ssh).
TLDR; This is absolutely not simple. It is absolutely not efficient.
Here’s a good article about it - https://www.warp.dev/blog/what-happens-when-you-open-a-terminal-and-enter-ls
All these features were originally provided by hardware terminals, which are big boxes with keyboards and screens, without computers:
ls --color)Also my understanding is that curses was actually extracted from Vi. These kind of apps also ran on hardware:
https://en.wikipedia.org/wiki/Curses_(programming_library)
You can think of it as isomorphic to the web:
There is a program/database called
termcapthat papers over the historical differences between all the terminals manufactured by different vendors.A big difference is that hardware terminals weren’t programmable, and software terminals today (xterm) aren’t programmable either. That is, there’s no equivalent of JavaScript for the terminal.
Also web browsers started out as software, but something like a ChromeBook makes it a little more like hardware. The Chromebook is intended to be stateless and even non-programmable, with all your apps on the server. Just like a terminal is stateless.
I’m not as well versed in, but I can give some input:
See for example the wikipedia entry for terminal emulator or computer terminals
Most of this is the background for what you call a terminal in combination with a shell. Which is also where the original color or symbol limits stem from. “Command Symbols” are basically just an answer to the question how you get your data together with a display-command in one stream. When all the system can do is shove bytes around and display them as they come in. From the origin of having many non-smart interfaces to one mainframe / system and multiple users and a serial connection. For example the VT100 is such an old piece of original hardware.
Then you built a ton of defacto standards which all new programs need to implement, to be actually compatible with what real programs expect, and you end up with what we have. (See also bash/sh) For example problems detecting the terminal capabilities.
This bit of hardware in particular:
https://en.m.wikipedia.org/wiki/VT100
Terminal emulator, the kernel, the shell, and the program running in a shell all pretend that there’s a “VT100 with more features”. Thats why we have terminal emulator / shell split, and the emulator part at all.
https://poor.dev/blog/terminal-anatomy/ explains how all that works specifically
Helix supports the enhanced keyboard protocol, so at least there is some work happening to reform the terminal, if not replace it: https://sw.kovidgoyal.net/kitty/keyboard-protocol/
Had I not been so stubborn about separating durden/pipeworld/console (wm models), core-arcan (desktop server), arcan-tui (the ncurses replacement), lash (shared shell building base) and cat9 (PoC shell) as interchangable layers but rather as an integrated vertical build you could have done so by now – with the usual nixpicking when it comes to input device and gpu access. I do not nix myself though there are a handful on the discord that do (including ehrm, fully statically linked, software rendering only on m1 mac..) and could perhaps be persuaded ..
On the context of the article and Kakounes integration ambition - one arcan dev/user uses zig/arcan-tui for that, see https://github.com/cipharius/kakoune-arcan . I do not think he has completed popups / windows as discrete objects yet though.
Yeah, after writing this comment, I realized that Arcan state in nixland is better these days. I was able to install&run arcan. I couldn’t get cat9 from nixpkgs work, but cloning the repo and following “not for the faint of heart” instructions worked. However, when launching cat9 I noticed that I can’t type
~symbol, and the key repeat was super slow (like 1 key press/second), at which point I ran out of my tinkering time quant.Key oscillation control is up to the WM side (durden needs to toggle it on / off contextually, console does not). It is set quite low so that a jumpy keyboard that performs their own repeat events and doesn’t tag them as such (several users had that problem) still can reach the runtime config.
There is a recent patch for the keymap script that other WMs that allows offline configuration of it.
I always suspected that nixos-nix and *nix-nix was confusion waiting to happen, but I’ve never seen it in practice until now.
Oils too, mentioned in the last post:
https://www.oilshell.org/blog/2023/05/release-0.15.0.html#guis-and-the-headless-shell
Linking back to the genesis in 2021:
https://www.oilshell.org/blog/2021/06/hotos-shell-panel.html#oils-headless-mode-should-be-useful-for-ui-research
The main reason it’s not more prominent is because I’m not “driving it” … but if you want it to happen, you should help us with it :)
I will try to publish a demo – there is actually working code on both sides, and it was recently translated to C++ (by Melvin), so I’d say the last remaining barrier is removed. (In the past, you could say Oils isn’t “production quality” because it was in Python, which was indeed slow for our use case. But now it’s pure native code.)
As mentioned in the post, Oils is different than Arcan as it doesn’t require boiling an ocean, only a large lake :) That is,
ls --color, Cargo, Rust compiler, and a bazillion other utilities still work in this model.The shell itself should not require a terminal, but most tools use some kind of terminal support, so they still need terminals.
Another comparison is the Warp terminal. The difference there is that they inject shell code to your bash and zsh, and then parse the terminal codes in order to get “blocks”. I think this probably works fine for most cases, but I’m not sure it’s correct in the presence of concurrency.
In contrast, Oils does this with terminal FD passing over Unix domain sockets, in a simple and principled way.
So I’d say that Oils is the ONLY shell that can be easily divorced from the terminal. (It will also be the only POSIX-compatible shell that also has structured JSON-like data types; Melvin is also working on translating this now, i.e. divorcing it from Python.)
The slogan is that the shell UI should HAVE a terminal, but it shouldn’t BE a terminal.
OK the original demo screenshots are probably worth a thousands words - https://github.com/subhav/web_shell
So it would be cool if someone (who knows how to program terminals and Unix domain sockets, or wants to learn) can help revive and test it against the fast native implementation of the protocol
The Go side is here - https://github.com/subhav/web_shell/blob/master/fanos.go
Oils C++ side is here - https://github.com/oilshell/oil/blob/master/cpp/fanos_shared.c
Python too: https://github.com/oilshell/oil/blob/master/client/py_fanos.py
The idea is that the GUI creates a terminal and passes through the shell to say Cargo or rustc. The shell itself does not use the terminal – it can send all its error messages and prompts and completion on a different channel, over the socket.
This gets you the “blocks” feature of Warp terminal very easily and naturally, along with shell history / completion / error messages OUTSIDE the terminal, in a GUI.
So in this way the shell is decoupled from the terminal, but the child processes can still use the terminal. Again, unlike in Arcan FE.
Arcan covers all permutations of the concept.
The same binary (afsrv_terminal) that sets up lash can be set to:
1,3,5 had been demoed before in the dream/dawn/day and pipeworld (https://arcan-fe.com/2021/04/12/introducing-pipeworld/) presentation and was thus left out of the lash#cat9 one.
OK, I read over the most recent blog post again
https://arcan-fe.com/2022/10/15/whipping-up-a-new-shell-lashcat9/
It does look pretty awesome … The “jobs” idea looks like the Warp “blocks”, which a FANOS-based GUI would also support naturally
Personally I’m spending most of my time on the new shell language, now called YSH (And that naturally goes into self-hosted / multi-cloud distributed git-like data and polyglot computation)
And I would like some kind of GUI for that, but don’t have time to build it myself (or the expertise, really)
But Oils/YSH is now pure C++ and requires only a shell to compile, not even Make
So it should be pretty trivial to drop into any environment
Another thought I had is that the demos still look grid-based and terminal-based to me, i.e. more like tmux than a Window manager
It’s probably easy for people to gloss over the difference when presented like that
I’m thinking more like something with GUI history like browser history, and then a GUI prompt like the address bar, GUI auto-complete, etc.
I feel like that would open people up the idea a bit more
Partly a backpressure problem - too many features to present already that there was no real point in going even further (about 1/4th of readers drop for each ‘section’ with a tail of ‘reads it all’ judging by the video downloads).
The non-uniform grid bits are also not developed enough to really show off, while border drawing now works as cell attributes rasterised outside the grid - there are more fancy text rendering nuances that were present when the shell process rasterised the text that had to be temporarily disabled when moving to server-side text rendering.
The interchange format (TPACK) allows for varying line properties (shaped, RTL) and other annotations. WM deferred popups, scrollbars etc. also works but better shown off in an article explicitly about the APIs rather than an example shell.
What about ACME from plan9 (originally)?
http://acme.cat-v.org/
Personally RSI rules out mouse (and heavy trackpad) use - but it should fit the description of doing something new vis-a-vis TTYs?
Maybe worth mentioning Smalltalk (in general) - and Cuis/Pharo as open source IDEs/editing environments too. And possibly (but maybe not) Racket.
Very interesting. As someone who only occasionally writes Go code, I had no idea about this issue.
That address bar search thing really took me a moment to notice on the video, but it’s really cool! Doesn’t work for me at all though. And I hope there is a key to toggle it between URL and search term…
Very cool!
Care to explain? I still don’t get what it’s supposed to show.
Here’s a screenshot showing the feature. I have searched ‘test one two three’ using the URL bar on DuckDuckGo. You’ll notice the search terms are there in the URL bar.
Only works for the default search engine, meh. Most www search engines already have sticky headers so this doesn’t add much.
It works for me on Nightly.
If you’re running 113 and it’s not showing, got to
about:configand enablebrowser.urlbar.showSearchTerms.enabledThat was enabled, but I’ve had to enable
browser.urlbar.showSearchTerms.featureGateactually. (WTF is it gated on, region/language or something?!)And yea, double pressing Esc gives me the URL. Very cool.
A brief (but not totally inclusive) summary of changes:
I always thought the elastic overscroll was part of cocoa. Any timeline on when it will be enabled on GTK?
I think you can enable it manually with
apz.overscroll.enabledinabout:config, but YMMVI’m so glad you’ve posted this. I stumbled across it earlier and wanted to share it, but was unable to as I’m a new user.
I think this is a really novel way to introduce newer and intermediate programmers to the concept of building an application by consuming APIs. I’ve spent a bit of time poking around the project and it looks to be quite expansive and robust, and there is an active community surrounding it. I think using this as the basis of a programming course in a class-based setting would be very interesting.
Doesn’t look simple to me but I get the point. I guess this will only be necessary when moving a future to another task so pretty rare in my case.
A couple of years ago I was rather excited about Rust but recently every time I read about async Rust I am just left puzzled by the amount of complexity. As a language user maybe I should just stop worrying…
I think the article posted here the other day perfectly sums up my feelings of referring to anything of this nature as simple.
Supporting asynchronicity under Rust’s standards of “memory safety without garbage collection, concurrency without data races, stability without stagnation” is acknowledged as complex, but, if one doesn’t need asynchronicity, one can choose libraries such as
ureqthat reject async and its complexity.Love this, absolutely something that could fit in my workflow. One issue I’ve found on an Android Phone (One Plus 7 Pro), is that in notification settings there is nothing for default notifications. I only have Max, High, Low and Minimum. This result in no proper notification (i.e no pop up or vibration) for default notifications. I’m not sure if this is an issue with my phone or the app. I downloaded it from F-Droid.
Edit: Using the Google Play version has solved the issue for me. I know there can be some funk with notifications and apps installed outside of Google Play, so not totally unexpected for the F-Droid version to be a little funky. I’ll stick to this version now.
That’s not something I have heard before, and there is no difference between F-Droid and Play in this regard. Odd. Feel free to file an issue if it’s a big issue.
I was under the impression that for proper push notifications, like your service is offering, that there was a dependency on Google Play Services? I’ve not developed for Android, so apologies if that is not correct. I seem to remember there being issues with Proton Mail on F-Droid that seemed to cite that reason before they fixed their notification issues.
Either way, since using the Google Play version it has all worked flawlessly. I will try the F-Droid version again at some point and just make sure it wasn’t something I’d done to stop it working properly.
I’m not certain I really agree with the article, but I do think it could prompt an interesting discussion here. Personally, I do not care about the “status” of software developers. I care about doing something I enjoy and earning a decent living. Even so, I actually don’t think our status has been eroded. On the contrary, I feel like it’s never been a more respected profession. The field is extremely accessible though, so you do end up with a very wide spectrum of developers and I think there is more of a divide between seasoned developers who learned in more traditional ways and career changers emerging in the field through bootcamps.
For those without and not wanting an account, here’s a version with that requirement removed.
I think ‘status’ is the wrong term, but respect for expertise is important. You could find a lot of opinions about the status of plumbing as an occupation but I don’t think you’d find much disagreement that the person planning and installing the system that removes sewage from your house has very important and valuable skills and the world would (locally, at least) be a much worse place if someone without those expertise tried to do the job.
The problem with a lot of the ‘eveyone can code’ movement is that it doesn’t differentiate between ‘everyone should be able to remove the u-bend under their sink and clean / replace it’ and ‘everyone should be able to plan a water and sewage system for a neighbourhood’ levels of skills. It’s pretty obvious to most people that the former is a useful life skill (you might still want a plumber to do it for you if the opportunity cost of you doing it is too high) whereas the latter is a specialised task that you shouldn’t expect someone to do just because they are able to do just because they fixed a leaky tap in the office kitchen that one time.
This lack of understanding and respect for expertise leads to a load of folks moving into management with the ability to do small programming tasks but without any understanding of software engineering.
Nice article. I’ve read many like this; typically featured on one of the major VPS providers; but I like that yours is rather concise whilst still explaining some of the less obvious aspects.
I also didn’t know about network.target not being a good way of determining that network interfaces are ready, so very glad I also came to the comments. Certainly a point worth updating in your article.
As a software developer, I adore having access to GPT3/4. Lately, I’ve been jumping around from language to language, and it’s a god-send when I know how to do something in one language and want to know how to do it in another. It’s not terribly difficult or time-consuming to do with a search engine, but with GPT I can ask follow-up questions to get an idea of idiomatic ways of doing things or about libraries that might exist in the new language to aid me.
How do you know it isn’t confabulating answers about idiomatic ways of doing things? Unlike the existence of libraries, it doesn’t seem like an easy thing to verify.
After using it for long enough you end up developing a pretty accurate intuition as to when to trust it and when to double-check.
Writing idiomatic code is something it’s actually really good at, because it’s seen a LOT of code in different languages and “idiomatic” basically means “in the most average way possible”. LLMs are really good at coming up with the most average way possible to express something!
That part just comes with experience I think. I don’t just blindly trust what it spits out. Often I’ll run similar prompts through it to check for consistency, and if I’m totally unfamiliar with a language I spend a bit of time on a forum/subreddit/channel for it initially just to get a feel.
I’d day it’s close to the mark more often that not though. To clarify, I’m not having it write code for me, but just having it give examples of how to get something done. Creating tests in an unfamiliar testing framework is something it really excels at in my experience.
I had a similar experience asking an LLM to generate some JavaScript code for me. The code was completely wrong, but it used all of the right APIs (I know enough of JavaScript-the-language to get things done, I haven’t looked at modern web APIs for ages) and so I could jump straight to the docs and write something correct in a few minutes.
Yes, this totally mirrors my experience. It getting the code wrong doesn’t matter, as we shouldn’t (yet) be relying on it for that. It’s a great tool for experienced developers to accelerate our workflows.