At a party, non-verbal communication (facial expressions, gestures) is crucial.
Maybe that explains why I as a blind person (well, low vision, but practically blind for this purpose) tend to be bored at parties, unless I manage to get into an interesting conversation with a person or small group. Somehow I never made that connection before.
That’s interesting. I do have some experience with how hearing-impaired people party, but I have no idea about blind people. Would you mind to share some details?
Fascinating. There mightbe some implications for informal team meetings here too, along with parties. (Or hell, they could be the same thing) The dynamic grouping thing, especially, caught my eye. If you think of a dinner party, you meet in small-ish groups, say 2-5, chat some on a topic, then wander off, listening and observing the room around to to see if anything else is going on that’s interesting.
This is sort of half-way between Hangout’s “show one person only” approach and some kind of tiled Brady Bunch approach. But those two approaches also have value! Would be fun to play with these ideas some more.
The main problem with subgroups, I think, is how to achieve certain isolation (so that you can, for example, gossip about other people at the party) but at the same time make it easy for people to wander around and join different subgroups.
Mozilla Hubs solves this by being a lightweight 3D environment (well, lightweight for 3D) where the sound is spatial:
Users embody 3D models […] Spatial audio means that you can break off into small groups and have conversations that broadcast your voice based on your position near others in the room. — Exploring Collaboration and Communication with Mozilla Hubs
I wonder whether there are any chat apps that also have spatial audio, but do the embodiment as a 2D room plan so as to be less CPU-intensive (and perhaps more mouse-friendly)? Although spatial audio is (or isn’t?) perhaps CPU-intensive on its own; I remember that when the audio-only game Papa Sangre came out, its binaural audio engine that ran on an iPhone processor was considered quite a technical feat.
You know, if you just had something like “three groups nearest us” with a text stream, even if it’s only a crappy speech-to-text deal, it might at least provide some stimulus to switch around.
Not only is the UI problematic, but if there’s any friction at all people are just going to camp out somewhere, defeating the goal.
You could do the data interchange thing without rewriting coreutils (and every other program). A structured shell could wrap each utility with a parser to its own object format. Adding support for new tools would be like writing completion scripts, and would not require any changes to the programs themselves.
That’s what the plan is for Oil. I’ve written a lot about parsing – not just because parsing shell itself is a problem, because I want the shell language to be good at parsing.
Parsing is the complement to using unstructured byte streams everywhere. Right now shell’s notion of parsing is $IFS and a subpar regex implementation, which can easily be improved.
Or rather, the plan is “both”, in the typical heterogeneous style of shell :) I do think some commands need structured variants, e.g. find and xargs, because they’re so central to the shell:
That looks pretty nice, and close to what I was thinking about for Oil.
I was thinking of calling it “TSV2”, which is basically TSV where if a field starts with " then it’s interpreted as a JSON string. Every language has a JSON library, so you can reuse that part.
Although now that I think about it, what @burntsushi said in this thread is right – JSON strings are problematic since they can’t represent arbitrary byte strings, only unicode code points.
I was also thinking of requiring tabs, which would have the nice property that you could validate the structure simply by counting tabs in a row, and you don’t have to parse the fields within a row. On the emitting side, tabs are easy to emit. The “align” and “trim” commands you implemented would still work for human readable output, of course.
I was also thinking of an optional JSON types in the header, with the default being string, like:
name num_bytes:number
foo.c 100234
foo.h 1234
This solves a common problem with working with converting CSVs to data frames in R and pandas. The builtin read.csv() functions try to guess the type of each column, and of course they do it wrong sometimes!
A few people e-mailed me saying they want Oil to have structured data, and we started a discussion on https://oilshell.zulipchat.com/ . Feel free to join if you’re interested! I laid out 3 concrete examples for structured data: the git log thing, my releases HTML page (which is really generated from a shell script), and Oil benchmarks, which use a combination of shell and R code, particularly xargs.
I think there needs to be some support in two places – in the shell itself, and in external tools. I also think xargs needs supports for “rows”, which I currently fake with xargs -n 7 for a row of 7 columns.
I suggest someone try following what separation kernels do: just describe the data format with a secure parser and plumbing autogenerated by a standardized tool. CAmkES is a field-deployed example. Cap n Proto might be ported to this use case for its balance of high speed and security.
Sadly that way lies PowerShell. It can maybe be done much better than that, but I’m skeptical mostly because if I want to do something that needs type-safety in a shell I’m leaning towards using a type-safe language as opposed to making my day-to-day shell require it.
Mostly that it feels like a bad compromise between bash and C#. e.g it’s got the bashism of “keep going when there’s an error” as default, but even if you switch it to “no, seriously, stop” you then have it’s exception handling, which doesn’t catch all exceptions, which is really confusing as the .Net work it’s based on would just fine…
I like it’s approach of “pipes of objects”, but it also has all the fun of having long names for shell builtins, and the deeply irritating script signing just makes it feel like it’s a lab experiment that escaped, and not in the good way.
What might work is making something with the usual design aesthetics of a typical unix shell, but having a type system around for “pipes of objects”, but it would need to make sure it’s still a usable shell at the end of things!
FWIW I don’t agree with the original criticism – you could have types in a shell without making it like PowerShell.
But here’s my impression of PowerShell and some feedback from others.
Not only is it tightly coupled to the .NET VM, all the “processes” it runs appear to be too. It prefers to run “cmdlets” which are like .NET programs running in the same VM, and not external processes, from my understanding.
Fundamentally Unix and Windows work in different ways, and PowerShell caters more to the design concepts of Windows. Windows is based on shared libraries and COM (binary protocols), and later .NET, while Unix is based on text.
I’ve heard some good feedback about PowerShell, but a lot of bad too.
PowerShell is my guilty pleasure of the computing world. Once you’ve piped strongly-typed objects around between your shell commands, text scraping seems barbaric by comparison. The problem, of course, is that you can’t use all that power for much, since the pool of PS-compatible tools and utilities is much shallower than it is for Unix shells.
Although I don’t think it makes sense to run a Windows shell on Unix, because you lose its advantages, it apparently does run there. And the VM issue apparently makes it very resource hungry:
PowerShell feels like it was built by people who had heard about command lines a long time ago and tried to recreate one based on stories passed down through generations of their ancestors.
When I look at the syntax, I tend to agree. It’s like they the cargo-culted a bunch of bad design decisions like the -eq operator without understanding that syntax isn’t why people use shell. It’s just legacy. It’s basically the worst of both worlds: it’s not compatible, AND it has legacy!
To me, the reason why people use shell is that it lets you integrate things that were NOT designed to be integrated. It’s glue which is vital to making real systems work. But if you couple everything to the .NET VM, then you’ve lost that property.
To me, the reason why people use shell is that it lets you integrate things that were NOT designed to be integrated. It’s glue which is vital to making real systems work. But if you couple everything to the .NET VM, then you’ve lost that property.
The irony is that Windows comes from OpenVMS heritage. VMS admins told me things such as data types and calling conventions were standardized to make cross-language development easier. That also makes integration by function calls easier. The .NET VM does something similar at the VM level. In theory, it could have a unique experience if a shell builds on that advantage. Instead, if your post is accurate, they threw away the best of VMS and Windows advantage to fail to emulate a UNIX advantage. Worst of both worlds.
This is a fun Unix-y idea, but even on tmpfs (which uses memory rather than disk/SSD), I suspect it will be around 2 orders of magnitude slower than a “normal” shell, and that does matter.
It looks like it turns every variable access into a file system access, which is at minimum an extra context switch. If you use FUSE, I believe it’s two context switches.
A variable access is basically a string hash lookup in most interpreters, including most shells, and that’s pretty fast. I suspect it takes 10-100 nanoseconds, i.e. you can do 10-100M / second.
Well I already have experience with a shell that’s 2 orders of magnitude too slow! That’s Oil, which I’m working on speeding up right now :) http://www.oilshell.org/blog/2019/02/05.html
Some people thought writing a shell in Python was a bad idea. Some people thought it would be perfectly fine and fast enough. I think both groups of people are wrong :) There are benefits to writing it in a high level language (6-8x fewer lines of code, which means fewer bugs). But it’s also too slow.
As I’ve written in a couple places, interactive completion functions are “CPU bound” and are already quite slow – easily milliseconds or tens or hundreds of ms, which is definitely human measurable. So you don’t want to slow those down by 100x.
So a shell doesn’t have to be fast for all shell scripts, since many of them are waiting on I/O. But it does have to be fast for SOME important scripts.
This is basically “orthogonal persistence” for a shell, and I see the appeal. But I do think there is a reason that those kinds of systems haven’t caught on, e.g. in particular because there is an important difference between ephemeral program state and the “contract” of your program. It’s useful to be able to change the ephemeral parts without worrying about breaking anyone.
I got only the last two right, and was within two orders of magnitude about half the time. Yikes, university does not prepare you with an intuition for actual orders of magnitude, as I suspected.
Yes, I definitely didn’t know these things coming out of university. Fixing performance bugs in “legacy” codebases is one good way to get intuition for these things. And I still don’t get 100% of them right, but I do make an effort to develop the intuition.
If you haven’t already seen them, it’s worth going over “latency numbers everyone should know”:
I think it’s worth memorizing at first, but as you develop software, you will encounter different systems that are bottlenecked by different things, and they will be made “real”, so you don’t have to memorize them.
And it’s worth adding other important operations to that hierarchy – particularly context swtiches (single digit microseconds) and interpreter bytecodes (ten to hundreds of ns)!
Honestly if you Google for this you’ll get a lot of disparate resources that aren’t that well presented… one thing I would like to do is distribute some shell scripts to measure these on a commodity Unix box! I think it is mostly possible.
The performance problem can be addressed by having an in-memory cache and saving to disk only when the interpreter exits.
As for the ephemeral part, I think you are talking about separation between API and implementation, right? That should still apply unless a third party starts messing with the content of the directory. My idea of how it would work was rather to create an ephemeral directory for each run of the program, a directory that would be private for the person running the program.
I guess I’m saying that it’s useful to have both memory and disk as separate things. I’ve read about a bunch of systems that have tried to get rid of the distinction, under the name “single level store” or “orthogonal persistence” / transparent persistence. Eros OS is one:
I see the appeal, but the I don’t think the benefits outweigh the costs.
I do think the fork() idea is interesting. In particular I wondered if it would be useful to fork across machines by saving your state, copying it to another machine, and then resuming on a different code path from that state, much like Unix fork(). I prototyped two versions of a format called “OHeap” that can do that.
I have shelved it for now, but it was supposed to replace Python’s marshal, pickle, and zipimport modules, which are mostly redundant.
I think that simply having primitives to start fresh processes on other machines is more important than being able to use fork(). There is a conceptual appeal but I could not find much practical motivation.
I’ve read about a bunch of systems that have tried to get rid of the distinction, under the name “single level store” or “orthogonal persistence” / transparent persistence.
Other examples that may be better known for SLS, many of which were used in practical systems:
Palm OS (for classic Palm PDAs, databases exist in memory)
Aegis (the Apollo Domain kernel, actually not “persistent’ SLS - it’s more everything is an mmaped file instead of persistent objects)
Genera (the OS for Symbolics Lisp Machines)
Phantom OS (Russian research/hobbyist OS)
IBM i (the OS of the AS/400, pointers also function as capabilities. Probably the most famous example of SLS)
it’s a bug—EBADF should not be happening. That it is, is a bug. Once fixed, it should not happen. So EFAULT, ENOTSOCK, and EISCONN all fall under this category.
It’s fixable outside the scope of the program—EACCESS, ENETUNREACH and EAGAIN are examples here. Report, and exit, not much else to do.
Resource exhaustion, things are probably going bad quickly—EAGAIN might also be this category. ENOMEM definitely is, but that’s not a possible error for connect().
Expected and should be handled by the program—ETIMEDOUT, ECONNREFUSED, EINTR (if using signal()), maybe ENETUNREACH could be considered here as well. Things that can normally happen, and there is some overlap with the second category here.
It’s now a bit simpler—just check for expected conditions and handle; everything else should be logged (and possibly terminate the program, depending upon the condition).
[1] On that page I list three ways to handle errors. Since then, my thinking has changed somewhat on that but I’ve yet to write it up.
I like the categorization. It makes easier to think about the errors.
One thing that comes to mind is: Can we deal with some of those categories automatically? For example, I’ve never seen ENOMEM handled in a reasonable way. While in theory is looks like it can be handled, thigs like memory overcommitment and OOM killer make it futile. Maybe we’ve given up any chance of handling OOM errors back in 1960’s when we’ve replaced static invocation records by call stack. Anyway, maybe returning ENOMEM makes no sense at all. Instead OOM killer should just kill the process. But I never done embedded programming, so who am I to tell?
One pattern that I’ve noticed is that many web developers don’t get rigorous error handling, perhaps because the context in which we’re developing does so much work for us. If you:
Don’t share mutable state between requests
Don’t eat exceptions
Have a transactional data store
there’s a limited amount of damage that you can do by being lazy with error handling. The worst case is that you fail to make progress in an instance where you could’ve continued past some exception, but you’ll never end up in an inconsistent state.
Yes. Also, if anything goes wrong, user will just hit “reload page” button. No big harm done. I think GUI applications are ones that rarely need rigorous error handling.
A function may succeed. Or it may, for example, fail because of disconnected backend. Or it may time out. And that’s it. There are only two failure modes and they are documented as a part of the API. Once you have that, the error handling becomes obvious.
I must be missing something because it really feels like there are plenty of other ways for a function to fail. Is this limited to a specific context? If it’s only for infrastructure, it still seems woefully pidgeonholed.
As already mentioned, classic exceptions are the worst.
I’m not clear on why they are “the worst”.
The discussion does hit on something that makes sense to me: think about and document the error conditions. Frankly, if you have that, the methodology of reporting the error becomes less of a hassle. But still, error handling is plauged by the fact that it is often something non-local that is affecting the computation and there is rarely any useful information or language constructs that make dealing with it anything short of a massive chore. (Correcting it usually means interacting or “conversing” with some other entity to gain the knowledge to proceed.)
I must be missing something because it really feels like there are plenty of other ways for a function to fail. Is this limited to a specific context? If it’s only for infrastructure, it still seems woefully pidgeonholed.
POSIX is quite a good example of how it could work. Every function can return few possible error codes and that’s it. The idea is that implemeter of the function deals with the complexity and factors all possible error conditions into a small neat set of error codes that makes sense from the user’s point of view.
The rule here should be: If you don’t know what to do with an error condition, don’t just pass it to the caller. The caller understands the problem domain even less than you do.
But still, error handling is plauged by the fact that it is often something non-local that is affecting the computation and there is rarely any useful information or language constructs that make dealing with it anything short of a massive chore.
The point is to use encapsulation for errors as well as for normal functionality. If something non-local causes an error somewhere down the stack, the layer that deals with the thing (and every layer above it) should convert it into an error that makes sense in the local context.
If something non-local causes an error somewhere down the stack, the layer that deals with the thing (and every layer above it) should convert it into an error that makes sense in the local context.
When said this way, I understand the point better. I did not get that from the original post. I think that’s a reasonable way to deal with things, although I don’t think it precludes exceptions as the mechanism for doing it.
True, but exceptions make it super easy to screw it up. Just forget a catch block in one function and the raw low-level exception escapes up the stack. In C/Golang style of error handling you have to at least pass it up manually which will, hopefully, make you consider whether it’s a good idea in the first place.
(Correcting it usually means interacting or “conversing” with some other entity to gain the knowledge to proceed.)
That’s why, even though it is relatively heavy-weight for an API, it seems that passing a callback to be called on error is one of the most versatile things you can do.
The callback can correct the error and allow the call to proceed or just throw an exception. At deeper level, doing this allows you to interact with context at the point of detection not the point where you express your intention: the initial call that led to the error.
I think this is the closest we can come to approximating Lisp’s condition system in languages without those constructs.
Signals and restarts are wonderful things. It’s such a shame no other language or programming system (to my knowledge) has made a serious effort to emulate it, let alone build on it. Callbacks are the best we can do – or what we’re willing to abide – it seems.
I would like to try this out, but I can’t auth the shopping list demo to Github because it wants access to both private and public repos, and those of my employer. I understand this might be a limit on Github’s side (maybe the permissions options aren’t granular enough) but unfortunately that makes it a hard pass for me.
There’s a scope to authorize the app only for public repos. But that in turn would make it impossible to use on private repos. It’s starts to look like GitHub API, as it doesn’t support granting per-repo access, is not designed to support this kind of scenario (application running in brower, impersonating the user).
I’ve wanted something like this for a while - the idea of mixing documentation/checklists and a record of going through them really appeals to me.
Not sure about GitHub-as-storage, but I can see how other approaches might be more work. I was expecting it to be more like a wiki (self-hosted, database backend) on first glance, but I’m not sure if that would actually be an improvement rather than just what I’m used to.
Yes, checklisted documentation is helpful, but also notice the context-collection capabilities via comments on steps. The motivation use case: A person is working on a complex process for several months, then leaves the company. Another person takes over and has to get up speed quickly. By looking at the graph they can immediately understand what have been done, what haven’t been done and what’s blocking what. By looking at individual unfinished step they can read the comments and understand what was already done.
I was thinking of storing everything, including the comments in a git instance, which would work independently of what git frontend you are using, but then I would have to speak git protocol from the browser which sucks. I may have a look at git.js
“I’ve been asking Github to enable CORS headers to their HTTPS git servers, but they’ve refused to do it. This means that a browser can never clone from github because the browser will disallow XHR requests to the domain.”
Anything self-hosted would be viable, but everything on git would be even better, although probably more complicated. We use gerrit at work (which sucks at several levels), and mostly anything third-party is very much disallowed. Maybe you could create an abstraction that would speak Github API to github and git protocol to other servers where this would work?
The other possibility could be a sort of optional backend/proxy, so, if the git server doesn’t have CORS, you could spin that optional server.
After thinking about it some more, there’s a lot that GitHub offers that I would have to reimplement myself. Authentication, for one thing. If it was used in a stand-alone mode in enterprise, some kind of authentication would be still needed. People would probably want SSO. Then there are notifications. GitHub sends you an email when you are mentioned in a bug. I would have to somehow interact with company’s mail server. And so on. This is my hobby project and I don’t really have time to go into that amount of complexity.
I’ve written ZeroMQ and nanomsg once. This is part of my years-long project to make writing such applications tractable. And by that I mean being able to write them without falling into either callback hell or state machine hell.
On that topic, what is the status of Nanomsg? Is libdill your main focus, or do you grow these projects in parallel? I’ve watched this projects without using them in practice, but I really like the approach of trying to find the right abstractions and patterns for expressive and efficient network programming.
One challenge is that one can always read all the pages. It would be great flavor if this shipped with like… 20 times as many pages. Things like random letters, accounting statements. So you can also experience the whole “sifting through a lot of stuff” thing, and perhaps landing on an interesting bit somewhere. Maybe even having this actually ship as several distinct sets of books, for example.
EDIT: You might want to check out Her Story for some interesting ideas in there as well. A bit harder to execute upon on paper, but is a very interesting mechanism for non-linear storytelling.
I like the idea of adding cruft to confuse the reader. Another option would be to hide different chapters at different physical locations and references would be instruction of how to get to the next chapter. But the it ceases to be a book, of course.
EDIT: I’ve added a comment to README along the lines of your comment. I hope you don’t mind. One modification though. Adding 20x more content isn’t feasible for a printed book. So, instead, the pulp should look as a legitimate content to waste reader’s time by solving nonsesical puzzles etc.
If we accept the premise that there is a lot of value in the uniform morphology, Esperanto could be an option suited for ASCII (in the orthography with «x» instead of diacritics, of course). Then there is also Toki Pona. Many people prefer just to combine enough separate English words to get the point across.
But I think there is another linguistical problem to consider: naming things in programming is hard because programming is an activity where minor semantic distinctions often matter. Maybe a uniform morphology would help by reducing verbosity and allowing to put more meaningful roots in the name of a given length; but anything general enough to be universally useful would have to be vague enough to be subtly misleaing in the specific cases anyway.
The problem is not just to remember the words — «reading with a dictionary» is a skill older than programming. The problem is that too much details are needed for defining even a single word.
Maybe. But you can also look at it from the other side: If morphology is standardized, people, being pattern-loving animals, would try to use the constructs consistently, i.e. try would try to make relationship between “parse” and “parser” be similar to relationship between “scan” and “scanner”. Eventually, the constructs could come to represent something like “design patterns”, something that you can assume to work in some specific way.
Well, the design pattern called Factory definitely has its own «-Factory» suffix. And predicates often get an affix of one or two characters («is», «-p», «?»). And Hungarian notation was used.
My fear is that humans are actually too good at pattern matching, so if all you have is «-er», you will get «parser» regardless of whether it tries to parse a prefix or requires a whole-string match.
Do you hope that using a spoken language with a lot of morphological modifiers as a base will affect the culture to create enough new modifiers for smaller patterns? I mean enough to avoid combining any dangerous-to-combine notions. I find this plausible, but not guaranteed; I guess naming things in Esperanto could be a way to try.
I don’t know really, but it might be worth a try. At least when a programmer talks to another programmer in person, they use natural language to get through the idea. This is often (at least in my experience) superior to just reading the code. So, maybe, if we were able to take a bit of this person-to-person communication in convey it via the code, it would help to some extent.
Well, in person-to-person communication there is not only different naming (I would be surprised if some structured morphology not used in the variable naming would arise), there are different protocols for manging the level of details. You can get an overview that is not just «N levels deep in the call tree». Sometimes abstractions are also intentionally lossy, which you are usually not allowed to do in code.
Some things depend on feedback, there is some research into allowing zoom in/zoom out for abstraction levels, but improving state of art in the area of zooming out the programming abstractions would definitely be valuable.
My point was also that we currently have more tools for lexical part than for grammatical part. Is morphology still where the best return on effort is? (I honestly do not know)
I don’t think this would help with the tooling. However, it would decrease the amount of lexical baggage which in turn could help with, say, keeping the learning curve flat, or, maybe, returning to old code years later, remembering just the core concepts and being able to get up to speed immediately.
I meant tools in a wider sense including conceptual tools.
Intuitively, a flat learning curve is not something you can achieve in an experiment (the morphology has to be learnt first). So this part is hard to know (getting data from Esperanto taking off only for code identifiers and comments sounds a bit optimistic).
It would be of course interesting if there were some subset that you could try with moderate effort and then tell a success story.
That’s the very crux of the problem. How would you shared knowledge without writing code? Well, there’s still an option to write academic papers, but given the rift between compsci academia and practicioners of programming I would expect it not to be very efficient.
The idea is a meme. The code is its reproductive organ. The code is ‘useful’ so that it can lure its prey (a living human brain). Once the code is used the idea is repeatedly injected into the brain.
Compare that to talking to people where the idea is basically let floating in the space to be voluntarily accepted or not.
Ideas spread fine on their own. For example I’m about to convince you of this without a single line of code. There’s no need to push things into formal language when they make sense in nonformal language. I don’t need to tell you the steps of how to build a boat for you to realize that some method of traveling over water is good. In fact I’d argue that if I told everyone the exact steps to build a boat most would miss the point about what the boat is for. They’d get caught up in the details and fail to capture the bigger picture.
English descriptions with formal specifications and/or pseudocode accompanying them in a formalism that’s simple. That was the standard for high-assurance security. It worked consistently so long as the formalism could handle what they were describing. The caveat that the teams needed at least one specialist to train them on and help them with the formalism. If we’re talking CompSci, that could just become another 101 course where people are exposed to a few easy ones.
Maybe that explains why I as a blind person (well, low vision, but practically blind for this purpose) tend to be bored at parties, unless I manage to get into an interesting conversation with a person or small group. Somehow I never made that connection before.
That’s interesting. I do have some experience with how hearing-impaired people party, but I have no idea about blind people. Would you mind to share some details?
Fascinating. There mightbe some implications for informal team meetings here too, along with parties. (Or hell, they could be the same thing) The dynamic grouping thing, especially, caught my eye. If you think of a dinner party, you meet in small-ish groups, say 2-5, chat some on a topic, then wander off, listening and observing the room around to to see if anything else is going on that’s interesting.
This is sort of half-way between Hangout’s “show one person only” approach and some kind of tiled Brady Bunch approach. But those two approaches also have value! Would be fun to play with these ideas some more.
The main problem with subgroups, I think, is how to achieve certain isolation (so that you can, for example, gossip about other people at the party) but at the same time make it easy for people to wander around and join different subgroups.
Mozilla Hubs solves this by being a lightweight 3D environment (well, lightweight for 3D) where the sound is spatial:
I wonder whether there are any chat apps that also have spatial audio, but do the embodiment as a 2D room plan so as to be less CPU-intensive (and perhaps more mouse-friendly)? Although spatial audio is (or isn’t?) perhaps CPU-intensive on its own; I remember that when the audio-only game Papa Sangre came out, its binaural audio engine that ran on an iPhone processor was considered quite a technical feat.
You know, if you just had something like “three groups nearest us” with a text stream, even if it’s only a crappy speech-to-text deal, it might at least provide some stimulus to switch around.
Not only is the UI problematic, but if there’s any friction at all people are just going to camp out somewhere, defeating the goal.
You could do the data interchange thing without rewriting coreutils (and every other program). A structured shell could wrap each utility with a parser to its own object format. Adding support for new tools would be like writing completion scripts, and would not require any changes to the programs themselves.
I’m kind of tempted by this idea, actually.
That’s what the plan is for Oil. I’ve written a lot about parsing – not just because parsing shell itself is a problem, because I want the shell language to be good at parsing.
Parsing is the complement to using unstructured byte streams everywhere. Right now shell’s notion of parsing is
$IFSand a subpar regex implementation, which can easily be improved.Or rather, the plan is “both”, in the typical heterogeneous style of shell :) I do think some commands need structured variants, e.g. find and xargs, because they’re so central to the shell:
https://github.com/oilshell/oil/issues/85 (help still wanted on ‘find’ if anyone’s interested)
But you shouldn’t be required to use new commands – scraping coreutils is also valid, and should be easier.
There are some more thoughts about structured data in Oil on Zulip:
https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/Structured.20Data (requires login, e.g. Github)
Here’s the canonical post about not “boiling the ocean”, e.g. imagining that every tool like git will add structured data:
Git Log in HTML: A Harder Problem and A Safe Solution
I’m surprised that most shells after Bash support scripting. I’d love to see a shell optimized for interactive use instead.
Well, fish is intended to be used interactively? I use it as my daily driver and it’s pretty good at smoothing some of the rough edges.
What makes fish better for interactive use than other shells?
Honestly? Not much, as far as I can tell. It has a nicely orthogonal syntax, and just in general is deeply more sane than POSIX shells.
I’ve tried to make something in that direction lately: https://github.com/sustrik/uxy
That looks pretty nice, and close to what I was thinking about for Oil.
I was thinking of calling it “TSV2”, which is basically TSV where if a field starts with
"then it’s interpreted as a JSON string. Every language has a JSON library, so you can reuse that part.Although now that I think about it, what @burntsushi said in this thread is right – JSON strings are problematic since they can’t represent arbitrary byte strings, only unicode code points.
I was also thinking of requiring tabs, which would have the nice property that you could validate the structure simply by counting tabs in a row, and you don’t have to parse the fields within a row. On the emitting side, tabs are easy to emit. The “align” and “trim” commands you implemented would still work for human readable output, of course.
I was also thinking of an optional JSON types in the header, with the default being string, like:
This solves a common problem with working with converting CSVs to data frames in R and pandas. The builtin
read.csv()functions try to guess the type of each column, and of course they do it wrong sometimes!A few people e-mailed me saying they want Oil to have structured data, and we started a discussion on https://oilshell.zulipchat.com/ . Feel free to join if you’re interested! I laid out 3 concrete examples for structured data: the git log thing, my releases HTML page (which is really generated from a shell script), and Oil benchmarks, which use a combination of shell and R code, particularly xargs.
I think there needs to be some support in two places – in the shell itself, and in external tools. I also think xargs needs supports for “rows”, which I currently fake with
xargs -n 7for a row of 7 columns.I suggest someone try following what separation kernels do: just describe the data format with a secure parser and plumbing autogenerated by a standardized tool. CAmkES is a field-deployed example. Cap n Proto might be ported to this use case for its balance of high speed and security.
Sadly that way lies PowerShell. It can maybe be done much better than that, but I’m skeptical mostly because if I want to do something that needs type-safety in a shell I’m leaning towards using a type-safe language as opposed to making my day-to-day shell require it.
What’s wrong with powershell?
Mostly that it feels like a bad compromise between bash and C#. e.g it’s got the bashism of “keep going when there’s an error” as default, but even if you switch it to “no, seriously, stop” you then have it’s exception handling, which doesn’t catch all exceptions, which is really confusing as the .Net work it’s based on would just fine…
I like it’s approach of “pipes of objects”, but it also has all the fun of having long names for shell builtins, and the deeply irritating script signing just makes it feel like it’s a lab experiment that escaped, and not in the good way.
What might work is making something with the usual design aesthetics of a typical unix shell, but having a type system around for “pipes of objects”, but it would need to make sure it’s still a usable shell at the end of things!
FWIW I don’t agree with the original criticism – you could have types in a shell without making it like PowerShell.
But here’s my impression of PowerShell and some feedback from others.
Not only is it tightly coupled to the .NET VM, all the “processes” it runs appear to be too. It prefers to run “cmdlets” which are like .NET programs running in the same VM, and not external processes, from my understanding.
Fundamentally Unix and Windows work in different ways, and PowerShell caters more to the design concepts of Windows. Windows is based on shared libraries and COM (binary protocols), and later .NET, while Unix is based on text.
I’ve heard some good feedback about PowerShell, but a lot of bad too.
https://news.ycombinator.com/item?id=12313179
Although I don’t think it makes sense to run a Windows shell on Unix, because you lose its advantages, it apparently does run there. And the VM issue apparently makes it very resource hungry:
https://news.ycombinator.com/item?id=12313969
https://news.ycombinator.com/item?id=17466305
https://news.ycombinator.com/item?id=12986621
Some harsh feedback from 2014:
https://medium.com/@octskyward/the-woes-of-powershell-8737e5346b1
PowerShell feels like it was built by people who had heard about command lines a long time ago and tried to recreate one based on stories passed down through generations of their ancestors.
When I look at the syntax, I tend to agree. It’s like they the cargo-culted a bunch of bad design decisions like the
-eqoperator without understanding that syntax isn’t why people use shell. It’s just legacy. It’s basically the worst of both worlds: it’s not compatible, AND it has legacy!To me, the reason why people use shell is that it lets you integrate things that were NOT designed to be integrated. It’s glue which is vital to making real systems work. But if you couple everything to the .NET VM, then you’ve lost that property.
The irony is that Windows comes from OpenVMS heritage. VMS admins told me things such as data types and calling conventions were standardized to make cross-language development easier. That also makes integration by function calls easier. The .NET VM does something similar at the VM level. In theory, it could have a unique experience if a shell builds on that advantage. Instead, if your post is accurate, they threw away the best of VMS and Windows advantage to fail to emulate a UNIX advantage. Worst of both worlds.
The question is whether you consider type-aware tab completion to be a useful feature. If so, then there’s an use case for type-awareness in shell.
This is a fun Unix-y idea, but even on tmpfs (which uses memory rather than disk/SSD), I suspect it will be around 2 orders of magnitude slower than a “normal” shell, and that does matter.
It looks like it turns every variable access into a file system access, which is at minimum an extra context switch. If you use FUSE, I believe it’s two context switches.
A variable access is basically a string hash lookup in most interpreters, including most shells, and that’s pretty fast. I suspect it takes 10-100 nanoseconds, i.e. you can do 10-100M / second.
https://computers-are-fast.github.io/ (first example is clocked at 68 M /s)
On the other hand a context switch is single digit microseconds, which is 2 orders of magnitude slower:
https://stackoverflow.com/questions/21887797/what-is-the-overhead-of-a-context-switch
Well I already have experience with a shell that’s 2 orders of magnitude too slow! That’s Oil, which I’m working on speeding up right now :) http://www.oilshell.org/blog/2019/02/05.html
Some people thought writing a shell in Python was a bad idea. Some people thought it would be perfectly fine and fast enough. I think both groups of people are wrong :) There are benefits to writing it in a high level language (6-8x fewer lines of code, which means fewer bugs). But it’s also too slow.
As I’ve written in a couple places, interactive completion functions are “CPU bound” and are already quite slow – easily milliseconds or tens or hundreds of ms, which is definitely human measurable. So you don’t want to slow those down by 100x.
So a shell doesn’t have to be fast for all shell scripts, since many of them are waiting on I/O. But it does have to be fast for SOME important scripts.
This is basically “orthogonal persistence” for a shell, and I see the appeal. But I do think there is a reason that those kinds of systems haven’t caught on, e.g. in particular because there is an important difference between ephemeral program state and the “contract” of your program. It’s useful to be able to change the ephemeral parts without worrying about breaking anyone.
I got only the last two right, and was within two orders of magnitude about half the time. Yikes, university does not prepare you with an intuition for actual orders of magnitude, as I suspected.
Yes, I definitely didn’t know these things coming out of university. Fixing performance bugs in “legacy” codebases is one good way to get intuition for these things. And I still don’t get 100% of them right, but I do make an effort to develop the intuition.
If you haven’t already seen them, it’s worth going over “latency numbers everyone should know”:
https://gist.github.com/jboner/2841832
https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html
I think it’s worth memorizing at first, but as you develop software, you will encounter different systems that are bottlenecked by different things, and they will be made “real”, so you don’t have to memorize them.
And it’s worth adding other important operations to that hierarchy – particularly context swtiches (single digit microseconds) and interpreter bytecodes (ten to hundreds of ns)!
Honestly if you Google for this you’ll get a lot of disparate resources that aren’t that well presented… one thing I would like to do is distribute some shell scripts to measure these on a commodity Unix box! I think it is mostly possible.
The performance problem can be addressed by having an in-memory cache and saving to disk only when the interpreter exits.
As for the ephemeral part, I think you are talking about separation between API and implementation, right? That should still apply unless a third party starts messing with the content of the directory. My idea of how it would work was rather to create an ephemeral directory for each run of the program, a directory that would be private for the person running the program.
I guess I’m saying that it’s useful to have both memory and disk as separate things. I’ve read about a bunch of systems that have tried to get rid of the distinction, under the name “single level store” or “orthogonal persistence” / transparent persistence. Eros OS is one:
https://scholar.google.com/scholar?cluster=13406830774672409828&hl=en&as_sdt=0,5&sciodt=0,5
Urbit is another recent one:
https://urbit.org/docs/learn/vere/runtime/
I see the appeal, but the I don’t think the benefits outweigh the costs.
I do think the fork() idea is interesting. In particular I wondered if it would be useful to fork across machines by saving your state, copying it to another machine, and then resuming on a different code path from that state, much like Unix fork(). I prototyped two versions of a format called “OHeap” that can do that.
http://www.oilshell.org/blog/2018/12/16.html#toc_3
I have shelved it for now, but it was supposed to replace Python’s marshal, pickle, and zipimport modules, which are mostly redundant.
I think that simply having primitives to start fresh processes on other machines is more important than being able to use fork(). There is a conceptual appeal but I could not find much practical motivation.
Other examples that may be better known for SLS, many of which were used in practical systems:
And http://libdill.org/, from the same authors, but without sticking to “go-style”.
Also libdill has support for structured concurrency, whereas libmill doesn’t.
This is why I simply love programming with golang. Everything is setup for you to handle errors somewhat properly, without much hassle.
But when you look at actual real-world code in Golang, what you often see is:
Even a sane system can be used in silly way.
At least the snippet checks for an
errvalue!That’s ok, if you don’t want to handle it there.
Worst is:
output, _ := fun()
That’s a joke, right?
Error handling is one of the few things I think go has comprehensively messed up. In no particular order, the builtin errors lack:
Different opinions, man. It’s not perfect but I think it’s pretty good, compared to all the languages I worked with.
Btw, internationalisation of error messages is dead simple to implement, even if it is not straight out of the box.
In my experience, there are four types of errors [1]. To summarize, using
connect()as the example:it’s a bug—
EBADFshould not be happening. That it is, is a bug. Once fixed, it should not happen. SoEFAULT,ENOTSOCK, andEISCONNall fall under this category.It’s fixable outside the scope of the program—
EACCESS,ENETUNREACHandEAGAINare examples here. Report, and exit, not much else to do.Resource exhaustion, things are probably going bad quickly—
EAGAINmight also be this category.ENOMEMdefinitely is, but that’s not a possible error forconnect().Expected and should be handled by the program—
ETIMEDOUT,ECONNREFUSED,EINTR(if usingsignal()), maybeENETUNREACHcould be considered here as well. Things that can normally happen, and there is some overlap with the second category here.It’s now a bit simpler—just check for expected conditions and handle; everything else should be logged (and possibly terminate the program, depending upon the condition).
[1] On that page I list three ways to handle errors. Since then, my thinking has changed somewhat on that but I’ve yet to write it up.
I like the categorization. It makes easier to think about the errors.
One thing that comes to mind is: Can we deal with some of those categories automatically? For example, I’ve never seen ENOMEM handled in a reasonable way. While in theory is looks like it can be handled, thigs like memory overcommitment and OOM killer make it futile. Maybe we’ve given up any chance of handling OOM errors back in 1960’s when we’ve replaced static invocation records by call stack. Anyway, maybe returning ENOMEM makes no sense at all. Instead OOM killer should just kill the process. But I never done embedded programming, so who am I to tell?
One pattern that I’ve noticed is that many web developers don’t get rigorous error handling, perhaps because the context in which we’re developing does so much work for us. If you:
there’s a limited amount of damage that you can do by being lazy with error handling. The worst case is that you fail to make progress in an instance where you could’ve continued past some exception, but you’ll never end up in an inconsistent state.
Yes. Also, if anything goes wrong, user will just hit “reload page” button. No big harm done. I think GUI applications are ones that rarely need rigorous error handling.
Bug report: looks like broken markdown around the libdill and POSIX
connect()links.Fixed. thanks!
I must be missing something because it really feels like there are plenty of other ways for a function to fail. Is this limited to a specific context? If it’s only for infrastructure, it still seems woefully pidgeonholed.
I’m not clear on why they are “the worst”.
The discussion does hit on something that makes sense to me: think about and document the error conditions. Frankly, if you have that, the methodology of reporting the error becomes less of a hassle. But still, error handling is plauged by the fact that it is often something non-local that is affecting the computation and there is rarely any useful information or language constructs that make dealing with it anything short of a massive chore. (Correcting it usually means interacting or “conversing” with some other entity to gain the knowledge to proceed.)
POSIX is quite a good example of how it could work. Every function can return few possible error codes and that’s it. The idea is that implemeter of the function deals with the complexity and factors all possible error conditions into a small neat set of error codes that makes sense from the user’s point of view.
The rule here should be: If you don’t know what to do with an error condition, don’t just pass it to the caller. The caller understands the problem domain even less than you do.
The point is to use encapsulation for errors as well as for normal functionality. If something non-local causes an error somewhere down the stack, the layer that deals with the thing (and every layer above it) should convert it into an error that makes sense in the local context.
When said this way, I understand the point better. I did not get that from the original post. I think that’s a reasonable way to deal with things, although I don’t think it precludes exceptions as the mechanism for doing it.
True, but exceptions make it super easy to screw it up. Just forget a catch block in one function and the raw low-level exception escapes up the stack. In C/Golang style of error handling you have to at least pass it up manually which will, hopefully, make you consider whether it’s a good idea in the first place.
That’s why, even though it is relatively heavy-weight for an API, it seems that passing a callback to be called on error is one of the most versatile things you can do. The callback can correct the error and allow the call to proceed or just throw an exception. At deeper level, doing this allows you to interact with context at the point of detection not the point where you express your intention: the initial call that led to the error.
I think this is the closest we can come to approximating Lisp’s condition system in languages without those constructs.
Signals and restarts are wonderful things. It’s such a shame no other language or programming system (to my knowledge) has made a serious effort to emulate it, let alone build on it. Callbacks are the best we can do – or what we’re willing to abide – it seems.
Have you heard of the Zen of Erlang? https://ferd.ca/the-zen-of-erlang.html
Related: Why Philosophers Should Care About Computational Complexity by Scott Aaronson.
+100
I can’t seem to get any of the examples to load workout errors.
Replied on the issue.
I would like to try this out, but I can’t auth the shopping list demo to Github because it wants access to both private and public repos, and those of my employer. I understand this might be a limit on Github’s side (maybe the permissions options aren’t granular enough) but unfortunately that makes it a hard pass for me.
There’s a scope to authorize the app only for public repos. But that in turn would make it impossible to use on private repos. It’s starts to look like GitHub API, as it doesn’t support granting per-repo access, is not designed to support this kind of scenario (application running in brower, impersonating the user).
I’ve wanted something like this for a while - the idea of mixing documentation/checklists and a record of going through them really appeals to me.
Not sure about GitHub-as-storage, but I can see how other approaches might be more work. I was expecting it to be more like a wiki (self-hosted, database backend) on first glance, but I’m not sure if that would actually be an improvement rather than just what I’m used to.
Yes, checklisted documentation is helpful, but also notice the context-collection capabilities via comments on steps. The motivation use case: A person is working on a complex process for several months, then leaves the company. Another person takes over and has to get up speed quickly. By looking at the graph they can immediately understand what have been done, what haven’t been done and what’s blocking what. By looking at individual unfinished step they can read the comments and understand what was already done.
Sounds interesting, might solve a problem I have at work, too bad it’s coupled to github =(
What would you prefer it to use as the underlying storage? (I am trying to understand what people actually want.)
https://gitea.io/en-us/
I was thinking of storing everything, including the comments in a git instance, which would work independently of what git frontend you are using, but then I would have to speak git protocol from the browser which sucks. I may have a look at git.js
Looking at git.js documentation :(
“I’ve been asking Github to enable CORS headers to their HTTPS git servers, but they’ve refused to do it. This means that a browser can never clone from github because the browser will disallow XHR requests to the domain.”
Anything self-hosted would be viable, but everything on git would be even better, although probably more complicated. We use gerrit at work (which sucks at several levels), and mostly anything third-party is very much disallowed. Maybe you could create an abstraction that would speak Github API to github and git protocol to other servers where this would work?
The other possibility could be a sort of optional backend/proxy, so, if the git server doesn’t have CORS, you could spin that optional server.
After thinking about it some more, there’s a lot that GitHub offers that I would have to reimplement myself. Authentication, for one thing. If it was used in a stand-alone mode in enterprise, some kind of authentication would be still needed. People would probably want SSO. Then there are notifications. GitHub sends you an email when you are mentioned in a bug. I would have to somehow interact with company’s mail server. And so on. This is my hobby project and I don’t really have time to go into that amount of complexity.
Sure, makes sense. It’s still a cool project, nonetheless, so, congrats =)
sounds like a job for the backend
The only issue that I have with it is sharing my organization details. Although you could do it manually, I’m always a bit annoyed about this.
This looks great. Is this library being written to be part of some large application?
I’ve written ZeroMQ and nanomsg once. This is part of my years-long project to make writing such applications tractable. And by that I mean being able to write them without falling into either callback hell or state machine hell.
On that topic, what is the status of Nanomsg? Is libdill your main focus, or do you grow these projects in parallel? I’ve watched this projects without using them in practice, but I really like the approach of trying to find the right abstractions and patterns for expressive and efficient network programming.
Banal question; libdill? why not just use go?
I love this concept.
One challenge is that one can always read all the pages. It would be great flavor if this shipped with like… 20 times as many pages. Things like random letters, accounting statements. So you can also experience the whole “sifting through a lot of stuff” thing, and perhaps landing on an interesting bit somewhere. Maybe even having this actually ship as several distinct sets of books, for example.
EDIT: You might want to check out Her Story for some interesting ideas in there as well. A bit harder to execute upon on paper, but is a very interesting mechanism for non-linear storytelling.
I like the idea of adding cruft to confuse the reader. Another option would be to hide different chapters at different physical locations and references would be instruction of how to get to the next chapter. But the it ceases to be a book, of course.
EDIT: I’ve added a comment to README along the lines of your comment. I hope you don’t mind. One modification though. Adding 20x more content isn’t feasible for a printed book. So, instead, the pulp should look as a legitimate content to waste reader’s time by solving nonsesical puzzles etc.
It’s an example of security through obscurity!
If we accept the premise that there is a lot of value in the uniform morphology, Esperanto could be an option suited for ASCII (in the orthography with «x» instead of diacritics, of course). Then there is also Toki Pona. Many people prefer just to combine enough separate English words to get the point across.
But I think there is another linguistical problem to consider: naming things in programming is hard because programming is an activity where minor semantic distinctions often matter. Maybe a uniform morphology would help by reducing verbosity and allowing to put more meaningful roots in the name of a given length; but anything general enough to be universally useful would have to be vague enough to be subtly misleaing in the specific cases anyway.
The problem is not just to remember the words — «reading with a dictionary» is a skill older than programming. The problem is that too much details are needed for defining even a single word.
Maybe. But you can also look at it from the other side: If morphology is standardized, people, being pattern-loving animals, would try to use the constructs consistently, i.e. try would try to make relationship between “parse” and “parser” be similar to relationship between “scan” and “scanner”. Eventually, the constructs could come to represent something like “design patterns”, something that you can assume to work in some specific way.
Well, the design pattern called Factory definitely has its own «-Factory» suffix. And predicates often get an affix of one or two characters («is», «-p», «?»). And Hungarian notation was used.
My fear is that humans are actually too good at pattern matching, so if all you have is «-er», you will get «parser» regardless of whether it tries to parse a prefix or requires a whole-string match.
Do you hope that using a spoken language with a lot of morphological modifiers as a base will affect the culture to create enough new modifiers for smaller patterns? I mean enough to avoid combining any dangerous-to-combine notions. I find this plausible, but not guaranteed; I guess naming things in Esperanto could be a way to try.
I don’t know really, but it might be worth a try. At least when a programmer talks to another programmer in person, they use natural language to get through the idea. This is often (at least in my experience) superior to just reading the code. So, maybe, if we were able to take a bit of this person-to-person communication in convey it via the code, it would help to some extent.
Well, in person-to-person communication there is not only different naming (I would be surprised if some structured morphology not used in the variable naming would arise), there are different protocols for manging the level of details. You can get an overview that is not just «N levels deep in the call tree». Sometimes abstractions are also intentionally lossy, which you are usually not allowed to do in code.
Some things depend on feedback, there is some research into allowing zoom in/zoom out for abstraction levels, but improving state of art in the area of zooming out the programming abstractions would definitely be valuable.
Yep. That’s why I said “a bit” :)
My point was also that we currently have more tools for lexical part than for grammatical part. Is morphology still where the best return on effort is? (I honestly do not know)
I don’t think this would help with the tooling. However, it would decrease the amount of lexical baggage which in turn could help with, say, keeping the learning curve flat, or, maybe, returning to old code years later, remembering just the core concepts and being able to get up to speed immediately.
I meant tools in a wider sense including conceptual tools.
Intuitively, a flat learning curve is not something you can achieve in an experiment (the morphology has to be learnt first). So this part is hard to know (getting data from Esperanto taking off only for code identifiers and comments sounds a bit optimistic).
It would be of course interesting if there were some subset that you could try with moderate effort and then tell a success story.
The next step, being code negative in other peoples software repositories via sharing knowledge alone :)
The bullet point on my resume that gets the most comments from interviewers says:
That’s the very crux of the problem. How would you shared knowledge without writing code? Well, there’s still an option to write academic papers, but given the rift between compsci academia and practicioners of programming I would expect it not to be very efficient.
Well you can talk to people.
Think about it in memetic terms.
The idea is a meme. The code is its reproductive organ. The code is ‘useful’ so that it can lure its prey (a living human brain). Once the code is used the idea is repeatedly injected into the brain.
Compare that to talking to people where the idea is basically let floating in the space to be voluntarily accepted or not.
The former approach is much more efficient.
Ideas spread fine on their own. For example I’m about to convince you of this without a single line of code. There’s no need to push things into formal language when they make sense in nonformal language. I don’t need to tell you the steps of how to build a boat for you to realize that some method of traveling over water is good. In fact I’d argue that if I told everyone the exact steps to build a boat most would miss the point about what the boat is for. They’d get caught up in the details and fail to capture the bigger picture.
English descriptions with formal specifications and/or pseudocode accompanying them in a formalism that’s simple. That was the standard for high-assurance security. It worked consistently so long as the formalism could handle what they were describing. The caveat that the teams needed at least one specialist to train them on and help them with the formalism. If we’re talking CompSci, that could just become another 101 course where people are exposed to a few easy ones.