There was a recent discussion about whether file-based APIs were better or worse the specific ones (https://lobste.rs/s/ckqzbn/why_filesystems_have_loose_coupling_your).

I think a lot of the pros and cons of that API choice are illuminated by unveil.

Things can always be implemented either way (layered on top of the generic mechanism, or given their own specific mechanism). But if you then enhance the general mechanism with a new feature, you get to use that for all of the functionality you have layered on top.

(As a side thought - it’s kind of a nice thought experiment: If the only syscalls were open/close/read/write and friends - how would you (a) provide an interface for all other system calls and (b) put them together in a filepath hierarchy so that unveil() grouped things nicely.)

For the ‘transfer repo’ use case, you can just tgz the repo (which will include the .git dir and thus the whole history).

I didn’t know you could use a bundle to package changesets though - that’s really useful.

The “who is client and who is server” question seems unfair.

There is a controlling level…analogous to the shell, which has the job of wiring up the inputs and outputs.

Strawman solution for RPC-over-HTTP…..each line-at-a-time service also accepts a destination url, which accepts line-at-a-time input. So every service reads the http op, performs its function, then pushed onto the next. The final service can be the ‘append to file X’ service.

If we’re mimicking Unix cmdline we can also have the services conspire to ‘batch’ (i.e. block buffer) lines for efficiency.

RPC may have several problems, but it doesn’t seem this is one of them (it seems no worse then Unix pipeline)

I don’t buy it because the real protocol is what you read and write from the file, not that you can read and write files. And if the “file” is a directory, what do the filenames you read and write from/to it mean?

So is there really any difference between open(read("/net/clone")) and net_clone();? The author seems to say the former is more loosely coupled than the latter because the only methods are open and read on the noun that is the file…. but really, you are stating exactly the same thing as the “verb” approach (if anything, I’d argue it is more loosely typed than loosely coupled). If a new version wants to add a new operation, what’s the difference between making it a new file that returns some random data you must write code to interpret, and a new method that returns some data you must write code to use?

On the EINTR/retry issue, it’s not clear to me why a simple ‘return to userspace and retry the instruction’ is harder to implement.

There is kernel-side state associated with the fd (seek position, network buffers etc) but these need to be maintained anyway for an EINTR return.

Put another way - what work can the kernel avoid in the ‘return EINTR and let the userspace application call the system call again’ scenario which is required in the ‘return to userspace at PC-1’?

Don’t all VCSs have tools to modify history? I think svnadmin does: http://oliverguenther.de/2016/01/rewriting-subversion-history/ (assuming there aren’t any blockchain-based VCSs. I daren’t look)

If the distinction being drawn is ‘admin’ vs ‘user’ tooling, I guess - like workflow - git punts that to the surrounding culture and environment (as it does “which version is the ‘master’” - which is the same feature/bug of any DVCS).

I admit I like being able to say “v234” but really, what that means is “v234 of the (single) upstream repo which can change any time the upstream repo manager runs svnadmin”.

There’s nothing to stop github putting a sequential “v1, v2, v3, …” on commits to master or otherwise blessing some workflow.

I think the differences aren’t so much about features + capability and tooling as culture.

This problem (test assertion not running) is a general problem in most testing frameworks which require you to call a ‘fail’ (directly or indirectly) as the only way to signal a problem. Sometimes you fail to call fail.

Perl produced a test system called TAP (Test Anything Protocol) - now ported to other environments: https://testanything.org/

This uses an approach where:

• both positive (ok) and negative (not ok) assertions are reported
• the test begins with a statement of the total number of assertions expected (you can finish with this too)
• a test whose number of assertions doesn’t match the plan (too many or too few) has failed

There is a small piece of extra work (when you add a test, you have to increase the planned number), but this is minor and the approach guards against tests failing to run for any reason (if your test fails so drastically that it can’t even emit a plan, the test harness will/should report that).

(Here’s one perl API to emit TAP: http://perldoc.perl.org/Test/More.html)

I think I’ve read this paper a half dozen times now after seeing someone or other wax lyrical about it online. And I just don’t get why people like it so much. Which means either I’m too much of an insider and this is old news, or I don’t appreciate what I don’t know.

Part of the problem is that it seems to overstate its contributions. It isn’t really “identifying” causes of complexity. Fred Brooks pointed out essential vs accidental complexity back in 1975, and there’s been a steady stream of articulation ever since. To any Haskeller the point that complexity comes from state is old news. Relational algebra is ancient. At best OP is laying things out in a new way.

This time I figured I’d speed up my rereading time by instead chasing down past threads. And I discovered a decent summary. This was helpful, because it helped me to focus on the ‘causes of complexity’ portion of OP.

Causes of complexity according to OP:

• State (difficulty of enumerating states)
• Control (difficulty of choosing an ordering)
• Concurrency
• Volume
• Duplicated/dead code, unnecessary abstraction

Compare the list I made a couple of years ago:

• Compatibility. Forcing ourselves to continue supporting bad ideas.
• Vestigial features. Continuing to support stuff long after it’s been obsoleted by circumstances. Simply because it’s hard to detect obsolescence.
• People churn. Losing institutional fingerspitzengefühl about the most elegant place to make a given change. Or knowing when some kludge has become obsolete.

Comparing these two lists, it looks like there’s a tension between the top-down and bottom-up views of software management. In the bottom-up view people seem to think about software like physics, trying to gain insight about a system by studying the atoms and forces between atoms. You tend to divide complexity into state and order, essential and accidental. Reductionism is the occupational hazard.

In my top-down view I tend to focus on the life cycle of software. The fact that software gets more complex over time, in a very tangible way. If we could avoid monotonically adding complexity over time, life would be much better. Regardless of how many zeroes the state count has. In this worldview, I tend to focus on the stream of changes flowing into a codebase over time, alongside the stream of changes happening to its environment. This view naturally leads me to categorize complexity based on its source. Is it coming from new feature requirements, or changes to the operating environment? How can I keep my raft slender as I skim the phase transition boundary between streams?

The blind spot of the bottom-up view is that it tends to end up at unrealistic idealizations (spherical cows as @minimax put it in this thread). The blind spot of the top-down view is that there’s a tendency to under-estimate the complexity of even small systems. Blub. The meme of the dog saying “this is fine” while surrounded by flames.

It seems worth keeping both sides in mind. In my experience the top-down perspective doesn’t get articulated as often, and remains under-appreciated.

Huh, what would a federated message board look like? I guess I could see a reddit-like one where each sub could be on a different server, but you’d have a shared account around them all. Still one server per forum, so you can have consistent ordering of stories and comments. I’m not really sure what the benefit is to anyone of having a shared account among a ton of federated board servers, though. It just preserves reddit weirdness like sharing massively different karma amounts between joke boards and deep research boards.

Lobsters is meant to have one main page though. How would you do consistent ordering of the front page if stories were federated?

The offhand ‘even perl’ in there struck me as unfair. It reminds me that perl is actually pretty fast (specifically at startup, but my recollection was also that it runs quickly):

$ time for i in `seq 1 1000`; do perl < /dev/null; done

real    0m2.786s
user    0m1.337s
sys     0m0.686s

$ time for i in `seq 1 1000`; do python < /dev/null; done

real    0m19.245s
user    0m9.329s
sys     0m4.860s

$ time for i in `seq 1 1000`; do python3 < /dev/null; done

real    0m48.840s
user    0m30.672s
sys     0m7.130s

Seen your update including perl. You ask:

Then the next question is: does it make programming easier (i.e. thinking more about programming and less about naming) or harder (i.e. naming things is documentation).

and the answer to that is that if you’re writing big, maintained perl programs, you typically use explicitly named lexicals. Except where the scope is very short. Not least because the un-named abbreviations have dynamic not lexical scope:

#!/usr/bin/perl
use 5.10.0;

for (qw/foo bar baz/) {
  do_stuff();
  say;      # output $_
}

sub do_stuff() {
  say "wtf $_" if /z/;
}  

$ perl tt.pl
foo
bar
wtf baz
baz

It’s not fashionable, but perl took from Larry Wall’s linguist background the concept of ‘it’ - i.e. the “thing we are talking about”.

It’s spelt “$_” explicitly in perl, but also many operations use $_ implicitly if no arg is given (e.g. regex/sub, print, etc). Also the looping constructs (for/map/grep) bind $_ to the ‘current element’. So you can:

print for @lines;

and have:

• the for loop loop over each element in the array/list
• bind ‘$_’ as a (mutable!) alias to each list element
• invoke print (which defaults to $_ in the absence of other args)

Or:

while (<STDIN>) {   # Loop over each line on input
  chomp;            # remove trailing \n
  s/foo/bar/;       # regex-and-replace
  say;              # Print to stdout with \n
}

The schwartzian transform (https://en.wikipedia.org/wiki/Schwartzian_transform) uses this, and also the convention/feature that in a block/lambda passed to ‘sort’, the two items under comparison are ‘$a’ and ‘$b’. Which are hence sufficiently magic that you should never use them for any other purpose in perl :-)

Is the birthday paradox correct in this case? We’re not looking for any two coins the same colour, we’re looking for a coin the same colour as the one we already have?

Does that kill the idea? Perhaps I’ve misunderstood, but is it saveable with the notion that the system can know:

• the time a config file was written/last read
• the version of the language on the system in use at that time
• language constructs which have changed since then

Think about something like ‘go fix’ which programmatically move you from one version to another.

Either at language upgrade time, or at config file load time, you could run language upgrade steps and checks.

Thanks for showing this.

I had fun recently writing my first CPU emulator (for the Z80) in go, which morphed into an assembler, disassembler and ultimately a simple ZX spectrum emulator. https://github.com/jbert/zog

Got as far as a few key games running well. It is the most fun I’ve had with a compiler since the cryptopals challenges.

To which I should probably return since there are more and I should have a go at learning some rust.

I think there’s a lesson here, but mining nuggets out of random email posts isn’t the best delivery mechanism.

I question the assertion that the interpreting slowdown is exponential; whether it takes a few dozen cycles per statement (as in a bytecode interpreter) or a few thousand (as in a pure AST interpreter), the same number of statements would need to be executed, and so the slowdown is roughly linear no matter how deep the call stack gets. The exception to that would be if the interpreter were itself getting interpreted for a subroutine call, and I can’t see any reason that an implementation would do such a thing.

That said, one interesting place you might look for purely interpreted applications is in the Forth family; it’s fairly common to implement forth using some form of threaded code, which is effectively an AST modulo the fact that neither the S nor the T really apply to forth. You don’t see many programs written in forth directly (though apparently Chuck Moore has written a VLSI layout package), but postscript is fairly common and every implementation I’ve seen has been a pure interpreter.

Another possibility, also outside the realm of traditional languages, is TeX, which seems to be completely uncompileable (even to bytecode); I’d definitely consider LaTeX to be a large application written entirely in TeX.

Is it:

• operations in a program are implicitly sequenced by their syntactic order (execute this line, then the next)
• it is not obvious whether that ordering is logically necessary (e.g. due to a data dependency) or not (this just the order I expressed the list of things which need to happen)
• concurrency is the explicit relaxation of that implicit syntactic ordering, leaving only the true dependencies (go foo(); go bar(ch); go baz(ch))

The benefits of this are increased clarity and possibly modularity, due to explicit expression of the real dependencies.

Parallelism is something you can do to take additional advantage of the concurrency. You can’t just take a normal program and execute every statement simultaneously, since you can run into the implicit dependencies. But you can take a program marked up for concurrency and execute the independent parts of it in parallel.

Something I think important in this kind of discussion is a bit more thought to what to do if your preconditions fail.

I’ve seen code in the past which seems to have evolved along the lines of:

• use this arg passed in
• have a crash report, arg was null
• wrap function body in “if (arg != null) { … }” (with no ‘else’)
• crash fixed, yay - commit and move on

Now rearranging that as a guard clause gets rid of one annoyance (code indent etc) but if the action taken if the guard clause fails is just “return”, we’ve missed the opportunity to fix something.

If some of the args don’t meet some criteria, that might be more reasonably an exception or panic() - it could well be a logic error in some other part of the program. Some times it makes sense for a function to be a no-op, but it’s pretty rare (the ‘draft timer’ in the article).

This is a pretty nice justification for the whole conventional belief system that leads to “abstraction”, “coupling vs cohesion”, “SOLID”, etc. But that whole memeplex is dangerously misguided. It has a kernel of truth, but it’s incomplete, and it’s dangerous because the vast majority of programmers seem to get brainwashed by it until they’re blind to anything else.

To see how it’s misleading, let’s focus on this sentence:

..a triangle is actually very highly interconnected for the number of nodes it has..

This is a telling sentence, because it highlights that this entire notion of ‘complexity’ is really about density of complexity. Which means that if you just took your triangle and started spliting the vertices into lots of nodes, so that it has lots of nodes along each edge of the triangle, the whole would seem a lot less complex. But you haven’t really made the whole any less complex, you’ve just spread the complexity out into a smooth thin layer. The inherent complexity of the system remains. It’s just harder to find; randomly selected nodes/modules seem simple, and you have to jump through a lot of hoops to find the ‘complex node’ that invariably gets the lion’s share of updates.

The whole thing takes me back to my childhood years, when I would shove my stuff into my closet to try to convince my mom I’d cleaned up my room.

Abstraction is useful, and it’s useful to think about the right places to draw module boundaries. But as a rule of thumb, mistrust anybody who tries to pontificate about the difference between simplicity and complexity merely by making reference to module/interface boundaries, without any attention to what the system does.

So much for tearing down somebody else who in fairness wrote very elegantly indeed. Can I be constructive instead? I’ll suggest an alternative memeplex for thinking about complexity that gets a lot less attention than it should. Complexity comes from the state space of inputs a system needs to handle, and the number of regimes this state space gets broken down into. (Imagine something like the characteristics of a transistor.) The more regimes a state space has, the more intrinsically complex the domain is.

If we could demarcate this state space, and make the regimes in the state space explicit in our representation of the system – rather than implicit in lines of code as happens today – I think we’d make far larger strides in controlling complexity than any number of attempts at redrawing internal boundaries between sub-systems. In particular, we’d be able to detect over-engineering and architecture astronomy: they would be situations where a code has significantly higher complexity than the domain it’s trying to address.

I think such a representation has to start at (no surprises for people here who’ve heard me ranting before) tests. Tests are great because they’re like a Fourier transform applied to the conventional representation of a program as code. Instead of focusing on what to do at each point in time for a program, looking at a program as a collection of tests tells you how the entire trajectory of processing needs to go for the major regimes in the input space. Tests allow you to get past what your current program does, and think about the essential things any program has to do to achieve the same ends.

(I’ve written about this idea before. Hopefully this latest attempt is clearer.)