Threads for chilton
-
jamesnvc
5 days ago
|
link
| on:
"...an audiophile forum started debating which versions of memcpy had the highest sound quality"
An interesting…if not rebuttal, counter-point to just mocking the audiophiles: https://fediscience.org/@marcbrooker/110017159893728862
-
For those not wanting to click on the link, the possible explanation was that different memcpy implementations use different pipelines and so may rigger different power states. This is possibly more plausible than you might at first imagine. I had a computer some years ago where I could hear SSE-heavy workloads because the EM leakage from the CPU hitting those pipelines was picked up in the sound card, so an SSE-based memcpy would give worse sound than one using integer pipelines (possibly - I doubt SSE loads and stores were actually audible on that machine). This is why sensible audiophiles that I’ve known recommend a cheap USB audio adaptor over any expensive sound card for analogue output: simply moving the DAC a metre away from the electrical noise does more good than anything else. Outputting optical audio to an external decoder and amplifier can be better because then your audio system can be completely electrically isolated from the computer (there are a lot of chips in a typical computer case that are little radio broadcasters and any nearby wire may accidentally act as an antenna receiving the signal).
The other plausible explanation that I considered was jitter. Probably not the case 10 years ago, but 20 years ago the ring buffers that sound cards could DMA from were quite small and you could quite clearly hear when the CPU wasn’t filling them fast enough. A slow memcpy being preempted at the wrong time may well have caused artefacts. If you can fill the buffer easily within a single scheduling quantum and yield then you’ll be priority boosted the next time and so you’ll easily keep up playing music. If you’re occasionally being preempted mid-copy, then you’ll suffer from weird artefacts as the card overtakes the CPU and plays a few ms of the previous sample instead of the new one.
I don’t think either of these were the case here though, and this is why sensible audiophiles do random double-blind tests on themselves: play the thing n times, where half are with one version, half with the other. If you don’t get a statistically significant number of guesses as which is the better one landing on the same implementation, you’re just kidding yourself (which is very easy to do with audio).
-
Those counterpoints are good. And he doesn’t much mock the audiophiles. High end audio is funny game. Since it involves chasing the limb of the diminishing returns curve, it naturally attracts eccentrics. The kind of people who take joy from the fact that a brake job on their Porsche cost ten times as much is does on a Toyota. For these folks, gold plated TOSLINK cables, $500 ethernet cables, and high fidelity audio grade ethernet switches are the stuff of which happy dreams are made. If that’s how eccentric people enjoy their money, more power to them.
What’s truly sad about this it ruins things for the people who have some sort of balance in their lives because there is point to be gained from climbing the diminishing returns curve in A/V equipment as a means of improving sound quality. The trick is that you don’t have to climb too far to greatly improve your experience.
-
I’m not involved in high-end audio, but it feels it has overlap with another field I’m tangentially aware of, high-end watch collection. If they’re similar, then the motivation driving participants is less the end goal of a perfect sound system or the perfect watch, but learning about the stuff that’s out there, trading for new used stuff, and generally being part of a community.
Specifically, it’s not that hard to incrementally increase ones collection by starting small, looking for used instances of the stuff you’re interested in, saving some discretionary income, buying up, etc. And for many people, that’s the actual fun part!
And while there’s an element of snake oil[1] and grift, in general both HE audio and watches are physical objects that can be inspected for quality. I.e. even if paying 6 figures or more for a turntable with a cast iron bed[2] is a lot of money in absolute terms, you are getting a physical object that has intrinsic worth, if only to other HE audio enthusiasts.
[1] I tried to find some examples and ran across this piece about expensive audio ethernet cables: https://arstechnica.com/gadgets/2015/07/gallery-we-tear-apart-a-340-audiophile-ethernet-cable-and-look-inside/. While excessively priced they are not outright fraudulent.
[2] price source: https://harpers.org/archive/2022/12/corner-club-cathedral-cocoon-audiophilia-and-its-discontents/
-
-
-
One good reason for having a low DNS TTL not mentioned in the article are DDNS setups.
Residential internet connections are flaky and sometimes a router cycles through multiple dynamic IPs in a matter of minutes, without anything the customer can do.
-
-
-
-
Anecdata: this is true in theory but I’m not sure in practice? Specifically, I used to get the same IPv4 address for weeks at a time - basically until my modem was rebooted. Then in 2014 ARIN entered phase 4 of their IPv4 exhaustion plan (triggered by them getting down to their last /8 block) and all of a sudden my modem’s IPv4 address refreshed far, far more often, IIRC every couple days.
I guess maybe this was not technically required though, and was potentially just my ISP overreacting? 🤷
CenturyLink in the Seattle area, FWIW.
-
At least here in Germany will have a different IPv4 address every 24 hours or on reconnect for far most of residential Internet access. The point with IPv6 is that you not only have one public. But again in Germany it‘s even hard to find a not regularly changing IPv6 prefix for residential Interner access. They think it‘s more Staat protection friendly… like cash. Crazy thoughts.
-
Ideally, a v6 provider would give you two subnets, one for inbound connections that remained stable, one for outbound connections that was changed frequently. Combined with the privacy extensions randomising the low 64 bits, this should make IP-based tracking difficult.
-
They think it‘s more Staat protection friendly…
The origin story of the 24h disconnect is that it used to be the differentiator between a leased line and a dial-up line, which belonged in different regulatory regimes (the most obvious aspect to customers has been cost but with a few backend differences, too). The approach has stuck since.
It’s also a (rather barebone) privacy measure against commercial entities, not the government: the latter can relatively easily obtain a mapping from IP to the address by giving a more-or-less refined reason.
-
-
-
-
-
-
-
That’s not going to be an easy problem to solve. We’ve embraced the current pseudo standards within home internet connectivity, maybe static, maybe dynamic IP, asymmetric speeds, CGNAT, no IPv6, etc. for so long that many people think that these are real industry standards with cost structures and cost savings behind them and we must suffer with them if we want cost effective internet connectivity at all. A lot of home ISP customers suffer from Stockholm syndrome.
-
-
I think that DynDNS.org is using a 60s timeout on A records so for a DynDNS/residential setup I think that the article author would approve of something like this:
foo.mydomain.com 14400 IN CNAME bar.my-dyndns-server.com…
bar.my-dyndns-server.com 60 IN A 192.168.1.100Specifically, I don’t think that the original author is complaining about the 60s TTL on records at
my-dyndns-server.comsince that company has to deal with the lack of caching in their DNS zones. He finds sub 1hr TTLs in the CNAME records to be a problem. And he finds CNAME TTLs shorter than the A record TTLs to be a bigger problem. Honestly even Amazon does this 60s TTL on dynamic resources trick.
-
-
Am I wrong here in assuming that ARM64 is big endian? If it is, would running Emacs on Raspberry Pi against an X server on my Intel mac will require this switch?
Update: According to quora ARM can be either but runs little endian.
-
To add a little context: ARM32 and 64 can support either but since iOS and Android became massive drivers in ARM development, basically all the software for them has been little-endian. But it was not uncommon on ARM before that, for example the Gameboy Advance is little-endian.
-
64-bit Arm dropped support for a big endian mode, though it does provide some efficient byte swapping instructions in case anyone wants to define a big-endian ABI (no one has).
-
I hate endianness so much.
Little endian is the one that makes sense to me, but honestly if we could all just use the same thing everywhere I’d be happy.
What arguments do people have in favor of big endian?
-
The only one I can think of is hex dumps. In the west, we write our numbers, like two hundred, thirty-eight, in big-endian format: 238. [1] A big-endian dump of, say, addresses:
00000230: 00 00 01 15 00 00 00 12 00 00 00 2E 08 04 9D 68 ...............h 00000240: 00 00 00 04 00 17 00 11 00 00 00 92 08 04 9D 6C ...............lis slightly easier to read than little-endian:
00000230: 15 01 00 00 12 00 00 00 2E 00 00 00 68 9D 04 08 ............h... 00000240: 04 00 00 00 11 00 17 00 92 00 00 00 6C 9D 04 08 ............l...One could print the hex dump out “backwards”:
...h............ 08 04 9D 68 00 00 00 2E 00 00 00 12 00 00 01 15 :00000230 ...l............ 08 04 9D 6C 00 00 00 92 00 17 00 11 00 00 00 04 :00000240but then any text looks weird:
ulcni/rsu/...... 75 6C 63 6E 69 2F 72 73 75 2F 01 00 00 00 01 01 :00001660 /edulcni/rsu/.ed 2F 65 64 75 6C 63 6E 69 2F 72 73 75 2F 00 65 64 :00001670 cg/bil/rsu/.stib 63 67 2F 62 69 6C 2F 72 73 75 2F 00 73 74 69 62 :00001680 il-tahder-683i/c 69 6C 2D 74 61 68 64 65 72 2D 36 38 33 69 2F 63 :00001690 dulcni/6.4.3/xun 64 75 6C 63 6E 69 2F 36 2E 34 2E 33 2F 78 75 6E :000016A0Personally, I like big-endian over little-endian, but then, I grew up programming Motorola CPUs (big-endian) and found them much more fun than anything produced by Intel (x86—uhg), but over time I’ve gotten used to reading little-endian data in hex dumps (still hate it though).
[1] In the Arabic world, where we get our modern numbering system from, they write their numbers in little-endian format, but given that they write right-to-left, it ends up looking the same to us.
-
Yeah, debugging memory dumps in little-endian makes me go a little crazy. It’s much easier to visually parse big-endian (speaking as someone who started on the 6502, but did most of my heavy debugging work on PowerPC).
-
network pcaps are the same, hence it also being called ‘network byte order’ in the various RFCs + htons / nthos. The wrong endianness won.
-
-
-
Little endian hardware is a little easier to design for small weak slow systems, big endian hardware is a little easier to design for big fast powerful systems. These days, the differences are basically irrelevant to software people, it mattered a lot more in Ye Olden Days when hardware was 10,000x more expensive. Big endian was a lot more common among the mainframes and minicomputers and workstations of the 1960’s-80’s, but those have mostly been killed off by x86 and ARM, which grew out of small systems, so now 98% of everything uses little-endian.
-
Little endian hardware is a little easier to design for small weak slow systems
I’m curious about the details. At hardware level isn’t it basically the same, since you can literally swap connections to make things big or little endian for free? Or is there something that uses the opportunity to pipeline processing as bytes come in?
-
I’m also sceptical about this claim. I don’t see endian affecting hardware complexity at all. The main advantage of little endian is in software: you can truncate on load without needing address arithmetic and you don’t need to know how big a value is to load the low n bytes (you rarely want to load the high n bits).
-
From Wikipedia:
Some operations in positional number systems have a natural or preferred order in which the elementary steps are to be executed. This order may affect their performance on small-scale byte-addressable processors and microcontrollers. However, high-performance processors usually fetch multi-byte operands from memory in the same amount of time they would have fetched a single byte, so the complexity of the hardware is not affected by the byte ordering.
Addition, subtraction, and multiplication start at the least significant digit position and propagate the carry to the subsequent more significant position. On most systems, the address of a multi-byte value is the address of its first byte (the byte with the lowest address). The implementation of these operations is marginally simpler using little-endian machines where this first byte contains the least significant digit.
Comparison and division start at the most significant digit and propagate a possible carry to the subsequent less significant digits. For fixed-length numerical values (typically of length 1,2,4,8,16), the implementation of these operations is marginally simpler on big-endian machines.
I’ve also heard that little-endian also make it easier to write hardware that deals with byte-serial I/O like RS-232, ‘cause you don’t need to store much context. Like I said, these actual differences are now basically irrelevant.
-
Comparison and division start at the most significant digit and propagate a possible carry to the subsequent less significant digits. For fixed-length numerical values (typically of length 1,2,4,8,16), the implementation of these operations is marginally simpler on big-endian machines.
This is a weird thing for Wikipedia to assert and I strongly suspect that it’s nonsense. The order of bits within a register is completely independent of the byte order in memory. Modern processors are load-store architectures internally.
-
-
-
-
-
-
-
-
-
chilton
6 months ago
|
link
| on:
Schedule Cronjob for the First Monday of Every Month, the Funky Way
Too magic for me. You can add a date/time/day screen on the shell script behind a cron job in less than ten lines of shell. I’m not happy that you have to look both at the script and the crontab to figure it out but I’m happier with that. Best would probably be something like:
0 0 1-7 * * : 'My First Monday only cronjob' && [ $(date '+%a') -eq 'Mon' ] && run-mycronjob.shAssuming I have the fields right.
-
-
-
Absolutely true. Also still arguably better than the original solution. The original solution is probably native to the cron package that the author uses. It’s not guaranteed to work anywhere else. Further, like a lot of tools in Unix, cron is terse as a result of its heritage from systems that would today be considered puny. Put a different way, me in 1985 would be astounded at how much computing power I can buy in 2022 in the form of a Raspberry Pi Zero. You should not embrace that terseness. The Systemd/Timer solutions are much better than the cron solutions presented. And I would say that if you are on a box without systemd, you’d still be better off to implement the “only on the first Monday of the month check” in shell rather than through clever manipulation of the cron parameters.
-
In retrospect, I should have emphasized in the post that the described trick is primarily a trick, a hack, an interesting curiosity.
PS. It’s really hard to buy a Raspberry Pi Zero in 2022, unfortunately. I wanted one for a hobby hardware project, I ended up getting a Radxa Zero. But point stands, the price/performance/size of computers in 2022 is awesome.
-
-
-
-
-
The biggest problem with OpenSSH security in this regard is the fact that the locally managed PKI toolkits aren’t developed well enough yet. I would welcome an open source PKI tool that generated certificates such that:
- they were compatible with OpenSSH;
- they worked with OpenBSD’s ikev2 PKI infrastructure for IPSec;
- they could be used internally for https, ldaps, and other server identification problems.
This would more than adequately address the problem of SSH priv/pub keys by replacing them with certificates that have a manageable lifespan, yet remain as convenient as the key infrastructure that they replace. Also, note well that this is really a problem of the documentation around the potential new project more than anything else.
-
After years of using git, which I think is a result of your influence over a set of mutual colleagues, I can’t imaging using a VCS without git’s capabilities. However, the quote: “git is a version control toolbox masquerading as a complete VCS” is pretty accurate. I can easily imagine using different UI over the current git internals.
Have you tried
magit modein Emacs?
-
The interface of Git and its underlying data models are two very different things, that are best treated separately.
The interface is pretty bad. If I wasn’t so used to it I would be fairly desperate for an alternative. I don’t care much for the staging area, I don’t like to have to clean up my working directory every time I need to switch branches, and I don’t like how easy it is to lose commit from a detached HEAD (though there’s always
git reflogI guess).The underlying data model however is pretty good. We can probably ditch the staging area, but apart from that, viewing the history of a repository as a directed graph of snapshots is nice. Captures everything we need. Sure patches have to be derived from those snapshots, but we care less about the patches than we care about the various versions we saved. If there’s one thing we need to get right, it’s those snapshots. You get reproducible builds & test from them, not from patches. So I think Patches are secondary. I used to love DARCS, but I think patch theory was probably the wrong choice.
Now one thing Git really really doesn’t like is large binary files. Especially if we keep changing them. But then that’s just a compression problem. Let the data model pretend there’s a blob for each version of that huge file, even though in fact the software is automatically compressing & decompressing things under the hood.
-
-
-
Do you actually use it, or do you just do
git commit -p, which only happens to use the staging area as an implementation detail?And how do you test the code you’re committing? How do you make sure that the staged hunks aren’t missing another hunk that, for example, changes the signature the function you’re calling? It’s a serious slowdown in workflow to need to wait for CI rounds, stash and rebase to get a clean commit, and push again.
-
-
I
git add -pto the staging area and then diff it before generating the commit. I guess that could be done without a staging area using a different workflow but I don’t see the benefit (even if I have to check git status for the command every time I need to unstage something (-: )As for testing, since I’m usually using Github I use the PR as the base unit that needs to pass a test (via squash merges, the horror I know). My commits within a branch often don’t pass tests; I use commits to break things up into sections of functionality for my own benefit going back later.
-
Just to add on, the real place where the staging area shines is with
git reset -p. You can reset part of a commit, amend the commit, and then create a new commit with your (original) changes or continue editing. The staging area becomes more useful the more you do commit surgery. -
-
I guess that could be done without a staging area using a different workflow but I don’t see the benefit
I don’t see the cost.
My commits within a branch often don’t pass tests;
If you ever need to
git bisect, you may come to regret that. I almost never usegit bisect, but for the few times I did need it it was a life saver, and passing tests greatly facilitate it.-
I bisect every so often, but on the squashed PR commits on main, not individual commits within a PR branch. I’ve never needed to do that to diagnose a bug. If you have big PRs, don’t squash, or don’t use a PR-based workflow, that’s different of course. I agree with the general sentiment that all commits on main should pass tests for the purposes of bisection.
-
-
-
I use git gui for committing, (the built in git gui command) which let’s you pick by line not just hunks. Normally the things I’m excluding are stuff like enabling debug flags, or just extra logging, so it’s not really difficult to make sure it’s correct. Not saying I never push bad code, but I can’t recall an instance where I pushed bad code because of that so use the index to choose parts of my unfinished work to save in a stash (git stash –keep-index), and sometimes if I’m doing something risky and iterative I’ll periodically add things to the staging area as I go so I can have some way to get back to the last known good point without actually making a bunch of commits ( I could rebase after, yeah but meh).
It being just an implementation detail in most of that is a fair point though.
-
I personally run the regression test (which I wrote) to test changes.
Then I have to wait for the code review (which in my experience has never stopped a bug going through; when I have found bugs, in code reviews, it was always “out of scope for the work, so don’t fix it”) before checking it in. I’m dreading the day when CI is actually implemented as it would slow down an already glacial process [1].
Also, I should mention I don’t work on web stuff at all (thank God I got out of that industry).
[1] Our customer is the Oligarchic Cell Phone Company, which has a sprint of years, not days or weeks, with veto power over when we deploy changes.
-
-
Author of the Jujutsu VCS mentioned in the article here. I tried to document at https://github.com/martinvonz/jj/blob/main/docs/git-comparison.md#the-index why I think users don’t actually need the index as much as they think.
I missed the staging area for at most a few weeks after I switched from Git to Mercurial many years ago. Now I miss Mercurial’s tools for splitting commits etc. much more whenever I use Git.
-
Thanks for the write up. From what I read it seems like with Jujutsu if I have some WIP of which I want to commit half and continue experimenting with the other half I would need to commit it all across two commits. After that my continuing WIP would be split across two places: the second commit and the working file changes. Is that right? If so, is there any way to tag that WIP commit as do-not-push?
-
Not quite. Every time you run a command, the working copy is snapshotted and becomes a real commit, amending the precis working-copy commit. The changes in the working copy are thus treated just like any other commit. The corresponding think to
git commit -pisjj split, which creates two stacked commits from the previous working-copy commit, and the second commit (the child) is what you continue to edit in the working copy.Your follow-up question still applies (to both commits instead of the single commit you seemed to imagine). There’s not yet any way of marking the working copy as do-not-push. Maybe we’ll copy Mercurial’s “phase” concept, but we haven’t decided yet.
-
-
-
-
Way I see it, the staging area is a piece of state needed specifically for a command line interface. I use it too, for the exact reason you do. But I could do the same by committing it directly. Compare the possible workflows. Currently we do:
# most of the time git add . git commit # piecemeal git add -p . # review changes git commitWithout a staging area, we could instead do that:
# most of the time git commit # piecemeal git commit -p # review changes git reset HEAD~ # if the changes are no goodAnd I’m not even talking about a possible GUI for the incremental making of several commits.
-
Personally I use
git add -pall of the time. I’ve simply been burned by the other way too many times. What I want is not to save commands but to have simple commands that work for me in every situation. I enjoy the patch selection phase. More often than not it is what triggers my memory of a TODO item I forgot to jot down, etc. The patch selection is the same as reviewing the diff I’m about to push but it lets me do it incrementally so that when I’m (inevitably) interrupted I don’t have to remember my place.From your example workflows it seems like you’re interested in avoiding multiple commands. Perhaps you could use
git commit -amost of the time? Or maybe add acommit-allalias?-
Never got around to write that alias, and if I’m being honest I quite often
git diff --cachedto see what I’ve added before I actually commit it.I do need something that feels like a staging area. I was mostly wondering whether that staging area really needed to be implemented differently than an ordinary commit. Originally I believed commits were enough, until someone pointed out pre-commit hooks. Still, I wonder why the staging area isn’t at least a pointer to a
treeobject. It would have been more orthogonal, and likely require less effort to implement. I’m curious what Linus was thinking.-
-
Hmm, that’s a tough one. Strictly speaking, none. But I can see the benefits.
Take Monocypher for instance: now it’s pretty stable, and though it is very easy for me to type
make testevery time I modify 3 characters, in practice I may want to make sure I don’t forget to do it before I commit anything. But even then there are 2 alternatives:- Running tests on the server (but it’s better suited to a PR model, and I’m almost the only committer).
- Having a pre push hook. That way my local commits don’t need the hook, and I could go back to using the most recent one as a staging area.
-
-
-
-
-
# most of the time
git add .
Do you actually add your entire working directory most of the time? Unless I’ve just initialized a repository I essentially never do that.
Here’s something I do do all the time, because my mind doesn’t work in a red-green-refactor way:
Get a bug report
Fix bug in foo_controller
Once the bug is fixed, I finally understand it well enough to write an automated regression test around it, so go do that in foo_controller_spec
Run test suite to ensure I didn’t break anything and that my new test is green
Add foo_controller and foo_controller_spec to staging area
Revert working copy (but not staged copy!) of foo_controller (but not it’s spec)
Run test suite again and ensure I have exactly one red test (the new regression test). If yes, commit the stage.
If no, debug spec against old controller until I understand why it’s not red, get it red, pull staged controller back to working area, make sure it’s green.
—
Yeah, I could probably simulate this by committing halfway through and then doing some bullshit with cherry-picks from older commits and in some cases reverting the top commit but, like, why? What would I gain from limiting myself to just this awkward commit dance as the only way of working? That’s just leaving me to cobble together a workflow that’s had a powerful abstraction taken away from it, just to satisfy some dogmatic “the commit is the only abstraction I’m willing to allow” instinct.
-
Do you actually add your entire working directory most of the time?
Yes. And when I get a bug report, I tend to first reproduce the bug, then write a failing test, then fix the code.
Revert working copy (but not staged copy!) of foo_controller (but not it’s spec)
Sounds useful. How do you do that?
-
-
Yes. And when I get a bug report, I tend to first reproduce the bug, then write a failing test, then fix the code.
Right, but that was just one example. Everything in your working copy should always be committed at all times? I’m almost never in that state. Either I’ve got other edits in progress that I intend to form into later commits, or I’ve got edits on disk that I never intend to commit but in files that should not be git ignored (because I still intend to merge upstream changes into them).
I always want to be intentionally forming every part of a commit, basically.
Sounds useful. How do you do that?
git add foo_controller <other files>; git restore -s HEAD foo_controllerand then
git restore foo_controllerwill copy the staged version back into the working set.
-
-
-
TBH, I have no idea what “git add -p” does off hand (I use Magit), and I’ve never used staging like that.
I had a great example use of staging come up just yesterday. I’m working in a feature branch, and we’ve given QA a build to test what we have so far. They found a bug with views, and it was an easy fix (we didn’t copy attributes over when copying a view).
So I switched over to views.cpp and made the change. I built, tested that specific view change, and in Magit I staged that specific change in views.cpp. Then I commited, pushed it, and kicked off a pipeline build to give to QA.
I also use staging all the time if I refactor while working on new code or fixing bugs. Say I’m working on “foo()”, but while doing so I refactor “bar()” and “baz()”. With staging, I can isolate the changes to “bar()” and “baz()” in their own commits, which is handy for debugging later, giving the changes to other people without pulling in all of my changes, etc.
Overall, it’s trivial to ignore staging if you don’t want it, but it would be a lot of work to simulate it if it weren’t a feature.
-
-
What’s wrong with the staging area? I use it all the time to break big changes into multiple commits and smaller changes.
I’m sure you do – that’s how it was meant to be used. But you might as well use commits as the staging area – it’s easy to commit and squash. This has the benefit that you can work with your whole commit stack at the same time. I don’t know what problem the staging area solves that isn’t better solved with commits. And yet, the mere existence of this unnecessary feature – this implicitly modified invisible state that comes and crashes your next commit – adds cognitive load: Commands like
git mv,git rmandgit checkoutpollutes the state, thengit diffhides it, and finally,git commit --amendaccidentally invites it into the topmost commit.The combo of being not useful and a constant stumbling block makes it bad.
-
-
Using e.g.
hg splitorjj split. The former has a text-based interface similar togit commit -pas well as a curses-based TUI. The latter lets you use e.g. Meld or vimdiff to edit the diff in a temporary directory and then rewrites the commit and all descendants when you’re done.-
That temporary directory sounds a lot like the index – a temporary place where changes to the working copy can be batched. Am I right to infer here that the benefit you find in having a second working copy in a temp directory because it works better with some other tools that expect to work files?
-
The temporary directory is much more temporary than the index - it only exists while you split the commit. For example, if you’re splitting a commit that modifies 5 files, then the temporary directory will have only 2*5 files (for before and after). Does that clarify?
The same solution for selecting part of the changes in a commit is used by
jj amend -i(move into parent of specified commit, from working-copy commit by default),jj move -i --from <rev> --to <rev>(move changes between arbitrary commits) etc.
-
-
-
I use git revise. Interactive revise is just like interactive rebase, except that it has is a
cutsubcommand. This can be used to split a commit by selecting and editing hunks likegit commit -p.Before git-revise, I used to manually undo part of the commit, commit that, then revert it, and then sqash the undo-commit into the commit to be split. The revert-commit then contains the split-off changes.
-
-
I don’t know, I find it useful. Maybe if git built in mercurials “place changes into commit that isn’t the most recent” amend thing then I might have an easier time doing things but just staging up relevant changes in a patch-based flow is pretty straightforward and helpful IMO
I wonder if this would be as controversial if patching was the default
-
-
What purpose does it serve that wouldn’t also be served by first-class rollback and an easier way of collapsing changesets on their way upstream? I find that most of the benefits of smaller changesets disappear when they don’t have commit messages, and when using the staging area for this you can only rollback one step without having to get into the hairy parts of git.
-
The staging area is difficult to work with until you understand what’s happening under the hood. In most version control systems, an object under version control would be in one of a handful of states: either the object has been cataloged and stored in its current state, or it hasn’t. From a DWIM standpoint for a new git user, would catalog and store the object in its current state. With the stage, you can stage, and change, stage again, and change again. I’ve used this myself to logically group commits so I agree with you that it’s useful. But I do see how it breaks peoples DWIM view on how git works.
Also, If I stage, and then change, is there a way to have git restore the file as I staged it if I haven’t committed?
-
Also, If I stage, and then change, is there a way to have git restore the file as I staged it if I haven’t committed?
Git restore .
-
-
-
Try testing your staged changes atomically before you commit. You can’t.
A better design would have been an easy way to unstage, similar to git stash but with range support.
-
-
Interesting, that would solve the problem. I’m surprised I’ve not come across that before.
In terms of “what’s wrong with the staging area”, what I was suggesting would work better is to have the whole thing work in reverse. So all untracked files are “staged” by default and you would explicitly un-stage anything you don’t want to commit. Firstly this works better for the 90% use-case, and compared to this workaround it’s a single step rather than 2 steps for the 10% case where you don’t want to commit all your changes yet.
The fundamental problem with the staging area is that it’s an additional, hidden state that the final committed state has to pass through. But that means that your commits do not necessarily represent a state that the filesystem was previously in, which is supposed to be a fundamental guarantee. The fact that you have to explicitly stash anything to put the staging area into a knowable state is a bit of a hack. It solves a problem that shouldn’t exist.
-
The way I was taught this, the way I’ve taught this to others, and the way it’s represented in at least some guis is not compatible.
I mean, sure, you can have staged and unstaged changes in a file and need to figure it out for testing, or unstage parts, but mostly it’s
edit->stage->commit->push.That feels, to me and to newbies who barely know what version control is, like a logical additive flow. Tons of cases you stage everything and commit so it’s a very small operation.
The biggest gripe may be devs who forget to add files in the proper commit, which makes
bisecthard. Your case may solve that for sure, but I find it a special case of bad guis and sloppy devs who do that. Also at some point the fs layout gets fewer new files.-
Except that in a completely linear flow the distinction between edit and stage serves no purpose. At best it creates an extra step for no reason and at worst it is confusing and/or dangerous to anyone who doesn’t fully understand the state their working copy is in. You can bypass the middle state with
git add .; git commitand a lot of new developers do exactly that, but all that does is pretend the staging state doesn’t exist.Staging would serve a purpose if it meant something similar to pushing a branch to CI before a merge, where you have isolated the branch state and can be assured that it has passed all required tests before it goes anywhere permanent. But the staging area actually does the opposite of that, by creating a hidden state that cannot be tested directly.
As you say, all it takes is one mistake and you end up with a bad commit that breaks bisect later. That’s not just a problem of developers being forgetful, it’s the bad design of the staging area that makes this likely to happen by default.
-
I think I sort of agree but do not completely concur.
Glossing over the staging can be fine in some projects and dev sloppiness is IMO a bigger problem than an additive flow for clean commits.
These are societal per-project issues - what’s the practice or policy or mandate - and thus they could be upheld by anything, even using the undo buffer for clean commits like back in the day. Which isn’t to say you never gotta do trickery like that with Git, just that it’s a flow that feels natural and undo trickery less common.
Skimming the other comments, maybe
jjis more like your suggestion, and I wouldn’t mind “a better Git”, but I can’t be bothered when eg.gitlessiirc dropped the staging and would make clean commits feel like 2003.
-
-
-
-
-
-
It’s usually not that hard.
https://lobste.rs/s/yi97jn/is_it_time_look_past_git#c_ss5cj3
-
-
-
-
Everyone seem to suppose I would like to ditch the workflows enabled by the staging area. I really don’t. I’m quite sure there ways to keep those workflows without using a staging area. If there aren’t well… I can always admit I was wrong.
-
Well, what I prize being able to do is to build up a commit piecemeal out of some but not all of the changes in my working directory, in an incremental rather than all-in-one-go fashion (ie. I should be able to form the commit over time and I should be able to modify a file, move it’s state into the “pending commit” and continue to modify the file further without impacting the pending commit). It must be possible for any commit coming out of this workflow to both not contain everything in my working area, and to contain things no longer in my working area. It must be possible to diff my working area against the pending commit and against the last actual commit (separately), and to diff the pending commit against the last actual commit.
You could call it something else if you wanted but a rose by any other name etc. A “staging area” is a supremely natural metaphor for what I want to work with in my workflow, so replacing it hardly seems desirable to me.
-
How about making the pending commit an actual commit? And then adding the porcelain necessary to treat it like a staging area? Stuff like
git commit -p fooif you want to add changes piecemeal.-
No. That’s cool too and is what tools like
git reviseandgit absorbenable, but making it an actual commit would have other drawbacks: it would imply it has a commit message and passes pre-commit hooks and things like that. The staging area is useful precisely for what it does now—help you build up the pieces necessary to make a commit. As such it implies you don’t have everything together to make a commit out of it. As soon as I do I commit, then if necessary--ammend,--edit, orgit reviselater. If you don’t make use of workflows that use staging then feel free to use tooling that bypasses it for you, but don’t try to take it away from the rest of us.-
pre-commit hooks
Oh, totally missed that one. Probably because I’ve never used it (instead i rely on CI or manually pushing a button). Still, that’s the strongest argument so far, and I have no good solution that doesn’t involve an actual staging area there. I guess it’s time to change my mind.
-
I think the final word is not said. These tools could also run hooks. It may be that new hooks need to be defined.
Here is one feature request: run git hooks on new commit
-
I think you missed the point, my argument is that the staging area is useful as a place to stage stuff before things like commit related hooks get run. I don’t want tools like
git reviseto run precommit hooks. When I usegit revisethe commit has already been made and presumably passed precommit phase.-
For the problem that
git revise“bypasses” the commit hook when using it to split a commit, I meant the commit hook (not precommit hook).I get that the staging area lets you assemble a commit before you can run the commit hook. But if this was possible to do statelessly (which would only be an improvement), you could do without it. And for other reasons, git would be so much better without this footgun:
Normally, you can look at
git diffand commit what you see withgit commit -a. But if the staging area is clobbered, which you might have forgot, you also have invisible state that sneaks in!-
Normally, you can look at git diff and commit what you see with git commit -a.
Normally I do nothing of the kind. I might have used
git commit -aa couple times in the last 5 years (and I make dozens to hundreds of commits per day). The stattefullness of the staging area is exactly what benefits my workflow and not the part I would be trying to eliminate. The majority of the time I stage things I’m working on from my editor one hunk at a time. The difference between my current buffer and the last git commit is highlighted and after I make some progress I start adding related hunks and shaping them into commits. I might fiddle around with a couple things in the current file, then when I like it stage up pieces into a couple different commits.The most aggressive I’d get is occasionally (once a month?) coming up with a use for
git commit -u.A stateless version of staging that “lets you assemble a commit” sounds like an oxymoron to me. I have no idea what you think that would even look like, but a state that is neither the full contents of the current file system nor yet a commit is exactly what I want.
-
-
-
-
-
Why deliberately make a mess of things? Why make a discreet concept of a “commit” into something else with multiple possible states? Why not just use staging like it is now? I see no benefit to jurry rigging more states on top of a working one. If the point is to simplify the tooling you won’t get there by overloading one clean concept with an indefinite state and contextual markers like “if commit message empty then this is not a real commit”.
-
Empty commit message is how you abort a commit
-
-
-
-
Again, what’s the benefit?
Sure, you could awkwardly simulate a staging area like this. The porcelain would have to juggle a whole bunch of shit to avoid breaking anytime you merge a bunch of changes after adding something to the fake “stage”, pull in 300 new commits, and then decide you want to unstage something, so the replacement of the dedicated abstraction seems likely to leak and introduce merge conflict resolution where you didn’t previously have to worry about it, but maybe with enough magic you could do it.
But what’s the point? To me it’s like saying that I could awkwardly simulate
if,whileandforwithgoto, or simulate basically everything with enoughNANDs. You’re not wrong, but what’s in it for me? Why am I supposed to like this any better than having a variety of fit-for-purpose abstractions? It just feels like I’d be tying one hand behind my back so there can be one less abstraction, without explain why having N-1 abstractions is even more desirable than having N.Seems like an “a foolish consistency is the hobgoblin of little minds” desire than anything beneficial, really.
-
Again, what’s the benefit?
Simplicity of implementation. Implementing the staging area like a commit, or at least like a pointer to a
treeobject, would likely make the underlying data model simpler. I wonder why the staging area was implemented the way it is.At the interface level however I’ve had to change my mind because of pre-commit hooks. When all you have is commits, and some tests are automatically launched every time you commit anything, it’s pretty hard to add stuff piecemeal.
-
Yes, simplicity of implementation and UI. https://github.com/martinvonz/jj (mentioned in the article) makes the working copy (not the staging area) an actual commit. That does make the implementation quite a lot simpler. You also get backups of the working copy that way.
-
Simplicity of implementation.
No offence but, why would I give a shit about this? git is a tool I use to enable me to get other work done, it’s not something I’m reimplementing. If “making the implementation simpler” means my day-to-day workflows get materially more unpleasant, the simplicity of the implementation can take a long walk off a short pier for all I care.
It’s not just pre-commit hooks that get materially worse with this. “Staging” something would then have to have a commit message, I would effectively have to branch off of head before doing every single “staging” commit in order to be able to still merge another branch and then rebase it back on top of everything without fucking about in the reflog to move my now-burried-in-the-past stage commit forward, etc, etc. “It would make the implementation simpler” would be a really poor excuse for a user hostile change.
-
If “making the implementation simpler” means my day-to-day workflows get materially more unpleasant, the simplicity of the implementation can take a long walk off a short pier for all I care.
I agree. Users shouldn’t have to care about the implementation (except for minor effects like a simpler implementation resulting in fewer bugs). But I don’t understand why your workflows would be materially more unpleasant. I think they would actually be more pleasant. Mercurial users very rarely miss the staging area. I was a git developer (mostly working on
git rebase) a long time ago, so I consider myself a (former) git power user. I never miss the staging area when I use Mercurial.“Staging” something would then have to have a commit message
Why? I think the topic of this thread is about what can be done differently, so why would the new tool require a commit message? I agree that it’s useful if the tool lets you provide a message, but I don’t think it needs to be required.
I would effectively have to branch off of head before doing every single “staging” commit in order to be able to still merge another branch and then rebase it back on top of everything without fucking about in the reflog to move my now-burried-in-the-past stage commit forward
I don’t follow. Are you saying you’re currently doing the following?
git add -p git merge <another branch> git rebase <another branch>I don’t see why the new tool would bury the staging commit in the past. That’s not what happens with Jujutsu/jj anyway. Since the working copy is just like any other commit there, you can simply merge the other branch with it and then rebase the whole stack onto the other branch after.
I’ve tried to explain a bit about this at https://github.com/martinvonz/jj/blob/main/docs/git-comparison.md#the-index. Does that help clarify?
-
Mercurial users very rarely miss the staging area.
Well, I’m not them. As somebody who was forced to use Mercurial for a bit and hated every second of it, I missed the hell out of it, personally (and if memory serves, there was later at least one inevitably-nonstandard Mercurial plugin to paper over this weakness, so I don’t think I was the only person missing it).
I’ve talked about my workflow elsewhere in this thread, I’m not really interested in rehashing it, but suffice to say I lean on the index for all kinds of things.
Are you saying you’re currently doing the following? git add -p git merge
I’m saying that any number of times I start putting together a commit by staging things on Friday afternoon, come back on Monday, pull in latest from main, and continue working on forming a commit.
If I had to (manually, we’re discussing among other things the assertion that you could eliminate the stage because it’s pointless, and you could “just” commit whenever you want to stage and revert the commit whenever they want to unstage ) commit things on Friday, forget I’d done so on Monday, pull in 300 commits from main, and then whoops I want to revert a commit 301 commits back so now I get to back out the merge and etc etc, this is all just a giant pain in the ass to even type out.
Does that help clarify?
I’m honestly not interested in reading it, or in what “Jujutsu” does, as I’m really happy with git and totally uninterested in replacing it. All I was discussing in this thread with Loup-Vaillant was the usefulness of the stage as an abstraction and my disinterest in seeing it removed under an attitude of “well you could just manually make commits when you would want to stage things, instead”.
-
I’m honestly not interested in reading it, or in what “Jujutsu” does
Too bad, this link you’re refusing to read is highly relevant to this thread. Here’s a teaser:
As a Git power-user, you may think that you need the power of the index to commit only part of the working copy. However, Jujutsu provides commands for more directly achieving most use cases you’re used to using Git’s index for.
-
What “jujutsu” does under the hood has nothing whatsoever to do with this asinine claim of yours, which is the scenario I was objecting to: https://lobste.rs/s/yi97jn/is_it_time_look_past_git#c_k6w2ut
At this point I’ve had enough of you showing up in my inbox with these poorly informed, bad faith responses. Enough.
-
I was claiming that the workflows we have with the staging area, we could achieve without. And Jujutsu here has ways to do exactly that. It has everything to do with the scenario you were objecting to.
Also, this page (and what I cited specifically) is not about what jujutsu does under the hood, it’s about its user interface.
-
-
-
-
-
-
No offence but, why would I give a shit about [simplicity of implementation]?
It’s because people don’t give a shit that we have bloated (and often slow) software.
-
-
-
-
-
-
-
-
-
Sublime Merge is the ideal git client for me. It doesn’t pretend it’s not git like all other GUI clients I’ve used so you don’t have to learn something new and you don’t unlearn git. It uses simple git commands and shows them to you. Most of git’s day-to-day problems go away if you can just see what you’re doing (including what you’ve mentioned).
CLI doesn’t cut it for projects of today’s size. A new git won’t fix that. The state of a repository doesn’t fit in a terminal and it doesn’t fit in my brain. Sublime Merge shows it just right.
-
-
I didn’t know about GitUp but I have become a big fan of gitui as of late.
-
-
-
-
-
See Game of Trees for an alternative to the git tool that interacts with normal git repositories.
Have to agree with others about the value of the staging area though! It’s the One Big Thing I missed while using Mercurial.
-
Well, on the one hand people could long for a better way to store the conflict resolutions to reuse them better on future merges.
On the other hand, of all approaches to DAG-of-commits, Git’s model is plain worse than the older/parallel ones. Git is basically intended to lose valuable information about intent. The original target branch of the commit often tells as much as the commit message… but it is only available in reflog… auto-GCed and impossible to sync.
-
-
Honestly I have completely the opposite feeling. Back in the days before git crushed the world, I used Mercurial quite a lot and I liked that Mercurial had both the ephemeral “throw away after use” model (bookmarks) and the permanent-part-of-your-repository-history model (branches). They serve different purposes, and both are useful and important to have. Git only has one and mostly likes to pretend that the other is awful and horrible and nobody should ever want it, but any long-lived project is going to end up with major refactoring or rewrites or big integrations that they’ll want to keep some kind of “here’s how we did it” record to easily point to, and that’s precisely where the heavyweight branch shines.
And apparently I wrote this same argument in more detail around 12 years ago.
-
-
-
-
If you go further, there are many directions to extend what you can store and query in the repository! And of course they are useful. But even the data Git forces you to have (unlike, by the way, many other DVCSes where if you do not want a meaningful name you can just have multiple heads in parallel inside a branch) could be used better.
-
-
I can’t imagine a scenario where the original branch point of a feature would ever matter, but I am constantly sifting through untidy merge histories that obscure the intent.
Tending to your commit history with intentionality communicates to reviewers what is important, and removes what isn’t.
-
It is not about the point a branch started from. It is about which of the recurring branches the commit was in. Was it in quick-fix-train branch or in update-major-dependency-X branch?
-
The reason why this isn’t common is because of GitHub more than Git. They don’t provide a way to use merge commits that isn’t a nightmare.
When I was release managing by hand, my preferred approach was rebasing the branch off HEAD but retaining the merge commit, so that the branch commits were visually grouped together and the branch name was retained in the history. Git can do this easily.
-
-
-
-
I never understood the hate for Git’s CLI. You can learn 99% of what you need to know on a daily basis in a few hours. That’s not a bad time investment for a pivotal tool that you use multiple times every day. I don’t expect a daily driver tool to be intuitive, I expect it to be rock-solid, predictable, and powerful.
-
This is a false dichotomy: it can be both (as Mercurial is). Moreover, while it’s true that you can learn the basics to get by with in a few hours, it causes constant low-level mental overhead to remember how different commands interact, what the flag is in this command vs. that command, etc.—and never mind that the man pages are all written for people thinking in terms of the internals, instead of for general users. (That this is a common failing of man pages does not make it any less a problem for git!)
One way of saying it: git has effectively zero progressive disclosure of complexity. That makes it a continual source of paper cuts at minimum unless you’ve managed to actually fully internalize not only a correct mental model for it but in many cases the actual implementation mechanics on which it works.
-
Its manpages are worthy of a parody: https://git-man-page-generator.lokaltog.net
-
-
Its predecessors CVS and svn had much more intuitive commands (even if they were was clumsy to use in other ways). DARCS has been mentioned many times as being much more easy to use as well. People migrating from those tools really had a hard time, especially because git changed the meanings of some commands, like checkout.
Then there were some other tools that came up around the same time or shortly after git but didn’t get the popularity of git like hg and bzr, which were much more pleasant to use as well.
-
I think the issues people have are less about the CLI itself and more about how it interfaces with the (for some developers) complex and hard to understand concepts at hand.
Take rebase for example. Once you grok what it is, it’s easy, but trying to explain the concept of replaying commits on top of others to someone used to old school tools like CVS or Subversion can be a challenge, especially when they REALLY DEEPLY don’t care and see this as an impediment to getting their work done.
I’m a former release engineer, so I see the value in the magic Git brings to the table, but it can be a harder sell for some :)
-
-
The interface is pretty bad.
I would argue that this is one of the main reasons for git’s success. The CLI is so bad that people were motivated to look for tools to avoid using it. Some of them were motivated to write tools to avoid using it. There’s a much richer set of local GUI and web tools than I’ve seen for any other revision control system and this was true even when git was still quite new.
I never used a GUI with CVS or Subversion, but I wanted to as soon as I started touching the git command line. I wanted features like PRs and web-based code review, because I didn’t want to merge things locally. I’ve subsequently learned a lot about how to use the git CLI and tend to use it for a lot of tasks. If it had been as good as, say, Mercurial’s from the start then I never would have adopted things like
gitx/gitgand GitHub and it’s those things that make the git ecosystem a pleasant place to be. -
The interface of Git and its underlying data models are two very different things, that are best treated separately.
Yes a thousand times this! :) Git’s data model has been a quantum leap for people who need to manage source code at scale. Speaking as a former release engineer, I used to be the poor schmoe who used to have to conduct Merge Day, where a branch gets merged back to main.
There was exactly one thing you could always guarantee about merge day: There Will Be Blood.
So let’s talk about looking past git’s god awful interface, but keep the amazing nubbins intact and doing the nearly miraculous work they do so well :)
And I don’t just mean throwing a GUI on top either. Let’s rethink the platonic ideal for how developers would want their workflow to look in 2022. Focus on the common case. Let the ascetics floating on a cloud of pure intellect script their perfect custom solutions, but make life better for the “cold dark matter” developers which are legion.
-
I would say that you simultaneously give credit where it is not due (there were multiple DVCSes before Git, and approximately every one had a better data model, and then there are things that Subversion still has better than everyone else, somehow), and ignore the part that actually made your life easier — the efforts of pushing Git down people’s throat, done by Linus Torvalds, spending orders of magnitude more of his time on this than on getting things right beyond basic workability in Git.
-
-
Well, Bazaar is technically earlies. Monotone is significantly earlier. Monotone has quite interesting and nicely decoupled data model where the commit DAG is just one thing; changelog, author — and branches get the same treatment — are not parts of a commit, but separately stored claims about a commit, and this claim system is extensible and queriable. And of course Git was about Linus Torvalds speedrunning implementation of the parts of BitKeeper he really really needed.
It might be that in the old days running on Python limited speed of both Mercurial and Bazaar. Rumour has it that the Monotone version Torvalds found too slow was indeed a performance regression (they had one particularly slow release at around that time; Monotone is not in Python)
Note that one part of things making Git fast is that enables some optimisations that systems like Monotone make optional (it is quite optimistic about how quickly you can decide that the file must not have been modified, for example). Another is that it was originally only intended to be FS-safe on ext3… and then everyone forgot to care, so now it is quite likely to break the repository in case of unclean shutdown mid-operation. Yes, I have damaged repositories that way to a state where I could not find advice on how to avoid re-cloning to get even partially working repository.
As of Subversion, it has narrow checkouts which are a great feature, and DVCSes could also have them, but I don’t think anyone properly has them. You kind of can hack something with remote-automate in Monotone, but probably flakily.
-
-
-
-
Let the data model pretend there’s a blob for each version of that huge file, even though in fact the software is automatically compressing & decompressing things under the hood.
Ironically, that’s part of the performance problem – compressing the packfiles tends to be where things hurt.
Still, this is definitely a solvable problem.
-
I used to love DARCS, but I think patch theory was probably the wrong choice.
I have created and maintains official test suite for pijul, i am the happiest user ever.
-
Hmm, knowing you I’m sure you’ve tested it to death.
I guess they got rid of the exponential conflict resolution that plagued DARCS? If so perhaps I should give patch theory another go. Git ended up winning the war before I got around to actually study patch theory, maybe it is sounder than I thought.
-
Pijul is a completely different thing than Darcs, the current state of a repository in Pijul is actually a special instance of a CRDT, which is exactly what you want for a version control system.
Git is also a CRDT, but HEAD isn’t (unlike in Pijul), the CRDT in Git is the entire history, and that is not a very useful property.
-
-
-
-
-
-
You can’t describe Git without discussing rebase and merge: these are the two most common operations in Git, yet they don’t satisfy any interesting mathematical property such as associativity or symmetry:
-
Associativity is when you want to merge your commits one by one from a remote branch. This should intuitively be the same as merging the remote HEAD, but Git manages to make it different sometimes. When that happens, your lines can be shuffled around more or less randomly.
-
Symmetry means that merging A and B is the same as merging B and A. Two coauthors doing the same conflictless merge might end up with different results. This is one of the main benefits of GitHub: merges are never done concurrently when you use a central server.
-
Well, at least this is not the fault of the data model: if you have all the snapshots, you can deduce all the patches. It’s the operations themselves that need fixing.
-
My point is that this is a common misconception: no datastructure is ever relevant without considering the common operations we want to run on it.
For Git repos, you can deduce all the patches indeed, but merge and rebase can’t be fixed while keeping a reasonable performance, since the merge problem Git tries to solve is the wrong one (“merge the HEADs, knowing their youngest common ancestor”). That problem cannot have enough information to satisfy basic intuitive properties.
The only way to fix it is to fetch the entire sequence of commits from the common ancestor. This is certainly doable in Git, but merges become O(n) in time complexity, where n is the size of history.
The good news is, this is possible. The price to pay is a slightly more complex datastructure, slightly harder to implement (but manageable). Obviously, the downside is that it can’t be consistent with Git, since we need more information. On the bright side, it’s been implemented: https://pijul.org
-
no datastructure is ever relevant without considering the common operations we want to run on it.
Agreed. Now, how often do we actually merge stuff, and how far is the common ancestor in practice?
My understanding of the usage of version control is that merging two big branches (with an old common ancestor) is rare. Far more often we merge (or rebase) work units with just a couple commits. Even more often than that we have one commit that’s late, so we just pull in the latest change then merge or rebase that one commit. And there are the checkout operations, which in some cases can occur most frequently. While a patch model would no doubt facilitate merges, it may not be worth the cost of making other, arguably more frequent operations, slower.
(Of course, my argument is moot until we actually measure. But remember that Git won in no small part because of its performance.)
-
I agree with all that, except that:
-
the only proper modelling of conflicts, merges and rebases/cherry-picking I know of (Pijul) can’t rely on common ancestors only, because rebases can make some future merges more complex than a simple 3-way merge problem.
-
I know many engineers are fascinated by Git’s speed, but the algorithm running on the CPU is almost never the bottleneck: the operator’s brain is usually much slower than the CPU in any modern version control system (even Darcs has fixed its exponential merge). Conflicts do happen, so do cherry-picks and rebases. They aren’t rare in large projects, and can be extremely confusing without proper tools. Making these algorithms fast is IMHO much more important from a cost perspective than gaining 10% on a operation already taking less than 0.1 second. I won’t deny the facts though: if Pijul isn’t used more in industry, it could be partly because that opinion isn’t widely shared.
-
some common algorithmic operations in Git are slower than in Pijul (
pijul creditis much faster thangit blameon large instances), and most operations are comparable in speed. One thing where Git is faster is browsing old history: the datastructures are ready in Pijul, but I haven’t implemented the operations yet (I promised I would do that as soon as this is needed by a real project).
-
-
-
-
-
-
-
I see bellcore… did it use MGR? That was a neat little windowing system with a bunch of interesting ideas.
-
There was definitely an X port. I can’t remember if MGR was available or not though I wouldn’t be surprised.
-
Coherent got X11 before it got TCP/IP. The X11 programs were compiled to use a library that provided IP over named pipes rather than using the network. I don’t remember there ever being a commercial networking stack for the product from Mark Williams. Someone did successfully add TCP/IP after Mark Williams went out of business. I dimly remember an AT&T MGR port but I could be wrong.
I wish that the effort that went into making X11 work had been used for TCP/IP networking instead. I believe that X11 would have been easier to port if TCP/IP was available and the delay would have also meant working with a later version of XFree86. XFree86 was improving rapidly at that time so the delay would have meant better quality code all around. It didn’t work out that way for a lot of reasons. The X11 vs networking issue seems like a big blunder but looking back with a realistic eye toward the fog of war reveals that the decision was not controversial at all. It should have been clear that TCP/IP would be a big player in networking between 1993 and 1997 but earlier on it wasn’t obvious that TCP/IP would subsume and replace all other networks.
The mid ‘90s were a weird time in Unix. There were a handful of sanctioned AT&T Unix OS’s for the i386. I remember SCO, Interactive Unix, and Xenix to mention 3. Bill Jolitz and his wife had just released 386BSD. There was Coherent which, ran on both the 286 and the 386, and wasn’t based on AT&T code. Coherent had been reviewed and vetted as such by Dennis Ritchie. And obviously, Linux had just appeared.
-
-
-
MGR
AFAIK, no.
There was a 3rd party X server, for X11R5: https://www.autometer.de/unix4fun/coherent/third_party.html
A relatively recent look at Coherent: https://virtuallyfun.com/wordpress/2017/12/19/coherent-3-0/
Later the company did its own version – here’s the manual: https://www.nesssoftware.com/home/mwc/doc/coherent/x/pdf/X11.pdf
-
-
I’m of the opinion that IPv6 should be mostly static. If rebooting my router doesn’t change my IPv4 address, it shouldn’t change my IPv6 address either. I’ve retained the same IPv4 address since February. Paying for business class internet with static IP is also an option. I hope that this is a temporary situation. I have Verizon FiOS in my condo. Verizon’s IPv6 rollout is a big project. They are working their way up the east coast and I anticipate that they will be in Boston/Cambridge in a month or two. I’ll probably keep my HE tunnel for a few months after I get native v6 until the dust shakes out.
-
-
The article praises the decision to expose buffers to the end-user, and I agree that it’s powerful to be able to use the same commands in basically every context… but it leads to a lot of confusion and awkwardness that I think needs to be addressed as well. My case in point: Several times a week, I need to rename a file I’m editing. Here’s what it looks like:
M-x rename-file- Choose the file I want to rename
- Type in the new name and confirm
M-x rename-buffer- Enter the new name
In a different text editor, it looks like this:
- (some keybinding or sequence)
- Enter the new name
It could look like that in Emacs as well, but if you go looking around you find that it’s not in the core command set, and you have to dig up some script someone has already put together and then shove that into your
init.el. Only then can you have workflow #2.Emacs knows which buffers are file-backed, and could very well offer a command like
rename-file-buffer. I don’t know why it doesn’t, in the Year of Our Lord 2022. Maybe some bikeshedding over what counts as a file-backed buffer, or naming the function, or some internals thing I don’t know about. But it probably has something to do with “everything’s a buffer” and not trying too hard to smooth over that.-
While I agree with you about frustrations on awkward interfaces surrounding buffers, I’m not sure that I follow your example. For your example, I’d normally do
-
C-x C-w(orM-x write-file) - Enter the new name
It seems like it follows your desired path, accomplishes your objectives, and only uses built-in commands and default keybindings? Is there something that I’m missing?
-
-
-
True but in this is a case where the functionality is all of obvious, easy to code, and absent from the core product. I figure that the reason this reature is absent is because core emacs already has two better ways to get the workflow done. I don’t remember when I discovered the
write-filemethod. I’d bet that it was early on in my use of emacs though so we’re talking early ’90s. I came todired modepretty late but learned very quickly how powerful it was.
-
-
-
-
-
-
That’s absolutely true and it’s interesting that they haven’t done this already.
How much do you want to bet there there aren’t a million and one “rename-buffer-and-file” functions floating around in an equal number of .emacs files? :)
For me, while I really truly do appreciate the power and wisdom of having my editor also be an capable programming environment unto itself, I think exactly this kind of lack of polish is going to continue to erode emacs adoption over the long haul.
-
Emacs not only knows when a buffer is attached to a file, it also does the right thing when it performs operations on the file from dired mode. I have a file I want to rename. I open the directory that it’s in with dired mode by pressing:
c-x c-f enterfrom the buffer visiting the file. I press'R'then fill out the new_filename in dialog. After the rename is finished, i press'F'and I’m taken back to the buffer visiting the file. Emacs renames the file and automatically renames the buffer as you intended. Also note that the buffer never changes. Your new position is exactly the same as the old.
-
-
-
Ah! It sounds like
diredis the thing I should have been using. I always wrote it off as a “power tool” for when you need to do heavier rearranging of files and directories – multi-renames, whatever – but maybe that’s what all the experience users are actually doing for renames? -
-
-
This doesn’t address the larger point, but it does the pain in your workflow. You can achieve the same in one logical step by using dired’s write mode (Wdired) to do the rename in the dired buffer.
- C-x C-j (opens dired in the directory the current buffer)
- C-x C-q
- The file in question inside the dired buffer
- C-c C-c
As to why rename-file-buffer doesn’t also rename the buffers that are visiting that file, I’m guessing it is because it is written in C, and the code is already hairy enough to complicate it further with additional responsibilities.
Especially as there are edge cases. Buffers don’t have to have the same name as the file they are visiting. For example when you use
clone-indirect-buffer-other-window, which some people use heavily in conjunction with narrowing. Should we rename all buffers visiting the file only where there is an exact match between the buffer and file name? what about when the file is part of the buffer name ej.foo.txt<3>orfoo.txt/fun-name<2>? I think it is a reasonable choice to have rename-file do only one thing and let users implement a more dwim version by themselves. -
I wrote a function to do that (“some script someone has already put together”). Once my work signs Emacs’s employer disclaimer of rights, I’m going to try to get this into Emacs proper.
-
This doesn’t address your actual point, but adding just in case it’s useful to someone. Pretty sure I stole this from Steve Yegge years ago
(defun rename-file-and-buffer (new-name) "Rename both file and buffer to NEW-NAME simultaneously." (interactive "sNew name: ") (let ((name (buffer-name)) (filename (buffer-file-name))) (if (not filename) (message "Buffer '%s' is not visiting a file." name) (if (get-buffer new-name) (message "A buffer named '%s' already exists." new-name) (progn (rename-file name new-name 1) (rename-buffer new-name) (set-visited-file-name new-name) (set-buffer-modified-p nil)))))) -
Reading this made me realize that I can add a little function to my
.emacsto do this (my strategy has tended to be to do it in avtermsession and then re-open the file when I need to do this once every blue moon).I do think there should be “a thing” (though stuff like editing of remote files have to be answered). I do wonder how open GNU Emacs is to simple QoL improvements like that.
-
New England Winter: Shovelling Snow. If time allows setting up new WiFi on Ubiquiti hardware.
-
That should be common knowledge since ~10 years
-
-
Yes, ~20 years of pain with no upside. Yikes! I use Postgres when I have a choice, personally.
-
-
-
-
I’ve never seen this until today. Probably because I stopped using MySQL and MariaDB over ten years ago. Still, if you say your encoding is
utf8, I do believe you you have created an implicit contract with the consumers of your work. If you’re definition ofutf8is only a subset what a programmer would want when he choosesUTF-8I do think that it’s on your to call that out in your documentation. After reading the MariaDB documentation on Unicode, I don’t think that’s been done.-
Your comment made me look up what the current versions are doing.
The MySQL docs mark utf8 as deprecated. The language around utf8mb3 is not as clear as I’d like (I would say “never use this it is broken”) but if you read carefully the info you need is there.
MariaDB seems to have redefined “utf8” to be a working UTF-8 encoding. See OLD MODE for details.
-
I read the MariaDB docs before I wrote my comment. This sentence stood out: “From MariaDB 10.6, utf8 is an alias for utf8mb3, but this can changed to ut8mb4 by changing the default value of the old_mode system variable.” Note well that as stated, the problem in caused when using three byte unicode rather than four byte unicode. The gist of it is that even in current MariaDB to get four byte unicode you have either explicitly ask for “utf8mb4” encoding in your database or change “old_mode” from it’s default setting to make “utf8” mean four byte utf-8 encoding. These changes only started with MariaDB 10.5, April 2021. That’s only about eight or nine months ago.
-
-
I took this sentence from the documentation: “If set, the default, utf8 is an alias for utf8mb3. If not set, utf8 would be an alias for utf8mb4.” to mean that
UTF8_IS_UTF8MB3is set by default inOLD_MODEmeaning that in MariaDB 10.6.1 “utf8” means “utf8mb3”. Had I written that passage, I would have more clearly stated the effect of the tunable, and it’s default setting by splitting those two pieces of information into two sentences. Regardless, If I go back to my high school english classes and take the meaning of that sentence to mean By default, utf8 is an alias for utf8mb3.-
Wow, what a mess, you’d think folks would care more about being able to store text correctly in a database. The docs sure are confusing.
Thanks for correcting my mistake up above; I’d naively assumed the default was sensible. It seems to be broken though, from the latest changes: “Default value changed in MariaDB Community Server 10.6.1 from “” (empty) to UTF8_IS_UTF8MB3”.
-
-
-
-
-
-
-
The indifferent approach to correctness is the reason that I would never use MySQL again. I know that today it’s less haphazard, but I lost so much data to bad defaults, incomprehensible limitations, and pure documentation fails that I am unalterably biased against it.
Of course, I work at Shopify, where we have a huge MySQL installation. Ah, but I don’t have to touch it.
-
-
If you’re using gitlabs, and/or your main concern is the code repository, you can automate backup creation with cron. The command is:
sudo gitlab-rake gitlab:backup:create ...This has been in gitlabs since at least 11.x.y. The downside to this backup is that it’s version locked meaning that a backup of gitlabs from a machine running gitlabs 14.6.0 will only restore to a fresh machine running gitlabs 14.6.0. I would find it nice if I could test upgrades by backing up from 14.6.0 and restoring to 14.7.10 obviously within reason.As the files are stored in a single file, an strategy for backup can look like this:
- Deploy a gitlabs machine in AWS.
- Create an AMI of the gitlabs machine.
- Arrange push for your main working repository to the AWS gitlabs machine on a project-by-project basis.
- Backup the community edition machine daily, hourly, whatever you need or can afford.
- Push the single backup file into AWS S3.
Note well:
- You aren’t creating another general purpose gitlabs machine here. The AWS machine will have a subset of users from your working git instance. This is done to reduce the maintenance required by this solution.
- By using a repository available 24/7, you can backup from any instance of git that can automate a push to an ssh or an https hosted repository.
Your absolute dead recovery process looks like this:
- Create a new gitlabs machine from the template.
- Fish the appropriate backup out of S3.
- Restore it.
- Reestablish the state of your main working repository with the recovered state from backup.
All of this is heavily dependent on the cloud but in an absolute disaster, you only need to be able to fish the appropriate backup out of AWS S3 and build a gitlabs community edition machine on the appropriate gitlabs-ce version.
This strategy was designed to store a backup of your entire gitlabs machine in a different region from your working gitlabs machine in the most bandwidth friendly way possible.
-
I commented earlier but I would add this. Today a lot of programmers don’t know email. I occasionally still host things on real linux boxes and one of the steps in my ansible install is to configure postfix to relay mail local mail through an internal smart smtp relay. The tell question is: What email server and user/password should I use for this project. The tell answer is the look on the questioner’s face when I answer: pick one of
127.0.0.1:25,[::1]:25, orlocalhost:25, no username or password.-
Yes, I feel this way. l do not know email, but feel like I should and I am trying to learn it. I really just want to know how to send registration and password reset emails without having to signup for another service. Then I’ll work on learning more from there.
-
The programming APIs for sending email are usually well document. The questions are: “What server should I use?”, “What port?”, “What username?” and “What password?” If your administrator can and has setup local relaying properly, the answers are: (“localhost”, 25, None, None)
-
If your administrator can and has setup local relaying properly
That is what I want to know how to do. Is there any good documentation on how to do that?
-
The setup I use has one system setup to relay all mail from a group of systems. I use postfix and there are two elements to this setup:
- One system, the relay, knows how to handle mail addressed from your domain to the list of destinations to which you want to send mail. And will handle mail for anything in your fleet by IP address.
- The other systems, the clients, will accept mail via localhost only and with use SMTP to handshake inbound mail, not destined for local delivery, through the relay.
To configure the clients, you want to have
relayhost = [therelay.example.com]:25in postfix’s main.cf. Googling forpostfix local mail relay smart hostshould be helpful here.Configuration of the relay largely depends on who you need to get mail to and what mechanisms you have to deliver mail. Googling for
postfix smart hostorpostfix smart relayshould get you started. Note well that any reasonable tutorial will include settings/entries for:-
inet_protocols, inet_interfaces: These postfix settings determine how your relay listens for incoming mail to relay or directly deliver. -
mynetworks_style, mynetworks, smtpd_relay_restrictions: These settings control the relaying aspects of postfix. By default postfix will reject mail if it doesn’t think that it’s the destination for that mail. By tuning these three settings, you can limit who can use your postfix machine as a mail relay. These settings are important because tuning them the wrong way can make your system an open mail relay which is very bad. Usually these will be setup such that the relay will accept mail from IP addresses its own LAN without question. On the other hand clients will set this up so the only valid source for mail is localhost. -
mydestination: This postfix setting determines what domains you consider to be your responsibility for delivery. Ex, a machine whose destination includesexample.comwill expect to be or to know the delivery port for mail tochip@example.com.
The biggest problem is usually how to get mail out to the internet. I think that my earlier comments trivialized that fact. There are generally two main ways to do it. Either, your ISP will give you the IP address and port of a system which will relay mail to the rest of the internet for you. Sometimes they will require you to use a username/password here also. The postfix tutorials that mention SASL cover this. If you run your own mailserver, and this is a much harder thing to do securely, you’ll need to get permission from your provider, either ISP or Virtual Private Server, to directly contact servers via port 25 on the internet.
I run a small fleet of Linux boxes on an otherwise Microsoft centric network for my company. We handle our mail via O365. I got permission for one of my systems to relay mail through the corporate SMTP gateways . This is my relay. I’ve configured the rest of the fleet to use my relay as a smart host for all messages.
-
-
-
-
-
I’m tempted to side with the autofs/automounter guys here as I think this is something that automounters are designed to handle. I don’t know systemd well enough to know what it will do when it tries to nfs mount a file system from dead, missing, or otherwise broken NFS server. The automounter is designed to gracefully handle this specific condition. I say this after having Intel dual gigabit network card fail in a TrueNAS server. From experience, a hard coded mount in /etc/fstab would cause a server to fail to come up on reboot. An autofs configuration would soft fail here with the server coming up. What should I expect systemd to do? I’m not sure. From what I understand of systemd’s design I think that the system would come up with a failed unit. Still, I prefer the automount solution here as it will release the mount when it’s not being used.
Finally, I would note that the root cause of problems here isn’t the automounter or systemd, it’s NFS. NFS’ issue is that nfs clients tend to behave very badly when a mounted filesystem disappears because the server became unavailable. The automounter fits these situations better by limiting the lifetime of NFS mounts.
-
Today, many email providers have some kind of method of contacting them, but I have never once received a response or even evidence of action due to one of these messages… both for complaints of abuse on their end and deliverability problems
Strangely, one thing I was surprised about was how it was possible to get in touch with Postmasters when we had issues with blacklisting. It generally took a while to track down the proper way to contact them, but I think there’s only been one mailhost (Apple) who we never heard back from – although we were removed from their blacklist the day after. When blacklisted by Microsoft (Outlook, Hotmail, Live, etc.) I even got to chat with a helpful real person; an actual named person no less!
-
-
This was my experience also. A couple of years ago, I was trying to change my Warby Parker password after a breach on their end. The process failed at the send a nonce to Chris via email step. I could see that the cause of the failure was that I hadn’t added warbyparker.com to my greylister so I wasn’t going to get immediate delivery. Even after bypassing warbyparker.com I wasn’t getting the nonce. A letter to hostmaster@warbyparker.com got a response but I didn’t need glasses at the time. I honestly didn’t know it was fixed until today when I remembered as a result of this thread.
-
-
Maybe I’m just being in too bad of a mood! Most of this comment dates back to when I was working in university IT 6-7 years ago and spent a long time battling blacklisting problems. I don’t think I ever got responses from Google, and while they did start accepting mail from us again each time it was long enough later (a few days) that it was unclear whether someone had read my message or the blocklist entry had just hit an expiration date. The worst problems we had in the other direction were abuse coming from Google (spam and fraud email, calendar invites, and phishing via Google Forms) and I don’t think I ever saw any evidence of their acting on abuse complaints. We started blacklisting certain gmail addresses at the institutional level because they were sending so much mail to us that just had very long address lists pasted into the To/Cc. The irony is that SpamAssassin was picking this stuff up and we had to manually bump the domain reputation of gmail.com to avoid it learning to spamcan every marginal email coming from Google.
I have not seen this kind of problem since I worked there (although I haven’t run as large of a mailserver since), but I would assume Google has tightened up their controls at some point because it was remarkably brazen behavior to continue from a major email provider for, as I recall, around a year. I’d guess this was 2013 or so. This situation actually lead somewhat directly to that institution switching to Google Workspace later because it fostered a perception that the in-house operation was incompetent, which is sort of ironic. It really got at one of those no-win problems in security: we were getting phished via Google Forms on a weekly basis, but whenever we tried taking active measures like dropping email with Google Forms links it turned into a huge upset and we had to back off. When I worked in incident response in 2015-2017 phishing via Google Forms continued to be a top-ten problem, but at that organization we had somewhat better tools available to combat it and ended up using our outbound web filters to inject a redirect to a warning page the user had to click through (we were fortunate enough to have TLS interception). Google now provides some additional measures to mitigate this problem but they of course require that you be a Google Workspaces customer. I assume they’ve also stepped up detection and enforcement a lot because in more recent cases where I’ve seen Google Forms phishing, the link has often been dead by the time I saw it. In any case the whole thing left me with a very negative perception of how Google runs their email operation (which was boosted when I was involved in the pre-sales process on the switch to G-Suite, which was amazingly incompetent).
And I shouldn’t let Google color my perception of the whole industry, my recollection is that Microsoft was pretty easy to get a hold of on hotmail/outlook.com issues, although I think we may have leveraged our existing relationship with them to get a better contact than what they have publicly.
With smaller organizations it’s always been a lot easier, of course being a university much of our interaction was with other universities and postmaster@ almost always got a response from someone helpful, whether we were the source or victim of the problem. Unfortunately this situation has become worse over the years as more and more institutions and businesses are moving over to cloud services that they have limited control over and knowledge of. In my current position, where I don’t even really deal with email, I’ve run into a few cases of “something is blocking email from you but we can’t figure out what.” It’s almost always a cloud email security gateway they use, which have a tendency to be very opaque.
-
-
-
-
I run my own mailserver. I go against a couple of the suggestions on both sides of this issue. You can do it, it’s not hard but the learning curve is tedious and long. I built my first internet mailserver for just myself back in 1996 or so. I sent and received mail via hybrid SMTP/UUCP from ripco communications in Chicago. I expanded the solution to my whole domain in about 2000 when I bought a SparcStation 20 and ran my mail on OpenBSD. I also transitioned from UUCP to pure SMTP at that time. My CATV company put a stop to that when they shut down inbound port 25 in like 2001. If I had it all to do over again, today, I’d probably just delegate my mail to someone else.
-
I assume some people don’t like Facebook, so I reformatted the text and included it here:
This is written by Jon “maddog” Hall
This is the long-promised Christmas present to all those good little girls and boys who love GNU/Linux. It was November of 1993 when I received my first CD of what was advertised as "A complete Unix system with source code for 99 USD". While I was dubious about this claim (since the USL vs BSDi lawsuit was in full swing) I said "What the heck" and sent away my 99 dollars, just to receive a thin booklet and a CD-ROM in the mail. Since I did not have an Intel "PC" to run it on, all I could do was mount the CD on my MIPS/Ultrix workstation and read the man(1)ual pages. I was interested, but I put it away in my filing cabinet. About February of 1994 Kurt Reisler, Chair of the UNISIG of DECUS started sending emails (and copying me for some reason) about wanting to bring this person I had never heard about from FINLAND (of all places) to talk about a project that did not even run on Ultrix OR DEC/OSF1 to DECUS in New Orleans in May of 1994. After many emails and no luck in raising money for this trip I took mercy on Kurt and asked my management to fund the trip. There is much more to this story, requiring me to also fund a stinking, weak, miserable Intel PC to run this project on, but that has been described elsewhere. Now I was at DECUS. I had found Kurt trying to install this "project" on this stinking, weak, miserable Intel PC and not having much luck, when this nice young man with sandy brown hair, wire-rim glasses, wool socks and sandals came along. In a lilting European accent, speaking perfect English he said "May I help you?" and ten minutes later GNU/Linux was running on that stinking, weak, miserable Intel PC. I sat down to use it, and was amazed. It was good. It was very, very good. I found out that later that day Linus (for of course it was Linus Torvalds) was going to give two talks that day. One was "An Introduction to Linux" and the other was "Implementation Issues in Linux". Linus was very nervous about giving these talks. This was the first time that he was giving a talk at a major conference (19,000 people attended that DECUS) to an English-speaking audience in English. He kept feeling as if he was going to vomit. I told him that he would be fine. He gave the talks. Only forty people showed up to each one, but there was great applause. The rest of the story about steam driven river boats, strong alcoholic drinks named "Hurricanes", massive amounts of equipment and funding as well as engineering resources based only on good will and handshakes have been told before and in other places. Unfortunately the talks that Linus gave were lost. Until now. As I was cleaning my office I found some audio tapes made of Linus' talk, and which I purchased with my own money. Now, to make your present, I had to buy a good audio tape playback machine and capture the audio in Audacity, then produce a digital copy of those tapes, which are listed here. Unfortunately I do not have a copy of the slides, but I am not sure how many slides Linus had. I do not think you will need them. Here is your Christmas present, from close to three decades ago. Happy Linuxing" to all, no matter what your religion or creed. And if you can not hear the talks, you are probably using the wrong browser:Introduction to Linux:
https://drive.google.com/file/d/1H64KSduYIqLAqnzT7Q4oNux4aB2-89VE/view?usp=sharing
Implementation Issues with Linux:
https://drive.google.com/file/d/1Y3EgT3bmUyfaeA_hKkv4KDwIBCjFo0DS/view?usp=sharing
-
Thanks!
Also I mirrored this on archive.org so people can find this after google no doubt caps the downloads.
-
Thanks! I really appreciate you posting the text.
It’s not so much that I don’t like Facebook, as that I literally cannot read things that are posted there, because it requires login and I don’t have an account. In my professional opinion as a privacy expert, neither should anyone else, but I realize that most people feel there isn’t really a choice.
-
-
Someone also linked the slides in the archive.org link :)
-
Does anyone have links to the referenced anecdotes “described elsewhere”?
-
-
The parent put the quote in a code block instead of in a blockquote.
-
The link that @neozeed posted to archive.org has the same text and is much easier to read on a mobile device.
-
-
-
Like many others who went to engineering school in the ‘80s, I appreciate high fidelity audio gear to the point of being a little bit of an audiophile. Also, like many people who knew electrical engineers from that era, the amount of bad advice that you can find in “audiophile” circles is astounding. And don’t get me wrong, I’ll tell you to your face that popular music sounds better today on vinyl rather than CD. What I won’t do is try to convince you that the reason for this is because digital is an inferior technology incapable of meeting audiophile standards because that’s not even close to the reason.
(What’s the real reason? :-))
As far as your ears are concerned, louder sounds better. CD and digital have far greater fidelity to the original wave form than vinyl does. Vinyl also has less dynamic range, the difference between the loudest sound that you can record and the softest sound that you can record. When music was regularly sold in both formats in the late eighties, the mastering process for popular music was the same for both formats. In the early ‘90s people discovered that the common mastering process for CD and LP was leaving a lot of dynamic range unused when the music was pressed onto CDs. As the vinyl LP was falling out of favor, people discovered that you if you reduce the dynamic range of the music you can master it at a higher sound level or loudness on CD without generating distortion. To people listening to the music these louder pressings initially sound better. Every rock and pop artist on the planet wants their song to sound the best when played alongside other music. So artist started asking for this as part of the mastering process. The problem with doing this is that everything begins to get a wall of sound feeling to it. By making the soft parts louder and the loud parts softer so you can make the whole thing louder, you take some of the impact that the music would have with its original dynamic range. When vinyl records starting coming back in to favor, the music destined for LP was mastered the way they used to for vinyl back in the ‘80s. If you listen to two versions of a song, one mastered for LP, and the other mastered for CD, the LP will sound better in the first few plays before vinyl’s entropic nature ruins it, because the LP version will have more dynamic range. The same is true of two pressings of Pink Floyd’s “Dark Side of the Moon” if you are comparing an early 1980’s issue CD to a late ’90s CD reissue. This is really only true of Rock, Pop, and R&B. Classical and Jazz were unaffected by the Loudness War because fans of those genres put fidelity highest in their desired traits.
Summarizing: when you say you prefer vinyl over CD, you say that you prefer 80’s style mastering over the overly compressed end-90s+ mastering.
It’s interesting that the extra headroom on CDs sparked the loudness war, instead of resulting in better dynamics. And now that people expect music to have a certain loudness, I guess we can’t go back.
Perhaps one day we could get a new wave of artists mastering their 320kbps mp3s 80s-stlye?
A loud mix makes sense if you’re listening to music in a noisy environment. (On your commute, say.) But I’d rather have the ability to compress the dynamic range at the time of playing, so I can adjust it to suit the environment.
I used to have a decent, though inexpensive, stereo system setup. Back when I would sit down just to listen to music, with no other distractions, like the Internet.
But when was the last time I really sat down to listen to music? For me it is usually in the car, or through a pair of earbuds. Or maybe washing the dishes.
The extra headroom mostly provided the opportunity, alongside the fidelity and lack of physical limitations of a CD: on a vinyl if you try to brickwall you end up with unusable media.
What sparked the loudness war is the usual prisoner’s dilemna, where producers ask for more volume in order to stand out, leading the next producer to do the same, until you end up with tons of compression and no dynamic range left. Radio was a big contributor, as stations tend(ed?) to do peak normalisation[0], so if you leverage dynamic range widely you end up very quiet next to the pieces played before and after.
To an extent it’s already been happening for about a decade: every streaming service does loudness normalisation[1], so by over-compressing you end up with a track that’s no louder than your neighbours, but it clips and sounds dead.
Lots of “legacy” media companies (software developers, production companies, distributors, …) have also been using loudness normalisation for about a decade following the spread of EBU R 128 (a european recommendation for loudness normalisation), for the same reason.
[0] where the highest sound of every track is set to the same level
[1] where the target is a perceived overall loudness for the entire track
That’s me. When I buy rock music on LP, I’m purchasing music mastered to the 1980’s LP standards. I do that because Rock music works well with the 60 dB or so of dynamic range that vinyl LP offers.
It really is quite shocking to take a CD mastered in the early 90s and another in the late 90s-early 2000s and play them at the same volume settings.
This is an excellent explanation. Being able to explain things clearly without hiding behind technical terms like “compression” is a strong indicator to me that you are a true expert in this field.
For drummers and bassists, “compression” is a well-known term, because compressing dynamic range is almost required in order to record them faithfully. The typical gigging bassist will have a compressor pedal in their effects chain for live performance, too.
I do appreciate it when digital releases are mastered in way that preserves dynamic range, and playing it after any typical digital release in the affected genres, it will sound really quiet.
Some bands have demonstrated to me that you can be a loud rock band with dynamic range mostly intact.
I think it’s fun to watch the record spin around. :-)
I listen to 90% of stuff on vinyl, and I have no rational explanation beyond yours as to why I like it more than streaming.
I heard on the radio that Metallica are investing in their own vinyl-pressing plant.
I stream way more than I use my turntable, basically for the same reasons @fs111 mentions. But I definitely prefer vinyl because while streaming is pure consumption, vinyl is participatory. I enjoy handling the vinyl and really taking care of it (cleaning it when I get it/before I play it, taking care of the jacket, etc.). It makes me feel like a caretaker of music that’s important to me - a participant in the process, instead of just a consumer.
On my phone I listen to music. On my turntable I play music.
I like the physicality of it, too, and I also love the actual artifacts, the records and their sleeves and such.
While I can see the appeal most of my music consumption is while working. I would not like getting up constantly to switch records.
Possibly: I’ve read that over time many pop songs get remastered with more and more dynamic range compression. This makes all parts of the song sound similar in loudness, but also removes some musical features (dynamics) and depending on the DRC method (fast attack/decay, slow attack/decay, manual envelope adjustment) can introduce audible distortion.
Older vinyl record and CD releases are from earlier masters. Albeit some records are newly manufactured, so some will be based on newer remasters anyway.
Cannot confirm or deny, I don’t buy or listen to pop :/
This is called the loudness war. This site collects the dynamic range for albums: https://dr.loudness-war.info
In addition to the loudness wars people have been talking about, certain technical restrictions limit what can accurately be recorded on vinyl. This leads to a subtle “sound” that people get used to and prefer. This could be reproduced when mastering for digital audio formats, but people either don’t do that processing or “audiophiles” claim that it gets lost in translation somehow.