PUSH was a failure. That was one RTT saving too far. When SPDY/H2 were being designed, WebSockets were hot, and everyone assumed HTTP would eventually be bi-directional somehow. PUSH looked less like an architectural violation, and more like an MVP for the future two-way HTTP. Fortunately, PUSH is dead. We’ve got Early Hints now, which are 1xx responses, so they’re even compatible with the good’ol HTTP/1.
Like PHK, I used to be grumpy about HTTP/2 being “too complex”, and mourned loss of being able to type HTTP requests in telnet by hand. However, unlike PKH, I came around to like HTTP/2, and I think it’s the good default now. Proper parsing of HTTP/1, one that is both robust and compatible with browsers, is actually too complex. This is a syntax that you implement in an evening, but patch edge cases forever (did you know Content-Length is not a number?). H/2 framing and HPACK seems to have more complexity, but once you implement it, you’re done.
DoS protection always required extra work beyond what the protocol seemed at face value. H/1 has been hit with slow loris and many types of request smuggling. The recently (re)discovered issue is not that different from TCP SYN flood attacks. We did not disable TCP due to this attack, we’ve hardened implementations instead. H/2 is getting hardened too.
HTTP/2 prioritization wasn’t well implemented for a long time, so naive H/2 servers weren’t winning in benchmarks on pages designed for HTTP/1. But once prioritization was taken more seriously, it actually enabled servers to optimize for fast page rendering (e.g. send JS and fonts before images). People were also inventing hacks for sending low-res image thumbnails first, and replacing them with high-res images later after the page loads. It turns out this can be implemented in H/2 without any JS hacks or changes to the page.
Efficiency of H/2 allowed the crates.io registry to migrate from syncing the whole database over git (which became as large and slow as you can imagine) to fetching only what’s needed over HTTP. The protocol that discovers the required set of data regularly needs to make 500 requests, but with HTTP/2’s single connection and HPACK it’s actually pretty fast and efficient.
Likewise, a sender MUST NOT forward a message with a Content-Length header field value that does not match the ABNF above, with one exception: a recipient of a Content-Length header field value consisting of the same decimal value repeated as a comma-separated list (e.g, “Content-Length: 42, 42”) MAY either reject the message as invalid or replace that invalid field value with a single instance of the decimal value, since this likely indicates that a duplicate was generated
or something else? Would be interested to hear more.
The protocol that discovers the required set of data regularly needs to make 500 requests, but with HTTP/2’s single connection and HPACK it’s actually pretty fast and efficient.
freebsd-update and portsnap often need to download considerably more than that, and achieve it through a minimal pipelined HTTP/1.1 client.
It would be interesting to see of HTTP/2 or 3 made a noticeable difference.
Browser vendors have wasted years trying to make HTTP/1.1 pipelining work, but eventually given up on it — even when talking to known-good servers there were proxies in between screwing it up, and H/1 has no reliable way to identify the problem so you get data corruption. I presume freebsd gets away with this because they control both ends.
I have mostly learnt not to let my OCD get the best of me with respect to the state of my $HOME, but yeah, it would be nice if everybody respected XDG. In a way the situation is now worse than when everything was going straight to $HOME/.myapp.
What did get my blood boiling in the past were proprietary apps that dropped a non-dot directory right into my $HOME for their three directories nested, five line XML configuration file. I imagine the situation in the Windows land is so bad nobody even cares anymore and professionals just designate a whole separate partition for any serious data they wish to keep and treat the rest as a radioactive wasteland. I felt really offended to encounter something like that on Linux.
But a single app has beaten them all. Czech ID card companion app (wine only, obviously) has dropped C:\something.log into my $HOME. That has really gotten me lost for words.
I imagine the situation in the Windows land is so bad nobody even cares anymore
I was a Windows dev for a few years (~ 2002 to ~ 2011). The situation was ludicrous even back then.
Many programs failed to run on Vista when it came out (in ~ 2007) because they were used to writing their configuration into \Program Files. In *NIX terms that’s like putting your config under /usr/local/bin.
Microsoft had been telling folks not to do that ever since \Program Files became a thing back in (IIRC) Windows 95, but … YOLO I guess.
(Parenthetically, that YOLO culture seemed much stronger in Windows-land than it did in *NIX, ever since I started using Linux back in 1995. I don’t know whether it still is; I haven’t programmed on Windows since ~ 2011 and have no desire ever to again.)
I vaguely remember Vista doing something like an overlay file system to prevent apps writing to Program Files while still retaining user-specific data the application thought it saved there, did it not? I guess everybody was used to file system without any permissions since Windows 98 were still using FAT and many applications were carried over. Windows NT/2000 users probably dealt with the issue by using elevated privileges.
I wonder how many collective hours have been wasted thanks to the “Python only initializes default parameters when the function is first evaluated” behavior. Such a nasty trap.
The only reason I was linked this article myself is that a colleague’s PyCharm warned him that my use of f(x: list[str] = []) was dangerous (Pyright rightfully said nothing of the sort). Digg v4 was launched in 2010 so maybe it wasn’t a footgun people were as-aware of, or as well-checked-for as it should’ve been.
I dunno, does it show no supervision? It’s really hard to see such a thing in code review. Assuming they did at all because we’re back in the pre-Git days here and such practices weren’t as easy to come by.
I never spot the Go loop iteration pointer pitfall in code review, and I’ve been reviewing Go code for years. There are some things that are just hard to see.
It’s really hard to see such a thing in code review. Assuming they did at all because we’re back in the pre-Git days here and such practices weren’t as easy to come by.
I’ve been working since 2009 and have never seen a team not doing reviews.
If you’re not editing the list things are totally fine, and it saves a None check later on by maintaining a clean interface.
There is a regularity to this behavior, but it is definitely a thing where you do want some sort of mechanism to say “no I really mean this” so that we could lint it properly (especially given how typed code with None is a bit of a pain in the butt if the first thing you do is initialize the list when you need it)
What I usually want is a sort of way to say “this is nullable up to a point” and then after that point to basically redefine it as non-nullabe. Stuff like Rust lets you just shadow names to get you to this effect, though I think Python (well… mypy) doesn’t. I might be wrong on this front, though
That’s a pretty good trick actually! I could type the value as a sequence (allowing for stuff like lists to be passed in), and of course the default value would not be written to. Love that as a trick
If you’re not editing the list things are totally fine, and it saves a None check later on by maintaining a clean interface.
There is a regularity to this behavior, but it is definitely a thing where you do want some sort of mechanism to say “no I really mean this” so that we could lint it properly (especially given how typed code with None is a bit of a pain in the butt if the first thing you do is initialize the list when you need it)
However, it’s also one you hit after having done actual coding in Python for more than 10 lines of code. You never forget to use “None” as the default parameter instead of “[]” or “{}” after the first time it bites you and you have to debug it.
I have to agree with the poster who said this is indicative of a lack of adult supervision. Any linter worth the CPU heat it generates will flag this as a dead obvious bug. Any Python coding experience beyond 30 days will do similar.
And it matches the article: “Senior engineers ghosted out the door, dampening productivity and pulling their remaining friends”
Nice article, very clearly written. A few nitpicks:
The code doesn’t check whether malloc or strdup failed. This isn’t an issue on most “real OSs”, which will likely kill the process long before it runs out of address space, but it’s a very real situation in embedded systems, or retro environments like DOS or the classic Mac OS.
You don’t initialize the h->buckets array, so it contains garbage. Since its items are pointers, that’s very likely to cause crashes. It’s pretty lucky that the code runs at all; it’s probably because new heap pages are zero filled when the process gets them from the kernel. You really, really need to call calloc for that allocation. (The others are best left as malloc.) FYI your platform or dev tools probably have a mode that initializes all malloc blocks with a garbage value like 0x55; this is super useful when testing. (On MacOS do “setenv MallocScribble” or turn on scribble in the build scheme diagnostic settings in Xcode.)
For your next article you might want to implement open hashing. It’s slightly more complex, but it’s more efficient because there’s less allocation and the buckets are closer together in memory.
You don’t initialize the h->buckets array, so it contains garbage. Since its items are pointers, that’s very likely to cause crashes. It’s pretty lucky that the code runs at all; it’s probably because new heap pages are zero filled when the process gets them from the kernel.
And, just as with your previous comment, the zero-fill of freshly allocated memory is typically a feature of “real OSs”. On various embedded systems, it is likely you’ll see garbage values instead.
Which is also why calloc exists. I definitely did get non-zeroes in malloc at some point in time.
And the standard lib malloc isn’t guaranteed to be the one used during runtime.
This isn’t an issue on most “real OSs”, which will likely kill the process long before it runs out of address space
While most platforms overcommit memory and so won’t fail allocations explicitly by default, they’ll still do so if resource limits are configured, so it’s still something to handle one way or another — even if it’s just a malloc wrapper that aborts with an error.
Fun party fact: for an array of pointers, calloc does not guarantee the behavior you want it to, because an all-zeros bit pattern is not guaranteed to be a null pointer. C++ fixed this with new/new[]. I’m not sure whether null-is-all-zeros is baked as an assumption into LLVM or not.
Really? I recall NULL and 0 being used interchangeably in old C code. And it’s still very common to use a pointer as a Boolean condition, like if(ptr)… Are those technically wrong?
Converting between integers and pointers magically treats NULL as 0 at the C language level, even if at the machine level NULL is not the same sequence of bytes as uintptr_t 0
By having it outside the loop, there is only a single long lived process of ts. If you had put in right after the echo inside the loop, you’d get a lot of short lived processes. This might matter a lot if this is a loop with a lot of iterations and the speed between iterations are of importantance.
One of my first Rust projects was a similar tool inspired by ts, rtss, which focuses on relative timestamps.
It’s very similar to ts -i, but is more efficient, has more than 1s of precision, and can run programs itself, including through a pty for things that would otherwise buffer their output:
❯ (while true; do sleep 1; echo hello;echo hi; done) | rtss --pty head -6
1.03s 1.03s | hello
1.03s | hi
2.10s 1.06s | hello
2.10s | hi
3.16s 1.06s | hello
3.16s | hi
3.16s exit status: 0
“default – the default queue, does what it says on the tin” didn’t really help me much!
I’m also confused by the difference between the ingress queue and the pull queue.
From keeping an eye on the Sidekiq dashboard on my own (single user) instance I’ve noticed that the push queue is by far the busiest, followed by the pull queue. I would expect that those should get a lot more processes/threads than the others, though maybe multiuser servers have different queue usage patterns.
default All tasks that affect local users push Delivery of payloads to other servers mailers Delivery of e-mails pull Lower priority tasks such as handling imports, backups, resolving threads, deleting users, forwarding replies scheduler Doing cron jobs like refreshing trending hashtags and cleaning up logs ingress Incoming remote activities. Lower priority than the default queue so local users still see their posts when the server is under load
Usually calling println! inside a hot loop is a pretty common performance footgun. Switching out to ,write!(.., "{}\n", ...); on a std::io::Stdout removes the penalty almost entirely. This is because println! performs a syscall and locks and flushes stdout each time. Whereas other methods allow you to buffer your writes and perform the flushes in batches. Using writeln! can avoid some of the syscalls and doesn’t require locking stdout each time, but it still flushes the buffer on each call. Using write! does not auto flush on each call and gives full control over locking and flushing.
I haven’t tested on this code but I wouldn’t be surprised if it had some impact even though I’m fairly sure the performance is largely bound by other items.
I sat down and made those changes (with some other minor ones as well) to see if there was much of an impact, the end result was on my machine runtime was reduced from 35:28 to 22:58. Here’s the details…
The primary changes I made were:
Using write! on a std::io::BufWriter wrapping std::io::Stdout
Ipv4Addr can natively be converted to a u32, so there isn’t a need to do all the multiplication by octets. Just change .octects() to into() and the variable binding to let ipv4: u32 = .... This also reduces the bounds checking performance hit, so it’s a win win. (A fun tidbit if you did need to do the multiplication its usually faster to bitshift, so that code would have become (ipv4[0] << 24) + (ipv4[1] << 16) + (ipv4[2] << 8) + ipv4[3]…is it worth the readability hit? That’s subjective).
Amortizing line allocations. BufReader::lines allocates a new String for each line. We can reduce that by using a String allocation and passing that as a buffer to BufReader::read_line. This means many lines won’t need to allocate at all. The downside is you have to remember to clear the line buffer before calling read_line() again.
Finally, if you know the machine that you’ll be running the code on supports AVX or AVX2, you can tell Rust to use those instructions via RUSTFLAGS='-Ctarget-feature=+av2,+avx' cargo build --release. Alternatively, if you’ll be running the code on the same machine you compile on you can just use RUSTFLAGS='-Ctarget-cpu=native' cargo build --release.
For writing, you can also slightly reduce the overhead of each write by locking it ahead of time, though I doubt it matters much with a BufWriter.
For reading, you might consider bstr’s for_byte_line or ripline, which in addition to avoiding unnecessary allocations, also avoids unnecessary copying.
Absolutely, and the links have some good discussion (as well as they themselves linking to other good resources) on faster solutions or ways to reduce copies even more, but in this case I wanted to keep it as close(ish) to the original as possible since one of the authors notes was how readable/understandable the Rust code was coming from Python.
Elapsed times should be measured with a monotonic clock. time(3) can jump around and even go backwards if the system clock is changed, but clock_gettime(2) with CLOCK_MONOTONIC will (bugs aside) never go backwards and at worst might get skewed a bit by ntp.
My terminal sat empty for hours. There were no changes – the process was running, but there was no feedback. I was nervous.
What if it failed silently?
How can I check?
What should I do?
On a BSD, hit Ctrl-T.
On Linux, check to see if there’s a SIGUSR1 handler (there is), and send it that. Don’t do this blind, as the default handler will terminate the process.
To my horror, stats printed to the screen: the backup had been 90% complete, and I had stopped it. Convinced I had ruined the backup completely, I deleted the partial backup from Tarsnap and started again from scratch.
Oof. If you’d just have left the partial, tarsnap would have skipped uploading 90% of the next archive using timestamps and dedupe - you’d have lost very little in interrupting it.
But why? Ain’t broken, do not fix it. Better cd? Seriously? Better cat? Seriously??? I wish I had the time to rewrite such trivial things in Rust for no good reason. Literally zero value is added by this.
I am sure that this matches significant fraction of usages when people type cat. I’d even say that viewing files + “priming the pipe” (for which <in.txt prog is the “proper” tool) represent majority of cat usages in the wild.
So, I don’t find “cat is for concatenation, nothing else” statement useful, as it doesn’t describe the reality.
it does for me. I never use cat to view files, I either use less (sometimes more, if I am on a very restricted system) or vim. I think you could remove cat from my system an I would not even notice, that is how often I use it. Most cli heavy people I know tend to be similar.
less is less convenient to me. I do not want to have to type q to close it, and I do want the file to be visible while I type further commands.
On a meta level, I just don’t get these prescriptivist comments. I would understand “less is more convenient than cat” framing, but what is the value of “cat should be used only for concatenation” statement eludes me.
cat(1) is also, depending on your platform, for line numbering, non-blank-line numbering, the removal of runs of multiple blank lines, the explicit display of end-of-lines, the explicit display of tabs and form feeds, converting all non-printables to a human-readable escaped form, and for line ending conversion.
And we might actually be better off that way. At least, a lot less likely to drive half the planet to extinction, and probably not significantly less happy (hedonic treadmill, etc).
Bat is awesome. It’s not really a cat replacement, but a “terminal file viewer with syntax highlighting”. I would never use bat in, say, a script to concatenate files. Two tools for two different jobs.
I haven’t tried loc or tokei, but cloc is a giant ball of slightly buggy perl. Run cloc on a big enough project and you’re bound to encounter issues.
Cut can only split based on single-byte delimiters. Lots of tools try to do some form of alignment by using multiple space characters, or they use a syntax like key: value, so cut usually doesn’t cut it. Having to reach for awk just to get the third column of a thing is unfortunate, so a better cut seems like a good idea.
Sudo is security critical. Every now and then, there are new memory issues found in sudo which couldn’t have existed if sudo was written in a safer language.
People like writing system monitors. Htop, bashtop, bpytop, gotop, the list goes on. I’ve even written a few myself. It only makes sense that someone would eventually write system monitors in Rust.
Hyperfine isn’t really a replacement for time. Rather, hyperfine is a fairly complete benchmarking solution which does multiple timed runs, warmup runs, detects outliers, and can compare multiple commands against each other. It’s become my ad-hoc benchmarking tool of choice.
Zoxide is certainly not just a cd replacement. It’s a completely different way to navigate the filesystem, learning which paths you usually go to so that you can write part of a path and have zoxide take you to a frequently used directory which matches your pattern. If you’re gonna complain about zoxide, complain that it’s a reimplementaiton of something like autojump (Python) or z.lua (lua).
I don’t have any experience with the rest of these projects, but I would bet that the same pattern holds for most of them. It seems like, actually, a lot of value is added by many of these projects.
How sure are you that the existing pieces of software are in fact not broken (in the sense of having some memory-safety-related security vulnerability that might get discovered and exploited tomorrow and become the next Heartbleed)?
I’m using a Hexgears K705A hot-swap board with Kailh BOX White switches and /dev/tty MT3 keycaps. Nice curvy, deep-dish profile, and acceptably clicky switches.
I also have a vintage NTC Alps White board, which has survived a lot of abuse. Those click-leafs, yum. They don’t make them like they used to.
// We're seeing panics across various platforms where consecutive calls
// to `Instant::now`, such as via the `elapsed` function, are panicking
// as they're going backwards. Placed here is a last-ditch effort to try
// to fix things up. We keep a global "latest now" instance which is
// returned instead of what the OS says if the OS goes backwards.
Not that the current time seeming to freeze for minutes or hours seems particularly great either.
yeah, IIRC we saw this in production for the dropbox sync client on macOS and added our own mitigation. interestingly the rust stdlib includes macOS on its Instant::actually_monotonic list…
That’s silly. A commit was made in January. It’s easy enough to do a personal build. It’s not ideal, but it’s certainly not the only project like that. Plus many times, a user might want HEAD build anyway. Ideally once a year release would be nice, but as long as commits are still happening, that’s not “languishing”
Seems appropriate to me, it’s been 59 weeks since the last release, 32 weeks since the last commit, and there’s a fair few fixes for quite annoying issues that have piled up since.
Someone suggesting building your own from a fork of a repository is unlikely to require lecturing on the ease of building your own.
PUSH was a failure. That was one RTT saving too far. When SPDY/H2 were being designed, WebSockets were hot, and everyone assumed HTTP would eventually be bi-directional somehow. PUSH looked less like an architectural violation, and more like an MVP for the future two-way HTTP. Fortunately, PUSH is dead. We’ve got Early Hints now, which are 1xx responses, so they’re even compatible with the good’ol HTTP/1.
Like PHK, I used to be grumpy about HTTP/2 being “too complex”, and mourned loss of being able to type HTTP requests in
telnetby hand. However, unlike PKH, I came around to like HTTP/2, and I think it’s the good default now. Proper parsing of HTTP/1, one that is both robust and compatible with browsers, is actually too complex. This is a syntax that you implement in an evening, but patch edge cases forever (did you knowContent-Lengthis not a number?). H/2 framing and HPACK seems to have more complexity, but once you implement it, you’re done.DoS protection always required extra work beyond what the protocol seemed at face value. H/1 has been hit with slow loris and many types of request smuggling. The recently (re)discovered issue is not that different from TCP SYN flood attacks. We did not disable TCP due to this attack, we’ve hardened implementations instead. H/2 is getting hardened too.
HTTP/2 prioritization wasn’t well implemented for a long time, so naive H/2 servers weren’t winning in benchmarks on pages designed for HTTP/1. But once prioritization was taken more seriously, it actually enabled servers to optimize for fast page rendering (e.g. send JS and fonts before images). People were also inventing hacks for sending low-res image thumbnails first, and replacing them with high-res images later after the page loads. It turns out this can be implemented in H/2 without any JS hacks or changes to the page.
Efficiency of H/2 allowed the crates.io registry to migrate from syncing the whole database over git (which became as large and slow as you can imagine) to fetching only what’s needed over HTTP. The protocol that discovers the required set of data regularly needs to make 500 requests, but with HTTP/2’s single connection and HPACK it’s actually pretty fast and efficient.
I’ll bite! Are you referring to:
or something else? Would be interested to hear more.
Yes. Header merging allows lists. This came up when new HTTP RFC wanted to define strongly-typed headers.
In browsers IIRC the rule is even relaxed and they take the last number in the list.
freebsd-update and portsnap often need to download considerably more than that, and achieve it through a minimal pipelined HTTP/1.1 client.
It would be interesting to see of HTTP/2 or 3 made a noticeable difference.
Browser vendors have wasted years trying to make HTTP/1.1 pipelining work, but eventually given up on it — even when talking to known-good servers there were proxies in between screwing it up, and H/1 has no reliable way to identify the problem so you get data corruption. I presume freebsd gets away with this because they control both ends.
I doubt there is or ever will be a web server deployment that supports HTTP/2 but doesn’t support HTTP/1.1.
One thing that I didn’t see mentioned here -
httpxships with RBS type signatures!I just added these to my tiny monotime gem and uncovered two bugs missed by the test suite in the process. Yay type checking.
A reminder that if you’re measuring durations like this you’ll probably want a monotonic clock.
std::chrono::steady_clock, in this case.Other languages:
clock_gettime(CLOCK_MONOTONIC, &t);System.nanoTimeSystem.Diagnostics.Stopwatchstd::time::InstantProcess.clock_gettime(Process::CLOCK_MONOTONIC)time.monotonicTime::HiRes::clock_gettime(Time::HiRes::CLOCK_MONOTONIC)hrtime(true)performance.now()(Go doesn’t need anything special,
time.Now()includes both real and monotonic times)I have mostly learnt not to let my OCD get the best of me with respect to the state of my
$HOME, but yeah, it would be nice if everybody respected XDG. In a way the situation is now worse than when everything was going straight to$HOME/.myapp.What did get my blood boiling in the past were proprietary apps that dropped a non-dot directory right into my
$HOMEfor their three directories nested, five line XML configuration file. I imagine the situation in the Windows land is so bad nobody even cares anymore and professionals just designate a whole separate partition for any serious data they wish to keep and treat the rest as a radioactive wasteland. I felt really offended to encounter something like that on Linux.But a single app has beaten them all. Czech ID card companion app (
wineonly, obviously) has droppedC:\something.loginto my$HOME. That has really gotten me lost for words.I was a Windows dev for a few years (~ 2002 to ~ 2011). The situation was ludicrous even back then.
Many programs failed to run on Vista when it came out (in ~ 2007) because they were used to writing their configuration into
\Program Files. In *NIX terms that’s like putting your config under/usr/local/bin.Microsoft had been telling folks not to do that ever since
\Program Filesbecame a thing back in (IIRC) Windows 95, but … YOLO I guess.(Parenthetically, that YOLO culture seemed much stronger in Windows-land than it did in *NIX, ever since I started using Linux back in 1995. I don’t know whether it still is; I haven’t programmed on Windows since ~ 2011 and have no desire ever to again.)
I vaguely remember Vista doing something like an overlay file system to prevent apps writing to
Program Fileswhile still retaining user-specific data the application thought it saved there, did it not? I guess everybody was used to file system without any permissions since Windows 98 were still using FAT and many applications were carried over. Windows NT/2000 users probably dealt with the issue by using elevated privileges.That would be UAC Virtualization, triggered for legacy 32-bit apps only.
I wonder how many collective hours have been wasted thanks to the “Python only initializes default parameters when the function is first evaluated” behavior. Such a nasty trap.
It’s bad default, but to me it also shows that they were operating without any adult supervision, because it’s a pretty well known pitfall.
The only reason I was linked this article myself is that a colleague’s PyCharm warned him that my use of
f(x: list[str] = [])was dangerous (Pyright rightfully said nothing of the sort). Digg v4 was launched in 2010 so maybe it wasn’t a footgun people were as-aware of, or as well-checked-for as it should’ve been.I remember learning this in 2013 but it had to be told to me as a fresh Python dev. I never would have known.
I dunno, does it show no supervision? It’s really hard to see such a thing in code review. Assuming they did at all because we’re back in the pre-Git days here and such practices weren’t as easy to come by.
I never spot the Go loop iteration pointer pitfall in code review, and I’ve been reviewing Go code for years. There are some things that are just hard to see.
I’ve been working since 2009 and have never seen a team not doing reviews.
Oh sweet summer child
Foot guns are not solved with more adult supervision
Seeing the countless function definitions on GitHub that include mutable default arguments makes me wonder how many are using the behavior purposefully, and how many were written by folks not familiar with this footgun.
If you’re not editing the list things are totally fine, and it saves a
Nonecheck later on by maintaining a clean interface.There is a regularity to this behavior, but it is definitely a thing where you do want some sort of mechanism to say “no I really mean this” so that we could lint it properly (especially given how typed code with
Noneis a bit of a pain in the butt if the first thing you do is initialize the list when you need it)Honestly if there was a way to say “I’m not going to modify this” in a type hint, that would help a lot here and elsewhere.
Coming from Ruby I really miss being able to
.freezestuff in Python.You could use an empty tuple.
I guess typing it as
x: collections.abc.Sequence = ()works, that could be a good idea.What I usually want is a sort of way to say “this is nullable up to a point” and then after that point to basically redefine it as non-nullabe. Stuff like Rust lets you just shadow names to get you to this effect, though I think Python (well… mypy) doesn’t. I might be wrong on this front, though
You should use a tuple if you want an immutable sequence.
That’s a pretty good trick actually! I could type the value as a sequence (allowing for stuff like lists to be passed in), and of course the default value would not be written to. Love that as a trick
If you’re not editing the list things are totally fine, and it saves a
Nonecheck later on by maintaining a clean interface.There is a regularity to this behavior, but it is definitely a thing where you do want some sort of mechanism to say “no I really mean this” so that we could lint it properly (especially given how typed code with
Noneis a bit of a pain in the butt if the first thing you do is initialize the list when you need it)It is a footgun.
However, it’s also one you hit after having done actual coding in Python for more than 10 lines of code. You never forget to use “None” as the default parameter instead of “[]” or “{}” after the first time it bites you and you have to debug it.
I have to agree with the poster who said this is indicative of a lack of adult supervision. Any linter worth the CPU heat it generates will flag this as a dead obvious bug. Any Python coding experience beyond 30 days will do similar.
And it matches the article: “Senior engineers ghosted out the door, dampening productivity and pulling their remaining friends”
It’s exactly the kind of thing which screws you up when you’re all overtired and in a hurry.
It’s just good for different algorithms than you’d expect, particularly for dynamic programming. Here’s a classic example computing Fibonacci numbers:
Thanks, I hate it
Nice article, very clearly written. A few nitpicks:
For your next article you might want to implement open hashing. It’s slightly more complex, but it’s more efficient because there’s less allocation and the buckets are closer together in memory.
Thanks for your comment, it was helpful! I updated the post to address these points a bit better: https://github.com/robinovitch61/the-leo-zone/commit/4a2e291cfde0a535fc49afebe8a68499899c9024
And, just as with your previous comment, the zero-fill of freshly allocated memory is typically a feature of “real OSs”. On various embedded systems, it is likely you’ll see garbage values instead.
Freshly-allocated pages are zeroed by the kernel, but memory returned by
malloc()will probably be recycled without being zeroed.Which is also why calloc exists. I definitely did get non-zeroes in malloc at some point in time.
And the standard lib malloc isn’t guaranteed to be the one used during runtime.
While most platforms overcommit memory and so won’t fail allocations explicitly by default, they’ll still do so if resource limits are configured, so it’s still something to handle one way or another — even if it’s just a malloc wrapper that aborts with an error.
Fun party fact: for an array of pointers, calloc does not guarantee the behavior you want it to, because an all-zeros bit pattern is not guaranteed to be a null pointer. C++ fixed this with new/new[]. I’m not sure whether null-is-all-zeros is baked as an assumption into LLVM or not.
“Null-is-all-zeros” is baked into POSIX (a requirement). It’s not a requirement for C.
Really? I recall NULL and 0 being used interchangeably in old C code. And it’s still very common to use a pointer as a Boolean condition, like if(ptr)… Are those technically wrong?
Converting between integers and pointers magically treats NULL as 0 at the C language level, even if at the machine level NULL is not the same sequence of bytes as
uintptr_t0It looks great!
As an alternative, if you just want to add timestamps to the output of a command, I can recommend piping to
ts:[0]:
tsis part ofmoreutilswhich has a lot of cool stuff: https://joeyh.name/code/moreutils/.Is there any reason for piping the whole loop through
tsinstead of justecho? (I’m not a shell expert)By having it outside the loop, there is only a single long lived process of
ts. If you had put in right after theechoinside the loop, you’d get a lot of short lived processes. This might matter a lot if this is a loop with a lot of iterations and the speed between iterations are of importantance.That’s neat, I wasn’t familiar with
ts. I was going to recommend piping tolnav -twhich is similar.One of my first Rust projects was a similar tool inspired by ts, rtss, which focuses on relative timestamps.
It’s very similar to
ts -i, but is more efficient, has more than 1s of precision, and can run programs itself, including through a pty for things that would otherwise buffer their output:Anyone know what the default queue does?
“default – the default queue, does what it says on the tin” didn’t really help me much!
I’m also confused by the difference between the ingress queue and the pull queue.
From keeping an eye on the Sidekiq dashboard on my own (single user) instance I’ve noticed that the push queue is by far the busiest, followed by the pull queue. I would expect that those should get a lot more processes/threads than the others, though maybe multiuser servers have different queue usage patterns.
https://docs.joinmastodon.org/admin/scaling/#sidekiq-queues
So
ingressis incoming-push, whilepullis locally originatedUsually calling
println!inside a hot loop is a pretty common performance footgun. Switching out to ,write!(.., "{}\n", ...);on astd::io::Stdoutremoves the penalty almost entirely. This is becauseprintln!performs a syscall and locks and flushes stdout each time. Whereas other methods allow you to buffer your writes and perform the flushes in batches. Usingwriteln!can avoid some of the syscalls and doesn’t require locking stdout each time, but it still flushes the buffer on each call. Usingwrite!does not auto flush on each call and gives full control over locking and flushing.I haven’t tested on this code but I wouldn’t be surprised if it had some impact even though I’m fairly sure the performance is largely bound by other items.
I sat down and made those changes (with some other minor ones as well) to see if there was much of an impact, the end result was on my machine runtime was reduced from 35:28 to 22:58. Here’s the details…
The primary changes I made were:
write!on astd::io::BufWriterwrappingstd::io::StdoutIpv4Addrcan natively be converted to au32, so there isn’t a need to do all the multiplication by octets. Just change.octects()tointo()and the variable binding tolet ipv4: u32 = .... This also reduces the bounds checking performance hit, so it’s a win win. (A fun tidbit if you did need to do the multiplication its usually faster to bitshift, so that code would have become(ipv4[0] << 24) + (ipv4[1] << 16) + (ipv4[2] << 8) + ipv4[3]…is it worth the readability hit? That’s subjective).BufReader::linesallocates a newStringfor each line. We can reduce that by using aStringallocation and passing that as a buffer toBufReader::read_line. This means many lines won’t need to allocate at all. The downside is you have to remember to clear the line buffer before callingread_line()again.RUSTFLAGS='-Ctarget-feature=+av2,+avx' cargo build --release. Alternatively, if you’ll be running the code on the same machine you compile on you can just useRUSTFLAGS='-Ctarget-cpu=native' cargo build --release.There’s some good discussion on if
Stdoutshould use block buffering by default which mentions theprintln!footgun in the Rust CLI book as well as a Reddit post asking about parsing a multi-gb fileThere is probably still things that could be done, but for a few line change that’s a decent speedup.
For writing, you can also slightly reduce the overhead of each write by locking it ahead of time, though I doubt it matters much with a BufWriter.
For reading, you might consider bstr’s for_byte_line or ripline, which in addition to avoiding unnecessary allocations, also avoids unnecessary copying.
Absolutely, and the links have some good discussion (as well as they themselves linking to other good resources) on faster solutions or ways to reduce copies even more, but in this case I wanted to keep it as close(ish) to the original as possible since one of the authors notes was how readable/understandable the Rust code was coming from Python.
Really insightful. Thanks for sharing these improvements. I rolled a few of them into the post and left a credit.
It’s inspired by it, sitting on top of it would be tricky given Lucene is a Java library.
Thanks. I cleared that up.
Elapsed times should be measured with a monotonic clock.
time(3)can jump around and even go backwards if the system clock is changed, butclock_gettime(2)withCLOCK_MONOTONICwill (bugs aside) never go backwards and at worst might get skewed a bit by ntp.C++ has
std::chrono::steady_clock, Rust hasstd::time::Instant, .NET hasStopwatch, Ruby hasProcess.clock_gettime(Process::CLOCK_MONOTONIC)(or I have a slightly nicer interface to it in monotime), Python hastime.monotonic.TIL! Thank you for the feedback :)
On a BSD, hit Ctrl-T.
On Linux, check to see if there’s a SIGUSR1 handler (there is), and send it that. Don’t do this blind, as the default handler will terminate the process.
Oof. If you’d just have left the partial, tarsnap would have skipped uploading 90% of the next archive using timestamps and dedupe - you’d have lost very little in interrupting it.
But why? Ain’t broken, do not fix it. Better cd? Seriously? Better cat? Seriously??? I wish I had the time to rewrite such trivial things in Rust for no good reason. Literally zero value is added by this.
This is literally false. I use bat for its syntax highlighting, and it is a significant value add for me over cat.
But that is not what cat is for. cat is for concatenation, nothing else.
I used to use
catto quickly view short files in the terminal.batreplacedcatfor me:https://github.com/matklad/config/blob/f5a5a9db95b6e2123d2588fe51091fa53208c6e6/home/.config/fish/set_universals.fish#L28
I am sure that this matches significant fraction of usages when people type
cat. I’d even say that viewing files + “priming the pipe” (for which<in.txt progis the “proper” tool) represent majority ofcatusages in the wild.So, I don’t find “
catis for concatenation, nothing else” statement useful, as it doesn’t describe the reality.it does for me. I never use cat to view files, I either use less (sometimes more, if I am on a very restricted system) or vim. I think you could remove cat from my system an I would not even notice, that is how often I use it. Most cli heavy people I know tend to be similar.
Then you were misused it as you should have been using
lessfor viewing files. That is the purpose of pager.lessis less convenient to me. I do not want to have to typeqto close it, and I do want the file to be visible while I type further commands.On a meta level, I just don’t get these prescriptivist comments. I would understand “less is more convenient than
cat” framing, but what is the value of “catshould be used only for concatenation” statement eludes me.cat(1) is also, depending on your platform, for line numbering, non-blank-line numbering, the removal of runs of multiple blank lines, the explicit display of end-of-lines, the explicit display of tabs and form feeds, converting all non-printables to a human-readable escaped form, and for line ending conversion.
1983 called and wants its boring whinge back.
Right, and now we have a program for bonbatenation. It replaces cat and less. I don’t see the problem here.
To be clear, bat’s pager capabilities start and end at running an external pager if stdout is a tty. It supplements less, it doesn’t replace it.
I wonder how much stuff would break if you taught cat to do the same thing. Nothing springs to mind.
If humanity had followed the “if it ain’t broke, don’t fix it” maxim to the tee without exception, we would all still be living in the stone age.
And we might actually be better off that way. At least, a lot less likely to drive half the planet to extinction, and probably not significantly less happy (hedonic treadmill, etc).
key: value, socutusually doesn’t cut it. Having to reach for awk just to get the third column of a thing is unfortunate, so a better cut seems like a good idea.time. Rather, hyperfine is a fairly complete benchmarking solution which does multiple timed runs, warmup runs, detects outliers, and can compare multiple commands against each other. It’s become my ad-hoc benchmarking tool of choice.I don’t have any experience with the rest of these projects, but I would bet that the same pattern holds for most of them. It seems like, actually, a lot of value is added by many of these projects.
tokei replaced my use of sloccount, which is also a giant ball of Perl and assorted support programs.
It’s slightly faster than either. On FreeBSD /usr/src:
How sure are you that the existing pieces of software are in fact not broken (in the sense of having some memory-safety-related security vulnerability that might get discovered and exploited tomorrow and become the next Heartbleed)?
I have been using broot instead of tree and it is quite a joy.
I’m using a Hexgears K705A hot-swap board with Kailh BOX White switches and /dev/tty MT3 keycaps. Nice curvy, deep-dish profile, and acceptably clicky switches.
I also have a vintage NTC Alps White board, which has survived a lot of abuse. Those click-leafs, yum. They don’t make them like they used to.
Will be fixing the missing files on CVS repositories, once I work out why there are missing files on CVS repositories.
Should be fixed now.
Company prescribed me (and everyone) a week off. So, can’t complain. Will likely do „irc bot in rust“ nonsense, for fun.
Also, we just got two kittens from the shelter.
I’ve been doing the Rust IRC bot thing myself, definitely good fun!
For Windows users, I wrote Compactor as a faster alternative to CompactGUI.
Rust’s monotonic time interface defends against this somewhat, seemingly for good reason:
Not that the current time seeming to freeze for minutes or hours seems particularly great either.
But you left out my favorite part of that code comment. I chuckle whenever I stumble upon this file…
The resigned sigh of a programmer having to hack a fix for broken OS code. :D
yeah, IIRC we saw this in production for the dropbox sync client on macOS and added our own mitigation. interestingly the rust stdlib includes macOS on its
Instant::actually_monotoniclist…(former dropboxer who worked on the sync engine)
Or define the group in your interface config:
I have groups for all my interfaces so I don’t have to hardcode anything in pf.conf.
I wrote
cwas a hopefully-fastwcclone, with multiple code paths for various flag combinations and support for threading. Was a fun little exercise.Also note
exais currently languishing, and you probably want to use a fork for the time being.That’s silly. A commit was made in January. It’s easy enough to do a personal build. It’s not ideal, but it’s certainly not the only project like that. Plus many times, a user might want HEAD build anyway. Ideally once a year release would be nice, but as long as commits are still happening, that’s not “languishing”
Seems appropriate to me, it’s been 59 weeks since the last release, 32 weeks since the last commit, and there’s a fair few fixes for quite annoying issues that have piled up since.
Someone suggesting building your own from a fork of a repository is unlikely to require lecturing on the ease of building your own.