Why would [you] write a script to call git from the script, with the exact same arguments, rather than just symlink git to g? If you change the arguments, that’s understandable.
This could totally work, just not with my personal workflow. If git isn’t where I expect, the symlink breaks. git’s path differs on my macOS and Linux machines. It also differs if I decide to install git from Homebrew instead of using the system version.
That’s fair, it had not occurred to me cross OS issues. I actually use git on mac from nix so it would be in an even more different place than yours (e.g. /Users/gabeio/.nix-profile/bin/git).
Fedor Indutny pointed out that npm’s lockfile, package-lock.json, contains a checksum of the package file. The npm people couldn’t easily re-compress existing packages without breaking things.
Is… is it just me or does it seem like it’d be better for npm to have the checksum of the uncompressed file? Or maybe even better: a checksum of a deterministic archive format instead of a general tarfile?
If the checksum was uncompressed, then that would let npm swap to a better compressor seamlessly, or switch to a better compression algorithm like zstd for newer npm builds, or even recompress packages to gzip “on the fly” while silently using better compression behind the scenes
Is there a reason that npm’s lockfile uses a hash of the .tar.gz file? Is there some benefit, e.g. maybe for preventing “zip bomb” style attacks?
It’s a shame, I think this would be a win overall, however small. While I understand that old packages won’t be benefitting from this, common sense tells me that most downloaded packages are often most updated ones and that would make it significant.
At the end of the day, you’ll probably have more impact by convincing the biggest players to reduce their size (like React). I’m not sure I understood what was difficult about integrating the tool into the npm cli, or why slower publishing is a problem (it’s not a common workflow compared to the time spent creating).
Integrating Zopfli into the CLI would require WebAssembly, which the npm maintainers were reticent to use. And slower publishing makes a big difference for some packages—for example, when I tested it on the typescript package, it took 2.5 minutes.
2.5 minutes once a day is.. nothing? Especially when the savings are on the scale of terabytes of data (and that’s just the public registry). I imagine they’re waiting on CI builds for significantly longer than that.
Integrating Zopfli into the CLI would require WebAssembly
Why was wasm necessary over a native (via bindings) or JS implementation?
2.5 minutes once a day is.. nothing? Especially when the savings are on the scale of terabytes of data (and that’s just the public registry). I imagine they’re waiting on CI builds for significantly longer than that.
I think doing this has too many edge cases to be worth it, at least as a default. I imagine a package author on a slow machine, wondering why it takes half an hour to publish. I imagine someone wondering why there’s been a huge slowdown, and emailing npm support or complaining online. I imagine a large module where Zopfli takes a very long time to build.
There are ways around this. You could imagine a command line flag like npm publish --super-compression, or a timeout on Zopfli with a “regular” gzip fallback. But that complexity has to work for hundreds of thousands of package authors.
Why was wasm necessary over a native (via bindings) or JS implementation?
That’s true, wasm isn’t the only solution. Because the compression is CPU-bound, I’d be concerned that a JS solution would be too slow. And a native solution could work, but I recall the npm CLI team trying to keep native dependencies to a minimum to improve compatibility.
but I recall the npm CLI team trying to keep native dependencies to a minimum to improve compatibility.
I’m impressed to find out the cli is entirely implemented in JS. I didn’t do any additional digging to look at the first-party package dependencies, however.
The unlinking behavior makes sense from a security perspective, I think. However this is a good example of what happens when security causes usability problems: People will set up hacks to undermine your security measures.
Ironic, since Signal is pretty famous for having both a high level of security and a high level of usability at the same time. Clearly this is an area that needs more work.
First thought “well if Signal opens on login to the desktop then it’ll stay linked” but if it’s not actually used from that desktop then that’s basically equivalent to OP’s hack which circumvents the security feature.
Also, what if you simply don’t log in to the desktop that often.
Maybe the phone could show a reminder that the desktop app will be unlinked in X day?
It seems odd to fuss over things like Boolean parameters, or variable naming, but wave away the return codes of network API calls. They are important, and they may matter to a lot more people than those who look at your source code.
There are quite a few times I’ve had trouble with an HTTP-based service that returns the wrong status codes, and ended up having to kludge a workaround and/or file a bug report.
For example, there is a difference between status codes 401 “Unauthorized” and 403 “Forbidden”, but that distinction has never been relevant to my work; the computer doesn’t usually care.
It’s an important distinction when returned from a request sent without authorization, since it determines whether the client will bother to look for credentials to retry with. And if the request does have auth, a 401 is simply incorrect.
I’ve been working on a client API that will return 200 OK { data: { error: true } } on damn near everything and it greatly complicates handling HTTP responses.
Same here, it is a total pain. I couldn’t stress enough how important HTTP status codes are for interfacing with an API. Almost every HTTP request/response library relies on throwing errors based on status codes.
I watched an entire engineering team fuck this up in glorious fashion at a medium-sized place. Angry phone calls and war rooms, application teams tearing their hair out, CTO and directors on calls…and the devops group was like “lolwut, it’s green across the board?”
All because nobody managed to close the loop that by switching to GraphQL (instead of something more appropriate) we were dutifully returning 200 successful deliveries of errors.
A little while ago (ok, 14 years) I was working with a wonderful API which would do that, but sometimes (depending on the error path) it wouldn’t even be valid JSON — like {"data":{"error":true,"message":"Can't find "record"}} — because they were all just custom responses … /o\
If you use HTTP just as a pipe that transfers your messages or requests and responses, it rather simplifies everything. Because if you get anything than 200 on the HTTP layer, you know that the pipe is broken. Otherwise you always just parse your message, that is expected to be in well-defined format (I prefer meta-formats that support namespaces i.e. you know what exact type of data arrived and do not rely on mere duck-typing). You can transfer you data through another pipe (like SCTP, XMPP, some other messaging or RPC protocol, socket etc.) and everything works the same.
So the question is whether you want to tie your application to the intrinsics of the HTTP protocol or make it protocol-independent. (I do not say what is generally „better“ – it depends on your use case).
P.S. Obviously, if server differentiates between 401 and 403, the client should handle them appropriately. i.e. if server is tied to HTTP, the client should be too – if server is protocol independent, the client should work the same way.
I think if you’re transacting over HTTP, it makes sense to conform to HTTP semantics. I’m intrigued though, in what cases would you want to transfer data in a protocol agnostic way?
At some point I commented on this site that status codes are for clients you don’t control. If someone else’s client will do something different based on the status code (e.g. a proxy will cache a response, a spider will stop trying to crawl a page) then change the code to make them do the thing. If you control the client, it doesn’t matter because you already have a ton of different ways to communicating your intent to the client.
In some contexts this can change. I think if there’s any reasonable likelihood that a project’s success will lead to additional REST clients, one might as well use tools and build habits that embrace the REST idiom, and that definitely includes using the correct status codes.
A good rule of thumb is to look at the RFC 9110 HTTP semantics section on status codes and see if the status code mentions any request or response headers. Don’t use a status code if you aren’t implementing the logic specified in RFC 9110 related to that code and its associated headers.
For example, 401 Unauthorized relates to the Authorization request header and WWW-Authenticate response header. 403 Forbidden has no such requirements so an application should use that instead for its own permissions errors.
And 412 Precondition Failed relates to precondition headers such as If-None-Match. 409 Conflict has no such requirements so an application should use that instead.
If you don’t want to debate nor want to check a couple of paragraphs of RFC 9110, use 400 Bad Request or 500 Internal Server Error as appropriate. Never pick a status code because it has a name that vaguely suggests roughly the meaning you want.
Not wrong, just different layer. The 200 on the HTTP layer means “we got your request, here is our response” i.e. the communication channel is working, nothing else. It is like receiving a letter inside an envelope – the message is in the letter, not on the envelope.
It is just important that the client and server understand and use the layers in the same way.
The 401 v 403 thing is a pet peeve of mine. Tons of people get it wrong (the naming is, to be fair, terrible), and specifics of the environment I work in cause the difference to matter quite a bit - returning a 401 in response to an auth’d request causes browsers to display the body of the first 401 response, not the last. That means you get something like “please enter a username/password” instead of “you can’t access this because you don’t have X permission”. Only one of those is useful in helping the user solve their problem.
Speaking of pet peeves, another one is 404s from search endpoints to indicate an empty search result set. An empty result set is not the same thing as a non-existent one. From an integrator’s standpoint, returning 404 makes having the wrong URL indistinguishable from having the correct URL and no results.
A lot of apps built in the early aughts and 2010s used 200 (OK) for everything because IE forced them to. Doing so caused over a decade of developer frustration.
Similarly, if you use 301 (Moved) instead of 302 (Found), you better be damn sure that the original URL is never coming back, because once your clients cache that 301, you’re in for a world of hurt if you have to reverse course.
Do fuss over HTTP status codes. But over all I thought the list was great.
There’s now enough IPv6 deployment that you can do most things with a pure IPv6 stack, but a bunch of big services are still missing. If you run a dual-stack implementation then you end up with a big attack surface. Being able to remove IPv4 support eliminates that and also significantly reduces the administrative overhead for managing a network (IPv6 is easier than v4, but v4 + v6 is a lot more complex). The sooner it’s plausible to run a v6-only network, the happier a lot of people will be.
More concretely: For CI systems, a bunch of hosting providers give a discount for machines with only v6 connectivity. For example, the cheapest Vultr node you can buy is $2.50/month with IPv6-only support, $3.50/month for v4 as well. If you want to run your own CI (or mirror of a git repo, or canonical git repo that you mirror to GitHub, or any other thing that integrates with GitHub) then you can save a chunk of money if you can reach GitHub via IPv6.
Is the attack surface of v6 only really smaller than v4 only? A quick review of some vulns I could find has quite a few more affecting v6 than v4, but I didn’t spend much time searching.
The packet parsing code is simpler. I’ve recently been poking at the FreeRTOS network stack for CHERIoT RTOS and so had to read up on all of the various packet formats. If I wanted to write a network stack from scratch and be confident in its correctness and security, and had to pick v4 or v6, I would defined pick v6. For IoT devices, we may be able to ditch v4 sooner than most things, which would be great.
In my experience that part of the network stack is pretty well tested. So I’m not sure if we’re really winning much. DHCPv6 is also a thing, not exactly for IP assignment, but stuff like (s)NTP servers.
IP4 address access is a bit like the water supply – you ignore it until it’s gone. Last year I attempted IPv6 transition and discovered that Github lack of ipv6 broke my deployments. Thus I had to revert to IPv4 NAT access.
There are workarounds that carry the same complexity and cost burden as a proxy or vpn.
Alas with IPv6 the weak link breaks the chain. You’ll need all of your dependencies to support ipv6 and a majority of internet apps depend on Github.
Because there are 8 billion people but only 4 billion IPv4 addresses. Here’s a quote from a github user in Brazil:
New ISPs in my country are IPv6-only because there is no new IPv4 space to be provided to them. They do have a over-shared IPv4 address by CGNAT but due to the oversharing, it is unstable and not rare to be offline. For these companies, the internet access is stable only in IPv6.
Thinking about the server-side, some cloud providers are making extra charges for IPv4 addresses (e.g.: Vultr.com) so most of the servers in my company are IPv6-only. Cloning github repositories is very cumbersome due to the lack of IPv6 support and this issue affects me and my team mates on a daily basis.
Has anyone done ‘git torrent’ yet? Decouple the naming of hashes from their storage+distribution? Basically just use bittorrent as the sha-1 addressed blob storage backing a git client.
Yeah, CJB built a prototype and explains how it works here. I try to package it every few months, but it’s resisted me so far; it depends on a patched DHT library. I would be amenable to building something new, particularly something which doesn’t have cryptocurrency integration.
On my site, I use the smallest PNG as an image placeholder before JS can load the final images. It’s included on the page as a base64 data URL. It looks like <img src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==">.
I want the space to be held by the image even before the JS kicks in and decides what to put there. I could just wait and add the img tag in JS, but it’s easier to have the img tag already there and then update it. If the src is blank, the page is invalid and the alt text shows through, so I need it to have a valid image at load time.
Related, it bothers me very slightly that PNG has such big headers that it’s larger than GIF for very small images even though the compression is better for anything substantial.
I use the smallest PNG as an image placeholder before JS can load the final images.
This shit is the bane of the modern Web. Why is it that so many webdevs are seemingly incapable of putting text and media on a webpage without forcing users to run their shitty scripts? It is poor a11y and takes more effort than not doing it.
I agree that JS is often overused (I disable it by default when browsing the web), but I think it’s wrong to call this “the bane of the modern Web” without knowing the specifics of the product. One of many possibilities: the server cannot know what the src is and it can only be known by the client.
You know nothing about my work, and my site loads in under a second while using a vanishingly small amount of JS because I expend a huge amount of personal effort to not just do the easy thing and include some dumb ass giant script that from Google or whatever, so kindly get bent. ☺️
My favorite quote: “what’s good and bad in a programming language is largely a matter of opinion rather than fact, despite the certainty with which many people argue about even the most trivial features of Go or any other language.”
I was just coming here to say something similar. Coincidentally, I was struck by that bit after seeing the debate about whether readability in code is subjective or objective in this other current thread.
This is such a convenient and contrived example. What if the argument actually should be between 7 and 42? There’s no way really to remove all the validation checks at initialization time, due to various constraints. Later on, specialization might arise for argument values of 8 or 16. In some languages you could specialize in the static types, but the extra complexity in the type level programming would soon out grow that actual usefulness.
I agree. This was a trivial example that doesn’t need much context—maybe good for a short blog post, but bad for demonstrating how a type system can help shorten your code in more realistic cases. I’ll think about making an edit of some kind.
Something about this example (if not necessarily the sentiment, I like static types) rubbed me the wrong way, and I think looking at the real code may have helped nail down why.
You raise a TypeError in the real code, and if you’re trying to simulate “type is an unsigned byte” then it seems a given that it is shorter in a statically typed language.
Can you really have a length of 0 for a Squid? Seems error prone. If not, the code should be checking for min_length > 0.
If it was a different range, say 0-100 you wouldn’t have a type that happened to conveniently overlap. Other comments are mentioning languages with support for defining types like this, which is definitely useful but I don’t know if you’re still making things “shorter” then.
Imagine a handy attr_byte, in the same way Crystal has a handy UInt8:
attr_byte :min_length
def initialize(min_length)
self.min_length = min_length
end
Still dynamic, still a runtime check, still short.
On the flip side we can imagine a statically typed language, with only signed bytes (Java), where typing the argument doesn’t get you the range of 0-255 no matter what type you choose.
This is what I noticed, too: it works because the value happened to be exactly a uint8. It’s trickier when you want a non 2^n range (and you’re not using Ada).
I agree. This was a trivial example that doesn’t need much context—maybe good for a short blog post, but bad for demonstrating how a type system can help shorten your code. I’ll think about making an edit of some kind.
If you are able to delete the runtime checks from the Ruby code and replace them with static types, then the runtime checks were never necessary in the first place. The semantic equivalent to a type-check would be an assert statement, not a check and raise. So either the Ruby code is wrong or your translation is wrong. You should only need to validate data at the program’s boundaries with untrusted sources. At the interior of the program, it should expect data to already be valid. The superfluous runtime checks in the Ruby code are indicative of a lack of discipline around what data is trusted or not trusted in the program and that means it’s likely there are security vulnerabilities present.
The discipline argument has always been odd to me. Have you worked on large projects with multiple different people? How do you expect everyone to keep all the assumptions about what data has been validated (and how) in their heads, how do you synchronize it even?
Offloading a hugely amount of tedious work that a computer can do quickly and automatically to humans is, in my opinion, a ridiculous proposition.
How do you expect everyone to keep all the assumptions about what data has been validated (and how) in their heads, how do you synchronize it even?
I recognize this is an OO idealist perspective, but like sandi metz says, if statements are type checks.. don’t type check.
Parse, don’t validate applies to dynamic languages just as much as static, note that these ideas becomes easier when you work with immutable data so you only have to check invariants in constructors.
After you define your types (classes, records), you write your code that depends on those assumptions as methods, IMO this is better in languages like clojure where you can define these away from type definitions via a protocol.
This to me is the core idea that people miss who either do OO wrong, or knock it.
I’m not saying its easy, I think in fact it still does require a lot of discipline, but its a well defined discipline.
What is interesting to me is this form of classification + polymorphism, and importantly the kind of dynamic code that the author was trying to replace with static typed code wouldn’t exist in this paradigm.
To be clear, my whole point is that this debate is actually bigger than static v dynamic, or FP v OO, its about generic code & there are really good examples of that in all these paradigms.
You’re ranting sufficiently hard that it’s difficult to tell what you’re trying to say.
It sounds like you’re contrasting assert(condition) with runtime checks, but assert(condition)is a runtime check. It’s executed at runtime, not at compile-time, and it checks whether condition is true. The difference between the two is whether the error is expected to be caught or not, I think?
My best guess at what you’re actually trying to say is:
For all internal facing interfaces, assume that no mistakes are made and all data is well formed. If you’re going to check at all, check only using assert(condition) statements in unit tests. (That’s what the link you shared says, though you didn’t mention unit tests.)
For all external facing interfaces, don’t trust your users and put runtime checks on everything.
Except I’m not sure why you’re attacking this article (“superfluous”, “lack of discipline”, etc), because it seems perfectly likely that the interface it discusses is external facing, in which case it is following your advice.
Responding to your actual point, that’s a very good approach and I try to follow it too, with one very large exception. If you’re working in a medium to large codebase (say 10,000+ LOC), it’s unrealistic to expect all data to be well formed and all function arguments to be correct. Thus I aim to organize code into modules (which means different things in different languages) that are impossible to misuse. If a “module” needs to maintain an invariant, it must either (i) have an interface that makes it impossible to violate that invariant, no matter what the calling code does, or (ii) check that the invariant is obeyed in its boundary, and error if it’s violated. This way every “module” guarantees that its invariants are obeyed within it, and if they’re not it’s a bug in the module (and you don’t have to look elsewhere).
check only using assert(condition) statements in unit tests.
assert statements are not only for unit tests. They are also enabled in debug builds.
Except I’m not sure why you’re attacking this article (“superfluous”, “lack of discipline”, etc),
Not sure what you mean by “attacking.” If you mean that my intent was to personally demean or disrespect the author, then that’s incorrect. My intent was to accurately describe what I observed in the article. Those are neutral words in my vocabulary when discussing engineering.
it’s unrealistic to expect all data to be well formed and all function arguments to be correct.
This statement is equivalent to saying that it’s unrealistic to produce working code.
If a “module” needs to maintain an invariant, it must either (i) have an interface that makes it impossible to violate that invariant, no matter what the calling code does, or (ii) check that the invariant is obeyed in its boundary, and error if it’s violated.
This is false. You’re expressing a personal API design philosophy or engineering discipline but not a hard reality. For a topical example, the standard multiplication operator does neither (i) or (ii) and freely allows incorrect usage that results in overflow. Often it’s not feasible to always check at runtime that the operands of the multiplication operator don’t result in an overflow nor have some special paired number type that guarantees that its members don’t overflow when multiplied. The reasonable approach is to structure the program such that it’s an invariant that the operands at the multiplication site don’t overflow, with a debug assertion. Not only is this the reasonable approach, this is the approach Zig and Rust take.
Not sure if you’re deliberately trolling… if you’re not, know that (i) you have misunderstood what I said, and (ii) I still don’t know what you’re advocating for, and it would be helpful if you clarified that instead of EDIT: arguing against your repeated misunderstandings of other people’s positions.
Can you explain what it is you believe I have misunderstood?
I don’t believe that would be productive. What would be productive is if you simply wrote down what you’re advocating for. I made a guess above, is that accurate, modulo that you would put asserts (which execute at runtime in debug mode but not in production) in the code as well as in unit tests?
I don’t believe I was “attacking” you per my previous comment.
Apologies, I should have said “arguing against” instead of “attacking”. A lot of your phrasing makes it sound like you’re angry, though I realize now you’re not. Edited.
I made a guess above, is that accurate, modulo that you would put asserts (which execute at runtime in debug mode but not in production) in the code as well as in unit tests?
Your initial guess was correct, sans the implication that asserts should only be run in unit tests.
Ok. First of all, please realize that what you were suggesting wasn’t clear until now. Actually I’m still not entirely clear on how many assert()s you suggest having. Should most internal invariants be checked with assert()s in well chosen locations? Your first comment seems to me to say “yes” in some places and “no” in others, to that question.
Let me try to state the crux of the matter precisely. Say you’re working in a company with a hundred other employees and 100,000+ lines of code, and everyone follows your advice to the letter. What will happen is that you’ll end up with bugs that are extremely difficult to debug, because the place they manifest is far away from the actual bug.
For example, there’s a data type Account that requires a complicated invariant. If this invariant is violated, it doesn’t cause an error in the Account code. Instead it causes another invariant in a different class History to be violated (because the code that interlinks the two relies on the Account invariant), and then some code that uses History ends up accessing an array out of bounds. So now you see an array out of bounds error (hopefully you’re not working in C++, so it is at least guaranteed to stop there), and you need to work backwards to see that it came from a bad History, and then work backwards from there and see that it came from a bad Account, instead of one of the dozen other ways a History could have been corrupted if there was a bug elsewhere in the codebase. Oh, and this error wasn’t caught in your unit tests and only happened in production, once.
In contrast, if you consider the boundary of Account and of History to be “external facing” and check their invariants on their boundaries, even in production, then you’d get a very clear error message at the actual cause of the error: an invalid Account construction.
I’ve worked at a company where all the code assumed you wouldn’t make a mistake, and at a company where the code assumed you might make a mistake and would error if you used it wrong. Issues at the former company were so much harder to debug, it was a nightmare.
Actually I’m still not entirely clear on how many assert()s you suggest having.
You can have zero asserts or an unbounded amount of asserts. Assertions on input data are not significantly wordier than type declarations.
What will happen is that you’ll end up with bugs that are extremely difficult to debug, because the place they manifest is far away from the actual bug.
Then assert at the earliest location possible. There is usually little downside.
Then assert at the earliest location possible. There is usually little downside.
The downsides of runtime checks, according to you:
the runtime checks were never necessary in the first place
The superfluous runtime checks in the Ruby code are indicative of a lack of discipline around what data is trusted or not trusted in the program and that means it’s likely there are security vulnerabilities present.
Redundant checking of invariants is wasteful and sloppy.
However, you’re saying that you can assert with little downside. So this whole time you’ve been quibbling about the difference between:
unless min_length.is_a?(Integer) && min_length >= 0 && min_length <= 255
raise "Minimum length has to be between 0 and 255"
end
Assertions are semantically different than explicit checks that return errors or raise exceptions. An assertion means that the caller is required to make sure arguments satisfy certain properties, otherwise unspecified behavior may occur. An explicit check is a specification that in the event that arguments are invalid, the function will return an error or raise an exception.
Given the meaning of an assertion, it’s valid to disable them in production. That’s why this distinction was created.
You say this like it’s an unassailable fact, but it actually varies between languages and between developer conventions.
My Google searching suggests that Ruby doesn’t even have an assert statement outside of a unit testing module, so it’s a strange place to start this discussion.
Rust has both assert, which runs in both debug and production, and debug_assert which runs only in debug mode. You often need to use assert instead of debug_assert to uphold Rust’s safety guarantees. E.g. if a public non-unsafe function could cause memory corruption if passed invalid arguments, it is incorrect for that function to not check those arguments in production, according to Rust’s safety guarantee. Rust’s safety guarantee is taken pretty seriously: e.g. I think most Rust library authors would remove any dependency they knew violated it.
As far as developer conventions: it is entirely possible for developers to implement an assertion using an exception. An assertion (as you’ve defined it) is a semantic concept, which has many possible implementations. There are all sorts of reasons that a developer might choose to implement an assertion with an exception.
For example, here’s the code from the article, written using an assertion (remember, as you pointed out, assertion is a semantic concept, so it’s perfectly fine to implement it using raise):
// Initialize with a length between 0 and 255, inclusive.
// The behavior is unspecified if the length is not in this range.
def initialize(min_length)
unless min_length.is_a?(Integer) && min_length >= 0 && min_length <= 255
raise "Minimum length has to be between 0 and 255"
end
....
end
The difference is that I added a comment saying that the behavior is unspecified if the argument is invalid. (It should be in the function documentation; I don’t know the syntax for that in Ruby but pretend the comments are official documentation.) To make debugging easier, the “unspecified behavior” is implemented as an exception that will be raised, even in production.
otherwise unspecified behavior may occur
The central point that I and others have been trying to convey, is that unspecified behavior in large systems is really bad. It makes debugging a nightmare. You need some, but should keep it local.
At a high level I think your main point of contention is that redundant validity checks (so-called “defensive programming”) result in a more resilient system whereas in my opinion it’s a symptom of a system that was designed without proper discipline around trusted / validated values. These are conflicting principles.
The “defensive programming” principle, if taken to its logical conclusion, results in a system where every argument is validated upon every function call and its not clear which part of the program is responsible for the primary validation of input data. Long-term maintenance and refactoring becomes difficult because it’s not clear who does what. Whereas with the “unspecified behavior” principle, if taken to its logical conclusion, results in a system that may start failing silently if there is a single bug in the program in a difficult-to-debug way, though it’s more clear what part of the system is responsible for validation and makes long-term maintenance and refactoring easier.
I think you want to bias towards building the latter system for hopefully obvious reason (e.g. explicitly checking a value you know to be valid over and over again serves no purpose and is confusing to people who read the program). To mitigate the issue of incorrect code / logical errors / “bugs”, asserts provide a way to materialize into code the abstract specifications about what functions expect. They serve as living executable documentation that can be selectively enforced depending on the paranoia of the programmer. If when engineering your system, you expect the invalid values or states to happen only when there are bugs in the program, then it’s reasonable that you enable assertions during testing and debugging and disable them once you are ready for release and all functional specifications of the program have been QA tested.
One issue is that lots of web programmers never see a proper “release” so the debug-enable / release-disable convention with assertions isn’t practical to them. The code is never done, there is no comprehensive QA, it’s never golden master. Therefore in those situations I’d recommend just always leaving assertions enabled, regardless of language used. I’d still recommend keeping the assertion concept distinct from normal logic in the program, if only to provide option value and serve as documentation for other programmers and perhaps static analysis tools.
Thanks for the chat, I enjoyed it and I hope the exposure to a different point of view was helpful. Perhaps you can suggest using asserts more liberally to your team the next time you find yourself working in a large codebase that has no mitigations for programmer error, as a sort of compromise to reduce the pain of debugging. Best wishes.
Sorry I can’t say the same, I found the conversation extremely frustrating. I can’t say I’ve learned anything, because even at this point I’m not sure if we have different views. Please try to learn from the fact that this was a huge thread with a lot of people being very confused at what you were saying. You need to give way more context and explanation up front.
If you don’t trust your code to operate correctly in production then leave assertions on. Use a process monitor to monitor when assertions fail and provide reasonable feedback to the user. Or use a more restrictive language.
That’s not my suggestion. There is a semantic difference between assertions and explicit checks. In simple terms, assertions are the runtime version of type-checks. Explicit checks are new behavior. Assertions afford you the ability to turn them off and indeed I run my production code with them off. I recommend that everyone build their systems such that they can feel confident turning off assertions in production.
My original comment pointed to the semantic inconsistency between the original Ruby version and the translated Crystal version. The article stated that the validation check was deleted in the translated version implying that it was redundant validation code. Redundant validation code signals a lack of discipline in terms of how trusted and untrusted values are handled throughout the code. TFA’s author clarified that the semantics of the translated code were in fact different from the original and that the validation was not simply deleted as originally stated in the article but moved.
If that’s the case, then the translation is not equivalent. The initialize method should take a wide integer type and then do the sanity checking for the correct range inside that method just as the Ruby code does. Since you replaced the argument type with a narrow integer type, that implies that you have moved the sanity checking for the value’s range to before the initialize method is called. That changes the contract of the translated method.
While you are technically correct in the sense that the domain of the two function is not identical, I think you’re being overly pedantic or disingenuous. The whole point is that static type checking allows you to statically restrict the domain in a way that is simply not possible in dynamic languages. The only reason why the domains are not identical, is because Ruby lacks the ability to restrict domains at all. As a result, the Ruby function is not total.
In other words, of course the Crystal version should not take a wider integer range as input. That way, library users cannot pass in arguments that result in an exception, a thing that is not possible in Ruby.
An accusation of being disingenuous requires an assumption of bad faith, which I think would be unreasonable and/or unwarranted in this context.
The only reason why the domains are not identical, is because Ruby lacks the ability to restrict domains at all.
Where input validation happens is incidental to whether the program is statically or dynamically typed. It’s not a given that the reason the input validation occurred in the initialize method originally is because Ruby lacks static typing. An alternative Ruby implementation could have been as follows:
class Foo
# min_length is expected to be integer-like in the range of 0 <= 255
def initialize(min_length)
@min_length = min_length
end
end
obj = Foo.new
min_length = read_min_length_from_external_source()
# make sure min_length is according to spec
unless min_length.is_a?(Integer) && min_length >= 0 && min_length <= 255
raise "Minimum length has to be between 0 and 255"
end
obj.initialize min_length
In other words, of course the Crystal version should not take a wider integer range as input. That way, library users will helpfully know how to actually use it correctly.
I don’t think that’s a given either. The input validation has to happen somewhere. In the original code the author says that it was specified to occur in the initialize method. When translating a program from one language to another I think it’s less error-prone to not change input validation sites and method contracts as much as possible. Every call site now needs to be aware of the new semantics of the initialize method.
Where input validation happens is incidental to whether the program is statically or dynamically typed. It’s not a given that the reason the input validation occurred in the initialize method originally is because Ruby lacks static typing.
As @EvanHahnalready pointed out, the initialize function is the library boundary. I therefore think it is reasonable to assume that input validation has to happen in the initialize method.
When translating a program from one language to another I think it’s less error-prone to not change input validate sites and method contracts as much as possible.
I don’t think that is unreasonable, but I also think it depends on the context. You could start out with a exact replica, make sure everything type checks, and then gradually add stricter constraints, letting the compiler help you maintain functionality along the way. If you aren’t taking advantage of the features of the new programming language, why are you translating your program at all?
Every call site now needs to be aware of the new semantics of the initialize method.
True, but the compiler will tell you. So there’s no chance you will accidentally misuse the initialize method. It is simply not possible.
As @EvanHahn already pointed out, the initialize function is the library boundary. I therefore think it is reasonable to assume that input validation has to happen in the initialize method.
Library boundaries are not necessarily trust boundaries. In fact, they often should not be since it’s usually only a small percentage of your program where you are accepting untrusted user input.
I… am not sure what you are saying here? Aren’t library boundaries literally the exact points in a library where you are accepting “untrusted user input”? Perhaps I am misunderstanding what you mean by trust boundary, or what kind of untrusted user input you mean?
To clarify, in the case of a library I use the word “user” to denote the developer using the library. The “input” is the arguments that the user passes to the library boundary functions. That user input is (and should be) untrusted by the library author.
I… am not sure what you are saying here? Aren’t library boundaries literally the exact points in a library where you are accepting “untrusted user input”? Perhaps I am misunderstanding what you mean by trust boundary, or what kind of untrusted user input you mean?
Untrusted user input means input coming from outside of the process, in particular the actual input to your program from its users. Any agent that your program can not trust to produce valid input.
There is no functional trust difference between library code and your own code, it’s simply code that is packaged separately. Regardless of how the library code is packaged, you must trust it does what it is specified to do and it must trust that you use it the way it is specified to be used. The same trust applies to your own code. If you cannot trust a library or the library cannot trust you, you cannot link to it liberally and expect your program to function correctly.
I’ve been the lone developer of a project since the project’s creator and my main collaborator passed away last year. They themselves were the lone developer of this project for a period.
They knew what was coming, and adopted what they coined ‘mortality driven development’. This manifested itself as most of the codebase being comments and tests, and a wealth of diagrams and documents explaining how things worked and the rationale behind them.
And so I never felt lost in their codebase, and sometimes even relied on them to help me write new features long after their death. I can’t say I’ve been half as diligent in my stewardship of the project — but this post helps remind me why it matters.
Description: Signal is a 501c3 nonprofit organization developing open source privacy technology that protects free expression and enables secure global communication. We’re a fully distributed/remote, small team looking to grow slightly (currently 30 people total/20 engineers). If you care about code quality as much as you care about user privacy, we’d love to talk with you.
Tech stack: Swift/Objective-C, Java/Kotlin, TypeScript, Java, Rust. We publish all our codebases @ https://github.com/signalapp, check ’em out!
Compensation: Competitive with Big Tech, without selling your soul or compromising your ethics. Full 401k match, health/vision/dental, parental leave, and self-managed time off.
Contact: workwithus at signal dot org (several of us staff that email alias). You can also DM me on here, or email evanhahn at signal dot org.
Why would [you] write a script to call git from the script, with the exact same arguments, rather than just symlink git to g? If you change the arguments, that’s understandable.
This could totally work, just not with my personal workflow. If
gitisn’t where I expect, the symlink breaks.git’s path differs on my macOS and Linux machines. It also differs if I decide to installgitfrom Homebrew instead of using the system version.That’s fair, it had not occurred to me cross OS issues. I actually use git on mac from nix so it would be in an even more different place than yours (e.g.
/Users/gabeio/.nix-profile/bin/git).Similar discussion here.
Is… is it just me or does it seem like it’d be better for npm to have the checksum of the uncompressed file? Or maybe even better: a checksum of a deterministic archive format instead of a general tarfile?
If the checksum was uncompressed, then that would let npm swap to a better compressor seamlessly, or switch to a better compression algorithm like zstd for newer npm builds, or even recompress packages to gzip “on the fly” while silently using better compression behind the scenes
Is there a reason that npm’s lockfile uses a hash of the
.tar.gzfile? Is there some benefit, e.g. maybe for preventing “zip bomb” style attacks?I agree. Users will almost always be unpacking the archive anyway, so you’d still need to protect against “zip bombs” and potentially other issues.
Maybe there’s something I’m missing, though.
In any case, such a change would be backwards incompatible.
It’s a shame, I think this would be a win overall, however small. While I understand that old packages won’t be benefitting from this, common sense tells me that most downloaded packages are often most updated ones and that would make it significant.
At the end of the day, you’ll probably have more impact by convincing the biggest players to reduce their size (like React). I’m not sure I understood what was difficult about integrating the tool into the npm cli, or why slower publishing is a problem (it’s not a common workflow compared to the time spent creating).
Integrating Zopfli into the CLI would require WebAssembly, which the npm maintainers were reticent to use. And slower publishing makes a big difference for some packages—for example, when I tested it on the
typescriptpackage, it took 2.5 minutes.https://www.npmjs.com/package/typescript?activeTab=versions
2.5 minutes once a day is.. nothing? Especially when the savings are on the scale of terabytes of data (and that’s just the public registry). I imagine they’re waiting on CI builds for significantly longer than that.
Why was wasm necessary over a native (via bindings) or JS implementation?
I think doing this has too many edge cases to be worth it, at least as a default. I imagine a package author on a slow machine, wondering why it takes half an hour to publish. I imagine someone wondering why there’s been a huge slowdown, and emailing npm support or complaining online. I imagine a large module where Zopfli takes a very long time to build.
There are ways around this. You could imagine a command line flag like
npm publish --super-compression, or a timeout on Zopfli with a “regular” gzip fallback. But that complexity has to work for hundreds of thousands of package authors.That’s true, wasm isn’t the only solution. Because the compression is CPU-bound, I’d be concerned that a JS solution would be too slow. And a native solution could work, but I recall the npm CLI team trying to keep native dependencies to a minimum to improve compatibility.
I’m impressed to find out the cli is entirely implemented in JS. I didn’t do any additional digging to look at the first-party package dependencies, however.
https://github.com/npm/cli
In alphabetical order, here are a few that come to mind:
entr: run a command when files changeffmpeg: convert audio/video (see also: ffmpeg buddy)ncdu: analyze disk spacepwgen: generate passwordsshellcheck: lint shell scriptstig:git loginterface I prefer (among other features)tmux: windows and splits in your terminal (among other features)tree: print directory trees nicelyyt-dlp: download audio/video from many sourceszopfli: make gzip-compatible files that are smaller thangzip. Also seezopflipngncduis one I can never remember when I need to go byte hunting. I’ll add it to my nixos config this time :)Consider trying duc : http://duc.zevv.nl/
I love this idea. A very similar blog post: “Don’t use booleans”.
The unlinking behavior makes sense from a security perspective, I think. However this is a good example of what happens when security causes usability problems: People will set up hacks to undermine your security measures.
Ironic, since Signal is pretty famous for having both a high level of security and a high level of usability at the same time. Clearly this is an area that needs more work.
First thought “well if Signal opens on login to the desktop then it’ll stay linked” but if it’s not actually used from that desktop then that’s basically equivalent to OP’s hack which circumvents the security feature.
Also, what if you simply don’t log in to the desktop that often.
Maybe the phone could show a reminder that the desktop app will be unlinked in X day?
When I worked at Signal years ago, this was something we discussed but never implemented. I don’t know the idea’s current status.
I think they do it now. I got a notification somewhere that my iPad was about to get unlinked.
It seems odd to fuss over things like Boolean parameters, or variable naming, but wave away the return codes of network API calls. They are important, and they may matter to a lot more people than those who look at your source code.
There are quite a few times I’ve had trouble with an HTTP-based service that returns the wrong status codes, and ended up having to kludge a workaround and/or file a bug report.
It’s an important distinction when returned from a request sent without authorization, since it determines whether the client will bother to look for credentials to retry with. And if the request does have auth, a 401 is simply incorrect.
I’ve been working on a client API that will return
200 OK { data: { error: true } }on damn near everything and it greatly complicates handling HTTP responses.Same here, it is a total pain. I couldn’t stress enough how important HTTP status codes are for interfacing with an API. Almost every HTTP request/response library relies on throwing errors based on status codes.
Must be a pain to monitor… everywhere I worked there was a metric on the number of 5xx returned…
Monitoring, now there’s an idea!
GraphQL is infamous for this. CouchDB has a few APIs that do this too.
I watched an entire engineering team fuck this up in glorious fashion at a medium-sized place. Angry phone calls and war rooms, application teams tearing their hair out, CTO and directors on calls…and the devops group was like “lolwut, it’s green across the board?”
All because nobody managed to close the loop that by switching to GraphQL (instead of something more appropriate) we were dutifully returning 200 successful deliveries of errors.
A little while ago (ok, 14 years) I was working with a wonderful API which would do that, but sometimes (depending on the error path) it wouldn’t even be valid JSON — like
{"data":{"error":true,"message":"Can't find "record"}}— because they were all just custom responses … /o\This depends on your point of view.
If you use HTTP just as a pipe that transfers your messages or requests and responses, it rather simplifies everything. Because if you get anything than 200 on the HTTP layer, you know that the pipe is broken. Otherwise you always just parse your message, that is expected to be in well-defined format (I prefer meta-formats that support namespaces i.e. you know what exact type of data arrived and do not rely on mere duck-typing). You can transfer you data through another pipe (like SCTP, XMPP, some other messaging or RPC protocol, socket etc.) and everything works the same.
So the question is whether you want to tie your application to the intrinsics of the HTTP protocol or make it protocol-independent. (I do not say what is generally „better“ – it depends on your use case).
P.S. Obviously, if server differentiates between 401 and 403, the client should handle them appropriately. i.e. if server is tied to HTTP, the client should be too – if server is protocol independent, the client should work the same way.
I think if you’re transacting over HTTP, it makes sense to conform to HTTP semantics. I’m intrigued though, in what cases would you want to transfer data in a protocol agnostic way?
Related discussion: https://lobste.rs/s/cdkkp6/software_engineers_are_not_should_not_be#c_cwmmqa
Perhaps this a belief I will change.
At some point I commented on this site that status codes are for clients you don’t control. If someone else’s client will do something different based on the status code (e.g. a proxy will cache a response, a spider will stop trying to crawl a page) then change the code to make them do the thing. If you control the client, it doesn’t matter because you already have a ton of different ways to communicating your intent to the client.
In some contexts this can change. I think if there’s any reasonable likelihood that a project’s success will lead to additional REST clients, one might as well use tools and build habits that embrace the REST idiom, and that definitely includes using the correct status codes.
I take Evan’s point to have been that if the “correct” status code isn’t obvious, then it doesn’t matter:
SOAP did everything with POST returning 200. That was wrong, but debating 410 Conflict vs 412 Precondition Failed is also a waste of time.
A good rule of thumb is to look at the RFC 9110 HTTP semantics section on status codes and see if the status code mentions any request or response headers. Don’t use a status code if you aren’t implementing the logic specified in RFC 9110 related to that code and its associated headers.
For example, 401 Unauthorized relates to the Authorization request header and WWW-Authenticate response header. 403 Forbidden has no such requirements so an application should use that instead for its own permissions errors.
And 412 Precondition Failed relates to precondition headers such as If-None-Match. 409 Conflict has no such requirements so an application should use that instead.
If you don’t want to debate nor want to check a couple of paragraphs of RFC 9110, use 400 Bad Request or 500 Internal Server Error as appropriate. Never pick a status code because it has a name that vaguely suggests roughly the meaning you want.
Yes, this was my intent.
Not wrong, just different layer. The 200 on the HTTP layer means “we got your request, here is our response” i.e. the communication channel is working, nothing else. It is like receiving a letter inside an envelope – the message is in the letter, not on the envelope.
It is just important that the client and server understand and use the layers in the same way.
The 401 v 403 thing is a pet peeve of mine. Tons of people get it wrong (the naming is, to be fair, terrible), and specifics of the environment I work in cause the difference to matter quite a bit - returning a 401 in response to an auth’d request causes browsers to display the body of the first 401 response, not the last. That means you get something like “please enter a username/password” instead of “you can’t access this because you don’t have X permission”. Only one of those is useful in helping the user solve their problem.
Speaking of pet peeves, another one is 404s from search endpoints to indicate an empty search result set. An empty result set is not the same thing as a non-existent one. From an integrator’s standpoint, returning 404 makes having the wrong URL indistinguishable from having the correct URL and no results.
The problem, IMO, is thinking about status codes primarily as numbers. Better to think of them as the associated cat memes.
302 Found
Location: https://http.cat/
A lot of apps built in the early aughts and 2010s used
200 (OK)for everything because IE forced them to. Doing so caused over a decade of developer frustration.Similarly, if you use
301 (Moved)instead of302 (Found), you better be damn sure that the original URL is never coming back, because once your clients cache that301, you’re in for a world of hurt if you have to reverse course.Do fuss over HTTP status codes. But over all I thought the list was great.
I agree. After I learned that “kilobyte” was an ambiguous term, I use “kibibyte” every time I want to be exact.
…unless you really mean kilobye, in which case the correct term is the ambiguous one.
Which is an argument to use kibibytes whenever possible.
Just say 0.9765625 KiB :)
If you need to refer to a decimal number of bytes you can just say e.g. 3000 bytes or 3e3 bytes or whatever.
I know very little about networking. Why do people care about this?
There’s now enough IPv6 deployment that you can do most things with a pure IPv6 stack, but a bunch of big services are still missing. If you run a dual-stack implementation then you end up with a big attack surface. Being able to remove IPv4 support eliminates that and also significantly reduces the administrative overhead for managing a network (IPv6 is easier than v4, but v4 + v6 is a lot more complex). The sooner it’s plausible to run a v6-only network, the happier a lot of people will be.
More concretely: For CI systems, a bunch of hosting providers give a discount for machines with only v6 connectivity. For example, the cheapest Vultr node you can buy is $2.50/month with IPv6-only support, $3.50/month for v4 as well. If you want to run your own CI (or mirror of a git repo, or canonical git repo that you mirror to GitHub, or any other thing that integrates with GitHub) then you can save a chunk of money if you can reach GitHub via IPv6.
Is the attack surface of v6 only really smaller than v4 only? A quick review of some vulns I could find has quite a few more affecting v6 than v4, but I didn’t spend much time searching.
The packet parsing code is simpler. I’ve recently been poking at the FreeRTOS network stack for CHERIoT RTOS and so had to read up on all of the various packet formats. If I wanted to write a network stack from scratch and be confident in its correctness and security, and had to pick v4 or v6, I would defined pick v6. For IoT devices, we may be able to ditch v4 sooner than most things, which would be great.
theres a whole slew of DHCP and NAT code that is not going to be needed anymore that should be factored in to your search.
That: and a lot of code on switches since the routing tables of big iron internet routers is so fragmented now that it needs additional handholding.
In my experience that part of the network stack is pretty well tested. So I’m not sure if we’re really winning much. DHCPv6 is also a thing, not exactly for IP assignment, but stuff like (s)NTP servers.
Dual stack is definitely much harder to maintain.
Because people want to move from IPv4 to IPv6
I’ve been complaining about this for eight years and I’m in disbelief it could actually be happening.
IP4 address access is a bit like the water supply – you ignore it until it’s gone. Last year I attempted IPv6 transition and discovered that Github lack of ipv6 broke my deployments. Thus I had to revert to IPv4 NAT access.
There are workarounds that carry the same complexity and cost burden as a proxy or vpn.
Alas with IPv6 the weak link breaks the chain. You’ll need all of your dependencies to support ipv6 and a majority of internet apps depend on Github.
Because there are 8 billion people but only 4 billion IPv4 addresses. Here’s a quote from a github user in Brazil:
Has anyone done ‘git torrent’ yet? Decouple the naming of hashes from their storage+distribution? Basically just use bittorrent as the sha-1 addressed blob storage backing a git client.
(Huh - someone had a similar idea, but no code: https://github.com/KurtRudolph/gitorrent)
Yeah, CJB built a prototype and explains how it works here. I try to package it every few months, but it’s resisted me so far; it depends on a patched DHT library. I would be amenable to building something new, particularly something which doesn’t have cryptocurrency integration.
not torrent, but kinda similar: I think the IPFS project had something like that based on IPFS
I’m familiar with Radicle and the tools outlined in this post, though I’ve never used any of them.
On my site, I use the smallest PNG as an image placeholder before JS can load the final images. It’s included on the page as a base64 data URL. It looks like
<img src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==">.You might be able to instead do
<img src='data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg"/>'>which is a little shorter and more human-readable.
Why? Why not leave it blank?
I want the space to be held by the image even before the JS kicks in and decides what to put there. I could just wait and add the img tag in JS, but it’s easier to have the img tag already there and then update it. If the src is blank, the page is invalid and the alt text shows through, so I need it to have a valid image at load time.
I’m assuming you meant to say you use the smallest GIF?
D’oh, you’re right.
Related, it bothers me very slightly that PNG has such big headers that it’s larger than GIF for very small images even though the compression is better for anything substantial.
This shit is the bane of the modern Web. Why is it that so many webdevs are seemingly incapable of putting text and media on a webpage without forcing users to run their shitty scripts? It is poor a11y and takes more effort than not doing it.
I agree that JS is often overused (I disable it by default when browsing the web), but I think it’s wrong to call this “the bane of the modern Web” without knowing the specifics of the product. One of many possibilities: the server cannot know what the
srcis and it can only be known by the client.You know nothing about my work, and my site loads in under a second while using a vanishingly small amount of JS because I expend a huge amount of personal effort to not just do the easy thing and include some dumb ass giant script that from Google or whatever, so kindly get bent. ☺️
A better way to do this is by using the
<picture>element and by ensuring the images use interlaced or progressive encoding.My favorite quote: “what’s good and bad in a programming language is largely a matter of opinion rather than fact, despite the certainty with which many people argue about even the most trivial features of Go or any other language.”
I was just coming here to say something similar. Coincidentally, I was struck by that bit after seeing the debate about whether readability in code is subjective or objective in this other current thread.
is this a way of leveraging ‘make illegal state unrepresentable’ (Yaron Minsky)?
I think so!
This is such a convenient and contrived example. What if the argument actually should be between 7 and 42? There’s no way really to remove all the validation checks at initialization time, due to various constraints. Later on, specialization might arise for argument values of 8 or 16. In some languages you could specialize in the static types, but the extra complexity in the type level programming would soon out grow that actual usefulness.
I agree. This was a trivial example that doesn’t need much context—maybe good for a short blog post, but bad for demonstrating how a type system can help shorten your code in more realistic cases. I’ll think about making an edit of some kind.
Something about this example (if not necessarily the sentiment, I like static types) rubbed me the wrong way, and I think looking at the real code may have helped nail down why.
You raise a TypeError in the real code, and if you’re trying to simulate “type is an unsigned byte” then it seems a given that it is shorter in a statically typed language.
Can you really have a length of 0 for a Squid? Seems error prone. If not, the code should be checking for min_length > 0.
If it was a different range, say 0-100 you wouldn’t have a type that happened to conveniently overlap. Other comments are mentioning languages with support for defining types like this, which is definitely useful but I don’t know if you’re still making things “shorter” then.
Imagine a handy attr_byte, in the same way Crystal has a handy UInt8:
Still dynamic, still a runtime check, still short.
On the flip side we can imagine a statically typed language, with only signed bytes (Java), where typing the argument doesn’t get you the range of 0-255 no matter what type you choose.
This is what I noticed, too: it works because the value happened to be exactly a
uint8. It’s trickier when you want a non 2^n range (and you’re not using Ada).I agree. This was a trivial example that doesn’t need much context—maybe good for a short blog post, but bad for demonstrating how a type system can help shorten your code. I’ll think about making an edit of some kind.
If you are able to delete the runtime checks from the Ruby code and replace them with static types, then the runtime checks were never necessary in the first place. The semantic equivalent to a type-check would be an assert statement, not a check and raise. So either the Ruby code is wrong or your translation is wrong. You should only need to validate data at the program’s boundaries with untrusted sources. At the interior of the program, it should expect data to already be valid. The superfluous runtime checks in the Ruby code are indicative of a lack of discipline around what data is trusted or not trusted in the program and that means it’s likely there are security vulnerabilities present.
EDIT: A classic mailing list entry on how to properly sanity check arguments in Ruby: https://blade.ruby-lang.org/ruby-talk/100511
The discipline argument has always been odd to me. Have you worked on large projects with multiple different people? How do you expect everyone to keep all the assumptions about what data has been validated (and how) in their heads, how do you synchronize it even?
Offloading a hugely amount of tedious work that a computer can do quickly and automatically to humans is, in my opinion, a ridiculous proposition.
I recognize this is an OO idealist perspective, but like sandi metz says, if statements are type checks.. don’t type check.
Parse, don’t validate applies to dynamic languages just as much as static, note that these ideas becomes easier when you work with immutable data so you only have to check invariants in constructors.
After you define your types (classes, records), you write your code that depends on those assumptions as methods, IMO this is better in languages like clojure where you can define these away from type definitions via a protocol.
This to me is the core idea that people miss who either do OO wrong, or knock it.
I’m not saying its easy, I think in fact it still does require a lot of discipline, but its a well defined discipline.
I don’t think that has anything to do with OO? The same argument can be used for statically typed functional languages like Haskell.
Yes, that’s actually my point.
What is interesting to me is this form of classification + polymorphism, and importantly the kind of dynamic code that the author was trying to replace with static typed code wouldn’t exist in this paradigm.
To be clear, my whole point is that this debate is actually bigger than static v dynamic, or FP v OO, its about generic code & there are really good examples of that in all these paradigms.
I referenced this in my comment. Use an assert(). Redundant checking of invariants is wasteful and sloppy.
You’re ranting sufficiently hard that it’s difficult to tell what you’re trying to say.
It sounds like you’re contrasting
assert(condition)with runtime checks, butassert(condition)is a runtime check. It’s executed at runtime, not at compile-time, and it checks whetherconditionis true. The difference between the two is whether the error is expected to be caught or not, I think?My best guess at what you’re actually trying to say is:
assert(condition)statements in unit tests. (That’s what the link you shared says, though you didn’t mention unit tests.)Except I’m not sure why you’re attacking this article (“superfluous”, “lack of discipline”, etc), because it seems perfectly likely that the interface it discusses is external facing, in which case it is following your advice.
Responding to your actual point, that’s a very good approach and I try to follow it too, with one very large exception. If you’re working in a medium to large codebase (say 10,000+ LOC), it’s unrealistic to expect all data to be well formed and all function arguments to be correct. Thus I aim to organize code into modules (which means different things in different languages) that are impossible to misuse. If a “module” needs to maintain an invariant, it must either (i) have an interface that makes it impossible to violate that invariant, no matter what the calling code does, or (ii) check that the invariant is obeyed in its boundary, and error if it’s violated. This way every “module” guarantees that its invariants are obeyed within it, and if they’re not it’s a bug in the module (and you don’t have to look elsewhere).
assert statements are not only for unit tests. They are also enabled in debug builds.
Not sure what you mean by “attacking.” If you mean that my intent was to personally demean or disrespect the author, then that’s incorrect. My intent was to accurately describe what I observed in the article. Those are neutral words in my vocabulary when discussing engineering.
This statement is equivalent to saying that it’s unrealistic to produce working code.
This is false. You’re expressing a personal API design philosophy or engineering discipline but not a hard reality. For a topical example, the standard multiplication operator does neither (i) or (ii) and freely allows incorrect usage that results in overflow. Often it’s not feasible to always check at runtime that the operands of the multiplication operator don’t result in an overflow nor have some special paired number type that guarantees that its members don’t overflow when multiplied. The reasonable approach is to structure the program such that it’s an invariant that the operands at the multiplication site don’t overflow, with a debug assertion. Not only is this the reasonable approach, this is the approach Zig and Rust take.
Not sure if you’re deliberately trolling… if you’re not, know that (i) you have misunderstood what I said, and (ii) I still don’t know what you’re advocating for, and it would be helpful if you clarified that instead of EDIT: arguing against your repeated misunderstandings of other people’s positions.
Can you explain what it is you believe I have misunderstood? I thought my previous post was sufficiently comprehensive.
I don’t believe I was “attacking” you per my previous comment.
I don’t believe that would be productive. What would be productive is if you simply wrote down what you’re advocating for. I made a guess above, is that accurate, modulo that you would put
asserts (which execute at runtime in debug mode but not in production) in the code as well as in unit tests?Apologies, I should have said “arguing against” instead of “attacking”. A lot of your phrasing makes it sound like you’re angry, though I realize now you’re not. Edited.
Your initial guess was correct, sans the implication that asserts should only be run in unit tests.
Ok. First of all, please realize that what you were suggesting wasn’t clear until now. Actually I’m still not entirely clear on how many
assert()s you suggest having. Should most internal invariants be checked withassert()s in well chosen locations? Your first comment seems to me to say “yes” in some places and “no” in others, to that question.Let me try to state the crux of the matter precisely. Say you’re working in a company with a hundred other employees and 100,000+ lines of code, and everyone follows your advice to the letter. What will happen is that you’ll end up with bugs that are extremely difficult to debug, because the place they manifest is far away from the actual bug.
For example, there’s a data type
Accountthat requires a complicated invariant. If this invariant is violated, it doesn’t cause an error in theAccountcode. Instead it causes another invariant in a different classHistoryto be violated (because the code that interlinks the two relies on theAccountinvariant), and then some code that usesHistoryends up accessing an array out of bounds. So now you see an array out of bounds error (hopefully you’re not working in C++, so it is at least guaranteed to stop there), and you need to work backwards to see that it came from a badHistory, and then work backwards from there and see that it came from a badAccount, instead of one of the dozen other ways aHistorycould have been corrupted if there was a bug elsewhere in the codebase. Oh, and this error wasn’t caught in your unit tests and only happened in production, once.In contrast, if you consider the boundary of
Accountand ofHistoryto be “external facing” and check their invariants on their boundaries, even in production, then you’d get a very clear error message at the actual cause of the error: an invalidAccountconstruction.I’ve worked at a company where all the code assumed you wouldn’t make a mistake, and at a company where the code assumed you might make a mistake and would error if you used it wrong. Issues at the former company were so much harder to debug, it was a nightmare.
You can have zero asserts or an unbounded amount of asserts. Assertions on input data are not significantly wordier than type declarations.
Then assert at the earliest location possible. There is usually little downside.
The downsides of runtime checks, according to you:
However, you’re saying that you can assert with little downside. So this whole time you’ve been quibbling about the difference between:
and
???
Assertions are semantically different than explicit checks that return errors or raise exceptions. An assertion means that the caller is required to make sure arguments satisfy certain properties, otherwise unspecified behavior may occur. An explicit check is a specification that in the event that arguments are invalid, the function will return an error or raise an exception.
Given the meaning of an assertion, it’s valid to disable them in production. That’s why this distinction was created.
You say this like it’s an unassailable fact, but it actually varies between languages and between developer conventions.
My Google searching suggests that Ruby doesn’t even have an
assertstatement outside of a unit testing module, so it’s a strange place to start this discussion.Rust has both
assert, which runs in both debug and production, anddebug_assertwhich runs only in debug mode. You often need to useassertinstead ofdebug_assertto uphold Rust’s safety guarantees. E.g. if a public non-unsafe function could cause memory corruption if passed invalid arguments, it is incorrect for that function to not check those arguments in production, according to Rust’s safety guarantee. Rust’s safety guarantee is taken pretty seriously: e.g. I think most Rust library authors would remove any dependency they knew violated it.As far as developer conventions: it is entirely possible for developers to implement an assertion using an exception. An assertion (as you’ve defined it) is a semantic concept, which has many possible implementations. There are all sorts of reasons that a developer might choose to implement an assertion with an exception.
For example, here’s the code from the article, written using an assertion (remember, as you pointed out, assertion is a semantic concept, so it’s perfectly fine to implement it using
raise):The difference is that I added a comment saying that the behavior is unspecified if the argument is invalid. (It should be in the function documentation; I don’t know the syntax for that in Ruby but pretend the comments are official documentation.) To make debugging easier, the “unspecified behavior” is implemented as an exception that will be raised, even in production.
The central point that I and others have been trying to convey, is that unspecified behavior in large systems is really bad. It makes debugging a nightmare. You need some, but should keep it local.
At a high level I think your main point of contention is that redundant validity checks (so-called “defensive programming”) result in a more resilient system whereas in my opinion it’s a symptom of a system that was designed without proper discipline around trusted / validated values. These are conflicting principles.
The “defensive programming” principle, if taken to its logical conclusion, results in a system where every argument is validated upon every function call and its not clear which part of the program is responsible for the primary validation of input data. Long-term maintenance and refactoring becomes difficult because it’s not clear who does what. Whereas with the “unspecified behavior” principle, if taken to its logical conclusion, results in a system that may start failing silently if there is a single bug in the program in a difficult-to-debug way, though it’s more clear what part of the system is responsible for validation and makes long-term maintenance and refactoring easier.
I think you want to bias towards building the latter system for hopefully obvious reason (e.g. explicitly checking a value you know to be valid over and over again serves no purpose and is confusing to people who read the program). To mitigate the issue of incorrect code / logical errors / “bugs”, asserts provide a way to materialize into code the abstract specifications about what functions expect. They serve as living executable documentation that can be selectively enforced depending on the paranoia of the programmer. If when engineering your system, you expect the invalid values or states to happen only when there are bugs in the program, then it’s reasonable that you enable assertions during testing and debugging and disable them once you are ready for release and all functional specifications of the program have been QA tested.
One issue is that lots of web programmers never see a proper “release” so the debug-enable / release-disable convention with assertions isn’t practical to them. The code is never done, there is no comprehensive QA, it’s never golden master. Therefore in those situations I’d recommend just always leaving assertions enabled, regardless of language used. I’d still recommend keeping the assertion concept distinct from normal logic in the program, if only to provide option value and serve as documentation for other programmers and perhaps static analysis tools.
I’m strongly against redundant validation. Validation should happen in exactly one place. But let’s end this here.
Thanks for the chat, I enjoyed it and I hope the exposure to a different point of view was helpful. Perhaps you can suggest using asserts more liberally to your team the next time you find yourself working in a large codebase that has no mitigations for programmer error, as a sort of compromise to reduce the pain of debugging. Best wishes.
Sorry I can’t say the same, I found the conversation extremely frustrating. I can’t say I’ve learned anything, because even at this point I’m not sure if we have different views. Please try to learn from the fact that this was a huge thread with a lot of people being very confused at what you were saying. You need to give way more context and explanation up front.
You seem to miss the point that some bugs only surface in production.
If you don’t trust your code to operate correctly in production then leave assertions on. Use a process monitor to monitor when assertions fail and provide reasonable feedback to the user. Or use a more restrictive language.
So your suggestion is to write the Ruby example exactly as written in the original article? What are we even debating at this point?
That’s not my suggestion. There is a semantic difference between assertions and explicit checks. In simple terms, assertions are the runtime version of type-checks. Explicit checks are new behavior. Assertions afford you the ability to turn them off and indeed I run my production code with them off. I recommend that everyone build their systems such that they can feel confident turning off assertions in production.
So what is wrong with the next part of @justinpombrio’s reply?
My original comment pointed to the semantic inconsistency between the original Ruby version and the translated Crystal version. The article stated that the validation check was deleted in the translated version implying that it was redundant validation code. Redundant validation code signals a lack of discipline in terms of how trusted and untrusted values are handled throughout the code. TFA’s author clarified that the semantics of the translated code were in fact different from the original and that the validation was not simply deleted as originally stated in the article but moved.
I generally agree with this. In this library’s case, it is at a program boundary, so I think the validations are sensible.
If that’s the case, then the translation is not equivalent. The initialize method should take a wide integer type and then do the sanity checking for the correct range inside that method just as the Ruby code does. Since you replaced the argument type with a narrow integer type, that implies that you have moved the sanity checking for the value’s range to before the initialize method is called. That changes the contract of the translated method.
While you are technically correct in the sense that the domain of the two function is not identical, I think you’re being overly pedantic or disingenuous. The whole point is that static type checking allows you to statically restrict the domain in a way that is simply not possible in dynamic languages. The only reason why the domains are not identical, is because Ruby lacks the ability to restrict domains at all. As a result, the Ruby function is not total.
In other words, of course the Crystal version should not take a wider integer range as input. That way, library users cannot pass in arguments that result in an exception, a thing that is not possible in Ruby.
An accusation of being disingenuous requires an assumption of bad faith, which I think would be unreasonable and/or unwarranted in this context.
Where input validation happens is incidental to whether the program is statically or dynamically typed. It’s not a given that the reason the input validation occurred in the
initializemethod originally is because Ruby lacks static typing. An alternative Ruby implementation could have been as follows:I don’t think that’s a given either. The input validation has to happen somewhere. In the original code the author says that it was specified to occur in the
initializemethod. When translating a program from one language to another I think it’s less error-prone to not change input validation sites and method contracts as much as possible. Every call site now needs to be aware of the new semantics of theinitializemethod.As @EvanHahn already pointed out, the
initializefunction is the library boundary. I therefore think it is reasonable to assume that input validation has to happen in the initialize method.I don’t think that is unreasonable, but I also think it depends on the context. You could start out with a exact replica, make sure everything type checks, and then gradually add stricter constraints, letting the compiler help you maintain functionality along the way. If you aren’t taking advantage of the features of the new programming language, why are you translating your program at all?
True, but the compiler will tell you. So there’s no chance you will accidentally misuse the initialize method. It is simply not possible.
Library boundaries are not necessarily trust boundaries. In fact, they often should not be since it’s usually only a small percentage of your program where you are accepting untrusted user input.
I… am not sure what you are saying here? Aren’t library boundaries literally the exact points in a library where you are accepting “untrusted user input”? Perhaps I am misunderstanding what you mean by trust boundary, or what kind of untrusted user input you mean?
To clarify, in the case of a library I use the word “user” to denote the developer using the library. The “input” is the arguments that the user passes to the library boundary functions. That user input is (and should be) untrusted by the library author.
Untrusted user input means input coming from outside of the process, in particular the actual input to your program from its users. Any agent that your program can not trust to produce valid input.
There is no functional trust difference between library code and your own code, it’s simply code that is packaged separately. Regardless of how the library code is packaged, you must trust it does what it is specified to do and it must trust that you use it the way it is specified to be used. The same trust applies to your own code. If you cannot trust a library or the library cannot trust you, you cannot link to it liberally and expect your program to function correctly.
Aha, I’m glad we cleared up that confusion! I disagree with what your opinion regarding library code, but I recognize that it is a valid stance.
Yeah, the translation is not equivalent and the contract is different. A one-to-one port should do what you describe.
I’d also like to see this tag because I think it’d help me learn more about this topic, which interests me greatly.
I’ve been the lone developer of a project since the project’s creator and my main collaborator passed away last year. They themselves were the lone developer of this project for a period.
They knew what was coming, and adopted what they coined ‘mortality driven development’. This manifested itself as most of the codebase being comments and tests, and a wealth of diagrams and documents explaining how things worked and the rationale behind them.
And so I never felt lost in their codebase, and sometimes even relied on them to help me write new features long after their death. I can’t say I’ve been half as diligent in my stewardship of the project — but this post helps remind me why it matters.
Earthstar is very cool. Thanks for your hard work on it!!
For others wondering: https://earthstar-project.org/
https://github.com/earthstar-project/earthstar
To the OP, I would recommend putting more links to it on your blog – it’s difficult to find!
Company: Signal
Company site: https://signal.org
Position(s): Engineering: iOS (engineers and tech lead), Android, Desktop, Server/Infra, Calling & Core Libraries. Product Designers too!
Location: 100% remote (within US timezones)
Description: Signal is a 501c3 nonprofit organization developing open source privacy technology that protects free expression and enables secure global communication. We’re a fully distributed/remote, small team looking to grow slightly (currently 30 people total/20 engineers). If you care about code quality as much as you care about user privacy, we’d love to talk with you.
Tech stack: Swift/Objective-C, Java/Kotlin, TypeScript, Java, Rust. We publish all our codebases @ https://github.com/signalapp, check ’em out!
Compensation: Competitive with Big Tech, without selling your soul or compromising your ethics. Full 401k match, health/vision/dental, parental leave, and self-managed time off.
Contact: workwithus at signal dot org (several of us staff that email alias). You can also DM me on here, or email evanhahn at signal dot org.