I can’t for the life of me seem to figure out what to do with a clojure program after I have written it. How do I take a .clj file and turn it into a .class?
I don’t want to set up a leingen project or whatever, I don’t want to bundle it as a jar, all I want to do is write a .clj file in a text editor and turn it into a .class so I can run it like java ClassName
Author of the article here. Clojure is more commonly loaded and compiled on the fly, so the easiest way to run a single .clj file is to just do clojure your_file.clj. This uses the official Clojure CLI. Compiling to class files (AOT compilation) is usually reserved for production deployments, where you build an (uber)jar.
That said if you want to just compile a single namespace, you do that with Clojure’s compile function. It’ll look for the namespace on the classpath (defaults to src), and write the resulting classes to classes.
Why do you want it to run like java ClassName in the first place? To me that seems a bit arbitrary, and really doesn’t work unless you add the Clojure runtime on the classpath.
Though I assume you just want to run a .clj file as a command on the command line, and there Babaskha is the de-facto standard tool to that these days.
While the article states that there isn’t much to configure for SQLite, you should still configure it. I wouldn’t be surprised if setting work_mem for Postgres – and not cache_size and temp_store in SQLite – skew the results in favour of Postgres.
I find it weird that they bundle s3 and postgres support directly into a runtime instead of having them as libraries. Looking at the documentation for s3, I see
Bun aims to be a cloud-first JavaScript runtime. That means supporting all the tools and services you need to run a production application in the cloud.
and I get that it’s better to have a native implementation rather than a pure JavaScript one for speed (though I’m skeptic of their 5x claim). But I can’t see this approach scaling. If performance is a concern, then other services you communicate with also need to be fast. In particular Redis/memcached or whatever caching mechanism you prefer, but also ElasticSearch or maybe some Oracle database you need to talk to as well. Should those also be in the runtime? And how do you handle backwards compatibility for those libraries?
I guess this is just a long-winded way of saying “Why aren’t they implementing this as native add-ons and instead force this into the runtime?”. It feels much more modular and sensible to put it outside of the runtime.
My two biggest complaints are mostly to do with the API - one being that s3 is imported from bun, rather than bun/s3 or similar. Python provides a big stdlib, but the modules still have different names. You don’t do from python import DefaultDict, httplib, for example.
The other one is that the file function supports s3 URIs, which is nice from a minimal API perspective, but I also think it’s not ideal to treat s3 the same as local files. s3 has a lot of additional setup - e.g AWS credentials, handling auth errors, etc. So I think it makes sense to logically separate the behaviour for local vs remote files.
I don’t mind new takes on AWS / postgres sdks, though. The SDK is pretty decent compared to some others (e.g Google or Slack), but I think both their AWS and postgress examples there (other than the two issues I mentioned) are pretty nice.
I agree with your sentiment and I am also very confused why I would use the Bun s3 implementation over the whole AWS SDK that I have been using and accustomed to for years now. Sure there could be some performance gains (for just S3) but I don’t see the benefit.
I’ve run into contention on S3 in a Python backend, and it’s really not fun. It’s a very good feature to have this sorted and guaranteed to work fast, it means that Bun can stay competitive with compiled languages for more intensive workloads. To me, this is a production mindset: identify the key components, and optimise them so that they don’t get in your way.
Call me impatient, but the startup time of this setup was too slow for me. I quickly ended up with long-living and shared Postgres databases instead. In my case, I started up a Docker image with configuration options to keep as much stuff in memory instead of disk. Then, the tests just connected to a hardcoded pg connection and did whatever it was asked to test.
Sharing database has some additional benefits: You won’t just test the database from an empty state. If you accidentally forget to add the AND user_id = %3 expression to your query, you’ll quickly realize whenever the second run of the same test occurs. Though this also depends on you being able to generate new users and data on the fly.
Of course, there are nitty gritty details here. You’d like to clean up the database now and then so that it doesn’t blow up your memory use. And you probably need an isolated database from time to time. But that’s manageable, and the complexity is well worth it to make tests snappy imo.
(Also, if you can, this is just much easier to do with Sqlite)
In roughly 10 years of writing Go I got so accustomed to writing max the long way around that I had to do it several more times, out of habit, with a coworker reminding me “you can just use max for that”, before it finally sunk in.
Go’s lack of ?: and (former) lack of stuff like max is considered pretty harmless by most, and mostly it is, but sometimes you really just want an expression, not a statement. Which one of these do you prefer?
a bit contrived, but I’ve worked with real code that’s near enough to that. The old way of thinking was “this is logic, it’s making a decision, and you need a statement to codify the fact that you’re making a decision here”. The new way is “you know what, taking the greatest or least of two or more things is common and everyone knows what it means — not only is it okay for you to avoid writing the repetitive if, it’s even okay to inline it into a function’s arguments list or an aggregate initializer sometimes.”
If you pass constants in, you get a constant out. Since there is not constexpr equivalent in Go, normal functions cannot do this, you need a builtin for that.
Of course, there are solutions to reduce the amount of boilerplate. In Rust, you can sugar coat the ugly syntax with the ? macro. This just hides all the tedious checking, but you still need to do all the work. Additionally, your function now returns error codes, which means you have to change the public interface and recursively change all functions that call this function.
I’m not sure what the “you still need to do all the work” refers to here, but one issue I’ve had with Rust (not sure if that has changed recently) is that errors don’t compose: Say you have two APIs you integrate with, and both define an error enum to cover all the cases that can happen, then you’re forced to transform these two types into one somehow if you want to call one after the other and use the ? operator. I know it’s possible to use some Into trick here, but it still requires a new datatype and some code to map from one enum option to another.
OCaml, on the other hand, has composable error handling via polymorphic variants. Since I’m not currently writing any type signatures, this is fine for me, but I suspect it becomes tedious if I were to create .mli signatures. That being said, it seems OCaml libraries still very much prefers exceptions, as I’ve had to wrap a lot of them in e.g. JSON parsing and whatnot. Roc seems to lean completely into the polymorphic variants error handling, which I very much love. But from what I can tell, still has the problem of keeping these types up-to-date if I were to add a new error variant down the stack, and also want to manually write my type signatures.
… which I guess is a long-winded way of saying “yes, there’s a bit of overhead for the developer”, but it’s not as bad as it once was, and I think errors should be in your face. As for performance.. maybe? Not that its performance doesn’t matter, but your bottleneck surely lies elsewhere.
errors don’t compose: Say you have two APIs you integrate with, and both define an error enum to cover all the cases that can happen, then you’re forced to transform these two types into one somehow if you want to call one after the other and use the ? operator
If you don’t care about preserving the exact semantic of the error, you could use a trait object return type of Box<dyn std::error::Error>. Or, you’d have to be explicit and define your own errors.
I know it’s possible to use some Into trick here, but it still requires a new datatype and some code to map from one enum option to another.
Another option is just a newtype wrapper that implements Error and wraps those two enums of error for the cases you care about, like
Yeah, the latter is what I’m familiar with. But it causes a little bit of messy nesting imo, and it’s less granular than I’d like it to be. Assume, for example, that I handle all error types of a particular kind from TheirError1 – I still have to pattern match on it as a consumer, unless I wrap every other error.
The composable error issue is interesting to me. I think a weakness-that-is-a-strength of Go is that errors are not exhaustive enums, but just an open ended interface, so that caller knows that they probably have to be prepared to handle unknown error types as well as known types.
In application-level code consider using e.g. anyhow or eyre, they are a much more powerful alternative to Box<dyn std::error::Error> and make error handling easy if the exact error type is not needed. (eyre has some additional niceties e.g. integrating with tracing via the tracing_error crate)
Immutability is the default, but you can use types that are mutable if you need/want to
That’s not quite what I see. Ocaml does not have immutable arrays. In contrast, Clojure and Haskell do have immutable arrays (with fast indexing and update). I guess that is consistent with your conclusion that Ocaml is not up to the standards of other languages.
? Haskell immutable arrays don’t have fast updates, they copy the whole array on update. This is fine if you plan to update all or most of the entries in the array anyway, but if you want to do point updates then you probably want a tree or trie or something.
There’s a differential array library but I’ve been told that the constant factors are terrible.
Well, Haskell has Data.Sequence, which appears to have similar performance properties to Clojure vectors. Data.Sequence is based on finger trees, which according to Wikipedia has amortized O(1) put, get, pushing, reversing, popping, and O(log n) append, split. I don’t know the details of the Haskell implementation, or the constant factors. Clojure stores 32 items in each leaf node of a vector, leading to shallow trees, data locality and generally good performance.
Update: I also found persistent-vector in Hackage, which claims to implement the Clojure vector data structure, and claims “O(1) append element, indexing, updates, length, and slicing” with “Reasonably compact representation”.
I guess the need doesn’t really come up that much in practice
In Clojure, persistent immutable vectors have primitive syntax, you write [1 2 3] for a vector literal, for example. Now that’s “immutability by default”. Needing to get an immutable array implementation from a 3rd party github is not “immutability by default”.
And we have built-in immutable lists in OCaml which serve many of the same purposes as immutable vectors, so we also have ‘immutability by default’. I would encourage you to more carefully research the language before making comparisons out of context.
I am researching Ocaml, F#, Haskell, Clojure as possible languages to use for a future project. One of my requirements is a complete set of immutable persistent data structures, including arrays. I need to use these types pervasively throughout my code, to support rolling back to an earlier state, in several contexts. One of my concerns is quality of implementation. If persistent immutable data structures are built in to the language core, and everybody uses them as a matter of course, then the implementation is likely solid and performant and the APIs are likely complete. If I have to use a 3rd party library on github, then I have less confidence in the quality of the implementation and I’m more likely to need to vendor it and maintain it myself.
None of the languages I’ve looked at meet all my requirements, but on the topic of persistent immutable data structures, Clojure is first and Haskell is second.
Can you say a bit more the application domain, and what kind of rollback logic you need? (For example, after you backtrack from a given state, do you even come back to it again?) I’m interested in use-cases for persistent and semi-persistent structures.
By persistent arrays, do you need only random access and update of one element, or do you need to add and remove elements at the end of the array, or to split or concatenate arrays?
The constant factors for persistent catenable arrays, even with very good implementations, are fairly high, to the point that I am not sure they are often a rational choice performance-wise – they are very nice for reasoning. For OCaml you can try Sek that is a high-quality implementation. If you are happy with fixed-size arrays with updates and random access, I would use Parray from Jean-Christophe Filliatre which offers O(1) access with good constant factors.
My previous prototype was in C++, and I used reference counting plus copy-on-write to perform functional updates on data structures. For arrays, indexing was efficient but concatenation required copying all the elements. I did not enjoy writing that much C++, so for the next prototype, I wish to use a high level functional language. Probably I’ll have lower performance, but I’ll be more productive, the code will be shorter and simpler, easier to reason about, I and will have an opportunity to explore more of the design space I’m interested in.
The application domain is something like Emacs, except instead of editing text, you are viewing and editing rich hypertext and hypermedia. Instead of the extension language being Elisp, it is like a pure functional language or like an imperative language with value semantics (not reference semantics). The extension language is a DSL and is part of the project: I’m writing an interpreter for this language, not embedding an existing general purpose language. All data is represented by pure immutable values with no pointer values or reference semantics. There is an undo stack, and immutable persistent data structures seem like a good choice for this. Whenever a document is updated by an editing action, this is accomplished by evaluating a function call in the extension language that maps the old document state onto the new document state. Document updates are transactional: if the function fails in the middle of the update, then the partial changes are rolled back. This is notionally implemented using STM (software transactional memory). If I have to implement my own STM, then I’ll use persistent data structures to efficiently roll back a transaction.
I agree that implementing a general purpose persistent array with good performance across a wide range of use cases is challenging. Clojure vectors are persistent arrays, and they are used ubiquitously throughout the language. My guess is that if the performance characteristics of clojure vectors are good enough for clojure then they are good enough for my application. The implementation has probably been heavily tuned.
I see, thanks for the details! Sounds like fun. I agree that trying to just use Clojure maps is a good bet. Clojure maps support transient/ephemeral updates which can also help, you don’t need to pay the cost of persistence on internal edits inside a user action whose internal states will never be observed. (This is a more static approach to copy-on-write-like optimizations, where the code is explicit about critical sections where sharing is forbidden.)
First, I think we need to clear up some terminology. The behaviour you described earlier:
concatenation required copying all the elements.
…is not what I (and I think most people) would understand as ‘persistent’. Persistent data structures are designed such that doing an immutable update would reuse much of the existing data structure in memory. Eg, an OCaml list is a simple persistent data structure. If you prepend (a new head element) to the list, the original list does not need to be copied at all. It is immutable and just reused in the new list.
What you described in ‘copying all the elements’ of your array is a non-persistent immutable data structure. This is easy to do, of course, but not as memory-efficient (obviously). Here’s a simple example in OCaml using mutable arrays as the backing store:
module ImmutableArray : sig
type _ t
val make : int -> 'a -> 'a t
val length : 'a t -> int
val get : int -> 'a t -> 'a
val set : int -> 'a -> 'a t -> 'a t
val append : 'a t -> 'a t -> 'a t
val pp : 'a Fmt.t -> 'a t Fmt.t
(** Pretty-printer. Requires 'fmt' package. *)
end = struct
type 'a t = 'a array
(* Non-mutating operations can just be reused *)
let make = Array.make
let length = Array.length
let get idx arr = Array.get arr idx
let append = Array.append
(* Array.mapi creates a new array *)
let set idx elem arr = Array.mapi (fun i a -> if i = idx then elem else a) arr
(* Helper for displaying the arrays *)
let pp pp_el = Fmt.brackets (Fmt.array ~sep:Fmt.semi pp_el)
end
The custom module that we wrote here is used to present a fully immutable interface on top of a mutable array. It is possible to use this interface and then completely swap out the implementation with, say, a new persistent data structure like the one I linked earlier in the thread. Those are of course more complex implementation-wise.
Yeah, my old system used a method for efficiently updating immutable data structures, based on reference counting and copy-on-write, but it did not use persistent arrays. I think that my new system would benefit from persistent data structures, due to the need to store multiple versions of the same state (as I described), with small differences between each version. For this situation, persistent data structures would save memory.
Thanks for the example code, I understand your earlier comment now.
In some cases it might be useful to have a read-only array type. E.g. when you give it as an argument to a function that “promises” not to modify it. It is possible to write a wrapper module that only exposes get though.
Both OCaml and Haskell have immutable singly linked list you can pattern match on. I’m not sure what Haskell provides in the standard library at the moment, but afaik there are efficient immutable collections you can add as dependencies for both languages.
Saddens me to say it, but I think it’s fairly accurate. OCaml is a relatively old language that shines in its historical domains of application (formal logic tools, compilers of all sorts, and do-it-from-scratch servers on top of posix). There’s been a lot of effort to try and have it catch up to the “modern” web stacks, with async IOs, web frameworks, ocaml-to-javascript compilers, etc. but it’s never really stabilized[^1]. The fragmentation in the ecosystem is huge (I should know), releases break code, and opam is designed in such a way that it needs people to explicitly set lower and upper bounds for dependencies (which of course people don’t always do).
[^1]: dream is the de facto lead web framework, I think, but it’s been stuck just below 1.0 for a long time.
In fact it seems they’re encouraged to do the opposite: opam-repository maintainers have pushed me to get rid of my dependency upper bounds, claiming they can fix it later if the need arises. I think they spend a significant amount of time doing this kind of housekeeping. If it were me I’d have a CI check that would fail the PR build if all dependencies did not have semantic versioned upper bounds.
but it’s been stuck just below 1.0 for a long time.
One of the many unfortunate impacts (albeit a small one) of the Russia-Ukraine war. OSS maintainers need to eat…
In fact it seems they’re encouraged to do the opposite: opam-repository maintainers have pushed me to get rid of my dependency upper bounds, claiming they can fix it later if the need arises.
Wow, that’s very disappointing to hear. You would have thought by now that people would have learned that reproducibility is worth a little more work up front.
I think this whole article demonstrates that “fix it later” doesn’t work.
Having worked with JVM languages for too long I physically shudder every time I have to deal with dependency bounds. I will take reproducibility over automatic updates any day.
In fact it seems they’re encouraged to do the opposite: opam-repository maintainers have pushed me to get rid of my dependency upper bounds, claiming they can fix it later if the need arises
I don’t know the specifics of your exchange with the maintainers, but IMO this mischaracterizes the general policy and its rationale: the opam repository is curated and CI checks are run on all package submissions. The CI checks include tests of all reverse dependencies. What we aim to offer thereby is an ecosystem of packages in which we do not prematurely lock out compatible upstream upgrades. Instead, when a breaking change to an upstream package is submitted to the repo, we place an upper bound when it is needed, before merging the breaking change.
Of course, I’m sure there are some cases where packages can slip thru the cracks, and we have a lot to do to improve the process, but it is not just a flippant “we will let things break and fix it someday”.
So, in short, re:
If it were me I’d have a CI check that would fail the PR build if all dependencies did not have semantic versioned upper bounds
IMO, from the perspective of a user of the repo, the reverse dependency CI check we actually do is strictly superior to this overly constrained approach.
- File "lib/dream_html.ml", line 1006, characters 10-25:
- 1006 | Dream.not_found req)
- ^^^^^^^^^^^^^^^
- Alert deprecated: Dream.not_found
- Use
- fun _ -> Dream.empty `Not_Found
I will fix this as of dream-html 3.4.2, but all versions <3.4.2 will fail to compile with future versions of Dream after Dream.not_found is removed. How would the maintainers tackle this? Manually remove all versions <3.4.2 from the build matrix by setting some constraint somewhere? This doesn’t seem very scalable, right?
How would the maintainers tackle this? Manually remove all versions <3.4.2 from the build matrix by setting some constraint somewhere? This doesn’t seem very scalable, right?
Indeed, archiving superseded patch versions and marking them unavailable has been discussed. Tho, iiuc, we also endorse adding upper bounds to packages when they become necessary (and not preemptively): perhaps that is the best course here? Perhaps that recourse needs be better documented for package authors?
Adding upper bounds when needed is low hanging fruit for automation, and on our roadmap.
The main benefit of discouraging premature upper-bounding is to reduce the possibility of passages into dependency hell across the ecosystem. AFAICT, this is working quite well in OCaml compared to other ecosystems I’ve worked in.
I appreciate the followup. It is interesting to see the process in action! IMO, it is very cool that we have this revdep CI running and multiple people in the ecosystem collaborating over it to plan how to adapt to keep everything working well. I am also noting that several revdeps are showing up in the CI as broken. I wish we had fixed or diagnosed those before merging, but at least we can see the problem and can tidy in a followup.
That’s great, but this is a manual process. It works today because relatively speaking there is a tiny number of opam packages. Are the opam-repository maintainers going to scale this linearly as more packages are added over time?
That is the aim, yes. We have lots of room for low hanging improvements in the current process, and some ways to automate and semi-automate the process in the pipeline. That said, I believe it is an open question how far we’ll be able to scale this approach. But if we manage it, I think opam users are going to continue benefiting from a nice, stable ecosystem of packages. In any case, my point is to try to correct the apparent misconception some readers of your remark formed, that the attitude of the repo maintainer is to invite dependency hell by allowing in packages that break downstream deps.
I haven’t seen issues with the version ranges yet, so I can’t comment on that.
Apart from that, most of the issues seem surmountable, especially the hoops I had to jump through. Though my biggest fear is probably the lwt/async/eio split and if there ever will be a consolidation on eio at some point. It seems like the most confusing and scary thing for a newcomer: What happens if a library I want to use only has, say, async support?
OCaml might be old but the 2003ish ecosystem felt so different than the last 10 years that it might as well be a different language. The stuff described in the post as problems were not even ideas back then.
They represent the file or directory that they resolve to relative to the source file they’re in. If you coerce them to a string, that file or directory is “interned” into the nix store and the resulting absolute path is used as the string.
Nushell has this too and it’s a blessing for shell scripts.
It also has file sizes as a type, which can be noted down as 10mb, 0.01gb or whatever you like and converted to bytes using into int. The same thing for Durations would be great for normal languages
No, go does not actually have a proper Duration type. You can do time.sleep(123) expecting seconds, and forget the Duration const, expecting seconds but get nanoseconds, it’s not opaque.
Kotlin is a language with a proper duration type, and you have to do 123.seconds, which will convert the Int to a Duration. But it is not a built-in type, which is what I was pointing towards.
The EXPRESS modelling language has bags, and arguably, SQL has as well (when you don’t do an ORDER BY, that is).
NULL in SQL is quite unique in that very few languages have the same semantics on NULL. Many think of it as an “not there” value (and it is often used that way), but I think it’s better explained as a value “we don’t know” yet. In a sense, UNKNOWN would be a better name, as select (null = 1) is null returning true sounds quite confusing to people, whereas select (unknown = 1) is unknown is probably more understandable. Arguably, the absence of a value and a missing/unknown one are two different things, so we should have both.
Also, Postgres at least has both literal rows and literal tables. I don’t know whether or not that is in conformance to the standard or a local innovation.
If you don’t know what that even means, mariadb has some docs on what it enables you to do. Basically versioning for your data on the level of tables, such that you can go back in “time”/transactions.
I wonder how well things like GDPR are implemented or understood when using this. Though the problem of slowly changing dimension isn’t new.
Yeah, that’s a good starter – seems like I completely forgot explaining what it actually is.
For GDPR: I imagine you’ll still be able to utilise an ON DELETE CASCADE on the history table and then refer to the original user table. Though maybe you’ll have to do it indirectly through a user identity table if the triggers hinders you from doing so.
For GDPR’s right to be forgotten, you’d only need to delete anything that could identify a person (eg: names, addresses, even pseudonyms like an IP * timestamp). So as long as your data governance is up to scratch, you should only need to anonymise the tables containing PII, and the rest can stay.
(Although someone might argue that say, a pattern of purchases might constitute PII, it comes down to how good/evil your lawyers are).
The problem with the words easy and simple is that they are used by so many, and are in many cases interchangeable nowadays. If you look up the definition of them both there’s quite the overlap. So while I’ve enjoyed the “Simple vs. Easy” talk, I avoid using both words when describing a language because I don’t think most developers are linguists (Sorry Clojure people).
The post discusses a lot of things you have to think about when picking a programming language, but I’d like to focus on one thing that didn’t sit right with me. The post argues (indirectly) that Python abstracts certain things Go doesn’t, and that makes it faster to make an implementation of whatever you have. But abstractions are leaky, and at some point enough edge cases of that abstraction leaks through and will make Go more maintainable. That makes sense.
But then the post says Rust is not easy (fair enough) and not simple either, without really going into more detail than “I cannot look at a piece of Rust code and say with certainty that I understand what is going on there”. Which, well, isn’t really an indicator of whether something is “simple”.
I’d say Rust removes even more abstractions and at the same time attempts to prevent certain kinds of programmer errors. That comes with complexity both Go and Python abstracts or outright ignores at compile time (None/null pointers, memory management, preventing race conditions, the list goes on). Of course, forcing you to handle those can feel complex, but the alternative that Go and Python does is clearly worse if you need to build very performant and/or reliable software. So in a sense, you can say that – when it comes to abstractions – Rust is to Go as Go is to Python.
If you look up the definition of them both there’s quite the overlap.
I avoid using both words when describing a language because I don’t think most developers are linguists (Sorry Clojure people).
Its a good thing to periodically remind yourself that dictionaries are descriptive of how language is used not proscriptive about how language should be used. They are historical records not rule books.
So while I’ve enjoyed the “Simple vs. Easy” talk, I avoid using both words when describing a language because I don’t think most developers are linguists (Sorry Clojure people).
But all you have to do is preface every conference talk discussion with a couple dictionary definitions! 😛
Just a heads up: I was sure the original memoised Fibonacci version would blow up due to stack depths, and sure enough:
>>> fib(5000)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in fib
File "<stdin>", line 8, in fib
File "<stdin>", line 8, in fib
[Previous line repeated 995 more times]
File "<stdin>", line 5, in fib
RecursionError: maximum recursion depth exceeded in comparison
Since integer multiplication takes O(n log n log log n) time, the “linear” algorithm takes a bit over O(n²) time. Likewise, the “log” version grows some logs faster than O(n), though it is not as visible in this graph.
You shouldn’t. What happens is that you implement a minimal feature set to get your website working. Then, next time you want to add something to the website, you realize a necessary feature is missing, and instead of updating the website, you spend your time implementing the feature. This happens over and over again and you lose interest in updating the website because it’s so much work every time.
Last time I tried, I found that even choosing a static site generator from the seemingly infinite list of options in 2016 was arguably more work than just building one. We tacked on a build pipeline for images and javascript at some point, but other than that I don’t think any features were added during the life of the site. There are certainly features it didn’t have, but that’s just the power of knowing your requirements up front, I guess.
Not to mention keeping your 3rd party SSG up to date. “Oh, I can’t post to my blog because _____ isn’t compatible with the current version of Ruby/Python/etc. Guess I’ll update it. Oh, now it’s obsoleted some config options I’m using, better learn what the modern equivalent is. Oh, looks like the theme I installed isn’t compatible, is there an updated version? No? Time to look for a new theme. Oh, now I remember I patched the old theme to add a custom feature and my pages won’t render without it; how do I port that patch to the new theme? Wow, looks like the theme engine changed a lot, I don’t recognize half these tags…”
The SSG doesn’t interact with the internet or any kind of untrusted data. Hence it’s totally fine to keep a VM or container image with the old working version of the tools forever and never update anything. :)
100% - I tried to get something done and was looking at hugo after a few years and it was quicker to write my own before reading up and making it work.
I’m not sure it’s fair to make a generalization about this. I found that building my own blog (not actually a SSG) in Rust with Axum and Maud was actually quite comfortable and ergonomic since I was able to lean on the existing tools that exist (Tower for any HTTP stuff, Comrak for markdown parsing etc).
It was a fun learning experience, and I think a blog is simple enough to be doable as a learning exercise, but complex enough to expose you to a lot of language features you might otherwise have missed.
My experience with using an off the shelf SSG is that everything was fine and dandy until I wanted to do something I felt was pretty basic but which I ended up spending hours and hours on trying to get to work before finally giving up. Writing my own would not have this problem. I would just implement the basic features I actually need and then not touch it again.
This isn’t limited to homemade SSGs though. I’ve had to write quite a bit of Ruby and understand Jekyll internals to get some plugins to work together and in the way I want them to work, and the solutions feel rather hacky and are verbose liquid templates. I’m fairly certain this takes less time than making my own SSG, but from experience it’s much less fun than making it yourself.
And if you want a feature that’s not yet supported in some SSGs like e.g. Hugo, you’re stuck because there’s no plugin system at all.
I don’t think this is the case for everyone. I’m sure there are edge cases, but if you simply need your SSG to render posts/pages/RSS you will be fine. The average “simple” blog rarely needs new features.
Yeah I’ve had my own custom blog site since 2008 and this has never happened to be once. If anything I’ve removed features I realized were unnecessary.
But it’s fun work. At least for me, as I don’t work in the web world normally. I wrote my own blog engine and over the 24 year history of my blog, there have been features I’ve added (like automatically cross posting to InstaMyFaceMePinTikLinkedSpaceTotWeBookGramTrestIn) only to remove later (when the API inevitably change). I’m not worried about the language it’s written in changing too much, as it’s in C.
In contrast, the rest of my website is a static site, using xsltproc (quite the retro-future approach). I wrote the XSLT back (as far as I can tell) around 2003-2004 and there was only one time I had to update it due to changes in xsltproc. I do not recommend using XSLT—it works, but boy, is it verbose. Way more than Cobol (and it’s a pure functional language, which is even more head exploding).
What happens is that you implement a minimal feature set to get your website working
FWIW I did exactly that for https://www.oilshell.org/ . I wrote a shell script that used Gruber’s markdown.pl (packaged in Debian) to put up a single blog post, and then an index.html that linked to it.
People actually read and liked the blog, which I was slightly surprised by, given that I knew many people had “written off” shell as a language to be learned.
So now I knew it was worth spending some time on, and gradually added features over the years. The anti-pattern is to “plan” the blog up front.
Then, next time you want to add something to the website, you realize a necessary feature is missing, and instead of updating the website, you spend your time implementing the feature. This happens over and over again and you lose interest in updating the website because it’s so much work every time.
That didn’t happen in my case. It’s not too much effort to use shell and Python to add new features.
It may not look like it, but the site is pretty deep now
I also deleted Google analytics a few years ago, and now use a pipeline of Python + R + JS + shell to make my own graphs. (the thing that is rotting here is the spam detection – there are so many undeclared crawlers these days)
It also has topic tagging, and OpenGraph metadata for Twitter/Mastodon etc.
I would say that if it were only Python, it would be too much work. I would maybe have given up on doing everything myself.
I would have fallen into the “programmer blogger trap”. This kind of code can be very repetitive and non-composable in pure Python.
But gluing everything together with shell makes it feasible. It’s just less code overall. And it’s faster to iterate.
I mentioned this Unix philosophy / multi-language / multi-process factoring here:
And to be fair it took me a long time to build up those skills – making a website is not trivial !!! I failed to make websites from scratch before, for sure. Multiple times. The benefit has to be worth the effort, as you say, and the effort is often high.
But now that I finally have one, I view it as an asset.
I’ve struggled a bit with this the last couple of years: I’m a perfectionist, and I mean that in the negative sense. Knowing that any good system should have CI, good logging, proper error handling, tests, documentation, deploy steps, proper DB migrations, and so on is a curse when you feel that everything you make should have it. Whether you’re learning something new or want to make a small new thing, you’re forcing yourself to slog through the boring stuff instead of building and learning.
As an attempt to recover from that, I’ve started to participate in a daily art challenge during every October. It’s a bit freeing because I have no experience with it professionally, so I suck and I see progress quite quickly. Hopefully it’ll help me just building the darn things I want to instead of using time on all the unnecessary bureaucracy.
There’s been an attempt to distinguish between “degrees Celsius” (temperature) and “Celsius degrees” (temperature difference) for that reason, though they still use the same symbol. Though it’s usually not a problem for humans because we can deduce it from the context, it’d be neat to have °C and, say, C° to eliminate any possible confusion. Getting people to adopt that would be an entirely different matter though…
Very field dependent. Lab sciences often use Kelvin, but the climate change literature is almost always phrased in terms like “1.5 degrees C warming” (not only in popular communications but also in journal articles).
I’d assume physics default to Kelvin all over the place, but I have never heard someone casually saying that it’s 10 Kelvin hotter/colder today compared to yesterday. That’s either Celsius or Fahrenheit, as the article mentions.
I got one of these to prevent RSI. I haven’t experienced any, but wanted to do preventative measures after I got a light case of emacs pinky (resolved by remapping Caps Lock to Ctrl).
The Oryx tooling was horrible at that time for Scandinavian keycodes, at least Norwegian ones. I could add æøå, but since our symbol pairings are completely different, delimiters wouldn’t work, & turned into / etc… Which I kinda get: We use AltGr quite extensively, and that’s absent from US keyboard layouts AFAIK.
I didn’t bother to manually try and resolve this, so I instead learnt the US layout. I guess that’s a blessing in disguise, because I felt a lot of the symbol locations didn’t make sense to me. So I tweaked and adjusted them quite liberally in the beginning, leading to something that now works well for me. That also caused me to end up with home row mods and frankly made my setup unusable for keyboard gaming… I mean, I guess that also prevents RSI, right?
I got a Keyboardio Model 100 also, which I prefer because of the palm key and because it’s slightly easier to reach the top keys for me. And because the keys are curved it’s slightly easier to automatically align your hands, but that’s not that big of a deal.
As a Clojurist, I strongly disagree. Clojure libraries are simple, have one purpose, and then they’re completed and don’t balloon into gigantic frameworks with scope creep and an eventually mismatching name. This article just points out a flaw in modern programming approaches and not really a universal truth. Aim for composability and then you can name each level of abstraction a new thing if you need to.
Oh, yeah, sorry, forgot to clarify that I meant that Clojure libraries are unlikely to ever end up bloating to the point of names mismatching, but unfortunately, Clojure has a ton of silly named libraries and projects. Really, you can do this with any language where you can write nicely atomic, composable, purely functional libraries, that rely on common data structures for their interfaces. Like, JavaScript and Ruby mostly come to mind. But culturally, it’s popular to find witty names for projects nowadays, so it is what it is…
There’s a big difference between libraries and services though. For one, libraries usually compose much better than services, and second, the purpose of a single library is usually fixed in scope, whereas a service tends to grow with a company’s goals etc.
But also, there’s no reason to not name subcomponents either, especially in log messages. “Shelob has broken again” is so little descriptive than saying “Shelob’s crawler halted because we set the outbound firewall rules too strict again”.
If I don’t know why somebody would need this, can you point me to an introduction to the topic of “expression language”? I figure some of this stuff (abs, int, float, string, max, min) is already in Go’s standard library, or some kind of “math” module. Why not just use that? The home page shows a stack machine running. Is this like a business logic execution engine, where you need configuration, introspection, and logging, instead of just plain logic execution?
Embeddable expression languages are used to express ‘rules’ in external configuration files/UIs
In technical world – these are firewall rule configurations.
In infrastructure world, you have monitoring applications that set up ‘alerts’ when an email should be send if something bad happens. These alerts are ‘rules’ expressed in these kind of languages.
In Security world, there are multiple examples – intrusion detection alerts, XACML authorization rules, etc.
In gaming world, to express specific character behavior’s, etc – Lua is used, often.
This expression language is to be embedded in a Go application.
In java world, there are several of these: eg http://mvel.documentnode.com/ or
There is a list of users down in the README. I was initially a bit confused at first myself, but I can imagine this being super valuable when you want to give users the ability to input arbitrary expressions, but don’t want to give them a fully fledged programming environment for security or performance reasons (think infinite loops through recursion or whatnot).
I’ve thought about using it for a templating language. Give users something that they can do at runtime to execute some code in templates without having to write a dynamic language myself.
Many many years ago I wrote something very similar, it was used as an expression language in an in-browser template engine (back when such a thing was novel).
Any time you want to give users the ability to filter, eg “run action A on this set of machines” where you might use an expression language to select the machines. Think of the --filter flag for most cloud provider CLI tools.
These kinds of ultra-strong backwards-compatibility guarantees for programming languages and their standard libraries tend to always lead to the same few outcomes (or some combination of these outcomes):
The compatibility guarantee devolves into arguments from upstream about how “well, technically…” they didn’t break compatibility even though your code got broken, or
The language accumulates huge amounts of “here be dragons” cruft where APIs that are actively bad and wrong and harmful to use are still kept around, and everyone has to just learn lots of “oh, don’t use that, or that, or especially that, and whatever you do don’t use this…”, or
They give up and relax the backwards-compatibility policy.
So far, Go seems to do a fair amount of (1) and probably is getting to the age where (2) will be a significant factor soon.
The language accumulates huge amounts of “here be dragons” cruft where APIs that are actively bad and wrong and harmful to use are still kept around,
I’ve been thinking about this recently, and it seems that the practical perception of this problem is too much influenced by Python. Python is always touted as an example why batteries included might go wrong with time, but I think Python’s stdlib was unique randomly assembled in the first place. Like, stdlib modules like unittest don’t follow language’s naming convention! This has nothing to do with compatibility, and everything to do with the practice of deciding what goes into stdlib. I don’t know how that was decided in Python, but I have a strong suspicion that “anything goes” was pretty close to the truth.
Now, when I update back from that example, I sort-of feel that a properly curated stdlib would age well? If you are careful with what you accept, it shouldn’t be too hard to maintain compat, and it’s unlikely that things go sour with time.
I predict that in 2032 Go stdlib would be fine (I don’t know how good it actually is today, but, given that Deno modeled its API on Go, I have a high prior that it is), and in 2035 Rust stdlib would be fine, and in 2035 people would still be joking about unittest naming convention, heapq procedural APIs and three argument parsers in Python’s stdlib.
My go-to example of a “here be dragons” stdlib is actually Java.
Python pared down some of the older stuff in the 2->3 transition, and has a plan in place for how to gradually deprecate and remove things from its stdlib. But Java… the list of “don’t use that, even though it’s the most immediately obvious thing in the stdlib for what you want to do” stuff in Java is getting pretty long, and it’s entirely due to Java’s absolutist stance on compatibility.
Yeah, Java is a good example here, it’s much more of “that seemed like a good idea at that time” (eg, mutable DateTime objects). This actually makes me think that I was foolish with my Go prediction — there was a fair amount of Java API churn with the addition of generics…
there was a fair amount of Java API churn with the addition of generics…
Was there? Java’s generics are gimped specifically to avoid it, much of the churn predates the addition of generics, and instead dates back to Java 1.2’s Collections Framework which soft-deprecated a bunch of types e.g. Vector (for ArrayList), Hashtable (for HashMap), Dictionary (for Map), Enumeration (for Iterator), … With generics the existing class hierarchy just got “informative” generic parameters tacked on.
C# was the one which had a lot of churn with generics, because it uses reified generics and thus the non-generic classes and the generic classes were incompatible. Hence System.Collections and System.Collections.Generic.
Oh wow, looks like I didn’t do my Java history homework! I was thinking that Vector -> List was due to generics, but of course you are right, thanks for correction!
My go-to example of a “here be dragons” stdlib is actually Java.
Java 2 improved this a lot, and 1.1 a bit, but 1.0 was full of classes with totally different naming conventions. Java kept the old ones for quite a while with deprecated labels on them (I think some are still there, but I haven’t really looked at Java since Java 8) but made those wrappers around the consistently-named versions.
C++ is up there, too. Learning C++ is a process of figuring out which of the six ways to do any given thing is the correct way, and which five are old ways from the ’90s and ’00s that are now known to cause memory leaks or run exponentially slower. Maintaining your C++ knowledge is a process of regularly converting all of your code to the new seventh way that got added to the standard library last week because the last way was also bad.
I think any non-trivial code, including the stdlib, will include bad-in-retrospect parts. Bugs, mistakes in design, and stuff that can only improve as the rest of the world changes (e.g. supporting popular but bad protocols, like TLS 1.1).
If you want to change things and realise that however hard you try you’re going to break backwards compat sometimes for some people then I think the polite thing is to let people select the version of each dependency at a good level of granularity so that folks can upgrade components separately.
I think Rust does this well by having a small standard library with lots of core stuff available in crates. Julia did this less well by having quite a large standard library, but now you have to explicitly note your standard library imports in your project.toml, same as 3rd party imports, so you can version them.
If you let users pick and choose more versions then you get new problems, especially if breaking changes are common: like a single program depending on multiple versions of the same library, which can be confusing and bloats binary size; or requirements clashes if your language doesn’t allow multiple versions of the same library to be used together.
I think those are better problems to have than forcing users to adapt to all of the changes in the language and stdlib at once if they want to upgrade or alternatively having pseudo versioning where nothing is ever removed but you end up with a cluttered stdlib because things get replicated (urllib2, pathlib, strncat, etc).
I like the approach Go is going for here with the new godebug behaviour and features. It’s granular versioning for the stdlib (and maybe base language in future?) but with different trade-offs versus handling it all through semantic versioning and the package manager.
Python I think made a couple of really bad, but in my opinion also fairly obvious design decisions. Things that made me avoid Python both for the language and standard library for really long, until I first used it. It always baffled me that Python was advertised as that language being simple and minimal and we’ll and clearly designed when even your hello world is enough to discredit the claim with print not being a function.
As for the standard library for example dealing with any kind of timestamps were horrible.
And then how to deal with packages.
For all of these there has been better prior art.
The good thing is that most of this was fixed, but I don’t think Python is a good benchmark.
There is unfortunately no way to assert whether a given change to a user-facing API is or isn’t a breaking change. (Hyrum’s Law makes this clear.) Every change exists at some point on the spectrum of impact, so every change policy is necessarily subjective, and your option (1) is always going to apply, no matter what. The only question is where to draw that (subjective) line in the sand.
There have definitely been changes to Go which have broken user programs. But AFAIK they’ve almost exclusively been security-critical fixes to e.g. crypto packages, or enforcement of stuff asserted by the spec but not necessarily provided by the implementation(s). In all cases, AFAIK, the net impact has been very minimal, statistically zero.
There have definitely been changes to Go which have broken user programs. But AFAIK they’ve almost exclusively been security-critical fixes to e.g. crypto packages, or enforcement of stuff asserted by the spec but not necessarily provided by the implementation(s)
From the article:
ParseInt. For example, Go 1.13 added support for underscores in large numbers for readability. Along with the language change, we made strconv.ParseInt accept the new syntax. This change didn’t break anything inside Google, but much later we heard from an external user whose code did break. Their program used numbers separated by underscores as a data format. It tried ParseInt first and only fell back to checking for underscores if ParseInt failed. When ParseInt stopped failing, the underscore-handling code stopped running.
To me that’s a breaking change. As far as I’m aware, the spec for ParseInt was not ambiguous, and its prior rejection of values containing underscores was neither a bug nor a critical security vulnerability. So to argue that this didn’t violate the compatibility policy is interesting.
For sure that’s a breaking change, in the boolean/Hyrum’s Law sense. But I certainly consider it a narrow edge case. Code which depended on ParseInt rejecting strings with underscores was acting at least somewhat outside of the documented guarantees of the function.
Again, the compatibility policy – any compatibility policy – is a subjective judgment, not a boolean assessment.
Again, the compatibility policy – any compatibility policy – is a subjective judgment
To put it mildly, this drastically decreases the utility of the policy. In fact, would argue that it makes the policy effectively meaningless – if it’s possible to retroactively and subjectively declare that a given break is acceptable because it was “an edge case” or “Hyrum’s Law!” or whatever, then the policy fundamentally cannot be trusted, because if someone can subjectively argue their way into justifying breaking that guy over there’s code, they can just as well subjectively argue their way into justifying breaking my code.
And this is not some sort of way-super-out-there-ultra-hyper-mega-fringe case. It relied on the fact that Go’s definition of an integer literal didn’t include certain characters. Then, Go’s definition of an integer literal changed to include more characters. The correct backwards-compatible approach would be either to introduce a new flag to ParseInt allowing code to opt in to the new character set (keeping the old as the default), or to introduce a new function which would only support the new character set.
So this is a break, plain and simple, and I don’t see how Go can continue to pride itself on its backwards compatibility policy when it’s clear that policy is meaningless since it will just always be creatively interpreted as “sure, it was a break, but that one doesn’t count, nor that one, nor that one either…”
To put it mildly, this definition of a breaking change isn’t useful, because it captures, literally, any and every change to an implementation.
It relied on the fact that Go’s definition of an integer literal didn’t include certain characters
Go never defined an integer literal as not including certain characters. The strconv.ParseInt function expressed, via documentation, a certain set of (subjective) constraints on what it considered to be a valid input string. Over time, those constraints were narrowed.
Were those changes breaking changes? Strictly speaking, yes. Practically speaking, no. The changes didn’t violate invariants asserted by earlier versions of the function, they introduced new invariants over properties that were previously undefined.
Code that called ParseInt expecting that it would always reject strings containing _ made assumptions that were never guaranteed. And if a behavior isn’t guaranteed – by the type system, or by documentation, or by some mechanism which is understood to represent stability – then you can’t treat that behavior as a guarantee.
An API provides only the guarantees which it explicitly declares.
Code that called ParseInt expecting that it would always reject strings containing _ made assumptions that were never guaranteed. And if a behavior isn’t guaranteed – by the type system, or by documentation, or by some mechanism which is understood to represent stability – then you can’t treat that behavior as a guarantee.
ParseInt interprets a string s in the given base (0, 2 to 36) and bit size (0 to 64) and returns the corresponding value i.
The string may begin with a leading sign: “+” or “-”.
If the base argument is 0, the true base is implied by the string’s prefix following the sign (if present): 2 for “0b”, 8 for “0” or “0o”, 16 for “0x”, and 10 otherwise. Also, for argument base 0 only, underscore characters are permitted as defined by the Go syntax for integer literals.
The bitSize argument specifies the integer type that the result must fit into. Bit sizes 0, 8, 16, 32, and 64 correspond to int, int8, int16, int32, and int64. If bitSize is below 0 or above 64, an error is returned.
The errors that ParseInt returns have concrete type *NumError and include err.Num = s. If s is empty or contains invalid digits, err.Err = ErrSyntax and the returned value is 0; if the value corresponding to s cannot be represented by a signed integer of the given size, err.Err = ErrRange and the returned value is the maximum magnitude integer of the appropriate bitSize and sign.
This is under-defined enough that your approach effectively allows ParseInt to do some wild things, because it doesn’t provide an exhaustive list of what is and is not valid for each possible base, or how each specific potential invalid character might be handled. For example, a future Go version could declare that any ASCII alphanumeric that isn’t a digit of the base is ignored (“1a2b3c” -> 123), or replaced with a zero à la HTML5 color parsing (“1a2b3c” -> 102030). And by your argument this would not violate any backwards-compatibility guarantee, since the current documentation does not promise to reject those (it just says what the error will look like if there are invalid digits, without defining the set of invalid digits – the same exact loophole you seem to be using to argue that the underscore change was non-breaking).
Personally I would find that to be an absurdity, and when an argument leads to absurdity I reject it on those grounds. As I reject your argument.
The ParseInt change was a breaking one. Yelling “Hyrum’s Law!” over and over doesn’t change that. Making up one-off excuses doesn’t change that. Arguing that we have to subjectively judge whether a use case is important enough to count as breaking doesn’t change that. Arguing that ParseInt can’t break compatibility because it’s so underspecified doesn’t change that.
This is under-defined enough that your approach effectively allows ParseInt to do some wild things, because it doesn’t provide an exhaustive list of what is and is not valid for each possible base, or how each specific potential invalid character might be handled. For example, a future Go version could declare that any ASCII alphanumeric that isn’t a digit of the base is ignored (“1a2b3c” -> 123), or replaced with a zero à la HTML5 color parsing (“1a2b3c” -> 102030). And by your argument this would not violate any backwards-compatibility guarantee, since the current documentation does not promise to reject those (it just says what the error will look like if there are invalid digits, without defining the set of invalid digits – the same exact loophole you seem to be using to argue that the underscore change was non-breaking).
Correct!
Arguing that ParseInt can’t break compatibility because it’s so underspecified doesn’t change that [this change was breaking]
I mean, it literally does mean that this change isn’t breaking, right? Whether or not a change is breaking depends on the guarantees asserted by the relevant API, and the guarantees of an API are determined by it’s specification, not it’s behavior.
not sure why your tone is so contemptuous. i’ve been using golang almost exclusively at home & professionally for 3-4 years, and i’ve never experienced a breaking change.
i also maintain several python projects and… well. the difference is so obvious and palpable that it’s hard to take your argument seriously.
I don’t have problems with Python, and I’m a heavy user of it. But I also know that other people have different experiences than I do, and I try to understand that and respect it and offer constructive advice when appropriate.
I just wish Go people would show the same respect and understanding to people who run into problems with Go, rather than endlessly parroting “well I don’t have that problem” and then going back to bashing whatever language they’re looking down their noses at this week. The overwhelming response of this thread to someone whose perfectly reasonable code was broken by a change in Go has been to dismiss it and blame the programmer for writing such code, and that’s not a good look for Go.
This isn’t a useful definition of a breaking change. In an open ecosystem, literally every change has the potential to break programs. Compatibility necessarily has a narrower definition. How much narrower is up for discussion.
Functions likeParseInt are often used as a way seeing if some input is numeric. That’s a super common idiom in lots of languages. The exact flow varies, of course; in some languages you get an exception, in others you get a boolean and the parse is stored in an out param, etc. In Go, ParseInt returns an error when the value passed in contains “invalid digits”.
Except that, in Go, ParseInt has already changed its definition of valid and invalid at least once, and as a result caused a break in someone’s code because inputs that used to be rejected no longer were.
That is not some sort of crazy way-out-there fringe spacebar-heating type thing. Like it or not, whoever wrote that code was doing something perfectly reasonable – there was no bug or critical security vulnerability in ParseInt as a result of which things it did or didn’t consider to be valid “digits”, and so there was no reason to suspect that the definition of valid/invalid might actually be a fluid thing that could be changed on a whim.
As I already pointed out, the right thing – if Go were actually committed to compatibility to the degree its fans like to claim it is – would have been to either introduce a new flag to ParseInt to opt into an expanded valid character set, or keep the old ParseInt as-is forever and introduce a new function which would always use the expanded valid character set.
Instead of which, Go chose to break people’s code. And, again, not because of any bug or critical security issue, but so that numeric literals could be slightly easier on the eyes.
If Python – which doesn’t even make the kinds of guarantees Go tries to – had broken end-user code for something that frivolous we’d never hear the end of it. In fact, we still haven’t heard the end of it with the print versus print() change. But when Go does this, there are endless excuses and rationalizations and justifications to try to explain that it was good for Go to arbitrarily break someone’s code and that it was the programmer’s fault for naïvely not reading all the fine print in the backwards-compatibility policy and realizing it contained loopholes big enough to drive a fleet of trucks through.
If Python had broken end-user code for something that frivolous we’d never hear the end of it.
FWIW it did in exactly the same manner: PEP 515 applied to both source literals and int parameters.
I’d think it’s either that there was never such super hard BC promise on new convenience so if people got hit they just went “oh well” orint is not super convenient to validate decimal integers in the first place so it’s rarely used that way. For instance why would you use int to parse an underscore-separated format when you can use csv with a custom separator, or split things out with re.
I assert that if someone had reported that Python broke their code with this change, and the response from the Python core team and from Python fans on programming forums had been exactly the response being put out by Go and its fans, we would never hear the end of it, and it would be held up endlessly as evidence that Python makes frivolous breaking changes and cannot be trusted (unlike Go, which takes compatibility seriously).
And I don’t think anyone can deny that’s exactly what would happen.
Except that, in Go, ParseInt has already changed its definition of valid and invalid at least once, and as a result caused a break in someone’s code because inputs that used to be rejected no longer were.
You simply can’t rely on properties of ParseInt which aren’t guaranteed by the docs for ParseInt, or the language spec.
I would even say Go has one of the best “compatibility stories”, but it’s still on the spectrum, so to speak. To claim Go 2 will “never” break Go 1 code just reminds me of Gary promising he “will never die” in Team America.
The changes are, strictly speaking, “compatible” in the sense of the Go 1 document, but they still break programs.
How could that be “compatible”, though? Unless we just redefine the word?
I guess it’s going to be like Windows, where there’s a culture of insisting it has perfect backwards compatibility (for reasons, to be sure), yet I can’t run old EXEs.
I’m really arguing that I think the ParseInt change was a breaking change.
The linked article dismisses this because the author(s) seem to feel that this wasn’t important enough, or didn’t affect Google enough, to count as a break. People here seem to be rolling with a combination of it not being important enough, and not being specified enough, to count as a break.
Personally I think it was entirely reasonable that a string containing characters illegal for integer literals would be a string that someone would assume ParseInt should reject, and Go should just admit it was a violation of backwards compatibility. And if such breaks are to be allowed, I think Go should stop pretending to have such an absolute ironclad unbreakable policy of absoluteness.
Python doesn’t make those kinds of promises. In fact, quite the opposite: Python has always had deprecations and removals. The Python 2 -> 3 transition was just an especially big all-at-once break, rather than a one-time thing, and what came out of that was not a promise never again to do backwards-incompatible changes, but a promise never again to do it on that scale/in that way (i.e., a Python 3 -> 4 of comparable scope to the 2 -> 3 change). You can go look at any recent Python version’s release notes and find a list of what’s newly deprecated in that version, what’s due for removal in the near future, and what was removed in that version.
So I do prefer Python, because Python strikes me as much more honest; it tells me when something’s deprecated, and how long I have until that thing gets removed/broken. But Go insists that it will never break, then does break and gaslights me about it afterward.
The context for this thread is Go’s ParseInt errored on strings containing characters illegal for a Go integer literal, and people relied on that behavior, but then Go decided to change the set of legal characters for integer literals, which broke real-world code. And the overwhelming response in this thread from people who like Go is to blame the people whose code was broken, while the official article that started all this just says that the change didn’t break anything at Google, as if that’s somehow consolation.
I think that’s a bad way to do things and a bad way to treat people and is probably one of the reasons for the reputation Go and its dev team have.
The assumption was that the ParseInt change wouldn’t break anything. But we’re just going in circles now. Enjoy being “treated like an adult” while dealing with breaking changes every minor release.
Enjoy being “treated like an adult” while dealing with breaking changes every minor release.
Kind of my whole point is that Go imposes breaking changes on people, and that people have to deal with those breaking changes. Using Go does not magically get you out of having to deal with breaking changes! It just puts you in a world where on top of still having to deal with breaking changes, everyone tells you that it’s your fault for writing code that worked on version N and not in Version N+1, because you didn’t study the fine print of the compatibility policy well enough.
You’ve clearly never actually used Go so this might be hard for you to imagine. But the vast majority of users have never experienced any compatibility issues when updating their Go version.
And the vast majority of Python users don’t experience compatibility issues when updating their Python version.
But the people who do experience issues aren’t any less important because of that, and their issues shouldn’t be dismissed (and they tend to be quite loud on programming forums). And I don’t think you’d let a Python fan dismiss such issues by claiming that most people don’t experience them. So I’m not going to let a Go fan do it, either.
(also, I have tinkered with Go, and maintain enough working knowledge to be able to read and review code since some people I work with write Go, but for many reasons I do not personally choose to write it or build things in it)
It’s arguable that this one is a breaking change. It’s only for the “base 0” case, and the strconv.ParseInt docs say “for argument base 0 only, underscore characters are permitted as defined by the Go syntax for integer literals”. I think one should read that as, “if the Go spec changes / becomes more permissive, ParseInt with base 0 will too”. Notably, ParseInt does not accept underscores (and hasn’t changed behaviour from 1.0) when you give an explicit base.
every change policy is necessarily subjective, and your option (1) is always going to apply, no matter what.
Would you consider a policy “Never change an existing, stabilized API or its implementation” as a counterexample to this assertion, assuming that whether an API is stabilized is clearly marked and unambiguous?
If not, would you consider “Issue a stable release of the language and library and never afterward change the language or library in any way” as a counterexample?
Would you consider a policy “Never change an existing, stabilized API or its implementation” as a counterexample to this assertion, assuming that whether an API is stabilized is clearly marked and unambiguous?
Never change an existing stabilized API? Sure. Never change its implementation? Definitely not. An API provides only those guarantees which it asserts, consumers cannot rely on any arbitrary behavior which they observe.
If not, would you consider “Issue a stable release of the language and library and never afterward change the language or library in any way” as a counterexample?
Again, stability is not a boolean, it’s a spectrum. Software producers choose a point on that spectrum.
Perhaps I was unclear. You asserted that “every change policy is necessarily subjective”, and I meant to ask whether you would accept “Never change an existing, stabilized API or its implementation” as an example of a change policy that is not subjective.
I added “or its implementation” to head off any subjectivity in whether changing the implementation counts as changing the interface.
Never change an existing stabilized API? Sure. Never change its implementation? Definitely not.
I assume you do not really mean that adding “or its implementation” turns the policy from objective to subjective?
Again, stability is not a boolean, it’s a spectrum. Software producers choose a point on that spectrum.
Sure, I don’t think I said anything to the contrary, but I was proposing as another example of an objective “change policy” a very specific point on that spectrum: the total cessation of development activity. I’m not suggesting this is necessarily a useful policy, though it can be in some cases.
I see. Yes, my claim that “every change policy is necessarily subjective” does indeed assume a definition of a change policy that permits changes. If you define a change policy which does not permit changes – as is the case for your example – then my claim would not apply.
The GODEBUG flags also seem really smelly to me, especially when multiple flags will inevitably interact with each other in funny ways someday. The Rust editions approach seems much better, with major checkpoints of behavior bundled together all at once.
I also like the minimal standard library, deferring to an ecosystem of well-known of crates for common tasks. How “well-known” these crates are is up for debate, but there are good resources like blessed.rs.
Go import also makes adding new libraries extremely easy, so I’m not really sure why the standard library needed to be so big. I think Go could have had a lot more packages in golang.org/x and kept stuff out of the standard library. That way you wouldn’t be stuck with those packages once they have been eclipsed by packages in the wider ecosystem.
Go’s authors believe that third-party dependencies are inherently costly and should be minimized as much as is feasible. This motivates a relatively large standard library, even if it comes at the expense of needing to support APIs that are “eclipsed by packages in the wider ecosystem”. Forward stability of user programs is judged to be the most important property, everything else is secondary.
Needing to temporarily turn off an feature in the built-in HTTPS library is a thing I want configurable via an env var. I expect to need to turn those on or off in arbitrary combinations in production.
For things like “nondeterministic is a keyword now”, I think I would definitely prefer language editions.
APIs that are actively bad and wrong and harmful to use are still kept around, and everyone has to just learn lots of “oh, don’t use that, or that, or especially that, and whatever you do don’t use this…”
Would you consider including a prominent warning in an API’s documentation that “It’s bad, and here’s why, so don’t use it” to save users from “[having] to just learn” not to use it?
If not, what about (and I would prefer this latter) a machine-readable annotation in the source code that, in addition to affecting the documentation (example), makes the compiler (or interpreter) emit a warning if the API is used?
2. The language accumulates huge amounts of “here be dragons” cruft where APIs that are actively bad and wrong and harmful to use are still kept around, and everyone has to just learn lots of “oh, don’t use that, or that, or especially that, and whatever you do don’t use this…”, or
I don’t feel that this is something unique to “ultra-strong backwards-compatibility” languages, this happens to every language out there that is used by a big audience.
Go decided to include batteries, which over time will cause more dragons to pop up. But it’s not a net negative imo, as it also “strongarms” in certain idioms that would be much harder to convey with libraries on the side. For example, a thing that annoys me with OCaml is the LWT/Async bifurcation, and even if something subpar was shipped with OCaml, it would’ve made the concurrency story a bit more palatable for newcomers.
Also, I think deprecated functions and types is more of a documentation and tooling issue than anything else. Hide deprecated functions or put them in a separate section, and let linters or editors scream whenever you see on in the code.
I can’t for the life of me seem to figure out what to do with a clojure program after I have written it. How do I take a
.cljfile and turn it into a.class?I don’t want to set up a leingen project or whatever, I don’t want to bundle it as a jar, all I want to do is write a .clj file in a text editor and turn it into a .class so I can run it like
java ClassNameAuthor of the article here. Clojure is more commonly loaded and compiled on the fly, so the easiest way to run a single
.cljfile is to just doclojure your_file.clj. This uses the official Clojure CLI. Compiling to class files (AOT compilation) is usually reserved for production deployments, where you build an (uber)jar.That said if you want to just compile a single namespace, you do that with Clojure’s
compilefunction. It’ll look for the namespace on the classpath (defaults tosrc), and write the resulting classes toclasses.Then from the shell:
You’ll need the Clojure and Spec jars to be able to run this:
Why a
.class? I use this trick to make my.cljfile executable:https://gist.github.com/yogsototh/dc16e3213115c8305720f8c74d4a8297
If you’re using
deps.ednto build your clojure project you probably wanttools.buildto produce an uberjar. https://clojure.org/guides/tools_build#_compiled_uberjar_application_buildWhy do you want it to run like
java ClassNamein the first place? To me that seems a bit arbitrary, and really doesn’t work unless you add the Clojure runtime on the classpath.Though I assume you just want to run a .clj file as a command on the command line, and there Babaskha is the de-facto standard tool to that these days.
Here’s a raw way with cli-tools:
Obviously, if you want an easier way of setting up the classpath, then the tooling is there for you. But if you don’t want to use it…
While the article states that there isn’t much to configure for SQLite, you should still configure it. I wouldn’t be surprised if setting
work_memfor Postgres – and notcache_sizeandtemp_storein SQLite – skew the results in favour of Postgres.I find it weird that they bundle s3 and postgres support directly into a runtime instead of having them as libraries. Looking at the documentation for s3, I see
and I get that it’s better to have a native implementation rather than a pure JavaScript one for speed (though I’m skeptic of their 5x claim). But I can’t see this approach scaling. If performance is a concern, then other services you communicate with also need to be fast. In particular Redis/memcached or whatever caching mechanism you prefer, but also ElasticSearch or maybe some Oracle database you need to talk to as well. Should those also be in the runtime? And how do you handle backwards compatibility for those libraries?
I guess this is just a long-winded way of saying “Why aren’t they implementing this as native add-ons and instead force this into the runtime?”. It feels much more modular and sensible to put it outside of the runtime.
Deno is doing the exact same thing so they are likely following suit. This is probably one of the ways Oven intends on monetizing the runtime.
My two biggest complaints are mostly to do with the API - one being that s3 is imported from bun, rather than
bun/s3or similar. Python provides a big stdlib, but the modules still have different names. You don’t dofrom python import DefaultDict, httplib, for example.The other one is that the
filefunction supports s3 URIs, which is nice from a minimal API perspective, but I also think it’s not ideal to treat s3 the same as local files. s3 has a lot of additional setup - e.g AWS credentials, handling auth errors, etc. So I think it makes sense to logically separate the behaviour for local vs remote files.I don’t mind new takes on AWS / postgres sdks, though. The SDK is pretty decent compared to some others (e.g Google or Slack), but I think both their AWS and postgress examples there (other than the two issues I mentioned) are pretty nice.
I agree with your sentiment and I am also very confused why I would use the Bun s3 implementation over the whole AWS SDK that I have been using and accustomed to for years now. Sure there could be some performance gains (for just S3) but I don’t see the benefit.
I’ve run into contention on S3 in a Python backend, and it’s really not fun. It’s a very good feature to have this sorted and guaranteed to work fast, it means that Bun can stay competitive with compiled languages for more intensive workloads. To me, this is a production mindset: identify the key components, and optimise them so that they don’t get in your way.
Call me impatient, but the startup time of this setup was too slow for me. I quickly ended up with long-living and shared Postgres databases instead. In my case, I started up a Docker image with configuration options to keep as much stuff in memory instead of disk. Then, the tests just connected to a hardcoded pg connection and did whatever it was asked to test.
Sharing database has some additional benefits: You won’t just test the database from an empty state. If you accidentally forget to add the
AND user_id = %3expression to your query, you’ll quickly realize whenever the second run of the same test occurs. Though this also depends on you being able to generate new users and data on the fly.Of course, there are nitty gritty details here. You’d like to clean up the database now and then so that it doesn’t blow up your memory use. And you probably need an isolated database from time to time. But that’s manageable, and the complexity is well worth it to make tests snappy imo.
(Also, if you can, this is just much easier to do with Sqlite)
In roughly 10 years of writing Go I got so accustomed to writing
maxthe long way around that I had to do it several more times, out of habit, with a coworker reminding me “you can just usemaxfor that”, before it finally sunk in.Go’s lack of
?:and (former) lack of stuff likemaxis considered pretty harmless by most, and mostly it is, but sometimes you really just want an expression, not a statement. Which one of these do you prefer?or
or
a bit contrived, but I’ve worked with real code that’s near enough to that. The old way of thinking was “this is logic, it’s making a decision, and you need a statement to codify the fact that you’re making a decision here”. The new way is “you know what, taking the greatest or least of two or more things is common and everyone knows what it means — not only is it okay for you to avoid writing the repetitive
if, it’s even okay to inline it into a function’s arguments list or an aggregate initializer sometimes.”FWIW
maxis nice to have in the standard library, but the overhead of writing it yourself got very low when generics came out:I was somewhat surprised that they made it into a builtin and didn’t just define one in the
cmppackage instead.Agreed. I don’t know the exact reasoning behind making it a built-in function but I have no complaints about it.
If you pass constants in, you get a constant out. Since there is not constexpr equivalent in Go, normal functions cannot do this, you need a builtin for that.
Imagine you’re obliged to do leetcode on a timer in Go, and you have to do this.
I’m not sure what the “you still need to do all the work” refers to here, but one issue I’ve had with Rust (not sure if that has changed recently) is that errors don’t compose: Say you have two APIs you integrate with, and both define an error enum to cover all the cases that can happen, then you’re forced to transform these two types into one somehow if you want to call one after the other and use the
?operator. I know it’s possible to use someIntotrick here, but it still requires a new datatype and some code to map from one enum option to another.OCaml, on the other hand, has composable error handling via polymorphic variants. Since I’m not currently writing any type signatures, this is fine for me, but I suspect it becomes tedious if I were to create .mli signatures. That being said, it seems OCaml libraries still very much prefers exceptions, as I’ve had to wrap a lot of them in e.g. JSON parsing and whatnot. Roc seems to lean completely into the polymorphic variants error handling, which I very much love. But from what I can tell, still has the problem of keeping these types up-to-date if I were to add a new error variant down the stack, and also want to manually write my type signatures.
… which I guess is a long-winded way of saying “yes, there’s a bit of overhead for the developer”, but it’s not as bad as it once was, and I think errors should be in your face. As for performance.. maybe? Not that its performance doesn’t matter, but your bottleneck surely lies elsewhere.
If you don’t care about preserving the exact semantic of the error, you could use a trait object return type of
Box<dyn std::error::Error>. Or, you’d have to be explicit and define your own errors.Another option is just a newtype wrapper that implements
Errorand wraps those two enums of error for the cases you care about, likeYeah, the latter is what I’m familiar with. But it causes a little bit of messy nesting imo, and it’s less granular than I’d like it to be. Assume, for example, that I handle all error types of a particular kind from
TheirError1– I still have to pattern match on it as a consumer, unless I wrap every other error.Yes, that is true. I’m hoping this becomes easier/ergonomic with
Trytrait stabilization.The composable error issue is interesting to me. I think a weakness-that-is-a-strength of Go is that errors are not exhaustive enums, but just an open ended interface, so that caller knows that they probably have to be prepared to handle unknown error types as well as known types.
In application-level code consider using e.g. anyhow or eyre, they are a much more powerful alternative to
Box<dyn std::error::Error>and make error handling easy if the exact error type is not needed. (eyre has some additional niceties e.g. integrating with tracing via the tracing_error crate)The second paragraph says
That’s not quite what I see. Ocaml does not have immutable arrays. In contrast, Clojure and Haskell do have immutable arrays (with fast indexing and update). I guess that is consistent with your conclusion that Ocaml is not up to the standards of other languages.
? Haskell immutable arrays don’t have fast updates, they copy the whole array on update. This is fine if you plan to update all or most of the entries in the array anyway, but if you want to do point updates then you probably want a tree or trie or something.
There’s a differential array library but I’ve been told that the constant factors are terrible.
Well, Haskell has Data.Sequence, which appears to have similar performance properties to Clojure vectors. Data.Sequence is based on finger trees, which according to Wikipedia has amortized O(1) put, get, pushing, reversing, popping, and O(log n) append, split. I don’t know the details of the Haskell implementation, or the constant factors. Clojure stores 32 items in each leaf node of a vector, leading to shallow trees, data locality and generally good performance.
Update: I also found persistent-vector in Hackage, which claims to implement the Clojure vector data structure, and claims “O(1) append element, indexing, updates, length, and slicing” with “Reasonably compact representation”.
If third-party libraries are fair game, then OCaml has the containers library and a persistent array: https://c-cube.github.io/ocaml-containers/3.13/containers-data/CCPersistentArray/index.html
There’s nothing in OCaml that prevents anyone from implementing an immutable array, but I guess the need doesn’t really come up that much in practice.
In Clojure, persistent immutable vectors have primitive syntax, you write
[1 2 3]for a vector literal, for example. Now that’s “immutability by default”. Needing to get an immutable array implementation from a 3rd party github is not “immutability by default”.And we have built-in immutable lists in OCaml which serve many of the same purposes as immutable vectors, so we also have ‘immutability by default’. I would encourage you to more carefully research the language before making comparisons out of context.
I am researching Ocaml, F#, Haskell, Clojure as possible languages to use for a future project. One of my requirements is a complete set of immutable persistent data structures, including arrays. I need to use these types pervasively throughout my code, to support rolling back to an earlier state, in several contexts. One of my concerns is quality of implementation. If persistent immutable data structures are built in to the language core, and everybody uses them as a matter of course, then the implementation is likely solid and performant and the APIs are likely complete. If I have to use a 3rd party library on github, then I have less confidence in the quality of the implementation and I’m more likely to need to vendor it and maintain it myself.
None of the languages I’ve looked at meet all my requirements, but on the topic of persistent immutable data structures, Clojure is first and Haskell is second.
Can you say a bit more the application domain, and what kind of rollback logic you need? (For example, after you backtrack from a given state, do you even come back to it again?) I’m interested in use-cases for persistent and semi-persistent structures.
By persistent arrays, do you need only random access and update of one element, or do you need to add and remove elements at the end of the array, or to split or concatenate arrays?
The constant factors for persistent catenable arrays, even with very good implementations, are fairly high, to the point that I am not sure they are often a rational choice performance-wise – they are very nice for reasoning. For OCaml you can try Sek that is a high-quality implementation. If you are happy with fixed-size arrays with updates and random access, I would use Parray from Jean-Christophe Filliatre which offers O(1) access with good constant factors.
My previous prototype was in C++, and I used reference counting plus copy-on-write to perform functional updates on data structures. For arrays, indexing was efficient but concatenation required copying all the elements. I did not enjoy writing that much C++, so for the next prototype, I wish to use a high level functional language. Probably I’ll have lower performance, but I’ll be more productive, the code will be shorter and simpler, easier to reason about, I and will have an opportunity to explore more of the design space I’m interested in.
The application domain is something like Emacs, except instead of editing text, you are viewing and editing rich hypertext and hypermedia. Instead of the extension language being Elisp, it is like a pure functional language or like an imperative language with value semantics (not reference semantics). The extension language is a DSL and is part of the project: I’m writing an interpreter for this language, not embedding an existing general purpose language. All data is represented by pure immutable values with no pointer values or reference semantics. There is an undo stack, and immutable persistent data structures seem like a good choice for this. Whenever a document is updated by an editing action, this is accomplished by evaluating a function call in the extension language that maps the old document state onto the new document state. Document updates are transactional: if the function fails in the middle of the update, then the partial changes are rolled back. This is notionally implemented using STM (software transactional memory). If I have to implement my own STM, then I’ll use persistent data structures to efficiently roll back a transaction.
I agree that implementing a general purpose persistent array with good performance across a wide range of use cases is challenging. Clojure vectors are persistent arrays, and they are used ubiquitously throughout the language. My guess is that if the performance characteristics of clojure vectors are good enough for clojure then they are good enough for my application. The implementation has probably been heavily tuned.
I see, thanks for the details! Sounds like fun. I agree that trying to just use Clojure maps is a good bet. Clojure maps support transient/ephemeral updates which can also help, you don’t need to pay the cost of persistence on internal edits inside a user action whose internal states will never be observed. (This is a more static approach to copy-on-write-like optimizations, where the code is explicit about critical sections where sharing is forbidden.)
Thanks; I’m reading about Clojure transients right now.
This is so simple to do with OCaml modules that nobody would bother to use a third-party package for it.
See https://github.com/ocaml-multicore/kcas
Thanks for the pointer to kcas. How do OCaml modules support persistent immutable arrays?
First, I think we need to clear up some terminology. The behaviour you described earlier:
…is not what I (and I think most people) would understand as ‘persistent’. Persistent data structures are designed such that doing an immutable update would reuse much of the existing data structure in memory. Eg, an OCaml list is a simple persistent data structure. If you prepend (a new head element) to the list, the original list does not need to be copied at all. It is immutable and just reused in the new list.
What you described in ‘copying all the elements’ of your array is a non-persistent immutable data structure. This is easy to do, of course, but not as memory-efficient (obviously). Here’s a simple example in OCaml using mutable arrays as the backing store:
The custom module that we wrote here is used to present a fully immutable interface on top of a mutable array. It is possible to use this interface and then completely swap out the implementation with, say, a new persistent data structure like the one I linked earlier in the thread. Those are of course more complex implementation-wise.
Yeah, my old system used a method for efficiently updating immutable data structures, based on reference counting and copy-on-write, but it did not use persistent arrays. I think that my new system would benefit from persistent data structures, due to the need to store multiple versions of the same state (as I described), with small differences between each version. For this situation, persistent data structures would save memory.
Thanks for the example code, I understand your earlier comment now.
In some cases it might be useful to have a read-only array type. E.g. when you give it as an argument to a function that “promises” not to modify it. It is possible to write a wrapper module that only exposes get though.
Yeah I was talking about Data.Array.
Ocaml has all the Okasaki data structures available too.
Both OCaml and Haskell have immutable singly linked list you can pattern match on. I’m not sure what Haskell provides in the standard library at the moment, but afaik there are efficient immutable collections you can add as dependencies for both languages.
Saddens me to say it, but I think it’s fairly accurate. OCaml is a relatively old language that shines in its historical domains of application (formal logic tools, compilers of all sorts, and do-it-from-scratch servers on top of posix). There’s been a lot of effort to try and have it catch up to the “modern” web stacks, with async IOs, web frameworks, ocaml-to-javascript compilers, etc. but it’s never really stabilized[^1]. The fragmentation in the ecosystem is huge (I should know), releases break code, and opam is designed in such a way that it needs people to explicitly set lower and upper bounds for dependencies (which of course people don’t always do).
[^1]: dream is the de facto lead web framework, I think, but it’s been stuck just below 1.0 for a long time.
In fact it seems they’re encouraged to do the opposite: opam-repository maintainers have pushed me to get rid of my dependency upper bounds, claiming they can fix it later if the need arises. I think they spend a significant amount of time doing this kind of housekeeping. If it were me I’d have a CI check that would fail the PR build if all dependencies did not have semantic versioned upper bounds.
One of the many unfortunate impacts (albeit a small one) of the Russia-Ukraine war. OSS maintainers need to eat…
Wow, that’s very disappointing to hear. You would have thought by now that people would have learned that reproducibility is worth a little more work up front.
I think this whole article demonstrates that “fix it later” doesn’t work.
Having worked with JVM languages for too long I physically shudder every time I have to deal with dependency bounds. I will take reproducibility over automatic updates any day.
Because of those reproducibility difficulties, some OCaml people have started using Nix instead: https://priver.dev/blog/ocaml/bye-opam-hello-nix/
I don’t know the specifics of your exchange with the maintainers, but IMO this mischaracterizes the general policy and its rationale: the opam repository is curated and CI checks are run on all package submissions. The CI checks include tests of all reverse dependencies. What we aim to offer thereby is an ecosystem of packages in which we do not prematurely lock out compatible upstream upgrades. Instead, when a breaking change to an upstream package is submitted to the repo, we place an upper bound when it is needed, before merging the breaking change.
Of course, I’m sure there are some cases where packages can slip thru the cracks, and we have a lot to do to improve the process, but it is not just a flippant “we will let things break and fix it someday”.
So, in short, re:
IMO, from the perspective of a user of the repo, the reverse dependency CI check we actually do is strictly superior to this overly constrained approach.
Which is great, but how then would we fix errors when a dependency removes a deprecated item and it breaks an older version of a revdep?
For example: https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/b115ba96baec90c8135e1f7f8a821999485eacb1/variant/compilers,4.14,dream.1.0.0~alpha6,revdeps,dream-html.3.4.1
Warning:
I will fix this as of dream-html 3.4.2, but all versions <3.4.2 will fail to compile with future versions of Dream after
Dream.not_foundis removed. How would the maintainers tackle this? Manually remove all versions <3.4.2 from the build matrix by setting some constraint somewhere? This doesn’t seem very scalable, right?See https://github.com/ocaml/opam-repository/issues/23789 . I imagine your input there could be valuable.
Indeed, archiving superseded patch versions and marking them unavailable has been discussed. Tho, iiuc, we also endorse adding upper bounds to packages when they become necessary (and not preemptively): perhaps that is the best course here? Perhaps that recourse needs be better documented for package authors?
Adding upper bounds when needed is low hanging fruit for automation, and on our roadmap.
The main benefit of discouraging premature upper-bounding is to reduce the possibility of passages into dependency hell across the ecosystem. AFAICT, this is working quite well in OCaml compared to other ecosystems I’ve worked in.
I appreciate the followup. It is interesting to see the process in action! IMO, it is very cool that we have this revdep CI running and multiple people in the ecosystem collaborating over it to plan how to adapt to keep everything working well. I am also noting that several revdeps are showing up in the CI as broken. I wish we had fixed or diagnosed those before merging, but at least we can see the problem and can tidy in a followup.
That’s great, but this is a manual process. It works today because relatively speaking there is a tiny number of opam packages. Are the opam-repository maintainers going to scale this linearly as more packages are added over time?
That is the aim, yes. We have lots of room for low hanging improvements in the current process, and some ways to automate and semi-automate the process in the pipeline. That said, I believe it is an open question how far we’ll be able to scale this approach. But if we manage it, I think opam users are going to continue benefiting from a nice, stable ecosystem of packages. In any case, my point is to try to correct the apparent misconception some readers of your remark formed, that the attitude of the repo maintainer is to invite dependency hell by allowing in packages that break downstream deps.
I haven’t seen issues with the version ranges yet, so I can’t comment on that.
Apart from that, most of the issues seem surmountable, especially the hoops I had to jump through. Though my biggest fear is probably the lwt/async/eio split and if there ever will be a consolidation on eio at some point. It seems like the most confusing and scary thing for a newcomer: What happens if a library I want to use only has, say, async support?
Just fyi, Jane Street Async does not support Windows, but Eio and Lwt do. And Eio has Lwt integration support.
OCaml might be old but the 2003ish ecosystem felt so different than the last 10 years that it might as well be a different language. The stuff described in the post as problems were not even ideas back then.
Another: Nix has paths as a basis type.
They represent the file or directory that they resolve to relative to the source file they’re in. If you coerce them to a string, that file or directory is “interned” into the nix store and the resulting absolute path is used as the string.
Nushell has this too and it’s a blessing for shell scripts.
It also has file sizes as a type, which can be noted down as
10mb,0.01gbor whatever you like and converted to bytes using into int. The same thing for Durations would be great for normal languagesGo has that, actually. The durations are defined like so:
and outside the time package you can do
5 * time.Secondto denote 5 seconds, for example.No, go does not actually have a proper Duration type. You can do
time.sleep(123)expecting seconds, and forget the Duration const, expecting seconds but get nanoseconds, it’s not opaque.Kotlin is a language with a proper duration type, and you have to do
123.seconds, which will convert the Int to a Duration. But it is not a built-in type, which is what I was pointing towards.The EXPRESS modelling language has bags, and arguably, SQL has as well (when you don’t do an ORDER BY, that is).
NULL in SQL is quite unique in that very few languages have the same semantics on NULL. Many think of it as an “not there” value (and it is often used that way), but I think it’s better explained as a value “we don’t know” yet. In a sense, UNKNOWN would be a better name, as
select (null = 1) is nullreturning true sounds quite confusing to people, whereasselect (unknown = 1) is unknownis probably more understandable. Arguably, the absence of a value and a missing/unknown one are two different things, so we should have both.Also, Postgres at least has both literal rows and literal tables. I don’t know whether or not that is in conformance to the standard or a local innovation.
If you don’t know what that even means, mariadb has some docs on what it enables you to do. Basically versioning for your data on the level of tables, such that you can go back in “time”/transactions.
I wonder how well things like GDPR are implemented or understood when using this. Though the problem of slowly changing dimension isn’t new.
Yeah, that’s a good starter – seems like I completely forgot explaining what it actually is.
For GDPR: I imagine you’ll still be able to utilise an
ON DELETE CASCADEon the history table and then refer to the original user table. Though maybe you’ll have to do it indirectly through a user identity table if the triggers hinders you from doing so.For GDPR’s right to be forgotten, you’d only need to delete anything that could identify a person (eg: names, addresses, even pseudonyms like an IP * timestamp). So as long as your data governance is up to scratch, you should only need to anonymise the tables containing PII, and the rest can stay. (Although someone might argue that say, a pattern of purchases might constitute PII, it comes down to how good/evil your lawyers are).
Glad I didn’t choke on a beverage on this :P
The problem with the words easy and simple is that they are used by so many, and are in many cases interchangeable nowadays. If you look up the definition of them both there’s quite the overlap. So while I’ve enjoyed the “Simple vs. Easy” talk, I avoid using both words when describing a language because I don’t think most developers are linguists (Sorry Clojure people).
The post discusses a lot of things you have to think about when picking a programming language, but I’d like to focus on one thing that didn’t sit right with me. The post argues (indirectly) that Python abstracts certain things Go doesn’t, and that makes it faster to make an implementation of whatever you have. But abstractions are leaky, and at some point enough edge cases of that abstraction leaks through and will make Go more maintainable. That makes sense.
But then the post says Rust is not easy (fair enough) and not simple either, without really going into more detail than “I cannot look at a piece of Rust code and say with certainty that I understand what is going on there”. Which, well, isn’t really an indicator of whether something is “simple”.
I’d say Rust removes even more abstractions and at the same time attempts to prevent certain kinds of programmer errors. That comes with complexity both Go and Python abstracts or outright ignores at compile time (None/null pointers, memory management, preventing race conditions, the list goes on). Of course, forcing you to handle those can feel complex, but the alternative that Go and Python does is clearly worse if you need to build very performant and/or reliable software. So in a sense, you can say that – when it comes to abstractions – Rust is to Go as Go is to Python.
Its a good thing to periodically remind yourself that dictionaries are descriptive of how language is used not proscriptive about how language should be used. They are historical records not rule books.
Aside from avoiding words with an obviously high probability of offending or inducing undue discomfort to the intended audience, the best thing to do to avoid a descent into an argument about semantics is to define your terms at the outset. One of my favorite examples of this type of presentation is a segment of the 99% Invisible podcast.
But all you have to do is preface every
conference talkdiscussion with a couple dictionary definitions! 😛Just a heads up: I was sure the original memoised Fibonacci version would blow up due to stack depths, and sure enough:
For that reason it’s more common to use a bottoms up (DP) version instead to avoid any recursive calls. Curiously that’s what’s used for
fib(n)in the image.Anyway, what I find curious about Fibonacci is that it’s one of those things where the running time of integer addition and multiplication suddenly start to matter. If we benchmark the running time up to 1 million instead, we end up with this graph.
Since integer multiplication takes O(n log n log log n) time, the “linear” algorithm takes a bit over O(n²) time. Likewise, the “log” version grows some logs faster than O(n), though it is not as visible in this graph.
Indeed. I was a little surprised to not see this classic version, which apparently I memorized at some point:
You shouldn’t. What happens is that you implement a minimal feature set to get your website working. Then, next time you want to add something to the website, you realize a necessary feature is missing, and instead of updating the website, you spend your time implementing the feature. This happens over and over again and you lose interest in updating the website because it’s so much work every time.
Last time I tried, I found that even choosing a static site generator from the seemingly infinite list of options in 2016 was arguably more work than just building one. We tacked on a build pipeline for images and javascript at some point, but other than that I don’t think any features were added during the life of the site. There are certainly features it didn’t have, but that’s just the power of knowing your requirements up front, I guess.
Not to mention keeping your 3rd party SSG up to date. “Oh, I can’t post to my blog because _____ isn’t compatible with the current version of Ruby/Python/etc. Guess I’ll update it. Oh, now it’s obsoleted some config options I’m using, better learn what the modern equivalent is. Oh, looks like the theme I installed isn’t compatible, is there an updated version? No? Time to look for a new theme. Oh, now I remember I patched the old theme to add a custom feature and my pages won’t render without it; how do I port that patch to the new theme? Wow, looks like the theme engine changed a lot, I don’t recognize half these tags…”
I’ve been down that road several times.
The SSG doesn’t interact with the internet or any kind of untrusted data. Hence it’s totally fine to keep a VM or container image with the old working version of the tools forever and never update anything. :)
An advantage of a statically compiled SSG is you can just keep the binary around forever.
Sure, but that’s a bit inflexible.
I’d much rather have the whole compiler environment snapshotted in a container or VM image because then I can make changes to the SSG.
100% - I tried to get something done and was looking at hugo after a few years and it was quicker to write my own before reading up and making it work.
I’m not sure it’s fair to make a generalization about this. I found that building my own blog (not actually a SSG) in Rust with Axum and Maud was actually quite comfortable and ergonomic since I was able to lean on the existing tools that exist (Tower for any HTTP stuff, Comrak for markdown parsing etc). It was a fun learning experience, and I think a blog is simple enough to be doable as a learning exercise, but complex enough to expose you to a lot of language features you might otherwise have missed.
My experience with using an off the shelf SSG is that everything was fine and dandy until I wanted to do something I felt was pretty basic but which I ended up spending hours and hours on trying to get to work before finally giving up. Writing my own would not have this problem. I would just implement the basic features I actually need and then not touch it again.
This isn’t limited to homemade SSGs though. I’ve had to write quite a bit of Ruby and understand Jekyll internals to get some plugins to work together and in the way I want them to work, and the solutions feel rather hacky and are verbose liquid templates. I’m fairly certain this takes less time than making my own SSG, but from experience it’s much less fun than making it yourself.
And if you want a feature that’s not yet supported in some SSGs like e.g. Hugo, you’re stuck because there’s no plugin system at all.
I don’t think this is the case for everyone. I’m sure there are edge cases, but if you simply need your SSG to render posts/pages/RSS you will be fine. The average “simple” blog rarely needs new features.
Yeah I’ve had my own custom blog site since 2008 and this has never happened to be once. If anything I’ve removed features I realized were unnecessary.
But it’s fun work. At least for me, as I don’t work in the web world normally. I wrote my own blog engine and over the 24 year history of my blog, there have been features I’ve added (like automatically cross posting to InstaMyFaceMePinTikLinkedSpaceTotWeBookGramTrestIn) only to remove later (when the API inevitably change). I’m not worried about the language it’s written in changing too much, as it’s in C.
In contrast, the rest of my website is a static site, using
xsltproc(quite the retro-future approach). I wrote theXSLTback (as far as I can tell) around 2003-2004 and there was only one time I had to update it due to changes inxsltproc. I do not recommend usingXSLT—it works, but boy, is it verbose. Way more than Cobol (and it’s a pure functional language, which is even more head exploding).FWIW I did exactly that for https://www.oilshell.org/ . I wrote a shell script that used Gruber’s markdown.pl (packaged in Debian) to put up a single blog post, and then an
index.htmlthat linked to it.People actually read and liked the blog, which I was slightly surprised by, given that I knew many people had “written off” shell as a language to be learned.
So now I knew it was worth spending some time on, and gradually added features over the years. The anti-pattern is to “plan” the blog up front.
That didn’t happen in my case. It’s not too much effort to use shell and Python to add new features.
It may not look like it, but the site is pretty deep now
I rewrote the TOC generator once (it used to be JS, and is now static) - e.g. the yellow box http://www.oilshell.org/blog/2023/06/narrow-waist.html
I wrote a separate Flask + JS tool to make these little story link boxes, and then I copy and paste into Markdown - http://www.oilshell.org/blog/2023/06/narrow-waist.html#appendix-blog-backlog
I wrote a separate tool in PHP to serve resized images, to save bandwidth - http://www.oilshell.org/blog/2023/06/surrogate-pair.html
I also deleted Google analytics a few years ago, and now use a pipeline of Python + R + JS + shell to make my own graphs. (the thing that is rotting here is the spam detection – there are so many undeclared crawlers these days)
It also has topic tagging, and OpenGraph metadata for Twitter/Mastodon etc.
I would say that if it were only Python, it would be too much work. I would maybe have given up on doing everything myself.
I would have fallen into the “programmer blogger trap”. This kind of code can be very repetitive and non-composable in pure Python.
But gluing everything together with shell makes it feasible. It’s just less code overall. And it’s faster to iterate.
I mentioned this Unix philosophy / multi-language / multi-process factoring here:
https://lobste.rs/s/r2jawd/avoid_load_bearing_shell_scripts#c_xelmvc
And to be fair it took me a long time to build up those skills – making a website is not trivial !!! I failed to make websites from scratch before, for sure. Multiple times. The benefit has to be worth the effort, as you say, and the effort is often high.
But now that I finally have one, I view it as an asset.
Except this also happens with SSGs you didn’t write, and then it’s even harder.
I’ve struggled a bit with this the last couple of years: I’m a perfectionist, and I mean that in the negative sense. Knowing that any good system should have CI, good logging, proper error handling, tests, documentation, deploy steps, proper DB migrations, and so on is a curse when you feel that everything you make should have it. Whether you’re learning something new or want to make a small new thing, you’re forcing yourself to slog through the boring stuff instead of building and learning.
As an attempt to recover from that, I’ve started to participate in a daily art challenge during every October. It’s a bit freeing because I have no experience with it professionally, so I suck and I see progress quite quickly. Hopefully it’ll help me just building the darn things I want to instead of using time on all the unnecessary bureaucracy.
There’s been an attempt to distinguish between “degrees Celsius” (temperature) and “Celsius degrees” (temperature difference) for that reason, though they still use the same symbol. Though it’s usually not a problem for humans because we can deduce it from the context, it’d be neat to have °C and, say, C° to eliminate any possible confusion. Getting people to adopt that would be an entirely different matter though…
Kind of similar to the situation with “percent increase” and “percent point increase”
I think it’s more common to use Kelvin for temperature differences.
Very field dependent. Lab sciences often use Kelvin, but the climate change literature is almost always phrased in terms like “1.5 degrees C warming” (not only in popular communications but also in journal articles).
I’d assume physics default to Kelvin all over the place, but I have never heard someone casually saying that it’s 10 Kelvin hotter/colder today compared to yesterday. That’s either Celsius or Fahrenheit, as the article mentions.
I got one of these to prevent RSI. I haven’t experienced any, but wanted to do preventative measures after I got a light case of emacs pinky (resolved by remapping Caps Lock to Ctrl).
The Oryx tooling was horrible at that time for Scandinavian keycodes, at least Norwegian ones. I could add æøå, but since our symbol pairings are completely different, delimiters wouldn’t work,
&turned into/etc… Which I kinda get: We use AltGr quite extensively, and that’s absent from US keyboard layouts AFAIK.I didn’t bother to manually try and resolve this, so I instead learnt the US layout. I guess that’s a blessing in disguise, because I felt a lot of the symbol locations didn’t make sense to me. So I tweaked and adjusted them quite liberally in the beginning, leading to something that now works well for me. That also caused me to end up with home row mods and frankly made my setup unusable for keyboard gaming… I mean, I guess that also prevents RSI, right?
I got a Keyboardio Model 100 also, which I prefer because of the palm key and because it’s slightly easier to reach the top keys for me. And because the keys are curved it’s slightly easier to automatically align your hands, but that’s not that big of a deal.
As a Clojurist, I strongly disagree. Clojure libraries are simple, have one purpose, and then they’re completed and don’t balloon into gigantic frameworks with scope creep and an eventually mismatching name. This article just points out a flaw in modern programming approaches and not really a universal truth. Aim for composability and then you can name each level of abstraction a new thing if you need to.
“Leiningen.”
Makes perfect sense, it’s for coping with the situation when you have far far far far far too many ant invocations to deal with.
Oh, yeah, sorry, forgot to clarify that I meant that Clojure libraries are unlikely to ever end up bloating to the point of names mismatching, but unfortunately, Clojure has a ton of silly named libraries and projects. Really, you can do this with any language where you can write nicely atomic, composable, purely functional libraries, that rely on common data structures for their interfaces. Like, JavaScript and Ruby mostly come to mind. But culturally, it’s popular to find witty names for projects nowadays, so it is what it is…
There’s a big difference between libraries and services though. For one, libraries usually compose much better than services, and second, the purpose of a single library is usually fixed in scope, whereas a service tends to grow with a company’s goals etc.
But also, there’s no reason to not name subcomponents either, especially in log messages. “Shelob has broken again” is so little descriptive than saying “Shelob’s crawler halted because we set the outbound firewall rules too strict again”.
If I don’t know why somebody would need this, can you point me to an introduction to the topic of “expression language”? I figure some of this stuff (abs, int, float, string, max, min) is already in Go’s standard library, or some kind of “math” module. Why not just use that? The home page shows a stack machine running. Is this like a business logic execution engine, where you need configuration, introspection, and logging, instead of just plain logic execution?
It’s really common for rules engines that need to restrict what the user can do. Examples:
Standard configuration/rule syntax over multiple products: https://github.com/google/cel-spec
Embeddable expression languages are used to express ‘rules’ in external configuration files/UIs
In technical world – these are firewall rule configurations. In infrastructure world, you have monitoring applications that set up ‘alerts’ when an email should be send if something bad happens. These alerts are ‘rules’ expressed in these kind of languages.
In Security world, there are multiple examples – intrusion detection alerts, XACML authorization rules, etc.
In gaming world, to express specific character behavior’s, etc – Lua is used, often.
This expression language is to be embedded in a Go application. In java world, there are several of these: eg http://mvel.documentnode.com/ or
https://docs.spring.io/spring-framework/docs/4.0.0.RC2/spring-framework-reference/html/expressions.html
But also many use JVM-compatible languages such as groovy (although those may be too heavy for simple things).
There is a list of users down in the README. I was initially a bit confused at first myself, but I can imagine this being super valuable when you want to give users the ability to input arbitrary expressions, but don’t want to give them a fully fledged programming environment for security or performance reasons (think infinite loops through recursion or whatnot).
I’ve thought about using it for a templating language. Give users something that they can do at runtime to execute some code in templates without having to write a dynamic language myself.
exactly, for instance in https://ossia.io we use http://www.partow.net/programming/exprtk/ to allow the user to do simple one-liner computations involving basic math functions & operations
Many many years ago I wrote something very similar, it was used as an expression language in an in-browser template engine (back when such a thing was novel).
Any time you want to give users the ability to filter, eg “run action A on this set of machines” where you might use an expression language to select the machines. Think of the
--filterflag for most cloud provider CLI tools.These kinds of ultra-strong backwards-compatibility guarantees for programming languages and their standard libraries tend to always lead to the same few outcomes (or some combination of these outcomes):
So far, Go seems to do a fair amount of (1) and probably is getting to the age where (2) will be a significant factor soon.
I’ve been thinking about this recently, and it seems that the practical perception of this problem is too much influenced by Python. Python is always touted as an example why batteries included might go wrong with time, but I think Python’s stdlib was unique randomly assembled in the first place. Like, stdlib modules like
unittestdon’t follow language’s naming convention! This has nothing to do with compatibility, and everything to do with the practice of deciding what goes into stdlib. I don’t know how that was decided in Python, but I have a strong suspicion that “anything goes” was pretty close to the truth.Now, when I update back from that example, I sort-of feel that a properly curated stdlib would age well? If you are careful with what you accept, it shouldn’t be too hard to maintain compat, and it’s unlikely that things go sour with time.
I predict that in 2032 Go stdlib would be fine (I don’t know how good it actually is today, but, given that Deno modeled its API on Go, I have a high prior that it is), and in 2035 Rust stdlib would be fine, and in 2035 people would still be joking about unittest naming convention, heapq procedural APIs and three argument parsers in Python’s stdlib.
My go-to example of a “here be dragons” stdlib is actually Java.
Python pared down some of the older stuff in the 2->3 transition, and has a plan in place for how to gradually deprecate and remove things from its stdlib. But Java… the list of “don’t use that, even though it’s the most immediately obvious thing in the stdlib for what you want to do” stuff in Java is getting pretty long, and it’s entirely due to Java’s absolutist stance on compatibility.
Yeah, Java is a good example here, it’s much more of “that seemed like a good idea at that time” (eg, mutable DateTime objects). This actually makes me think that I was foolish with my Go prediction — there was a fair amount of Java API churn with the addition of generics…
Was there? Java’s generics are gimped specifically to avoid it, much of the churn predates the addition of generics, and instead dates back to Java 1.2’s Collections Framework which soft-deprecated a bunch of types e.g. Vector (for ArrayList), Hashtable (for HashMap), Dictionary (for Map), Enumeration (for Iterator), … With generics the existing class hierarchy just got “informative” generic parameters tacked on.
C# was the one which had a lot of churn with generics, because it uses reified generics and thus the non-generic classes and the generic classes were incompatible. Hence
System.CollectionsandSystem.Collections.Generic.Oh wow, looks like I didn’t do my Java history homework! I was thinking that Vector -> List was due to generics, but of course you are right, thanks for correction!
Java 2 improved this a lot, and 1.1 a bit, but 1.0 was full of classes with totally different naming conventions. Java kept the old ones for quite a while with deprecated labels on them (I think some are still there, but I haven’t really looked at Java since Java 8) but made those wrappers around the consistently-named versions.
C++ is up there, too. Learning C++ is a process of figuring out which of the six ways to do any given thing is the correct way, and which five are old ways from the ’90s and ’00s that are now known to cause memory leaks or run exponentially slower. Maintaining your C++ knowledge is a process of regularly converting all of your code to the new seventh way that got added to the standard library last week because the last way was also bad.
I think any non-trivial code, including the stdlib, will include bad-in-retrospect parts. Bugs, mistakes in design, and stuff that can only improve as the rest of the world changes (e.g. supporting popular but bad protocols, like TLS 1.1).
If you want to change things and realise that however hard you try you’re going to break backwards compat sometimes for some people then I think the polite thing is to let people select the version of each dependency at a good level of granularity so that folks can upgrade components separately.
I think Rust does this well by having a small standard library with lots of core stuff available in crates. Julia did this less well by having quite a large standard library, but now you have to explicitly note your standard library imports in your project.toml, same as 3rd party imports, so you can version them.
If you let users pick and choose more versions then you get new problems, especially if breaking changes are common: like a single program depending on multiple versions of the same library, which can be confusing and bloats binary size; or requirements clashes if your language doesn’t allow multiple versions of the same library to be used together.
I think those are better problems to have than forcing users to adapt to all of the changes in the language and stdlib at once if they want to upgrade or alternatively having pseudo versioning where nothing is ever removed but you end up with a cluttered stdlib because things get replicated (urllib2, pathlib, strncat, etc).
I like the approach Go is going for here with the new godebug behaviour and features. It’s granular versioning for the stdlib (and maybe base language in future?) but with different trade-offs versus handling it all through semantic versioning and the package manager.
Python I think made a couple of really bad, but in my opinion also fairly obvious design decisions. Things that made me avoid Python both for the language and standard library for really long, until I first used it. It always baffled me that Python was advertised as that language being simple and minimal and we’ll and clearly designed when even your hello world is enough to discredit the claim with print not being a function.
As for the standard library for example dealing with any kind of timestamps were horrible.
And then how to deal with packages.
For all of these there has been better prior art.
The good thing is that most of this was fixed, but I don’t think Python is a good benchmark.
There is unfortunately no way to assert whether a given change to a user-facing API is or isn’t a breaking change. (Hyrum’s Law makes this clear.) Every change exists at some point on the spectrum of impact, so every change policy is necessarily subjective, and your option (1) is always going to apply, no matter what. The only question is where to draw that (subjective) line in the sand.
There have definitely been changes to Go which have broken user programs. But AFAIK they’ve almost exclusively been security-critical fixes to e.g. crypto packages, or enforcement of stuff asserted by the spec but not necessarily provided by the implementation(s). In all cases, AFAIK, the net impact has been very minimal, statistically zero.
From the article:
To me that’s a breaking change. As far as I’m aware, the spec for
ParseIntwas not ambiguous, and its prior rejection of values containing underscores was neither a bug nor a critical security vulnerability. So to argue that this didn’t violate the compatibility policy is interesting.For sure that’s a breaking change, in the boolean/Hyrum’s Law sense. But I certainly consider it a narrow edge case. Code which depended on ParseInt rejecting strings with underscores was acting at least somewhat outside of the documented guarantees of the function.
Again, the compatibility policy – any compatibility policy – is a subjective judgment, not a boolean assessment.
To put it mildly, this drastically decreases the utility of the policy. In fact, would argue that it makes the policy effectively meaningless – if it’s possible to retroactively and subjectively declare that a given break is acceptable because it was “an edge case” or “Hyrum’s Law!” or whatever, then the policy fundamentally cannot be trusted, because if someone can subjectively argue their way into justifying breaking that guy over there’s code, they can just as well subjectively argue their way into justifying breaking my code.
And this is not some sort of way-super-out-there-ultra-hyper-mega-fringe case. It relied on the fact that Go’s definition of an integer literal didn’t include certain characters. Then, Go’s definition of an integer literal changed to include more characters. The correct backwards-compatible approach would be either to introduce a new flag to
ParseIntallowing code to opt in to the new character set (keeping the old as the default), or to introduce a new function which would only support the new character set.So this is a break, plain and simple, and I don’t see how Go can continue to pride itself on its backwards compatibility policy when it’s clear that policy is meaningless since it will just always be creatively interpreted as “sure, it was a break, but that one doesn’t count, nor that one, nor that one either…”
To put it mildly, this definition of a breaking change isn’t useful, because it captures, literally, any and every change to an implementation.
Go never defined an integer literal as not including certain characters. The strconv.ParseInt function expressed, via documentation, a certain set of (subjective) constraints on what it considered to be a valid input string. Over time, those constraints were narrowed.
Were those changes breaking changes? Strictly speaking, yes. Practically speaking, no. The changes didn’t violate invariants asserted by earlier versions of the function, they introduced new invariants over properties that were previously undefined.
Code that called ParseInt expecting that it would always reject strings containing
_made assumptions that were never guaranteed. And if a behavior isn’t guaranteed – by the type system, or by documentation, or by some mechanism which is understood to represent stability – then you can’t treat that behavior as a guarantee.An API provides only the guarantees which it explicitly declares.
Here is the entirety of the current documentation of
ParseInt:This is under-defined enough that your approach effectively allows
ParseIntto do some wild things, because it doesn’t provide an exhaustive list of what is and is not valid for each possible base, or how each specific potential invalid character might be handled. For example, a future Go version could declare that any ASCII alphanumeric that isn’t a digit of the base is ignored (“1a2b3c” -> 123), or replaced with a zero à la HTML5 color parsing (“1a2b3c” -> 102030). And by your argument this would not violate any backwards-compatibility guarantee, since the current documentation does not promise to reject those (it just says what the error will look like if there are invalid digits, without defining the set of invalid digits – the same exact loophole you seem to be using to argue that the underscore change was non-breaking).Personally I would find that to be an absurdity, and when an argument leads to absurdity I reject it on those grounds. As I reject your argument.
The
ParseIntchange was a breaking one. Yelling “Hyrum’s Law!” over and over doesn’t change that. Making up one-off excuses doesn’t change that. Arguing that we have to subjectively judge whether a use case is important enough to count as breaking doesn’t change that. Arguing thatParseIntcan’t break compatibility because it’s so underspecified doesn’t change that.Correct!
I mean, it literally does mean that this change isn’t breaking, right? Whether or not a change is breaking depends on the guarantees asserted by the relevant API, and the guarantees of an API are determined by it’s specification, not it’s behavior.
“Go: the language that promises nothing, so that it can break your code and blame you” is probably not the catchy slogan you want.
not sure why your tone is so contemptuous. i’ve been using golang almost exclusively at home & professionally for 3-4 years, and i’ve never experienced a breaking change.
i also maintain several python projects and… well. the difference is so obvious and palpable that it’s hard to take your argument seriously.
I don’t have problems with Python, and I’m a heavy user of it. But I also know that other people have different experiences than I do, and I try to understand that and respect it and offer constructive advice when appropriate.
I just wish Go people would show the same respect and understanding to people who run into problems with Go, rather than endlessly parroting “well I don’t have that problem” and then going back to bashing whatever language they’re looking down their noses at this week. The overwhelming response of this thread to someone whose perfectly reasonable code was broken by a change in Go has been to dismiss it and blame the programmer for writing such code, and that’s not a good look for Go.
This isn’t a useful definition of a breaking change. In an open ecosystem, literally every change has the potential to break programs. Compatibility necessarily has a narrower definition. How much narrower is up for discussion.
Functions like
ParseIntare often used as a way seeing if some input is numeric. That’s a super common idiom in lots of languages. The exact flow varies, of course; in some languages you get an exception, in others you get a boolean and the parse is stored in an out param, etc. In Go,ParseIntreturns an error when the value passed in contains “invalid digits”.Except that, in Go,
ParseInthas already changed its definition of valid and invalid at least once, and as a result caused a break in someone’s code because inputs that used to be rejected no longer were.That is not some sort of crazy way-out-there fringe spacebar-heating type thing. Like it or not, whoever wrote that code was doing something perfectly reasonable – there was no bug or critical security vulnerability in
ParseIntas a result of which things it did or didn’t consider to be valid “digits”, and so there was no reason to suspect that the definition of valid/invalid might actually be a fluid thing that could be changed on a whim.As I already pointed out, the right thing – if Go were actually committed to compatibility to the degree its fans like to claim it is – would have been to either introduce a new flag to
ParseIntto opt into an expanded valid character set, or keep the oldParseIntas-is forever and introduce a new function which would always use the expanded valid character set.Instead of which, Go chose to break people’s code. And, again, not because of any bug or critical security issue, but so that numeric literals could be slightly easier on the eyes.
If Python – which doesn’t even make the kinds of guarantees Go tries to – had broken end-user code for something that frivolous we’d never hear the end of it. In fact, we still haven’t heard the end of it with the
printversusprint()change. But when Go does this, there are endless excuses and rationalizations and justifications to try to explain that it was good for Go to arbitrarily break someone’s code and that it was the programmer’s fault for naïvely not reading all the fine print in the backwards-compatibility policy and realizing it contained loopholes big enough to drive a fleet of trucks through.This isn’t the first time this has happened. It probably, sadly, won’t be the last. I remember pretty clearly the time when someone literally got harassed and insulted so much by the Go partisans on this site they just gave up and quit trying to make their point.
I urge you to reflect on this and let it lead to changes in how you interact with people about Go.
FWIW it did in exactly the same manner: PEP 515 applied to both source literals and
intparameters.I’d think it’s either that there was never such super hard BC promise on new convenience so if people got hit they just went “oh well” or
intis not super convenient to validate decimal integers in the first place so it’s rarely used that way. For instance why would you useintto parse an underscore-separated format when you can usecsvwith a custom separator, or split things out withre.Then I’ll rephrase:
I assert that if someone had reported that Python broke their code with this change, and the response from the Python core team and from Python fans on programming forums had been exactly the response being put out by Go and its fans, we would never hear the end of it, and it would be held up endlessly as evidence that Python makes frivolous breaking changes and cannot be trusted (unlike Go, which takes compatibility seriously).
And I don’t think anyone can deny that’s exactly what would happen.
You simply can’t rely on properties of ParseInt which aren’t guaranteed by the docs for ParseInt, or the language spec.
Honestly, you’re not wrong.
I would even say Go has one of the best “compatibility stories”, but it’s still on the spectrum, so to speak. To claim Go 2 will “never” break Go 1 code just reminds me of Gary promising he “will never die” in Team America.
How could that be “compatible”, though? Unless we just redefine the word?
I guess it’s going to be like Windows, where there’s a culture of insisting it has perfect backwards compatibility (for reasons, to be sure), yet I can’t run old EXEs.
Are you really arguing that any observable change in a public API means you can’t claim backwards compatibility?
(Which is the definition of Hyrum’s Law)
I’m really arguing that I think the
ParseIntchange was a breaking change.The linked article dismisses this because the author(s) seem to feel that this wasn’t important enough, or didn’t affect Google enough, to count as a break. People here seem to be rolling with a combination of it not being important enough, and not being specified enough, to count as a break.
Personally I think it was entirely reasonable that a string containing characters illegal for integer literals would be a string that someone would assume
ParseIntshould reject, and Go should just admit it was a violation of backwards compatibility. And if such breaks are to be allowed, I think Go should stop pretending to have such an absolute ironclad unbreakable policy of absoluteness.Ok .. so if you don’t consider Go to have solid backwards compatibility, where does Python stand for you?
I think Go promises something it can’t deliver.
Python doesn’t make those kinds of promises. In fact, quite the opposite: Python has always had deprecations and removals. The Python 2 -> 3 transition was just an especially big all-at-once break, rather than a one-time thing, and what came out of that was not a promise never again to do backwards-incompatible changes, but a promise never again to do it on that scale/in that way (i.e., a Python 3 -> 4 of comparable scope to the 2 -> 3 change). You can go look at any recent Python version’s release notes and find a list of what’s newly deprecated in that version, what’s due for removal in the near future, and what was removed in that version.
So I do prefer Python, because Python strikes me as much more honest; it tells me when something’s deprecated, and how long I have until that thing gets removed/broken. But Go insists that it will never break, then does break and gaslights me about it afterward.
I think you’ve setup a bit of straw man for Go’s compatibility guarantee. Maybe you should read it https://go.dev/doc/go1compat
But I think it’s pretty telling that you’d prefer pythons backwards compatibility story. Maybe tone down the copium.
The context for this thread is Go’s
ParseInterrored on strings containing characters illegal for a Go integer literal, and people relied on that behavior, but then Go decided to change the set of legal characters for integer literals, which broke real-world code. And the overwhelming response in this thread from people who like Go is to blame the people whose code was broken, while the official article that started all this just says that the change didn’t break anything at Google, as if that’s somehow consolation.I think that’s a bad way to do things and a bad way to treat people and is probably one of the reasons for the reputation Go and its dev team have.
The assumption was that the ParseInt change wouldn’t break anything. But we’re just going in circles now. Enjoy being “treated like an adult” while dealing with breaking changes every minor release.
Kind of my whole point is that Go imposes breaking changes on people, and that people have to deal with those breaking changes. Using Go does not magically get you out of having to deal with breaking changes! It just puts you in a world where on top of still having to deal with breaking changes, everyone tells you that it’s your fault for writing code that worked on version N and not in Version N+1, because you didn’t study the fine print of the compatibility policy well enough.
You’ve clearly never actually used Go so this might be hard for you to imagine. But the vast majority of users have never experienced any compatibility issues when updating their Go version.
And the vast majority of Python users don’t experience compatibility issues when updating their Python version.
But the people who do experience issues aren’t any less important because of that, and their issues shouldn’t be dismissed (and they tend to be quite loud on programming forums). And I don’t think you’d let a Python fan dismiss such issues by claiming that most people don’t experience them. So I’m not going to let a Go fan do it, either.
(also, I have tinkered with Go, and maintain enough working knowledge to be able to read and review code since some people I work with write Go, but for many reasons I do not personally choose to write it or build things in it)
It’s arguable that this one is a breaking change. It’s only for the “base 0” case, and the strconv.ParseInt docs say “for argument base 0 only, underscore characters are permitted as defined by the Go syntax for integer literals”. I think one should read that as, “if the Go spec changes / becomes more permissive, ParseInt with base 0 will too”. Notably,
ParseIntdoes not accept underscores (and hasn’t changed behaviour from 1.0) when you give an explicit base.Would you consider a policy “Never change an existing, stabilized API or its implementation” as a counterexample to this assertion, assuming that whether an API is stabilized is clearly marked and unambiguous?
If not, would you consider “Issue a stable release of the language and library and never afterward change the language or library in any way” as a counterexample?
Never change an existing stabilized API? Sure. Never change its implementation? Definitely not. An API provides only those guarantees which it asserts, consumers cannot rely on any arbitrary behavior which they observe.
Again, stability is not a boolean, it’s a spectrum. Software producers choose a point on that spectrum.
Perhaps I was unclear. You asserted that “every change policy is necessarily subjective”, and I meant to ask whether you would accept “Never change an existing, stabilized API or its implementation” as an example of a change policy that is not subjective.
I added “or its implementation” to head off any subjectivity in whether changing the implementation counts as changing the interface.
I assume you do not really mean that adding “or its implementation” turns the policy from objective to subjective?
Sure, I don’t think I said anything to the contrary, but I was proposing as another example of an objective “change policy” a very specific point on that spectrum: the total cessation of development activity. I’m not suggesting this is necessarily a useful policy, though it can be in some cases.
I see. Yes, my claim that “every change policy is necessarily subjective” does indeed assume a definition of a change policy that permits changes. If you define a change policy which does not permit changes – as is the case for your example – then my claim would not apply.
The GODEBUG flags also seem really smelly to me, especially when multiple flags will inevitably interact with each other in funny ways someday. The Rust editions approach seems much better, with major checkpoints of behavior bundled together all at once.
I also like the minimal standard library, deferring to an ecosystem of well-known of crates for common tasks. How “well-known” these crates are is up for debate, but there are good resources like blessed.rs.
Go import also makes adding new libraries extremely easy, so I’m not really sure why the standard library needed to be so big. I think Go could have had a lot more packages in golang.org/x and kept stuff out of the standard library. That way you wouldn’t be stuck with those packages once they have been eclipsed by packages in the wider ecosystem.
Go’s authors believe that third-party dependencies are inherently costly and should be minimized as much as is feasible. This motivates a relatively large standard library, even if it comes at the expense of needing to support APIs that are “eclipsed by packages in the wider ecosystem”. Forward stability of user programs is judged to be the most important property, everything else is secondary.
They seem good for different problems I think.
Needing to temporarily turn off an feature in the built-in HTTPS library is a thing I want configurable via an env var. I expect to need to turn those on or off in arbitrary combinations in production.
For things like “
nondeterministicis a keyword now”, I think I would definitely prefer language editions.Would you consider including a prominent warning in an API’s documentation that “It’s bad, and here’s why, so don’t use it” to save users from “[having] to just learn” not to use it?
If not, what about (and I would prefer this latter) a machine-readable annotation in the source code that, in addition to affecting the documentation (example), makes the compiler (or interpreter) emit a warning if the API is used?
I don’t feel that this is something unique to “ultra-strong backwards-compatibility” languages, this happens to every language out there that is used by a big audience.
Go decided to include batteries, which over time will cause more dragons to pop up. But it’s not a net negative imo, as it also “strongarms” in certain idioms that would be much harder to convey with libraries on the side. For example, a thing that annoys me with OCaml is the LWT/Async bifurcation, and even if something subpar was shipped with OCaml, it would’ve made the concurrency story a bit more palatable for newcomers.
Also, I think deprecated functions and types is more of a documentation and tooling issue than anything else. Hide deprecated functions or put them in a separate section, and let linters or editors scream whenever you see on in the code.