Thanks for this summary! Despite doing a bunch of D3 ages ago I’ve barely used Observable – definitely the proprietary platform was a disincentive. It will be interesting to see what happens with Quarto (which I have been using a bit recently), since there’s a ton of overlap here and Quarto already supports Observable (1.0). I did just find an HN comment where mbostock says he sees ”[Observable] Framework more as an alternative to Quarto than a complement.” Great to have new tools in the space, regardless.
My comment from HN, possibly of interest to this community too:
I recently finished my first “in anger” project with an Observable notebook. This involved learning Observable Plot, Arquero, relearning bits of Javascript, and integrating it with a Rust based simulator that’s my data generation process.
It’s honestly been really wonderful. Learning all of those tools has taken some significant energy, and I’m missing some functionality I’d love around parameterizing my data generator, but the final notebook is beautiful and functional.
Using markdown and reactivity makes notebooks like this actually feel usable. Jupyter’s custom format made version control a giant pain and without reactivity your iteratively designed notebook easily becomes a write-only, stateful mess. I’ve also tried making this work using Quarto and their Observable integration and it was hacky and piecemeal.
Genuinely, this was the first time I’ve been pleasantly surprised and excited to write a notebook and share it with others. I’m sure there will be more sharp edges, but it’s become my first choice notebook tool after this project.
And I feel like this will lead to the classic situation of “every time I go to write a blog post, I have to upgrade the Ruby deps, or the Hugo version broke something”
I actually haven’t used npm in like 5+ years. Nothing I do relies on it
Does anyone want to say I’m irrational for avoiding NPM? That it is very stable if you follow some rules?
This seems like a very significant piece of software, and it’s going to have dependencies, of course … but I have bad experiences with NPM
And also I have a setup where all my web content builds fine with basically zero maintenance for the last 7+ years.
(Though to be honest I do have a bunch of package manager headaches with R, which is why only 1% of my web content uses it . Seems like every time I go to use R, something has changed in a breaking way. But I still use it because it is very capable and worth it.)
I’m completely with you on resistance to NPM stuff, for the same reason: I always expect my projects to be hard to run in the future as dependencies shift from under me.
In this particular case I’m willing to take the risk, mainly because I have deep trust in the Observable team to keep everything working - and because the capabilities I get from the software are worth it.
I suggest trying the tutorial and seing how it feels - I was pleasantly surprised at how little friction there was in getting it all working.
Package management for reproducible work in Node/JS is much better than in R because the default setup with node is an isolated environment per project and lock files. You don’t get either of those ergonomically in R (as of ~5 years ago there was only packrat, which was better than nothing but awkward and extremely slow).
Hm yeah lockfiles are definitely better. Do the lockfiles ensure that a project can be buildable “forever”?
I think the usual exception to that is C library changes – say libc changes or OpenSSL changes.
Basically no language package manager has solved that problem – they are disconnected from the OS package manager. So if something changes at the C level, you are forced to update.
But C libraries are usually fairly stable – Other than that, I wonder what the sources of breakages are if you use lock files? It does seem like that should eliminate a lot of instability
A common one in the R ecosystem is that packages put in constraints on the R version, and dependent packages can do that too. They will release a new version of a package, update a constraint for R > 3.5, and my distro has R 3.4, and now the package doesn’t work.
I have to search for a binary of a new version of R not supported by the distro.
But lockfiles should solve that problem. Because then you don’t consult the constraints every time.
Do you install node/npm from the distro or by downloading binaries?
Do the lockfiles ensure that a project can be buildable “forever”?
Yes, as long as a package you depend on hasn’t been unpublished. NPM-the-organization has gotten better about preventing that (for non-malicious packages) since left-pad, but I believe there are still some ways it can happen.
Do you install node/npm from the distro or by downloading binaries?
Always install using the official binaries (or using NVM, which will download the official binaries for you, and allow you to run multiple versions of Node side-by-side). Distro-provided builds of node are not maintained by the Node developers, often out of date, and as as you experienced, often distributed without NPM.
Depending on the system libraries is a bad idea for reproducible code. For any library that isn’t extremely stable you’re better off using e.g. an npm package that vendors and builds a particular version of the library.
Otherwise you need to bundle the whole project in a container or use Nix or whatever.
Same for your language toolchain. Install a particular version with mise/asdf or similar. Depending on “whatever Debian was shipping at the time” is not a rational or sensible thing to be doing for any project that you want to be reproducible or low maintenance. You could probably get away with it if you’re writing in some very stable language and don’t have strict reproducibility requirements, but why risk it when it is so easy not to?
Graphics and GUI libraries often need to link with the system
And there are almost always transitive dependencies with the system. Probably every piece of C code calls libc, even if you vendor it. libc can’t be vendored in general, on OS X or BSDs or even Linux.
I don’t know about graphics or GUI libs needing to link with the system. I imagine the actually required interface can be small, and from a practical point of view you can still target a fixed API version of e.g. SDL.
Thanks for this summary! Despite doing a bunch of D3 ages ago I’ve barely used Observable – definitely the proprietary platform was a disincentive. It will be interesting to see what happens with Quarto (which I have been using a bit recently), since there’s a ton of overlap here and Quarto already supports Observable (1.0). I did just find an HN comment where mbostock says he sees ”[Observable] Framework more as an alternative to Quarto than a complement.” Great to have new tools in the space, regardless.
My comment from HN, possibly of interest to this community too:
I recently finished my first “in anger” project with an Observable notebook. This involved learning Observable Plot, Arquero, relearning bits of Javascript, and integrating it with a Rust based simulator that’s my data generation process.
It’s honestly been really wonderful. Learning all of those tools has taken some significant energy, and I’m missing some functionality I’d love around parameterizing my data generator, but the final notebook is beautiful and functional.
Using markdown and reactivity makes notebooks like this actually feel usable. Jupyter’s custom format made version control a giant pain and without reactivity your iteratively designed notebook easily becomes a write-only, stateful mess. I’ve also tried making this work using Quarto and their Observable integration and it was hacky and piecemeal.
Genuinely, this was the first time I’ve been pleasantly surprised and excited to write a notebook and share it with others. I’m sure there will be more sharp edges, but it’s become my first choice notebook tool after this project.
Hm, I really want to use this, because right now I use Make/Ninja files and Python/R for similar things
But having some kind of GUI with dataflow/caching is nicer for developing, and interactivity can help visualizations.
And now they actually support arbitrary Unix commands! woohoo
But then I see
And I look at the deps here
https://github.com/observablehq/framework/blob/main/package.json
And I feel like this will lead to the classic situation of “every time I go to write a blog post, I have to upgrade the Ruby deps, or the Hugo version broke something”
I actually haven’t used npm in like 5+ years. Nothing I do relies on it
Does anyone want to say I’m irrational for avoiding NPM? That it is very stable if you follow some rules?
This seems like a very significant piece of software, and it’s going to have dependencies, of course … but I have bad experiences with NPM
And also I have a setup where all my web content builds fine with basically zero maintenance for the last 7+ years.
(Though to be honest I do have a bunch of package manager headaches with R, which is why only 1% of my web content uses it . Seems like every time I go to use R, something has changed in a breaking way. But I still use it because it is very capable and worth it.)
I’m completely with you on resistance to NPM stuff, for the same reason: I always expect my projects to be hard to run in the future as dependencies shift from under me.
In this particular case I’m willing to take the risk, mainly because I have deep trust in the Observable team to keep everything working - and because the capabilities I get from the software are worth it.
I suggest trying the tutorial and seing how it feels - I was pleasantly surprised at how little friction there was in getting it all working.
Package management for reproducible work in Node/JS is much better than in R because the default setup with node is an isolated environment per project and lock files. You don’t get either of those ergonomically in R (as of ~5 years ago there was only packrat, which was better than nothing but awkward and extremely slow).
Hm yeah lockfiles are definitely better. Do the lockfiles ensure that a project can be buildable “forever”?
I think the usual exception to that is C library changes – say libc changes or OpenSSL changes.
Basically no language package manager has solved that problem – they are disconnected from the OS package manager. So if something changes at the C level, you are forced to update.
But C libraries are usually fairly stable – Other than that, I wonder what the sources of breakages are if you use lock files? It does seem like that should eliminate a lot of instability
A common one in the R ecosystem is that packages put in constraints on the R version, and dependent packages can do that too. They will release a new version of a package, update a constraint for R > 3.5, and my distro has R 3.4, and now the package doesn’t work.
I have to search for a binary of a new version of R not supported by the distro.
But lockfiles should solve that problem. Because then you don’t consult the constraints every time.
Do you install node/npm from the distro or by downloading binaries?
It does seem like if you lock a particular binary here - https://nodejs.org/en/download/current
and if npm is builtin, and you use lockfiles, then it should be pretty stable.
But actually it seems like npm is a separate thing - https://docs.npmjs.com/downloading-and-installing-node-js-and-npm
Right now on my machine I have nodejs from Debian apt-get, but not npm.
Yes, as long as a package you depend on hasn’t been unpublished. NPM-the-organization has gotten better about preventing that (for non-malicious packages) since left-pad, but I believe there are still some ways it can happen.
Always install using the official binaries (or using NVM, which will download the official binaries for you, and allow you to run multiple versions of Node side-by-side). Distro-provided builds of node are not maintained by the Node developers, often out of date, and as as you experienced, often distributed without NPM.
Depending on the system libraries is a bad idea for reproducible code. For any library that isn’t extremely stable you’re better off using e.g. an npm package that vendors and builds a particular version of the library.
Otherwise you need to bundle the whole project in a container or use Nix or whatever.
Same for your language toolchain. Install a particular version with mise/asdf or similar. Depending on “whatever Debian was shipping at the time” is not a rational or sensible thing to be doing for any project that you want to be reproducible or low maintenance. You could probably get away with it if you’re writing in some very stable language and don’t have strict reproducibility requirements, but why risk it when it is so easy not to?
Graphics and GUI libraries often need to link with the system
And there are almost always transitive dependencies with the system. Probably every piece of C code calls libc, even if you vendor it. libc can’t be vendored in general, on OS X or BSDs or even Linux.
Libc is stable enough that it’s okay to link it, especially if you link with old headers.
See here for more: https://docs.binarybuilder.org/stable/
I don’t know about graphics or GUI libs needing to link with the system. I imagine the actually required interface can be small, and from a practical point of view you can still target a fixed API version of e.g. SDL.
Again, transitive dependencies. SDL can’t do anything interesting without the system. The kernel has a monopoly on I/O
But this is besides the point, my questions about npm are basically answered
I think you’re just skipping the part where I talked about libraries being stable or not.
It’s generally fine to depend on the system’s kernel abi, glibc or sdl2 because those are obsessively backwards compatible.