I’ve come around on Nix and I think I like it now that I really understand it. I’m gonna do some more research and make a write up of the things that changed my mind about it.
And then a 39 MB docker image is emitted that is mostly the image assets for the application.
nix-shell and direnv combined with lorri also are shockingly useful. This lets me have each project use its own tooling (compilers, helper tools, etc) without having to pollute my global environment with them. Consider this shell.nix from a Rust project:
When combined with direnv-mode, this gives my emacs install access to the nix-provided rustc, rustfmt tool, rust language server and the like without having to download gigabytes of data into ~/.rustup (though if you really want rustup you can nix-env -i rustup).
The biggest problem this has is documentation and tooling. The command line tools make emerge and pacman look user-friendly. But overall I think it’s a tool that’s worth learning for its own merits. It could be better, but so could nearly all of the tools and the like we use on a regular basis.
that allow you to pull in dependencies from other places
You are probably well aware there are many tools that can do this. So my question is: why are you actually using Nix? Just to play and discover? :) I feel like this reason alone is not enough to justify it. Just curious about your thoughts.
I love this question! It’s one I actually ask myself when I come to frustration when using nixos (purely to learn and play). nix-shell -p nodejs is nice to quickly get a throwaway env but then I’m left thinking I did something the “wrong” way.
Yes but Nix allows you to do this for more than just Node. I’m currently using it for Rust, Go, Elm and Haskell. This integration means that I don’t have to install n tools for all n languages that I use in personal and professional projects.
How is this any different from using Docker? The only thing I can think of is duplication at the byte level, but this can be fixed with some deduplication-enabled file system?
Nix is actually less complicated than docker for my use cases. I have a docker image with my setup, but compare this:
$ cd ~
$ which cargo
<exit code 1 because cargo isn't on the path>
$ cd ~/code/Xe/withinbot
direnv: loading ~/code/Xe/withinbot/.envrc
$ which cargo
/nix/store/lm64cjp5c5kc772rpmf88q22s8lr2zhi-rustc-1.40.0/bin/cargo
And I can pin exact versions of things like nodejs via nix-shell invocations:
$ nix-shell -p nodejs-10_x
[nix-shell]$ which node
/nix/store/n3d6gbfbzbdw65smn5cj83s8rb1jwld8-nodejs-10.19.0/bin/node
$ nix-shell -p nodejs-12_x
[nix-shell]$ which node
/nix/store/5ca56xq1p92pcpmzrw3lwls0pnqg83dx-nodejs-12.15.0/bin/node
$ nix-shell -p nodejs-13_x
[nix-shell]$ which node
/nix/store/0fj8xvbdk7j79pyhif44xz2mkl6vyv0b-nodejs-13.8.0/bin/node
Or in a project-local shell.nix:
let
pkgs = import <nixpkgs> {};
in
pkgs.mkShell {
buildInputs = [
pkgs.nodejs-10_x # or pkgs.nodejs-12_x or pkgs.nodejs-13_x
];
}
$ nix-shell
[nix-shell]$ which node
/nix/store/n3d6gbfbzbdw65smn5cj83s8rb1jwld8-nodejs-10.19.0/bin/node
And then I never have to modify my global $PATH or deal with a version manager, because the package manager handles that for me.
Instead of pulling down a full operating system, with Nix you can just pull down the exact cached binaries and dependencies you need, which are stored and accessed in such a way that you know they won’t interfere with other applications using the same deps.
So Nix doesn’t actually handle any of the process isolation/network bridging/volume sharing stuff that docker does, it’s aimed at a much simpler layer.
Nix can be a really potent tool when combined with Docker, either through generating docker images or using Nix to install and configure packages inside of Docker.
While superficially they share some use cases (e.g. I want to run this application and have some other app take care of the details), they operate at very different layers with different ideals in mind.
Docker and Nix will without doubt, have a set of intersecting collections of logic.
The question is, which intersects the other more? :)
Here I think Docker can do all that Nix can, and more. It just operates at a higher level.
Pulling down an operating system is a detail, just like however Nix manages things.
As I said: don’t be surprised if we eventually see a Docker-based Nix-like solution written in Python or Bash. There are ways to make a system like this efficient.
This discussion was interesting to read. The argument for Nix seems like the same argument against node_modules – every use case gets covered if you just unpack an already correct state of affairs into your environment every time you need it (Docker), but arguably it would be way less resource-heavy to not create identical copies of common dependencies every time (Nix). Presumably there’s some middle ground between Docker’s simplicity and Nix’s resource efficiency but other than those factors I can’t see problems with either approach.
I’m aware I can pull in things with any choice of tools. I’m using Nix for at least because it lets me deduplicate the dependency builds across multiple projects. This also lets me avoid git submodules or similar (I personally see git submodules as more pain than they’re worth).
If anything, I’m using Nix for this because I can and you can’t stop me (muahahahahahahaha, etc). But really it’s to evaluate it as a building block for deploying my stuff into production as docker images. The docker images Nix makes are really tiny.
[X] parsing function headers
[X] code-generating primitive instructions
[X] function calls
[X] arguments
[X] return values
[X] local variables and their reclamation
[X] register locals and shadowing
[X] blocks, loops and early exits
unsafe language with user-defined types
[X] compound types: `addr` and `array`
[ ] `type` for user-defined product types
[ ] creating types on the stack
[ ] `index` for accessing inside array types
[ ] `get` for accessing inside product types
[ ] `handle` for heap allocations (fat pointers)
safe language
[ ] types in primitive instructions
[ ] types in function calls
[ ] register checks
[ ] types in `get` instructions
[ ] `ref` in caller can map to `addr` in callee
[ ] register lifetime checks in conditionals
[ ] register lifetime checks in loops
[ ] check for function result initialization
[ ] code-generate runtime checks for array bounds
[ ] code-generate runtime checks for heap lookups
[ ] checks for the restricted '*' operator
I’m implementing a runtime for a scripting language for a text editor I’ve been working on. It’s a project I started a long time ago and I dropped for ages but having spent a bit too much time recently writing Rust and C++ I’ve been itching to write some plain C89 again.
I’ve spent a bit of time designing it and now it’s just a lot of careful C89 programming: strings, hash tables, green threading, etc. It’s amazing how productive one can be when working on a totally new codebase entirely on one’s own, not working around other people, not fighting bad tools, etc.
Retargeting a C compiler for a chip that doesn’t exist yet and has no simulator. Also, the documentation for the compiler framework is 20 year old and everyone who knows how to use it has left the company.
2D C99 game engine, 2900 LOC now. I pummeled the (first of probably many) backend renderer rewrite in less than a day over the weekend. The stage is set for font rendering, another system I’ve failed multiple times in the past to write. Once more unto the breech.
This week I’m working on an automated Photo prompt application for myself. It’s going to be emailing me a bi-weekly prompt, as well as have a tracker feature as well for accountability!
I guess it’s really more of a service than an application (semantics).
Bi-weekly a lambda function will fire off. It’s going to connect to a db of “prompts” and pick one based on a few variables I define. Next it will grab some inspirational photos from some API using that prompt as the keyword (I was thinking Unsplash API here, potentially).
Then, it will take that prompt and those pictures, and email them to me.
Example would be Sunday night, lambda connects to db and gets the “architecture” prompt, queries Unsplash for 3 photos with the keyword “architecture”, then sends me “Prompt: Architecture \n {Unsplash Pictures + Credit}”.
Part 2 is some tool where I can link that prompt with the pictures I published, which is less mapped out right now.
The database tables are interesting. I have some ideas for some other applications that could utilize them. Like a “give me a prompt” kind of website which outputs the same data the email does.
Or sitting an API layer in front of the “tracker” tables that can facilitate an SSG Portfolio.
Compiling java lambdas is hard work. The JDK contains what one might describe as a JIT compiler for lambdas, that is, a seeparate JIT compiler that runs in addition to the JIT compiler most people use in the JVM, and there are n levels of indirection. I’m really quite amazed. Even many quite simple java programs a) parse bits of java source at startup and b) compile to bytecode at startup. No wonder java servers often take half a minute to start listening for requests.
I am implementing lambdas in my java compiler. Mentioned twice before here, chief feature drastically lower memory usage. Lambdas has taken >1 week so far and I’m not done.
At work I’m changing how our Android plugin uses KML, and also finishing up some changes to the script we use to configure new dev environments.
Outside of work i’ve been making a Common Lisp library for using the Spotify API.
I’m also still reading “The Design and Evolution of C++”. The second half deep dives into a bunch of (mostly obscure) language details and how/why they came to be. It’s interesting, but a little tedious, and slower to read than the first half.
I’m continuing to improve the “no-code” platform I am working on [1]. In particular I am working on a view editor that allows a user to create custom views over a data model. If I can keep the design simple enough I think the result will be enough power to be useful but not over-complicated.
I’ll try to work on my site. I want to have one, I’ve even tried many times, but it rarely got past the half-done phase. I want to change it.
I have a million other things and priorities, personally and at work, so this might fail, again. That’s why I’m posting it, to “commit myself”, to tell myself that I actually am doing it this time.
Well, I use wintersmith as mentioned. For me, this part is yet irrelevant, though. I first want to figure out what to put there, content-wise. Like, I know I want an about page and a now page.
I also want a sort of a microblog - just for regular posts and smaller, random thoughts - and a deep-dive subsection, where I explore concepts (sorta like long-post multi-part blogs). And a photo gallery to upload some of my photos. A back-to-the-roots, own-your-data, indieweb sort of thing.
So where do I start with all of that? How to make a “hello world, my online now lives here” appearance? How do I start adding content? When do I finally say “hey people I wrote something, take a look”? You know, not to end up like one of those blogs with 2 articles, a “hello world” and the “first in the series” that then die off.
I think I basically want to work out my sitemap this week.
I am new to Zig and I am not very experienced with system programming in general so take what I say with a grain of salt.
Even if Zig is not mature yet and far from fully documented, the language is very concise and opinionated which make it easy and straightforward to learn and understand, no magic.
And I just love the syntax.
You can also work with C librairies without wrappers: you just import your header files and it just works like it was Zig code in the first place.
Finally, I really like Andrew’s approach and dedication, he offers us a great piece of engineering !
My developer pre-release PinePhone arrived! Which means:
Hacking around with it to see if I can get audio in voice calls working in Ubuntu Touch. If so, that means I can switch to it as my daily driver (Audio, WiFi, video all work. Camera, Bluetooth, still non-functional but I can deal with that for a while).
Experimenting with an interactive Common Lisp development environment for writing mobile apps - SBCL, Emacs, SLIME, McCLIM. Largely empty repo here: https://gitlab.com/duncan-bayne/mobili.
(B|Ph)logging about the above :)
Tirelessly evangelising the PinePhone among my developer friends, as an alternative to walled gardens (iOS) and advertising platforms (Android).
Point 2 is really where it’s at. Create a super powerful development environment that allows devs to be far more productive than with other mobile dev tools, and point 4 becomes a lot easier :) “Hey, watch this …”
Finishing up a static site generator in Nix for my new website and then finally getting around to writing a blog post about Emacs I’ve been thinking about for a while.
Specifically what I’m trying to do is write a post that highlights to people that Emacs is not a text editor (though it has a text editor) and is actually a platform for implementing applications with text-based UIs. This is not to convince people to use Emacs, but to engage with it enough to understand the paradigm and maybe become interested enough to investigate how that paradigm can apply to other stuff they’re working on.
$WORK: I finished revising my paper and submitted it and it’s completely done! I now have about 6 or 7 weeks to make and prepare a presentation for the conference, and also polish up the code for release.
I’m getting some more work done on my other paper about graph rendering. For this week, my focus is to get an E2E MVP built and start evaluating it.
$FUN: My website is passably working now: personal page and blog, all with appropriate tags to each other that can be automatically parsed.
I spent most of the weekend playing around with IndieWeb concepts: MicroPub and MicroSub, specifically. I’m excited for them because I’m hoping to build a “personal indexer” of sorts. I read lots of articles online and I always forget what I’ve read and where. It’d be nice if my browser automatically saved the contents of pages I visit, saving them in either a plain text or microformat way. Further, if someone is interested in the things I read about (or more likely, if I’m interested in what others wrote), they could subscribe to my personal archive and query it too. I suspect that it’d be possible to replace a lot of my Google searches with this, without sending ever more data to Big Google all the time.
Regular work. I also enrolled in an online course on Artificial Intelligence in edX just to refresh my knowledge. So, I’ll try to go through the course materials after work and hopefully make a good progress this week.
I’ve come around on Nix and I think I like it now that I really understand it. I’m gonna do some more research and make a write up of the things that changed my mind about it.
Who are you and what have you done to the real @cadey?!
Seriously though, I hope you find the time for the write-up. I’ve been following the discussions around nix and friends with interest.
The real magic comes from tools like niv that allow you to pull in dependencies from other places. For example a tarot reading program in Elm that I still haven’t finished depends on gruvbox-css normally, and it also depends on quickserv when run in a docker image.
This lets me have a derivation that’s like:
And then a 39 MB docker image is emitted that is mostly the image assets for the application.
nix-shell and direnv combined with lorri also are shockingly useful. This lets me have each project use its own tooling (compilers, helper tools, etc) without having to pollute my global environment with them. Consider this shell.nix from a Rust project:
When combined with direnv-mode, this gives my emacs install access to the nix-provided rustc, rustfmt tool, rust language server and the like without having to download gigabytes of data into
~/.rustup(though if you really want rustup you cannix-env -i rustup).The biggest problem this has is documentation and tooling. The command line tools make emerge and pacman look user-friendly. But overall I think it’s a tool that’s worth learning for its own merits. It could be better, but so could nearly all of the tools and the like we use on a regular basis.
You are probably well aware there are many tools that can do this. So my question is: why are you actually using Nix? Just to play and discover? :) I feel like this reason alone is not enough to justify it. Just curious about your thoughts.
I love this question! It’s one I actually ask myself when I come to frustration when using nixos (purely to learn and play).
nix-shell -p nodejsis nice to quickly get a throwaway env but then I’m left thinking I did something the “wrong” way.For that, I just use
nvm(node version manager)Yes but Nix allows you to do this for more than just Node. I’m currently using it for Rust, Go, Elm and Haskell. This integration means that I don’t have to install
ntools for allnlanguages that I use in personal and professional projects.How is this any different from using Docker? The only thing I can think of is duplication at the byte level, but this can be fixed with some deduplication-enabled file system?
Nix is actually less complicated than docker for my use cases. I have a docker image with my setup, but compare this:
And I can pin exact versions of things like nodejs via
nix-shellinvocations:Or in a project-local
shell.nix:And then I never have to modify my global
$PATHor deal with a version manager, because the package manager handles that for me.Your first example: I’d just create a Docker image with that cargo version using rustup.
Your second example: Again just select any nodejs version and swap between Docker envs.
As it is right now, I bet someone could create Nix-like behavior with just some shell and Docker.
Instead of pulling down a full operating system, with Nix you can just pull down the exact cached binaries and dependencies you need, which are stored and accessed in such a way that you know they won’t interfere with other applications using the same deps.
So Nix doesn’t actually handle any of the process isolation/network bridging/volume sharing stuff that docker does, it’s aimed at a much simpler layer.
Nix can be a really potent tool when combined with Docker, either through generating docker images or using Nix to install and configure packages inside of Docker.
While superficially they share some use cases (e.g. I want to run this application and have some other app take care of the details), they operate at very different layers with different ideals in mind.
Superficially?
Programs are collections of logic.
Docker and Nix will without doubt, have a set of intersecting collections of logic.
The question is, which intersects the other more? :)
Here I think Docker can do all that Nix can, and more. It just operates at a higher level.
Pulling down an operating system is a detail, just like however Nix manages things.
As I said: don’t be surprised if we eventually see a Docker-based Nix-like solution written in Python or Bash. There are ways to make a system like this efficient.
Interesting to think about :) Thanks
This discussion was interesting to read. The argument for Nix seems like the same argument against node_modules – every use case gets covered if you just unpack an already correct state of affairs into your environment every time you need it (Docker), but arguably it would be way less resource-heavy to not create identical copies of common dependencies every time (Nix). Presumably there’s some middle ground between Docker’s simplicity and Nix’s resource efficiency but other than those factors I can’t see problems with either approach.
I’m aware I can pull in things with any choice of tools. I’m using Nix for at least because it lets me deduplicate the dependency builds across multiple projects. This also lets me avoid git submodules or similar (I personally see git submodules as more pain than they’re worth).
If anything, I’m using Nix for this because I can and you can’t stop me (muahahahahahahaha, etc). But really it’s to evaluate it as a building block for deploying my stuff into production as docker images. The docker images Nix makes are really tiny.
I’m continuing to work on my safe programming language implemented in machine code, that maps mostly 1:1 to machine code. The core language for just integer types is done, and I’m now going to start working on user-defined types. Here’s what my todo list looks like:
unsafe language with just int types
unsafe language with user-defined types
safe language
Compare 5 weeks ago.
This is sick man. Please please continue on :) Higher order, typed assembly is something I’ve been waiting for someone to do for awhile :D
Back to building my first WebSocket API in Common Lisp.
Because this is CL, and I know you, are you
This is work, so it rules out #2!
Really just making an API for a service our system provides, in CL. Using Hunchensocket for now to handle the hard parts.
I’m trying to finish off a NumPy to OpenCL JIT compiler. I’ve dug myself into a hole, and I’m still digging
I’m implementing a runtime for a scripting language for a text editor I’ve been working on. It’s a project I started a long time ago and I dropped for ages but having spent a bit too much time recently writing Rust and C++ I’ve been itching to write some plain C89 again.
I’ve spent a bit of time designing it and now it’s just a lot of careful C89 programming: strings, hash tables, green threading, etc. It’s amazing how productive one can be when working on a totally new codebase entirely on one’s own, not working around other people, not fighting bad tools, etc.
Finishing a C++ OS installation system. At least, hopefully finishing it.
Retargeting a C compiler for a chip that doesn’t exist yet and has no simulator. Also, the documentation for the compiler framework is 20 year old and everyone who knows how to use it has left the company.
2D C99 game engine, 2900 LOC now. I pummeled the (first of probably many) backend renderer rewrite in less than a day over the weekend. The stage is set for font rendering, another system I’ve failed multiple times in the past to write. Once more unto the breech.
I am going to take a day off mid-week to recover from an overload of things.
There’s an In/Clojure 2020 (at Pune, India) Clojure Workshop on Friday where I am TA’ing, and the conference on Saturday.
Continuing reading through Designing Data Intensive Applications. It links to so many papers! 😱🤓
This week I’m working on an automated Photo prompt application for myself. It’s going to be emailing me a bi-weekly prompt, as well as have a tracker feature as well for accountability!
Wait, what? What kind of a photo prompt application? What will it be doing?
I guess it’s really more of a service than an application (semantics).
Bi-weekly a lambda function will fire off. It’s going to connect to a db of “prompts” and pick one based on a few variables I define. Next it will grab some inspirational photos from some API using that prompt as the keyword (I was thinking Unsplash API here, potentially).
Then, it will take that prompt and those pictures, and email them to me.
Example would be Sunday night, lambda connects to db and gets the “architecture” prompt, queries Unsplash for 3 photos with the keyword “architecture”, then sends me “Prompt: Architecture \n {Unsplash Pictures + Credit}”.
Part 2 is some tool where I can link that prompt with the pictures I published, which is less mapped out right now.
Cool, sounds like an elaborate attempt to do the job of making yourself doing more photos :)
That’s exactly what it is.
The database tables are interesting. I have some ideas for some other applications that could utilize them. Like a “give me a prompt” kind of website which outputs the same data the email does.
Or sitting an API layer in front of the “tracker” tables that can facilitate an SSG Portfolio.
Compiling java lambdas is hard work. The JDK contains what one might describe as a JIT compiler for lambdas, that is, a seeparate JIT compiler that runs in addition to the JIT compiler most people use in the JVM, and there are n levels of indirection. I’m really quite amazed. Even many quite simple java programs a) parse bits of java source at startup and b) compile to bytecode at startup. No wonder java servers often take half a minute to start listening for requests.
I am implementing lambdas in my java compiler. Mentioned twice before here, chief feature drastically lower memory usage. Lambdas has taken >1 week so far and I’m not done.
At work I’m changing how our Android plugin uses KML, and also finishing up some changes to the script we use to configure new dev environments.
Outside of work i’ve been making a Common Lisp library for using the Spotify API.
I’m also still reading “The Design and Evolution of C++”. The second half deep dives into a bunch of (mostly obscure) language details and how/why they came to be. It’s interesting, but a little tedious, and slower to read than the first half.
I’m continuing to improve the “no-code” platform I am working on [1]. In particular I am working on a view editor that allows a user to create custom views over a data model. If I can keep the design simple enough I think the result will be enough power to be useful but not over-complicated.
I hope everyone here has a great week!
[1] https://www.webase.com
I’ll try to work on my site. I want to have one, I’ve even tried many times, but it rarely got past the half-done phase. I want to change it.
I have a million other things and priorities, personally and at work, so this might fail, again. That’s why I’m posting it, to “commit myself”, to tell myself that I actually am doing it this time.
Wanna elaborate on what you are using to create your website?
Well, I use wintersmith as mentioned. For me, this part is yet irrelevant, though. I first want to figure out what to put there, content-wise. Like, I know I want an about page and a now page.
I also want a sort of a microblog - just for regular posts and smaller, random thoughts - and a deep-dive subsection, where I explore concepts (sorta like long-post multi-part blogs). And a photo gallery to upload some of my photos. A back-to-the-roots, own-your-data, indieweb sort of thing.
So where do I start with all of that? How to make a “hello world, my online now lives here” appearance? How do I start adding content? When do I finally say “hey people I wrote something, take a look”? You know, not to end up like one of those blogs with 2 articles, a “hello world” and the “first in the series” that then die off.
I think I basically want to work out my sitemap this week.
Are you crafting it by hand or using something like Wordpress or Hugo?
wintersmith. It’s similar, just JavaScript, as JavaScript is my hammer.
End of holidays, so I am getting back to work.
At home I will stop to code in Nim and take some time to learn Zig by coding a small game with SDL2 which is one of my month goal :)
I’ve heard of Zig, but don’t know much about it. What do you like about it?
Hey Johnblood,
I am new to Zig and I am not very experienced with system programming in general so take what I say with a grain of salt.
Even if Zig is not mature yet and far from fully documented, the language is very concise and opinionated which make it easy and straightforward to learn and understand, no magic. And I just love the syntax.
You can also work with C librairies without wrappers: you just import your header files and it just works like it was Zig code in the first place.
Finally, I really like Andrew’s approach and dedication, he offers us a great piece of engineering !
Ps: Road to Zig 1.0 give you a nice overview of Zig.
My developer pre-release PinePhone arrived! Which means:
Point 2 is really where it’s at. Create a super powerful development environment that allows devs to be far more productive than with other mobile dev tools, and point 4 becomes a lot easier :) “Hey, watch this …”
Finishing up a static site generator in Nix for my new website and then finally getting around to writing a blog post about Emacs I’ve been thinking about for a while.
Specifically what I’m trying to do is write a post that highlights to people that Emacs is not a text editor (though it has a text editor) and is actually a platform for implementing applications with text-based UIs. This is not to convince people to use Emacs, but to engage with it enough to understand the paradigm and maybe become interested enough to investigate how that paradigm can apply to other stuff they’re working on.
Your Emacs article sounds interesting. Looking forward to it.
(2 weeks ago: https://lobste.rs/s/gmbx5t/what_are_you_doing_this_week#c_s0pk5o)
$WORK: I finished revising my paper and submitted it and it’s completely done! I now have about 6 or 7 weeks to make and prepare a presentation for the conference, and also polish up the code for release.
I’m getting some more work done on my other paper about graph rendering. For this week, my focus is to get an E2E MVP built and start evaluating it.
$FUN: My website is passably working now: personal page and blog, all with appropriate tags to each other that can be automatically parsed.
I spent most of the weekend playing around with IndieWeb concepts: MicroPub and MicroSub, specifically. I’m excited for them because I’m hoping to build a “personal indexer” of sorts. I read lots of articles online and I always forget what I’ve read and where. It’d be nice if my browser automatically saved the contents of pages I visit, saving them in either a plain text or microformat way. Further, if someone is interested in the things I read about (or more likely, if I’m interested in what others wrote), they could subscribe to my personal archive and query it too. I suspect that it’d be possible to replace a lot of my Google searches with this, without sending ever more data to Big Google all the time.
Regular work. I also enrolled in an online course on Artificial Intelligence in edX just to refresh my knowledge. So, I’ll try to go through the course materials after work and hopefully make a good progress this week.
Hopefully, learn how to make pixel art and how to program. For those interested, I’m learning YAB for programming on Haiku.
Home: