I’ve been toying with encoding an ast as a database (prolog currently) and reading this is very inspiring. I think a lot of problems become a lot simpler when your code is stored in an easily queryable (with metadata) format, type checking & linting becomes significantly simpler for example.
But i can also imagine editors & plugin systems which could for example load sql db tables into the code database & trivialize safe sql dsls.
I read this Joe Armstrong post around 12 years ago when it was first published and I thought yes, definitely and I’m very happy that so many people are working on systems which do this. Thank you!
Yes, that’s how git operates: every git repository is a content-addressable store.
Note that, unlike Unison, I intend to identify equivalent functions and deduplicate them as well; when a programmer asks for an older and less efficient version of a function, they should automatically be given the latest and most efficient version with equivalent behavior.
I thought git used content + commit message + parent Sha + date + author email, which looks too much for your goal?
Also, are you serializing the AST of each function in a separate file?
The encoding is deeper than that. I’ve started documenting it here, and I’ll submit this as a top-level post once I’ve finished documenting what I’ve built.
This seems like a great talk for beginners. My feedback is not aimed at beginners, but at Nix evangelists.
Because there were no other options at the time, Nix uses its own programming language also named Nix to define packages.
This doesn’t quite sound right to me. While Eelco’s thesis doesn’t directly refute this claim (say, on p69), it does explain that the Nix expression language is the simplest possible language which has all of the necessary features, and it also explains why each individual feature (laziness, recursive definitions, attrsets, functions, etc.) are necessary for defining packages. The thesis also explains why languages like Make are unsuited to the task.
A NixOS module is a function that takes the current state of the world and returns things to change in it.
This is perhaps too imperative to be true. NixOS configuration is generally monoidal, and often commutative (or configuring underlying systems which commute, e.g. networking.firewall.allowedTCPPorts which is sorted by the iptables subsystem.) It’s important to not give folks the false impression that adding a NixOS module is like adding a SysV init script.
yeah a lot of good language design is saying no to features or syntactic decisions because of concerns that seem very abstract, because past experience has shown they have far-reaching negative impact
with is in that category and should be removed, imo. I find it super convenient but that does not outweigh the loss of referential transparency. there have been a few proposed replacements, some of which are kind of okay although nothing perfect, but at the end of the day it’s a damaging feature and even just removing it entirely ought to be on the table
it’s hard to do that when there’s existing code, but it’s possible, and it does pay dividends down the line
Right, I mean, I’m a known nix partisan as it were, but in my view I’d say that there is no better expression language for the thing nix is doing today, and there also wasn’t at the time it was new. :)
Sure, but these are written once somewhere deep inside lib, and then used declaratively. You won’t really be writing functions as you would in a regular PL, and I would argue that most package descriptions can actually be read by humans if you parse them more or less as you would a JSON, which is the important mental model.
Nix seems like a worthwhile investment to learn. Though (as mentioned with the comic sans slide) the ecosystem is somewhat confusing.
Among the things that add a barrier to entry is needing to learn yet another custom configuration language (funny that the post starts off by complaining that NGINX has one). I’m wondering if a wrapper in a lang I might already know, like Ruby or Rust, might ease that one small burden so I could focus on the outcome and not the syntax.
If you already know a programming language with some functional elements (Rust definitely fits that bill), I’d say Nix the language can be learned in an hour by taking the time to read Nix pills 4 and 5.
The language is definitely quirky, but it is also extremely small. Warts are an annoyance but not a complete blocker.
Then however you have to find out what all the functions in the lib thingy do (I think it’s provided by nixpgs and not Nix the language but I am still not sure), and this is IMO the big barrier to entry.
funny that the post starts off by complaining that NGINX has one
NixOS is surprisingly old: Wikipedia tells me that the initial release of NixOS was in 2003, while nginx was in 2004. Maybe nginx should have used nix the language for configuration from the beginning :P
That’s correct, lib is part of nixpkgs, not part of the language. Strictly speaking there’s one lib that’s used when writing packages and another that’s used when writing modules, but you’ll rarely need to worry about that distinction.
I really want guix to succeed, but fear it’s lack of network effect will continue to hamper it. nixpkgs just has so many more packages and people involved. Sadly, guix will continue to be even more niche than the niche nix as long as it’s tied to GNU. In the real world, purity doesn’t really work for operating systems any more than programming languages.
Eh, “pure” is a very overloaded term that has no meaning to the general computing public. I think it’s safer to say that they are more strictly focused on being repeatable (same inputs -> same outputs).
Yep. Only that one simple thing with nix, that I can just replicate this exact environment perfectly in another machine every single time is, for me, so valuable I kind of don’t mind some of the quirks in nix/nixpkgs.
I think you two interpret “pure” differently: Guix is pure in the FSF/GNU sense: free of proprietary software in its repositories, to the point where it is using linux-libre (a kernel stripped of binary blobs). This has the major disadvantage of not working very well on a lot of hardware that does require those binary blobs.
nixpkgs has plenty of unfree software in it, and even though you can’t install them without going through additional hoops, they’re there, thus, it’s not pure in the FSF/GNU sense. Like, GNU considers the linux kernel impure to begin with, and you can install that from nixos without any additional hoops.
Well, Nix is simply a decade older than Guix, so it’s understandable. For my personal use, I’ve found https://toys.whereis.みんな/ to be a nice package search across many channels, and you can add any channels you want as and when you find something that’s not packaged in the main repository. As for the last sentence, Nix is a pure programming language ;)
Scheme is just not doing it for me. I just had a look at how to create something nix-shell like in GUIX, and found this reference. Unfortunately the examples (see the --expression=expr section) is exactly the sort of Perl-style line noise I can’t abide. Like, I don’t want to have to remember which complex LISP thing @ or % stand for. What’s wrong with words?
Hmm, I have to say I vehemently disagree! I’m sorry if what follows feels like flamebait, tone is hard to convey over text and I just want to address some issues with what you said. You are of course entitled to any opinions you hold. The funny thing about what you said, that % is arcane, is that %has no special meaning, that’s just part of the variable name! @ is just splat, like in javascript or python, and neither of those are words either (whereas in scheme you can express it as a word, but using @ is shorter and spending a tiny amount of time is enough to understand that it’s shorthand). The examples are extremely clear and there is a 1-1 mapping between nix-shell and guix shell until you get to the one about --expression, which you would only use if you actually want to evaluate code to do something, which you can’t do with nix-shell.
Of course someone used to Scheme would disagree ;) I tried learning Scheme many years ago, and just got the worst cognitive dissonance ever from every single token being abbreviated to death, and from using obtuse technical jargon like car and cdr instead of short, simple, everyday synonyms like first and second, or even initial and subsequent. LISP clones have a massive usability issue in that it’s not just a new programming language, but an entirely new set of words which other programming languages have moved away from.
I would love to learn a LISP clone which chose usability over trying to look like 1960s code.
Wouldn’t a Nix vs. Guix syntax debate just go back to the split between ML & LISP? That’s been one of the main syntax dividing lines in FP for decades. Having used many MLs (PureScript, Elm, OCaml, Haskell) Nix feels cozy (albeit quirky) & my limited time with Clojure(Script) or Chicken Scheme for Tree-sitter feels alien & unergonomic.
Yeah, Flakes are very heavily used - that would still be the case even if every single person involved in Nix said “no, don’t use them, they’re not ready!”
The Flake design attempted to solve a fundamental usability and caching problem and it doesn’t do that perfectly, but it’s such an important problem that any partial solution is going to be popular.
I’ve added a precision footnote (instead of directly editing).
By done, I wanted to mean released, or available independently without feature flags, and feature complete. It is not meant to imply that every feature is incomplete or incorrect.
By that, I meant the “unopinionatable” meaning of released.
Which has a mainline QMK firmware implementation available. I could plug it into a Windows PC and also have mouse keys work there. The great thing is that I’ve programmed the various knobs up the top to act as scroll wheels.
Looking at the datasheet, it lists the operating temperatures for “charge” to be 0-50C, and “discharge” -20-+60 C, which makes me think the “Stable up to +130C” means “will not catch fire and burn your house down until the external temperature is 130C”, not “is perfectly fine and happy up to 130C”.
So… yeah. You’re probably fine. I would personally keep a good bucket of sand or an appropriate fire extinguisher nearby though, and possibly put some thought into an arrangement that lets it stay cool and have plenty of space for radiating heat. …maybe in a metal bucket or something. I’ve seen enough go bad that every lithium battery has to be treated as potentially hazardous.
That said, this looks straightforward enough that it is making me consider resurrecting my own failed attempt to do something similar…
Thanks, I’ll keep that in mind. It is on a metal balcony, so it should be okay, but still, the fire would be quite an unpleasant experience. So far, the simplest solution for higher temperature was to put the box more under the solar panel, so it is shielded from the sun. I found out, that the temperature dropped immediately into more comfortable numbers. In the long term, I definitely plan to build some inflammable enclosure, as mentioned in the article.
This reads like you’ve just did a Google about the topic and you’re now trying to give advice on it. I imagine the author knows a little bit about the topic.
The primary safety concern for most environments with lithium batteries is thermal runaway, caused by things like overcharging and short-circuiting. LiFePO4 batteries use non-flammable electrolytes, unlike other chemistries, which means that thermal runaway should only happen if it reaches a high temperature, when things turn into gases. These electrolytes can become flammable by introducing external things (e.g. water) but internally they should remain non-flammable.
In the video, there were multiple punctures to the battery over time. It looks like the second puncture created some sort of spark and ignited the vapours coming from the electrolyte. This was not a runaway event which other chemistries are capable of. For every video like this, there are probably 10 of people puncturing, drilling, dropping, etc. without any flames - you need a bit of luck to get vapours, sparks and oxygen in the right places.
So:
You can mistreat a LiFePO4 battery in terms of charging, discharging and short circuiting and not create flames (awesome!)
I don’t know anything about LiFePO4 batteries specifically but I’ve had training about handling lithium batteries in general at two different jobs, and had to put out a large fire once. It was enough to scare the shit out of me. I don’t care what the label says, treat any lithium battery the same way you would treat a none-too-durable jar of gasoline of equivalent size.
They were absolutely right to teach you to treat any lithium batter like a jar of gasoline and that 100% applies to LiFePO4 batteries, too. The details are in the numbers, as per Akin’s first law :-) and in that regard the author is right to be more confident in LiFePO4 batteries than in other types of Li-ion batteries.
Anything, including water, will trigger a fire if you pass a high enough current through it. LiFePO4 batteries doubly so because a high enough current will break the P-O bonds in the PO4 3- ions of the cathode and release oxygen.
That being said, LiFePO4 really are safer than other Li-ion batteries in widespread use in this regard. There’s literature that supports this, including flammability characteristics (see e.g. here, here, or here, or here – see that website with papers for the full text of some of those, you know which one :-P). There are some good reasons for that. For instance, the P-O bond is pretty strong. It takes more energy to break it than e.g. the Co-O bond in LCO batteries; this, along with other characteristics of LFP batteries, really does make thermal runaway harder to trigger (see here for some data). The flammability characteristics are also a lot milder, it’s… kind of like a smaller jar of gasoline.
Some of the chemistry behind it is way over my head (EE, not chem, so as soon as I get into anything that discusses the inner Helmholtz plane and crystaline lattices and whatever I rever to I viscerally understand some of those words via the nightmares they evoke from my sophomore year) but the 130C figure looks about right. As you point out it definitely does not mean it’s OK to use it at that temperature (I don’t recall the exact details but IIRC operating LFP batteries at higher temperatures than the indicated discharge range reduces their life span more steeply than for other types). The safe discharge range is up to 60 C, 130 C is the temperature up to which the manufacturer claims you won’t trigger a thermal runaway.
I’m not sure about the exact design implications here. When I learned about them in uni, LFP batteries were still kind of in their infancy (people were struggling to work around their really poor native conductivity figures) and/or patent-encumbered to hell, and by the time they were in active used I hadn’t actually done any real EE work in years. So I can’t comment on whether the author uses them correctly or not or how safe this particular application is, but in general it does make sense to treat them as safer than other Li-ion battery types.
Most lithium battery chemistries are worse than gasoline in many cases due to thermal runaway. But LiFePO4 is pretty different - needs a lot more than a spark.
This is a good idea in general, but I would be careful about just stopping the test once branch coverage is reached. The major benefit of property tests is that they check for more input data combinations than other testing approaches, and “correctness” means that a program works for all input data combinations.
By stopping once branch coverage is met, you miss out on additional data states that might find a bug.
I imagine in a lot of cases, this would result in more tests being run, not less. If you have a branch which only triggers for 1 value in a huge input space and you run 100 property tests by default, you’re not going to hit that branch most of the time. For example, a lot of property tests rely on many runs over time to eventually find failures.
I have a big pile of thoughts on this, maybe too much to fit into a lobsters comment.
This testing strategy is fine for what you get from property testing. Dan Luu’s excellent post on testing has a quote:
The QuickCheck paper mentions that it uses random testing because it’s nearly as good as partition testing and much easier to implement.
QuickCheck by itself is a sweet spot on the axes of “easy to implement” and “finds bugs”. The first version of QuickCheck was tiny, maybe two pages of Haskell?
Hooking that into code coverage is another sweet spot on those axes, my entire implementation supporting three property test frameworks is 76 lines of code.
Property tests are ‘spot checks’ where you might get a lucky hit on an input that finds a counterexample.
Another sweet spot I’ve considered is making sure that new generated inputs do not duplicate previously used inputs for this run, but I haven’t gotten there yet.
If you want to cover hard to reach parts of the code, I have a heavy duty solution in trynocular. In trynocular, the generated random input is wrapped in a “spy” that watches what parts are used, so you can discover how to increase code coverage. Describing that will be a much longer blog post. (But golly it’s gonna be fun)
… a sweet spot on the axes of “easy to implement” and “finds bugs”
I can definitely get behind this, because it’s an economical argument - running property tests for longer may find additional bugs, but the ROI is the number of bugs found per time period invested. And I think everyone agrees that there’s diminishing returns for longer time periods of random testing.
I still prefer to separate these “spot checks” from deep correctness checks, because they serve different and conflicting purposes. And there’s one additional cost I’d like to throw into the mix: the scaling cost, meaning the cost to scale the same test to a higher level of correctness, which here basically means running the test for longer. The scaling cost of property tests is pretty much zero: just run the test for longer. Property tests are also embarrassingly parallelizeable, since two tests processes require no coordination and we can just run more of them in parallel.
So my ideal strategy is to use “spot checks” before merge / release, and jack up the test durations after release to still try and find bugs before a user does.
If you want to cover hard to reach parts of the code….
Your trynocular tool looks really cool. I keep saying that we should really separate property testing from random data generation, since randomness is just one strategy. The idea of more targeted data generation strategies is a really interesting research area, and I support any ideas down that path. After all, the holy grail of testing is to find the minimum number of test cases that imply correctness.
I imagine in a lot of cases, this would result in more tests being run, not less
That’s right, but the set of values that you need to check for correctness is a superset of the values you need to check for branch coverage, so you need more tests to get there.
If you have a branch which only triggers for 1 value in a huge input space…
This is also my point. If we do what’s proposed here, the 1 value won’t get hit very often, so the branch also won’t get hit, so we’ll stop the test before the branch can be taken.
It’s still good to know what the branch coverage is during a property-based test, because more branch coverage is certainly better. I just would’t use it to stop looking for inputs.
I wrote a longer reply further down in this thread, but in short, I see property tests as spot checks. What you describe sounds more heavy duty. Do you have a way to measure data combinations? Perhaps trynocular does what you want?
I use NewPipe every day, have an adblocker on my computer and phone and am absolutely shocked when I watch YouTube at other places and see how many ads they are pushing. Even more shocking is how people have gotten used to it.
Of course Alphabet/YouTube has to finance itself somehow, but 11,99€/month for YouTube Premium is definitely overprized. If you consider that YouTube only makes a fraction of a cent per ad-view per person, they could probably be profitable with 0,99€/month, and I would be willing to pay that.
This would not resolve one other major issue I have with YouTube: The website is bloated as hell. NewPipe is smooth and native, runs on F-Droid and allows easy downloading (audio or video).
absolutely shocked when I watch YouTube at other places and see how many ads they are pushing
…
11,99€/month for YouTube Premium is definitely overprized.
Are you sure? ;-P
So, I pay for a family premium account, primarily to disable ads on stuff my kids watch. I have several motivations:
I want them to grow up assuming that it’s reasonable to pay for content (albeit indirectly in this case).
Ad content is so often mental poison.
I want them to grow up unused to omnipresent advertising. It should stand out, and feel intrusive and horrible.
Now that they’re starting to find content they really like (Explosions and Fire is great for kids, if a bit sweary) I’m going to fund them to choose their favourite creator each, via Patreon or similar.
Edited to add: I also run ad-blocking, at a network level. I have no qualms at all about ad-blocking in general, and think that ad-tech is a blight on the world. Please don’t mistake my willingness to pay for YouTube Red (or whatever it is called) as a criticism of adblocking.
Intentionally systematic do you think, or an unintended consequence of the technology?
I’ve not had a lot to do with video creators professionally. But when we did engage one to create some YouTube videos capturing our company and what it was like to work there, I was super impressed. Worth every cent and so much better than anything most amateurs could produce.
I don’t know that I’m interested in trying to divine if people intend to exploit or just accidentally help build systems which do. The purpose of a thing is what it does.
So when the thing does something you consider undesirable, how then do you determine whether to incrementally improve the thing, reform the thing, or burn the thing to the ground?
So you don’t consider what the intended purpose of a thing was, in that process? Even if only for the purpose (heh) of contemplating what to replace it with?
That would be a useful principle for me if you replaced the need to understand the intentionality of systems with understanding the material circumstances and effects of them. Spinoza, Marx, and the second-order cyberneticists had the most useful views on this in my opinion.
Ah, yeah - what I meant was, if you want YouTube content and loathe the ads, paying to remove them probably isn’t a rip-off.
At least, in the proximate transaction. I do wonder how much of that $11 or whatever goes to the content creators. I know it’s zero for some of the content my kids watch, because it turns out swearing Australian chemistry post-docs blowing things up isn’t “monetizable” :)
Hence paying the creators through a platform like Patreon. Although I’d rather not Patreon specifically.
(Edited to add: I remember how much I treasured my few Transformers toys as a child. Hours of joyous, contented, play alone and with others. Sure they were expensive for what they were, physically, and cartoon TV advertising was a part of what enabled that. It’s a similar deal with my kids and Pokemon and MTG cards … total rip-off from one angle (“it’s just a cardboard square”) but then you see them playing happily for hours, inventing entire narratives around the toys. Surely that’s no rip-off?)
This convinced me to give Newpipe a try and omg it is so much better than using Firefox even with uBlock Origin, let alone the Android Youtube app. Thank you so much for the recommendation!
Remember that YouTube Premium also comes with other things, like a music streaming service. Apparently YouTube tentatively thinks 6,99€ is around what the ads are worth.
Much like any creative endeavor, the platform/middleman almost certainly takes a big cut, and then what’s left is probably a Pareto-type distribution where a small number of top-viewed channels get most of the payout.
On the other hand, if you don’t pay for it and don’t watch any ads, then it’s likely that the people creating the videos get nothing as a result.
You can also take the work from postmarketOS and port it over to mobile-nixos, they work pretty similarly. I’ve now ported mobile-nixos to a few old devices and I love it. I now have an ebook reader, a games console and various phones running NixOS.
The user interfaces are never quite right, but it’s great to have a bunch of powerful ARM-based devices with full access to NixOS.
I’m a bit surprised no one has brought up the pinephone yet (or the pro, since that’s the one that’s got usable hardware specs). It has a keyboard case, is relatively modern ARM, and moderately good firmware/software support (YMMV).
One gripe I have with how we talk about “no silver bullet”:
There is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity.
That’s doesn’t happen in any field. When order-of-magnitude improvements happen it’s due to a bunch of different technological innovations that were all in progress for a long time, often interplaying with each other. Even Brooks himself admits this!
A disciplined, consistent effort to develop, propagate, and exploit them should indeed yield an order-of-magnitude improvement. There is no royal road, but there is a road.
I forgot about that second part, thanks for the reminder.
I also have an analogy of my own: while we may not have a silver bullet, we can most probably gather enough silver dust to forge one bullet or three. Yes, I think that up to three orders of magnitude improvement, compared to current practices, is attainable at least on principle. For instance:
The STEPS project demonstrated the ability to make a home computing system in 20K lines of code, including compilation toolchain, rendering & compositing, desktop publishing, network communication, drawing, spreadsheets… all while traditional systems (GNU/Linux, Windows, MacOS, and their associated software) approach or exceed 200 million lines of code to do the same, or 4 orders of magnitude more. Now STEPS does not include the kernel, but…
Casey Muratori pointed out in The Thirty-Million-Lines Problem that while kernels currently (well, in 2015) weighted around 15 million lines (but a fraction of the the aforementioned 200M lines), if hardware vendors got their act together (or I would add were forced by regulation) and agreed on reasonable standard interfaces (ISA) for their hardware and published the damn specs (all those webcams, wifi modules, USB thingies, and of course those fucking graphics cards), we could go back to have small kernels that would only require 20K lines of code or so.
Casey Muratori also pointed out that performance really got the short end of a stick, with programs who routinely runs 3 orders of magnitude slower than they could, to the point that users can actually notice it, despite the ludicrous performance of our computers (and I would include the weakest smartphone produced in 2023 in that list). Not only that, but there are low hanging fruits we could pick if we knew how. (A note on refterm being 3 orders of magnitude faster than the Windows terminal: a Windows terminal contributor confirmed to me that those speeds are achievable by the Windows terminal itself, it’s just a ton of work, and they’re working towards approaching it.)
Back before my day the Oberon system, by Niklaus Wirth (Pascal/Modula/Oberon languages inventor), was used at his university to do actual secretary and research work for a few years, and the entirety of the OS required no more than 10K lines of code. With a language arguably much less expressive than the STEPS’ languages, on a computer orders of magnitude weaker. This does include the kernel, and the hardware description itself takes up about 1K lines of Verilog.
@jerodsanto, instead of repeating yet again that there is no silver bullet in a way that is most likely to have us abandon all hope, I believe it would help more to highlight that massive improvements, while far from free, are absolutely possible.
Casey Muratori pointed out in The Thirty-Million-Lines Problem that while kernels currently (well, in 2015) weighted around 15 million lines (but a fraction of the the aforementioned 200M lines), if hardware vendors got their act together (or I would add were forced by regulation) and agreed on reasonable standard interfaces (ISA) for their hardware and published the damn specs (all those webcams, wifi modules, USB thingies, and of course those fucking graphics cards), we could go back to have small kernels that would only require 20K lines of code or so.
That presupposes that an operating system kernel needs to know everything about the computer’s hardware and serve as an intermediate for all communication with it. That isn’t true, the kernel could limit itself to scheduling and orchestrating access. Then the diverse hardware could be handled by similarly diverse set of separate daemons.
That’s a better approach than making a monolith that has to handle everything and then complaining that everything is too much.
I reckon your suggestion has benefits. Done well I expect it will increase reliability and security, probably without even sacrificing performance : computers are so fast nowadays that having components be inches apart makes them distributed systems to begin with.
But it wouldn’t really help the point I was making unfortunately. We need those drivers one way or another. To send you this comment I need at some point to talk to my Wi-Fi module on my laptop. If there are a gazillion different Wi-Fi modules out there we’ll collectively need a gazillion drivers. So what happens in practice is that hardware vendors, instead of giving us the specs, write a (proprietary) driver for the 1 most popular OS (or 2 if we’re lucky), and then let the other OSes to fend for themselves. With a standard ISA for all Wi-Fi modules, we could have one driver per OS, done.
Casey Muratori pointed out in The Thirty-Million-Lines Problem that while kernels currently (well, in 2015) weighted around 15 million lines (but a fraction of the the aforementioned 200M lines), if hardware vendors got their act together (or I would add were forced by regulation) and agreed on reasonable standard interfaces (ISA) for their hardware and published the damn specs (all those webcams, wifi modules, USB thingies, and of course those fucking graphics cards), we could go back to have small kernels that would only require 20K lines of code or so.
They’re not going to. They are simply never going to do that. Take it as a constraint and move on.
More to the point, this reeks of “We could have a Good, Moral, Simple System if All You Zombies acted right and accepted something that was easier for us to code!” The most successful variant of that basic philosophy was the original 128K Macintosh, and even Apple eventually had to bow to the real world and add enough functionality to keep the IBM PC and clones from steamrollering them utterly.
There’s room to simplify and improve, but I’m allergic to being strait-jacketed into someone else’s idea of a Moral System just so an implementer can brag about how few lines of code they needed.
They’re not going to. They are simply never going to do that. Take it as a constraint and move on.
I guess they’re not. Though One argument Casey put there here was that it might actually benefit the first hardware company who does that a great deal. One reason they don’t is because it’s different from the status quo, and different is risky.
There’s room to simplify and improve, but I’m allergic to being strait-jacketed into someone else’s idea of a Moral System just so an implementer can brag about how few lines of code they needed.
Not sure exactly what straightjacket you’re talking about, and how moral it really is. ISAs aren’t all that constraining, they didn’t stop Intel and AMD from updating their processors (they do pay the x86 decoding tax though). Also, the problem is less the complexity of any given ISA, and more the fact that there are so many different ISAs out there. A big driver towards the near insurmountable amount of code in kernels is that you need so many different drivers.
There is one thing however that I would have no qualm writing into law: when we sell computer or electronic hardware, all interfaces must be documented. The entire ISA, all the ways we might modify the operations of the device, everything one might need to use the hardware, up to and including writing a state of the art driver. No specs, no sell. Hardware vendors do write their own drivers, it’s not like they don’t already have their internal specs.
Hardware vendors are occasionally forced by regulation to agree on standard interfaces. It’s wrong to assume that corporations always get their way and are never democratically regulated.
I agree that we’re leaving tons of performance on the table, but I don’t agree with the implication that it’s some kind of moral failing. People optimize towards a goal. When they reach it, they stop optimizing. A purpose built system will always be simple and faster than a general purpose system, but general purpose systems can solve problems not anticipated by people higher up the stack. It’s all a trade off. As Moore’s Law dies off, we’ll see people start optimizing again because they can’t just wait around until hardware gets better, and we know the kinds of things we’re building towards now, so we can be a little less general and a little more purpose built.
Actually I’ll do more than imply the moral failing, I’ll outright state it: it is at some point a moral failing, especially when we have a non-trivial number of users, who would like to have less drain on their battery, or would like to wait less time on their software, or would like to suffer fewer bugs and inconveniences. What we might see as wasting a few seconds a day accumulate to much, much more when we have many users.
That, and more performance would allow us to get away with weaker, more economical, more durable, more ecological hardware. The computer and electronics industry is one of the most polluting out there, reducing that pollution (for instance by not expecting users to update their hardware all the time) would be nice.
We might be trading off other things for that performance, but to be honest even there I’m not sure. Writing reasonably performant software doesn’t require much more effort, if at all, than writing crap. I see it every day on the job, we can barely achieve simplicity. If we did that everything would be easier in the long run, including performance.
Actually I’ll do more than imply the moral failing, I’ll outright state it
Then perhaps Casey (and you) could direct some ire at the field of game dev. I know Casey and his fans like to promote game dev as the one true field where We Care About Performance™, but as I always point out that’s just not the truth.
And to prove it’s not the truth, I generally just go over to reddit, pull whatever the latest big game release is, and look at a few threads. Apparently right now it’s The Lord of the Rings: Gollum, which benchmarks atrociously even on ultra-high-end gaming hardware, and based on comment threads isn’t really any better on console (which is where the motte-and-bailey argument about game dev often retreats to), suffering degraded graphics in order to keep up any semblance of performance. One thread I glanced at included the savage comment “The Lord of the Rings books had better graphics”.
None of the people involved in pumping out these games seem to think that performance is a moral imperative, or that it’s morally wrong to waste cycles or memory. Yet so often it’s all the rest of us who are supposed to take lessons from them.
So I’ll outright state it: if I took to heart the way game dev does performance, I’d get fired. Casey and friends should spend some time on the beam in their own eyes before complaining about the mote in mine.
I know Casey and his fans like to promote game dev as the one true field where We Care About Performance™
His fans perhaps. Casey himself I actually don’t know. He happens to work in the game industry, but he’s on the record about being in it for the technical challenges, not really the game themselves. And if we’re going to scold him about not doing what he’s preaching, we should take a look at what he actually did: the Granny animation system, whatever he did for The Witness… Similarly, I would look at what Mike Acton’s participated in when he worked at Insomniac Games, or what he managed to contribute for Unity.
About games wasting way too many cycles, I know the list is long. Of the top of my head, I personally played Supreme Commander, who I think demanded way more from the hardware than it had any right to, and Elite Dangerous (in VR), whose Odyssey update had unplayable performance for a while, and is still reputedly quite crap.
I would also like to know why Factorio and The Witness take so long to boot. Factorio devs cared a huge deal about performance (and improved it by a couple orders of magnitude over the years), and Jonathan Blow ranted in 2017 about Photoshop boot times, so I guess there must be some explanation? Especially from Blow: I’d like to know why his boot times are somehow acceptable, when Photoshop’s are not.
No sources on me rn, but iirc computer hardware is actually low on the pollution totem pole. Things like concrete manufacturing, steel manufacturing, agriculture, and cars swamp everything else.
I can concede that. Computer stuff being really polluting is something I’ve heard I don’t remember where, and trusted then. Maybe I shouldn’t have. I also recall some graphics and numbers on greenhouse gas emissions (from Jean Marc Jancovici), and for these no single sector seemed to dominate any other: we need to reduce everything all around.
I do suspect however the mindset responsible for computer related waste is also responsible for much of the waste pretty much everywhere else as well.
It’s such a high bar. “Oh, this tool only doubles your productivity? Thats not an order of magnitude so not a silver bullet”
People really miss the point of the paper. Brooks’ argument was that programmers could make an order of magnitude difference and tools wouldn’t. People joke about 10x programmers but then quote the tool part of the paper.
you have to understand, at the time the book was written, science fiction and fantasy fandom was not yet mainstream, and realist, detail-oriented taxonomy of every fictional concept was not as far along. it feels silly saying it that way but … I get the feeling it might be hard to imagine what the world was like prior to the cultural victory of a certain segment of fandom, so I’m doing my best to describe what has changed.
this is all by way of saying that it wasn’t intended to be a fully worked-out metaphor. silver bullets are a thing that sounds cool. nobody cared that it’s specifically werewolves they’re usually discussed in relation to. that’s all there was to it.
and yes, software complexity was the obstacle being discussed
Of all the monsters who fill the nightmares of our folklore, none terrify more than
werewolves, because they transform unexpectedly from the familiar into horrors. For
these, we seek bullets of silver than can magically lay them to rest.
The familiar software project has something of this character (at least as seen by the
non-technical manager), usually innocent and straightforward, but capable of becoming a
monster of missed schedules, blown budgets, and flawed products. So we hear desperate
cries for a silver bullet, something to make software costs drop as rapidly as computer
hardware costs do.
Ah, so I was confused because I hadn’t read the original paper being commented and was only replying to @puffnfresh’s blog post(‘s URL). Serves me right, but I apologize for wasting you all’s time with my confusion!
I self-host and honestly I am not sure what I am doing differently because it doesn’t seem all that hard? I’ve barely touched my config in the past 5 years other than enabling TLS and the auth stuff at some point. I probably don’t even have some of that configured quite right.
Spam is not much of a problem these days. The only special thing I do there is using unique addresses for every service I sign up for. But very few of those addresses ever gets spam. Some have gotten the occasional “newsletter” but even that goes away when I hit the unsubscribe link. I’ve only blackholed about 10 addresses in the past few years.
I do keep a vanity gmail address for some uses because it’s easier to get the spelling right than my own domain. I fetch that into my local mail handling too though, I hardly ever touch the gmail UI. Now that I think about it the gmail address probably gets most of the spam I see.
I don’t run my own DNS these days because it didn’t seem worth the hassle but the flexibility I get with email is well worth the (minimal ongoing) effort.
Works pretty well but I haven’t configured my spam filtering properly - things are being soft rejected but they just show up as normal emails in my client. Need to figure that out.
My understanding on the whole “never host your own mail server” argument is that email is something you probably want near perfect uptime and as a regular person its hard to manage that. When I was looking at hosting myself there was a lot of talk about putting a caching server in front of your mail server to catch anything if you go down which seemed like a good solution. But I figured if I was going to do that I might as well just pay for a service.
Except that scrapscript is a scripting language for a decentralized social network, and has authentication and a payment system. (See link in my other post.)
When I read some of the Unison docs, I got the strong impression that names don’t matter, entities (types, values) are identified solely by their hash, and their name is not part of their identity.
In scrapscript, names do matter, it doesn’t have these Unison naming semantics. A scrap can have a name that is consistent across multiple versions of the scrap, which allows you to upgrade code to the latest version. I don’t know how this works in Unison.
The social network, authentication, and payment systems are the “killer apps” that I want to build with the language, but aren’t part of the core architecture :)
I’m not super familiar with unison either, but your description of scrapscript was spot on :) thanks!
To make things worse, Foldable is a higher-kinded type class.
This is not a bad thing! I’ve seen people new to Haskell understand higher kinds very easily, over and over. I even wrote about how I constantly see higher kinded types benefit learning:
I know the point is more about type-classes and yes, they need to come after the basics. That’s the problem with making a useful Prelude - what you want to use in production is not always the first thing you want to learn. This is not really a problem specific to Haskell.
In the early lectures, when I teach equational reasoning, I will tell students a small white lie and pretend that the evaluation strategy is strict.
I don’t know how that’s useful. Why do people care? Are they doing complexity analysis of functional algorithms? I don’t stress that much when teaching, usually it’s something quickly before moving onto the next exercise “btw that’s O(1), because laziness”
When I teach Haskell, I work around the concept of currying by telling students a white lie, that functions can have more than one argument.
I have people in classes who insist that Haskell functions take more than one argument and it creates a lot of problems for them. Once they eventually understand and agree, things become easier. So I always insist, from the first time we write a function, so that things are cleared up as soon as possible.
To make things worse, Foldable is a higher-kinded type class.
This is not a bad thing! I’ve seen people new to Haskell understand higher kinds very easily, over and over.
That’s the problem with making a useful Prelude - what you want to use in production is not always the first thing you want to learn.
I agree with you, but I also think the author makes a very good point.
The UX of map and fold[lr] is inconsistent.
Ideally, the Prelude should contain the most useful (generalised) versions,
while type-specific modules contain specialised variants.
So Data.List would export
foldr :: (a -> b -> b) -> b -> [a] -> b
map :: (a -> b) -> [a] -> [b]
This would also open an opportunity to introduce these concepts.
“Hey, List.map and Maybe.map and Either.map kinda do related things,
don’t they.”
Instead, map and fold[rl] are imported directly from Data.List.
By a disappointing historical accident,
we get a specialised map and a generalised fold.
Yeah map being specialised is definitely a problem. Prelude is really serving nobody well. It’s not good to use professionally and definitely not good to use for teaching.
It’s wonderful to see this finally release! Rules being fully defined in Starlark is great for integrating new languages, and seeing Shake’s influence here after so many years makes me happy.
tl;dr FB hired the author of Shake to productionise his research into build systems.
From the blog post: (emphasis added)
The goal of Buck2 was to keep what we liked about Buck1 (the core concepts and workflows), draw inspiration from innovations after Buck1 (including Bazel, Adapton, and Shake), and focus on speed and enabling new experiences.
As future work, we see value in investigating the feasibility to extend Buck in this direction, given the benefits offered by monadic build systems such as Shake.
The blog post also notes that Buck2 “[follows] the model in the paper, ‘Build Systems a la Carte’”. That paper was co-authored by the creator of Shake and one significant section within is entitled “Experience from SHAKE.”
Finally, as @puffnfresh noted, Neil Mitchell (author of Shake) is a co-author of Buck2.
Is it natural for me to have skeptical feelings for something like this?… The reasoning being, there has already been so many man hours poured into FP languages, by various experts, and it hasn’t been “paradigm changing”. On top of that, optimizing FP seems to take a lot of work and knowledge. You’re telling me they’ve got all the experts, and hammered something out (which itself sounds inadequate to achieve something great here), which is better than GHC or similar? It’s incredible if true. Like insanely incredible. I would love for someone to explain to me why Verse has a chance at all.
… The reasoning being, there has already been so many man hours poured into FP languages, by various experts, and it hasn’t been “paradigm changing”.
It definitely it has been, look around! Languages have been copying lots of things from FP tools. Algebraic data types, higher-order functions everywhere, pattern matching, immutable by default, futures and promises, generators, etc. These features are many decades old in Haskell, OCaml, etc.
FP and these experts have already changed the paradigm.
You’re telling me they’ve got all the experts, and hammered something out (which itself sounds inadequate to achieve something great here), which is better than GHC or similar?
No, it’s completely different. Verse is absolutely definitely not Haskell and never will be. The goal of GHC is totally different to what Epic Games is doing with Verse.
People in Epic Games have been thinking about this for decades, recently invested a bunch of time, hired a bunch of experts, made a language, decided to rewrite one of the most popular games in the world using the language they just made. So yeah, I think “Verse has a chance” at doing what it was exactly designed to do. Why not?
decided to rewrite one of the most popular games in the world using the language they just made
Verse’s current implementation compiles to Unreal Engine’s Blueprint VM, so I imagine the integration into Fortnite was simple and not a complete rewrite. The /Fortnite.com/ package contains a bunch of Fornite specific APIs that FII out to Fortnite code iiuc
Yes, the initial integration uses existing tools, but from Tim Sweeney (just after 33 minute mark):
We see ourselves at the very beginning of a 2 or 3 year process of rewriting all of the game play logic of Fortnite Battle Royale (which is now C++) in Verse…
Verse’s current implementation compiles to Unreal Engine’s Blueprint VM
I really hope the Open™ implementation is implemented on top of something a lot more efficient. Blueprint is horribly wasteful. The VM and bytecode has been largely the same since the UnrealScript days - for the curious, it’s essentially a glorified tree-walk interpreter (but with compact bytecode instead of heap-allocated nodes.) It really makes me wonder why Epic decided to continue using the same inefficient tech for UE4’s Blueprint.
The guy who presented the language on stage built a lot of JavaScriptCore / WebKit internals, so they certainly have the right people to build a very fast VM.
Languages have been copying lots of things from FP tools.
I should’ve been clearer in what I meant, but this is definitely true! Definitely true… What I meant was any particular FP language hooking everyone into the FP paradigm (or changing the FP paradigm to a degree of making it globally attractive).
Secondly, if it’s already changed then by the right people in various ways - what the heck does Verse have to offer? I think your insight further drives home the idea that Verse is just a “fun company project”, at least to me!
To be pedantic, generators are from CLU, which is where a lot of languages got them. I’d have to check my research notes but I think pattern matching is from Prolog? Definitely popularized by FP though.
Verse is a domain specific scripting language for UEFN, the Unreal Editor for Fortnite. It’s the language used to describe game logic. The goal is to allow game logic to run in a distributed fashion across many servers, where there could be a hundred thousand players in the same game.
This is a pretty extreme requirement, I don’t actually know how you can reasonably do this in a conventional language. It makes sense that they’ve chosen a logic programming language, because that allows you to express a set of requirements at a high level, without writing a ton of code to specifically take into account the fact that your logic is running in a distributed fashion. According to what I’ve read about Verse, the existing logic languages they looked at have non-deterministic evaluation, which make code unpredictable and hard to debug. So Verse is a deterministic logic language, and that is called out as a specific contribution in the research paper about the Verse calculus.
As for whether Verse will “succeed”. The only job it has to do is to describe game logic in UEFN, and the criteria for success is that it scales to allow very large game environments, in a way that Epic hasn’t succeeded in doing using C++. There’s lots of money available and they’ve hired the best people. so I think they have a good chance of succeeding.
Does Nix still have value if most of your stuff is statically linked? Memory and disk space are cheap, so more and more stuff seems to be going that way. Admittedly, it’s not as elegant to have lots of fat binaries around.
Yes! You still need to specify your build system, versions of the libraries that get statically linked in, etc. Dynamic linking is just a tiny piece of Nix.
What constitutes “your stuff”? The vast majority of what I’m using NixOS for is configuring a set of virtual machines running all sorts of software I didn’t write personally, and I have no idea whether most of it is statically linked or not - what does it even mean for, say, my random manga-reader app running in a Docker container and written in Crystal to be statically-linked anyway? Nonetheless, I need to configure it correctly or it breaks and I can’t read my manga anymore, and I’d like that configuration to be version-controlled, not break if I change something unrelated it depends on like my database machine’s address, etc.
Yeah lots of value: reproducibility, consistent build tooling, code reuse. The dynamic linking that nixpkgs configures is such a tiny part of using Nix, I really don’t care about it.
I’d like to say the Nix is ‘Hybrid Linking’ , Artefacts / outputs / Binaries (what ever you call them) , look like they are dynamically link, but nix-build’s them so they are static.
So it is neither Dynamically linked like every other Unix/Linux out there, it’s neither statically link (very popular with golang developers) …it’s both.
Some things are shared when possible, other things are not shared (like having two versions of program or dependency to keep the software ‘operational’).
Nix treats ‘the package manager’ as computer memory, it is allocated (malloc) , used, and then can be free (free)
it’s just treats the package manager (nix) and the operating system (nix/OS) as a program.
The fact that nix ‘builds’ the linux kernel, and put libc together, and then add all the other scripts and programs to make a ‘distribution’ is purely coincidental.
It could as easly build a freebsd kernel, or a solaris one, or anything in the past or the future.
It’s a build a system.
‘If you gave it a pile and bricks and some cement, it would nix-build a wall for you’ and then simple vanish.
Yes, in fact I think Nix solves a particular problem with static linking very well:
When you do need to upgrade a library everywhere, for example to fix a security issue, you need to rebuild everything using that library. Nix makes this really easy and automatic.
the thing is, even if you pay for them, they never seem to be satisfied. it’s also why the whole “if you don’t pay then you are the product” saying is basically false… (selling customer data is even more profitable when customers pay them to sell it)
Some people are just bad actors. But when you find a good provider, they’re much more likely to want to keep your data for you if they’re profiting from their work.
How could we know about Docker though? It’s not like the service is free, they have always had a revenue stream, and it’s the industry standard. You see that, look at the pricing model, and … what do you do when you see that their free tier happens to be enough for your use cases? Do you choose a different provider just because the free tier happens to be good enough and you don’t want to use something you’re not paying for?
It’s not like the service is free, they have always had a revenue stream,
Not enough of one. Docker’s lack of sustainable revenue has been discussed for years. Their business plan from the outside looked like ‘give away great tools, get bought by someone who needs them’.
and it’s the industry standard.
No, OCI is the industry standard, Docker is a company that sells the most polished tools for dealing with the industry standard.
You see that, look at the pricing model, and … what do you do when you see that their free tier happens to be enough for your use cases?
You look at their free tier and see that something free is going to have to go away if they aren’t bought by GitHub or similar (and then it may still go away, just be bundled with something you might already be paying for) and avoid making plans around it always being there for free.
I’m pretty sure other OCI implementations are significantly less used than Docker? It certainly was a few years ago when a lot of the decisions to use Docker were made.
“Industry standard” doesn’t mean a formal standard managed by a standards body.
You look at their free tier and see that something free is going to have to go away if they aren’t bought by GitHub or similar (and then it may still go away, just be bundled with something you might already be paying for) and avoid making plans around it always being there for free.
Okay, and how do you do that when there’s no indication at all about what a potential future price might be? Do you put your finger in the air and guess? What if it ends up being significantly more expensive than you guessed?
Okay, and how do you do that when there’s no indication at all about what a potential future price might be? Do you put your finger in the air and guess? What if it ends up being significantly more expensive than you guessed?
When you depend on a free offering from a company with an obviously unsustainable business model, you assume it will go away at some point and have a plan B. If they move it to a paid offering that is cheaper than your plan B, then you may choose to use that instead, but you should always assume it will disappear. Possibly with no notice if the company goes bust.
But how do you price that in when the industry-standard service is free for your use-case and they’re not giving any indication of what your use-case might cost in the future? How do you price in a completely unknown potential future price which may or may not materialise?
The price depends on what strategy you use to manage the risk
You pay to avoid the risk. You host your own docker registry, or you don’t use docker at all. The price is whatever you have to pay.
You don’t pay anything upfront and assume the risk. If it happen, it’s the end of the road, and so be it. The price is what you lose if it happens. If you have a small project you do for fun, this may be the easiest solution. If the project is your livelihood, this is a dangerous gamble. Maybe look at option 1 or 3 instead.
You pay for a contingency plan to reduce the impact of the risk. Maybe you keep a docker registry hosted somewhere, ready to take over when your main registry fail. Since you don’t pay the bandwidth, it remain pretty low cost until the risk happens. Or you always keep enough cash flow to switch to a paid plan if the free option disappear. The price is what you pay to keep the contingency available, plus the price to use it if it happens.
The solution is probably a service that makes hosting free but charges for downloads. For commercial products, increasing downloads increases revenue and so it’s completely fine to increase the costs to the publisher as the number of downloads goes up, to cover the traffic. For community-developed projects, most users pay nothing.
GitHub is probably in a good position to keep this kind of thing free because they want to monetise the things that pull containers. In particular, Actions and CodeSpaces are both very large consumers of containers and benefit from making it easy for people to expose a large number of container images to their customers.
The Nix programming language is a purely-functional programming language, not a configuration language. It’s used to program the Nix package manager in the same way as any other programming language would. It just happens to be good at configuration :^)
we’ve had this discussion before, it always comes full circle:
What is your solution to improve upon this shell piping ?
Verify Checksum: Coming from the same source, you have to trust them
download a .deb: Coming from the same source, you have to trust them and they have to provide a ton of different package formats
apt install: it’s new, there is no shipped version
docker: this isn’t made for docker - also docker doesn’t mean security
VM: neat, but how do I get from the “vm” testing step to my actual installation ? I could just pipe this shell command in the VM anyway, so no need to provide a VM image (which format?).
snap/flatpack/appimage.. which one of them, are they even useful for this installer ? also you’re still trusting the authors and now the snap servers too
You can only argue that downloading it to disk and then executing can leave a trace on the disk, so you can inspect the malware later on. We’re still downloading debian ISOs straight from the host, not that big of a difference. And I bet no relevant percentage checks the signatures or checksums.
Downloading it first or putting it at some reliable hosting (where you can’t change the download for specific people by random just through compromised developer hosts) is definitely one step better. But it doesn’t change much in terms of the core complaints: Running shell scripts from the internet.
Well, pretty much all free software boils down to “Running [executable code] from the internet”.
The question is whom you need to trust. The connection is HTTPS and you need to trust both, your TLS setup and determinate.systems anyway to use this installer, so not much is left.
Getting a an external signature/checksum would guard you against a compromised web server, but only if you can get that checksum out of band or from multiple, independent sources.
I’ve been toying with encoding an ast as a database (prolog currently) and reading this is very inspiring. I think a lot of problems become a lot simpler when your code is stored in an easily queryable (with metadata) format, type checking & linting becomes significantly simpler for example.
But i can also imagine editors & plugin systems which could for example load sql db tables into the code database & trivialize safe sql dsls.
I read this Joe Armstrong post around 12 years ago when it was first published and I thought yes, definitely and I’m very happy that so many people are working on systems which do this. Thank you!
Have you figured out deduplication yet? I had to iterate a few times, and currently I’m prototyping using git for deduplicated AST storage.
couldn’t you content-address the functions based on the the hash of their AST, like unison does?
Yes, that’s how git operates: every git repository is a content-addressable store.
Note that, unlike Unison, I intend to identify equivalent functions and deduplicate them as well; when a programmer asks for an older and less efficient version of a function, they should automatically be given the latest and most efficient version with equivalent behavior.
I thought git used content + commit message + parent Sha + date + author email, which looks too much for your goal? Also, are you serializing the AST of each function in a separate file?
The encoding is deeper than that. I’ve started documenting it here, and I’ll submit this as a top-level post once I’ve finished documenting what I’ve built.
This seems like a great talk for beginners. My feedback is not aimed at beginners, but at Nix evangelists.
This doesn’t quite sound right to me. While Eelco’s thesis doesn’t directly refute this claim (say, on p69), it does explain that the Nix expression language is the simplest possible language which has all of the necessary features, and it also explains why each individual feature (laziness, recursive definitions, attrsets, functions, etc.) are necessary for defining packages. The thesis also explains why languages like Make are unsuited to the task.
This is perhaps too imperative to be true. NixOS configuration is generally monoidal, and often commutative (or configuring underlying systems which commute, e.g.
networking.firewall.allowedTCPPortswhich is sorted by theiptablessubsystem.) It’s important to not give folks the false impression that adding a NixOS module is like adding a SysV init script.The
withkeyword certainly could’ve been removed from the language without removing expressiveness. I’d even say the language would be better for it.yeah a lot of good language design is saying no to features or syntactic decisions because of concerns that seem very abstract, because past experience has shown they have far-reaching negative impact
withis in that category and should be removed, imo. I find it super convenient but that does not outweigh the loss of referential transparency. there have been a few proposed replacements, some of which are kind of okay although nothing perfect, but at the end of the day it’s a damaging feature and even just removing it entirely ought to be on the tableit’s hard to do that when there’s existing code, but it’s possible, and it does pay dividends down the line
Right, I mean, I’m a known nix partisan as it were, but in my view I’d say that there is no better expression language for the thing nix is doing today, and there also wasn’t at the time it was new. :)
TBH if I was remaking Nix today, I’d probably use JavaScript as the expression language to make people’s lives easier
It’s basically JSON for all practical purposes. Also, you want to keep computations to a minimal, it is more of a data description language.
Functions are a super important part. Recursion is used heavily to do things like overrides. It’s not really JSON.
Sure, but these are written once somewhere deep inside lib, and then used declaratively. You won’t really be writing functions as you would in a regular PL, and I would argue that most package descriptions can actually be read by humans if you parse them more or less as you would a JSON, which is the important mental model.
Try it and see what happens. There’s several projects like Nickel and Zilch already exploring the space, too.
I understand the impulse but I still feel it would be the wrong call. it’s not like it’s something really exotic, it uses infix and curly braces, heh.
Thanks for pointing this out!
Nix seems like a worthwhile investment to learn. Though (as mentioned with the comic sans slide) the ecosystem is somewhat confusing.
Among the things that add a barrier to entry is needing to learn yet another custom configuration language (funny that the post starts off by complaining that NGINX has one). I’m wondering if a wrapper in a lang I might already know, like Ruby or Rust, might ease that one small burden so I could focus on the outcome and not the syntax.
If you already know a programming language with some functional elements (Rust definitely fits that bill), I’d say Nix the language can be learned in an hour by taking the time to read Nix pills 4 and 5. The language is definitely quirky, but it is also extremely small. Warts are an annoyance but not a complete blocker.
Then however you have to find out what all the functions in the
libthingy do (I think it’s provided by nixpgs and not Nix the language but I am still not sure), and this is IMO the big barrier to entry.Also about this bit:
NixOS is surprisingly old: Wikipedia tells me that the initial release of NixOS was in 2003, while nginx was in 2004. Maybe nginx should have used nix the language for configuration from the beginning :P
That’s correct,
libis part of nixpkgs, not part of the language. Strictly speaking there’s onelibthat’s used when writing packages and another that’s used when writing modules, but you’ll rarely need to worry about that distinction.Thanks for clarifying!
sure thing!
Guix isn’t quite as widely adopted as Nix, but uses scheme which is a big win for it. Anecdotally, it’s also much easier to use.
I really want guix to succeed, but fear it’s lack of network effect will continue to hamper it. nixpkgs just has so many more packages and people involved. Sadly, guix will continue to be even more niche than the niche nix as long as it’s tied to GNU. In the real world, purity doesn’t really work for operating systems any more than programming languages.
Nix and NixOS are pure, so that’s a very weird to say!
Eh, “pure” is a very overloaded term that has no meaning to the general computing public. I think it’s safer to say that they are more strictly focused on being repeatable (same inputs -> same outputs).
Same inputs, same outputs is purity.
Yes, but if you call it purity you start making people’s eyes glaze over because it starts to sound like Haskell ivory tower bullshit lol
That seems like “oh this isn’t pure, because that’s ivory tower, this is just strictly focused on being repeatable”
I prefer to say “oh purity is super practical, look”
Yep. Only that one simple thing with nix, that I can just replicate this exact environment perfectly in another machine every single time is, for me, so valuable I kind of don’t mind some of the quirks in nix/nixpkgs.
I think you two interpret “pure” differently: Guix is pure in the FSF/GNU sense: free of proprietary software in its repositories, to the point where it is using linux-libre (a kernel stripped of binary blobs). This has the major disadvantage of not working very well on a lot of hardware that does require those binary blobs.
nixpkgs has plenty of unfree software in it, and even though you can’t install them without going through additional hoops, they’re there, thus, it’s not pure in the FSF/GNU sense. Like, GNU considers the linux kernel impure to begin with, and you can install that from nixos without any additional hoops.
I can’t remember the FSF using the term “pure” in this sense (complete absence of non-free software). It’s possible other FLOSS projects do, however.
What’s the relevance to programming languages?
Well, Nix is simply a decade older than Guix, so it’s understandable. For my personal use, I’ve found https://toys.whereis.みんな/ to be a nice package search across many channels, and you can add any channels you want as and when you find something that’s not packaged in the main repository. As for the last sentence, Nix is a pure programming language ;)
Scheme is just not doing it for me. I just had a look at how to create something nix-shell like in GUIX, and found this reference. Unfortunately the examples (see the
--expression=exprsection) is exactly the sort of Perl-style line noise I can’t abide. Like, I don’t want to have to remember which complex LISP thing@or%stand for. What’s wrong with words?Hmm, I have to say I vehemently disagree! I’m sorry if what follows feels like flamebait, tone is hard to convey over text and I just want to address some issues with what you said. You are of course entitled to any opinions you hold. The funny thing about what you said, that % is arcane, is that
%has no special meaning, that’s just part of the variable name!@is just splat, like in javascript or python, and neither of those are words either (whereas in scheme you can express it as a word, but using @ is shorter and spending a tiny amount of time is enough to understand that it’s shorthand). The examples are extremely clear and there is a 1-1 mapping between nix-shell and guix shell until you get to the one about--expression, which you would only use if you actually want to evaluate code to do something, which you can’t do with nix-shell.Of course someone used to Scheme would disagree ;) I tried learning Scheme many years ago, and just got the worst cognitive dissonance ever from every single token being abbreviated to death, and from using obtuse technical jargon like
carandcdrinstead of short, simple, everyday synonyms likefirstandsecond, or eveninitialandsubsequent. LISP clones have a massive usability issue in that it’s not just a new programming language, but an entirely new set of words which other programming languages have moved away from.I would love to learn a LISP clone which chose usability over trying to look like 1960s code.
maybe you’d find clojure more amenable, it is very much a modern language and (I think) uses head and tail instead of car and cdr.
Wouldn’t a Nix vs. Guix syntax debate just go back to the split between ML & LISP? That’s been one of the main syntax dividing lines in FP for decades. Having used many MLs (PureScript, Elm, OCaml, Haskell) Nix feels cozy (albeit quirky) & my limited time with Clojure(Script) or Chicken Scheme for Tree-sitter feels alien & unergonomic.
I think of a wrapper language for Nix occasionally too, but I’ve been scared away by how difficult it would be to debug the errors.
I don’t know what problems they’re talking about, and they didn’t really give any examples.
This sounds like an unfounded opinion.
Yeah, Flakes are very heavily used - that would still be the case even if every single person involved in Nix said “no, don’t use them, they’re not ready!”
The Flake design attempted to solve a fundamental usability and caching problem and it doesn’t do that perfectly, but it’s such an important problem that any partial solution is going to be popular.
I think the concrete problem is that they are very heavily used, but aren’t documented in the reference manual beyond the CLI commands.
Good catch. Different interpretations of done.
I’ve added a precision footnote (instead of directly editing).
By that, I meant the “unopinionatable” meaning of released.
I run NixOS on my Kobo Clara 2E:
https://github.com/puffnfresh/nix-files/blob/master/machines/kobo-clara-2e/configuration.nix
I recently bought this device:
https://www.aliexpress.us/item/3256805230352371.html
Which has a mainline QMK firmware implementation available. I could plug it into a Windows PC and also have mouse keys work there. The great thing is that I’ve programmed the various knobs up the top to act as scroll wheels.
Sounds too good to be true.
Looking at the data sheet from the manufacturer’s website it does indeed say that, and does it appear that LiFePO4 batteries are significantly less pyromaniac-friendly than normal LiPO chemistries. (TIL!) But while they tend to smoulder rather than burn you can easily find youtube videos of them still producing respectable amounts of fire.
Looking at the datasheet, it lists the operating temperatures for “charge” to be 0-50C, and “discharge” -20-+60 C, which makes me think the “Stable up to +130C” means “will not catch fire and burn your house down until the external temperature is 130C”, not “is perfectly fine and happy up to 130C”.
So… yeah. You’re probably fine. I would personally keep a good bucket of sand or an appropriate fire extinguisher nearby though, and possibly put some thought into an arrangement that lets it stay cool and have plenty of space for radiating heat. …maybe in a metal bucket or something. I’ve seen enough go bad that every lithium battery has to be treated as potentially hazardous.
That said, this looks straightforward enough that it is making me consider resurrecting my own failed attempt to do something similar…
Thanks, I’ll keep that in mind. It is on a metal balcony, so it should be okay, but still, the fire would be quite an unpleasant experience. So far, the simplest solution for higher temperature was to put the box more under the solar panel, so it is shielded from the sun. I found out, that the temperature dropped immediately into more comfortable numbers. In the long term, I definitely plan to build some inflammable enclosure, as mentioned in the article.
This reads like you’ve just did a Google about the topic and you’re now trying to give advice on it. I imagine the author knows a little bit about the topic.
The primary safety concern for most environments with lithium batteries is thermal runaway, caused by things like overcharging and short-circuiting. LiFePO4 batteries use non-flammable electrolytes, unlike other chemistries, which means that thermal runaway should only happen if it reaches a high temperature, when things turn into gases. These electrolytes can become flammable by introducing external things (e.g. water) but internally they should remain non-flammable.
In the video, there were multiple punctures to the battery over time. It looks like the second puncture created some sort of spark and ignited the vapours coming from the electrolyte. This was not a runaway event which other chemistries are capable of. For every video like this, there are probably 10 of people puncturing, drilling, dropping, etc. without any flames - you need a bit of luck to get vapours, sparks and oxygen in the right places.
So:
Well ackshually…
I don’t know anything about LiFePO4 batteries specifically but I’ve had training about handling lithium batteries in general at two different jobs, and had to put out a large fire once. It was enough to scare the shit out of me. I don’t care what the label says, treat any lithium battery the same way you would treat a none-too-durable jar of gasoline of equivalent size.
They were absolutely right to teach you to treat any lithium batter like a jar of gasoline and that 100% applies to LiFePO4 batteries, too. The details are in the numbers, as per Akin’s first law :-) and in that regard the author is right to be more confident in LiFePO4 batteries than in other types of Li-ion batteries.
Anything, including water, will trigger a fire if you pass a high enough current through it. LiFePO4 batteries doubly so because a high enough current will break the P-O bonds in the PO4 3- ions of the cathode and release oxygen.
That being said, LiFePO4 really are safer than other Li-ion batteries in widespread use in this regard. There’s literature that supports this, including flammability characteristics (see e.g. here, here, or here, or here – see that website with papers for the full text of some of those, you know which one :-P). There are some good reasons for that. For instance, the P-O bond is pretty strong. It takes more energy to break it than e.g. the Co-O bond in LCO batteries; this, along with other characteristics of LFP batteries, really does make thermal runaway harder to trigger (see here for some data). The flammability characteristics are also a lot milder, it’s… kind of like a smaller jar of gasoline.
Some of the chemistry behind it is way over my head (EE, not chem, so as soon as I get into anything that discusses the inner Helmholtz plane and crystaline lattices and whatever I rever to I viscerally understand some of those words via the nightmares they evoke from my sophomore year) but the 130C figure looks about right. As you point out it definitely does not mean it’s OK to use it at that temperature (I don’t recall the exact details but IIRC operating LFP batteries at higher temperatures than the indicated discharge range reduces their life span more steeply than for other types). The safe discharge range is up to 60 C, 130 C is the temperature up to which the manufacturer claims you won’t trigger a thermal runaway.
I’m not sure about the exact design implications here. When I learned about them in uni, LFP batteries were still kind of in their infancy (people were struggling to work around their really poor native conductivity figures) and/or patent-encumbered to hell, and by the time they were in active used I hadn’t actually done any real EE work in years. So I can’t comment on whether the author uses them correctly or not or how safe this particular application is, but in general it does make sense to treat them as safer than other Li-ion battery types.
Most lithium battery chemistries are worse than gasoline in many cases due to thermal runaway. But LiFePO4 is pretty different - needs a lot more than a spark.
This is a good idea in general, but I would be careful about just stopping the test once branch coverage is reached. The major benefit of property tests is that they check for more input data combinations than other testing approaches, and “correctness” means that a program works for all input data combinations.
By stopping once branch coverage is met, you miss out on additional data states that might find a bug.
I imagine in a lot of cases, this would result in more tests being run, not less. If you have a branch which only triggers for 1 value in a huge input space and you run 100 property tests by default, you’re not going to hit that branch most of the time. For example, a lot of property tests rely on many runs over time to eventually find failures.
Is this the case @shapr?
I have a big pile of thoughts on this, maybe too much to fit into a lobsters comment.
This testing strategy is fine for what you get from property testing. Dan Luu’s excellent post on testing has a quote:
QuickCheck by itself is a sweet spot on the axes of “easy to implement” and “finds bugs”. The first version of QuickCheck was tiny, maybe two pages of Haskell?
Hooking that into code coverage is another sweet spot on those axes, my entire implementation supporting three property test frameworks is 76 lines of code.
Property tests are ‘spot checks’ where you might get a lucky hit on an input that finds a counterexample.
Another sweet spot I’ve considered is making sure that new generated inputs do not duplicate previously used inputs for this run, but I haven’t gotten there yet.
If you want to cover hard to reach parts of the code, I have a heavy duty solution in trynocular. In trynocular, the generated random input is wrapped in a “spy” that watches what parts are used, so you can discover how to increase code coverage. Describing that will be a much longer blog post. (But golly it’s gonna be fun)
I can definitely get behind this, because it’s an economical argument - running property tests for longer may find additional bugs, but the ROI is the number of bugs found per time period invested. And I think everyone agrees that there’s diminishing returns for longer time periods of random testing.
I still prefer to separate these “spot checks” from deep correctness checks, because they serve different and conflicting purposes. And there’s one additional cost I’d like to throw into the mix: the scaling cost, meaning the cost to scale the same test to a higher level of correctness, which here basically means running the test for longer. The scaling cost of property tests is pretty much zero: just run the test for longer. Property tests are also embarrassingly parallelizeable, since two tests processes require no coordination and we can just run more of them in parallel.
So my ideal strategy is to use “spot checks” before merge / release, and jack up the test durations after release to still try and find bugs before a user does.
Your trynocular tool looks really cool. I keep saying that we should really separate property testing from random data generation, since randomness is just one strategy. The idea of more targeted data generation strategies is a really interesting research area, and I support any ideas down that path. After all, the holy grail of testing is to find the minimum number of test cases that imply correctness.
That’s right, but the set of values that you need to check for correctness is a superset of the values you need to check for branch coverage, so you need more tests to get there.
This is also my point. If we do what’s proposed here, the 1 value won’t get hit very often, so the branch also won’t get hit, so we’ll stop the test before the branch can be taken.
It’s still good to know what the branch coverage is during a property-based test, because more branch coverage is certainly better. I just would’t use it to stop looking for inputs.
I wrote a longer reply further down in this thread, but in short, I see property tests as spot checks. What you describe sounds more heavy duty. Do you have a way to measure data combinations? Perhaps trynocular does what you want?
I use NewPipe every day, have an adblocker on my computer and phone and am absolutely shocked when I watch YouTube at other places and see how many ads they are pushing. Even more shocking is how people have gotten used to it.
Of course Alphabet/YouTube has to finance itself somehow, but 11,99€/month for YouTube Premium is definitely overprized. If you consider that YouTube only makes a fraction of a cent per ad-view per person, they could probably be profitable with 0,99€/month, and I would be willing to pay that.
This would not resolve one other major issue I have with YouTube: The website is bloated as hell. NewPipe is smooth and native, runs on F-Droid and allows easy downloading (audio or video).
Are you sure? ;-P
So, I pay for a family premium account, primarily to disable ads on stuff my kids watch. I have several motivations:
Now that they’re starting to find content they really like (Explosions and Fire is great for kids, if a bit sweary) I’m going to fund them to choose their favourite creator each, via Patreon or similar.
Edited to add: I also run ad-blocking, at a network level. I have no qualms at all about ad-blocking in general, and think that ad-tech is a blight on the world. Please don’t mistake my willingness to pay for YouTube Red (or whatever it is called) as a criticism of adblocking.
I will teach my children to know when they’re being ripped off and how they can protect themselves and their valuable time.
But they’re pretty obviously not being ripped off, if they want to watch the content, right?
They may not be “ripped off” but video creators will be either way, for a skillset that was once highly valued.
Can you elaborate on that? Do you mean, ripped off by adblocking, YouTube / other tech aggregators’ models in general, … ?
I am referring directly to the systematic devaluation of their otherwise professional labor.
Intentionally systematic do you think, or an unintended consequence of the technology?
I’ve not had a lot to do with video creators professionally. But when we did engage one to create some YouTube videos capturing our company and what it was like to work there, I was super impressed. Worth every cent and so much better than anything most amateurs could produce.
I don’t know that I’m interested in trying to divine if people intend to exploit or just accidentally help build systems which do. The purpose of a thing is what it does.
So when the thing does something you consider undesirable, how then do you determine whether to incrementally improve the thing, reform the thing, or burn the thing to the ground?
with its measurable circumstances and effects
So you don’t consider what the intended purpose of a thing was, in that process? Even if only for the purpose (heh) of contemplating what to replace it with?
Not interesting.
Consider Chesterton’s Fence.
That would be a useful principle for me if you replaced the need to understand the intentionality of systems with understanding the material circumstances and effects of them. Spinoza, Marx, and the second-order cyberneticists had the most useful views on this in my opinion.
As a child I wanted lots of plastic toys which cost a lot of money. Advertising works really well for doing that!
I wanted them and they were a rip off.
Ah, yeah - what I meant was, if you want YouTube content and loathe the ads, paying to remove them probably isn’t a rip-off.
At least, in the proximate transaction. I do wonder how much of that $11 or whatever goes to the content creators. I know it’s zero for some of the content my kids watch, because it turns out swearing Australian chemistry post-docs blowing things up isn’t “monetizable” :)
Hence paying the creators through a platform like Patreon. Although I’d rather not Patreon specifically.
(Edited to add: I remember how much I treasured my few Transformers toys as a child. Hours of joyous, contented, play alone and with others. Sure they were expensive for what they were, physically, and cartoon TV advertising was a part of what enabled that. It’s a similar deal with my kids and Pokemon and MTG cards … total rip-off from one angle (“it’s just a cardboard square”) but then you see them playing happily for hours, inventing entire narratives around the toys. Surely that’s no rip-off?)
True, but I hope you understood what I meant.
This convinced me to give Newpipe a try and omg it is so much better than using Firefox even with uBlock Origin, let alone the Android Youtube app. Thank you so much for the recommendation!
I’m glad my recommendation was helpful to you! :)
Remember that YouTube Premium also comes with other things, like a music streaming service. Apparently YouTube tentatively thinks 6,99€ is around what the ads are worth.
how much of that makes it back to the artists and people creating videos for google’s platform?
Much like any creative endeavor, the platform/middleman almost certainly takes a big cut, and then what’s left is probably a Pareto-type distribution where a small number of top-viewed channels get most of the payout.
On the other hand, if you don’t pay for it and don’t watch any ads, then it’s likely that the people creating the videos get nothing as a result.
Enough that they find it worth their time to do so, rather than… not.
Does anyone know of any platforms like this available now?
Yes, the MNT Pocket is pretty much a successor of the N900: https://mntre.com/media/reform_md/2022-06-20-introducing-mnt-pocket-reform.html
It’s open hardware, and the indie lab / company that produces them has a track record of delivering their promises.
PinePhone with the keyboard case is not too far off.
N900 with postmarketOS can do all of the “things your iphone can’t”, and it also has functional wifi, a modem, etc.
postmarketOS supports a bunch of devices, you might even have one laying around!
https://wiki.postmarketos.org/wiki/Devices
You can also take the work from postmarketOS and port it over to mobile-nixos, they work pretty similarly. I’ve now ported mobile-nixos to a few old devices and I love it. I now have an ebook reader, a games console and various phones running NixOS.
The user interfaces are never quite right, but it’s great to have a bunch of powerful ARM-based devices with full access to NixOS.
I’m a bit surprised no one has brought up the pinephone yet (or the pro, since that’s the one that’s got usable hardware specs). It has a keyboard case, is relatively modern ARM, and moderately good firmware/software support (YMMV).
modem is 3G, so it won’t work with U.S. carriers at least. there may be a few countries which still have 3G carriers but I don’t know which.
still a great suggestion though.
I love my N900, but it’s also really starved for RAM, even back in the day. I can’t imagine how well a modern Linux stack would run on it.
it runs alright as long as you don’t try to use the “modern” web :D
remiss not to mention the DragonBox Pyra, though “available” is a stretch:
https://pyra-handheld.com/boards/pages/pyra/
One gripe I have with how we talk about “no silver bullet”:
That’s doesn’t happen in any field. When order-of-magnitude improvements happen it’s due to a bunch of different technological innovations that were all in progress for a long time, often interplaying with each other. Even Brooks himself admits this!
I forgot about that second part, thanks for the reminder.
I also have an analogy of my own: while we may not have a silver bullet, we can most probably gather enough silver dust to forge one bullet or three. Yes, I think that up to three orders of magnitude improvement, compared to current practices, is attainable at least on principle. For instance:
The STEPS project demonstrated the ability to make a home computing system in 20K lines of code, including compilation toolchain, rendering & compositing, desktop publishing, network communication, drawing, spreadsheets… all while traditional systems (GNU/Linux, Windows, MacOS, and their associated software) approach or exceed 200 million lines of code to do the same, or 4 orders of magnitude more. Now STEPS does not include the kernel, but…
Casey Muratori pointed out in The Thirty-Million-Lines Problem that while kernels currently (well, in 2015) weighted around 15 million lines (but a fraction of the the aforementioned 200M lines), if hardware vendors got their act together (or I would add were forced by regulation) and agreed on reasonable standard interfaces (ISA) for their hardware and published the damn specs (all those webcams, wifi modules, USB thingies, and of course those fucking graphics cards), we could go back to have small kernels that would only require 20K lines of code or so.
Casey Muratori also pointed out that performance really got the short end of a stick, with programs who routinely runs 3 orders of magnitude slower than they could, to the point that users can actually notice it, despite the ludicrous performance of our computers (and I would include the weakest smartphone produced in 2023 in that list). Not only that, but there are low hanging fruits we could pick if we knew how. (A note on refterm being 3 orders of magnitude faster than the Windows terminal: a Windows terminal contributor confirmed to me that those speeds are achievable by the Windows terminal itself, it’s just a ton of work, and they’re working towards approaching it.)
Back before my day the Oberon system, by Niklaus Wirth (Pascal/Modula/Oberon languages inventor), was used at his university to do actual secretary and research work for a few years, and the entirety of the OS required no more than 10K lines of code. With a language arguably much less expressive than the STEPS’ languages, on a computer orders of magnitude weaker. This does include the kernel, and the hardware description itself takes up about 1K lines of Verilog.
@jerodsanto, instead of repeating yet again that there is no silver bullet in a way that is most likely to have us abandon all hope, I believe it would help more to highlight that massive improvements, while far from free, are absolutely possible.
That presupposes that an operating system kernel needs to know everything about the computer’s hardware and serve as an intermediate for all communication with it. That isn’t true, the kernel could limit itself to scheduling and orchestrating access. Then the diverse hardware could be handled by similarly diverse set of separate daemons.
That’s a better approach than making a monolith that has to handle everything and then complaining that everything is too much.
I reckon your suggestion has benefits. Done well I expect it will increase reliability and security, probably without even sacrificing performance : computers are so fast nowadays that having components be inches apart makes them distributed systems to begin with.
But it wouldn’t really help the point I was making unfortunately. We need those drivers one way or another. To send you this comment I need at some point to talk to my Wi-Fi module on my laptop. If there are a gazillion different Wi-Fi modules out there we’ll collectively need a gazillion drivers. So what happens in practice is that hardware vendors, instead of giving us the specs, write a (proprietary) driver for the 1 most popular OS (or 2 if we’re lucky), and then let the other OSes to fend for themselves. With a standard ISA for all Wi-Fi modules, we could have one driver per OS, done.
They’re not going to. They are simply never going to do that. Take it as a constraint and move on.
More to the point, this reeks of “We could have a Good, Moral, Simple System if All You Zombies acted right and accepted something that was easier for us to code!” The most successful variant of that basic philosophy was the original 128K Macintosh, and even Apple eventually had to bow to the real world and add enough functionality to keep the IBM PC and clones from steamrollering them utterly.
There’s room to simplify and improve, but I’m allergic to being strait-jacketed into someone else’s idea of a Moral System just so an implementer can brag about how few lines of code they needed.
I guess they’re not. Though One argument Casey put there here was that it might actually benefit the first hardware company who does that a great deal. One reason they don’t is because it’s different from the status quo, and different is risky.
Not sure exactly what straightjacket you’re talking about, and how moral it really is. ISAs aren’t all that constraining, they didn’t stop Intel and AMD from updating their processors (they do pay the x86 decoding tax though). Also, the problem is less the complexity of any given ISA, and more the fact that there are so many different ISAs out there. A big driver towards the near insurmountable amount of code in kernels is that you need so many different drivers.
There is one thing however that I would have no qualm writing into law: when we sell computer or electronic hardware, all interfaces must be documented. The entire ISA, all the ways we might modify the operations of the device, everything one might need to use the hardware, up to and including writing a state of the art driver. No specs, no sell. Hardware vendors do write their own drivers, it’s not like they don’t already have their internal specs.
Hardware vendors are occasionally forced by regulation to agree on standard interfaces. It’s wrong to assume that corporations always get their way and are never democratically regulated.
I agree that we’re leaving tons of performance on the table, but I don’t agree with the implication that it’s some kind of moral failing. People optimize towards a goal. When they reach it, they stop optimizing. A purpose built system will always be simple and faster than a general purpose system, but general purpose systems can solve problems not anticipated by people higher up the stack. It’s all a trade off. As Moore’s Law dies off, we’ll see people start optimizing again because they can’t just wait around until hardware gets better, and we know the kinds of things we’re building towards now, so we can be a little less general and a little more purpose built.
Actually I’ll do more than imply the moral failing, I’ll outright state it: it is at some point a moral failing, especially when we have a non-trivial number of users, who would like to have less drain on their battery, or would like to wait less time on their software, or would like to suffer fewer bugs and inconveniences. What we might see as wasting a few seconds a day accumulate to much, much more when we have many users.
That, and more performance would allow us to get away with weaker, more economical, more durable, more ecological hardware. The computer and electronics industry is one of the most polluting out there, reducing that pollution (for instance by not expecting users to update their hardware all the time) would be nice.
We might be trading off other things for that performance, but to be honest even there I’m not sure. Writing reasonably performant software doesn’t require much more effort, if at all, than writing crap. I see it every day on the job, we can barely achieve simplicity. If we did that everything would be easier in the long run, including performance.
Then perhaps Casey (and you) could direct some ire at the field of game dev. I know Casey and his fans like to promote game dev as the one true field where We Care About Performance™, but as I always point out that’s just not the truth.
And to prove it’s not the truth, I generally just go over to reddit, pull whatever the latest big game release is, and look at a few threads. Apparently right now it’s The Lord of the Rings: Gollum, which benchmarks atrociously even on ultra-high-end gaming hardware, and based on comment threads isn’t really any better on console (which is where the motte-and-bailey argument about game dev often retreats to), suffering degraded graphics in order to keep up any semblance of performance. One thread I glanced at included the savage comment “The Lord of the Rings books had better graphics”.
None of the people involved in pumping out these games seem to think that performance is a moral imperative, or that it’s morally wrong to waste cycles or memory. Yet so often it’s all the rest of us who are supposed to take lessons from them.
So I’ll outright state it: if I took to heart the way game dev does performance, I’d get fired. Casey and friends should spend some time on the beam in their own eyes before complaining about the mote in mine.
His fans perhaps. Casey himself I actually don’t know. He happens to work in the game industry, but he’s on the record about being in it for the technical challenges, not really the game themselves. And if we’re going to scold him about not doing what he’s preaching, we should take a look at what he actually did: the Granny animation system, whatever he did for The Witness… Similarly, I would look at what Mike Acton’s participated in when he worked at Insomniac Games, or what he managed to contribute for Unity.
About games wasting way too many cycles, I know the list is long. Of the top of my head, I personally played Supreme Commander, who I think demanded way more from the hardware than it had any right to, and Elite Dangerous (in VR), whose Odyssey update had unplayable performance for a while, and is still reputedly quite crap.
I would also like to know why Factorio and The Witness take so long to boot. Factorio devs cared a huge deal about performance (and improved it by a couple orders of magnitude over the years), and Jonathan Blow ranted in 2017 about Photoshop boot times, so I guess there must be some explanation? Especially from Blow: I’d like to know why his boot times are somehow acceptable, when Photoshop’s are not.
No sources on me rn, but iirc computer hardware is actually low on the pollution totem pole. Things like concrete manufacturing, steel manufacturing, agriculture, and cars swamp everything else.
I can concede that. Computer stuff being really polluting is something I’ve heard I don’t remember where, and trusted then. Maybe I shouldn’t have. I also recall some graphics and numbers on greenhouse gas emissions (from Jean Marc Jancovici), and for these no single sector seemed to dominate any other: we need to reduce everything all around.
I do suspect however the mindset responsible for computer related waste is also responsible for much of the waste pretty much everywhere else as well.
Where could one read about the STEPS project? The name is quite ungoogleable.
Oh, sorry, here it is:
It’s such a high bar. “Oh, this tool only doubles your productivity? Thats not an order of magnitude so not a silver bullet”
People really miss the point of the paper. Brooks’ argument was that programmers could make an order of magnitude difference and tools wouldn’t. People joke about 10x programmers but then quote the tool part of the paper.
https://brianmckenna.org/blog/softwerewolves
Are the 10x developers the werewolves, or… where do werewolves come in? I’m confused.
Edit: Oh, silver bullets… I’m still confused, though. The werewolves are… complexity in software?
you have to understand, at the time the book was written, science fiction and fantasy fandom was not yet mainstream, and realist, detail-oriented taxonomy of every fictional concept was not as far along. it feels silly saying it that way but … I get the feeling it might be hard to imagine what the world was like prior to the cultural victory of a certain segment of fandom, so I’m doing my best to describe what has changed.
this is all by way of saying that it wasn’t intended to be a fully worked-out metaphor. silver bullets are a thing that sounds cool. nobody cared that it’s specifically werewolves they’re usually discussed in relation to. that’s all there was to it.
and yes, software complexity was the obstacle being discussed
It’s in the original paper.
Ah, so I was confused because I hadn’t read the original paper being commented and was only replying to @puffnfresh’s blog post(‘s URL). Serves me right, but I apologize for wasting you all’s time with my confusion!
That always has been my assumption.
I self-host and honestly I am not sure what I am doing differently because it doesn’t seem all that hard? I’ve barely touched my config in the past 5 years other than enabling TLS and the auth stuff at some point. I probably don’t even have some of that configured quite right.
Spam is not much of a problem these days. The only special thing I do there is using unique addresses for every service I sign up for. But very few of those addresses ever gets spam. Some have gotten the occasional “newsletter” but even that goes away when I hit the unsubscribe link. I’ve only blackholed about 10 addresses in the past few years.
I do keep a vanity gmail address for some uses because it’s easier to get the spelling right than my own domain. I fetch that into my local mail handling too though, I hardly ever touch the gmail UI. Now that I think about it the gmail address probably gets most of the spam I see.
I don’t run my own DNS these days because it didn’t seem worth the hassle but the flexibility I get with email is well worth the (minimal ongoing) effort.
I use Simple NixOS Mailserver. The setup is a few DNS entries and a few lines of Nix.
https://nixos-mailserver.readthedocs.io/en/latest/index.html
Works pretty well but I haven’t configured my spam filtering properly - things are being soft rejected but they just show up as normal emails in my client. Need to figure that out.
My understanding on the whole “never host your own mail server” argument is that email is something you probably want near perfect uptime and as a regular person its hard to manage that. When I was looking at hosting myself there was a lot of talk about putting a caching server in front of your mail server to catch anything if you go down which seemed like a good solution. But I figured if I was going to do that I might as well just pay for a service.
Tbh this sounds a lot like unison
Isomorf was another similar thing.
Except that scrapscript is a scripting language for a decentralized social network, and has authentication and a payment system. (See link in my other post.)
When I read some of the Unison docs, I got the strong impression that names don’t matter, entities (types, values) are identified solely by their hash, and their name is not part of their identity.
In scrapscript, names do matter, it doesn’t have these Unison naming semantics. A scrap can have a name that is consistent across multiple versions of the scrap, which allows you to upgrade code to the latest version. I don’t know how this works in Unison.
The social network, authentication, and payment systems are the “killer apps” that I want to build with the language, but aren’t part of the core architecture :)
I’m not super familiar with unison either, but your description of scrapscript was spot on :) thanks!
IIUC, in Unison, names “matter” but are aliases to those content hashes.
Unless you were talking about structural / unique types. For structural types, the names don’t really matter, but they do for unique types.
This is not a bad thing! I’ve seen people new to Haskell understand higher kinds very easily, over and over. I even wrote about how I constantly see higher kinded types benefit learning:
https://brianmckenna.org/blog/higher_kinded_parametricity
I know the point is more about type-classes and yes, they need to come after the basics. That’s the problem with making a useful Prelude - what you want to use in production is not always the first thing you want to learn. This is not really a problem specific to Haskell.
I don’t know how that’s useful. Why do people care? Are they doing complexity analysis of functional algorithms? I don’t stress that much when teaching, usually it’s something quickly before moving onto the next exercise “btw that’s O(1), because laziness”
I have people in classes who insist that Haskell functions take more than one argument and it creates a lot of problems for them. Once they eventually understand and agree, things become easier. So I always insist, from the first time we write a function, so that things are cleared up as soon as possible.
I agree with you, but I also think the author makes a very good point. The UX of
mapandfold[lr]is inconsistent.Ideally, the Prelude should contain the most useful (generalised) versions, while type-specific modules contain specialised variants. So
Data.Listwould exportThis would also open an opportunity to introduce these concepts. “Hey,
List.mapandMaybe.mapandEither.mapkinda do related things, don’t they.”Instead,
mapandfold[rl]are imported directly fromData.List. By a disappointing historical accident, we get a specialised map and a generalised fold.Yeah
mapbeing specialised is definitely a problem. Prelude is really serving nobody well. It’s not good to use professionally and definitely not good to use for teaching.It’s wonderful to see this finally release! Rules being fully defined in Starlark is great for integrating new languages, and seeing Shake’s influence here after so many years makes me happy.
How so?
tl;dr FB hired the author of Shake to productionise his research into build systems.
From the blog post: (emphasis added)
Also within the blog post, the single incremental dependency graph link goes to Implementing Applicative Build Systems Monadically. In that paper, it describes Buck as a “monadic” build system and notes:
The blog post also notes that Buck2 “[follows] the model in the paper, ‘Build Systems a la Carte’”. That paper was co-authored by the creator of Shake and one significant section within is entitled “Experience from SHAKE.”
Finally, as @puffnfresh noted, Neil Mitchell (author of Shake) is a co-author of Buck2.
Neil Mitchell is the author of Shake.
Is it natural for me to have skeptical feelings for something like this?… The reasoning being, there has already been so many man hours poured into FP languages, by various experts, and it hasn’t been “paradigm changing”. On top of that, optimizing FP seems to take a lot of work and knowledge. You’re telling me they’ve got all the experts, and hammered something out (which itself sounds inadequate to achieve something great here), which is better than GHC or similar? It’s incredible if true. Like insanely incredible. I would love for someone to explain to me why Verse has a chance at all.
It definitely it has been, look around! Languages have been copying lots of things from FP tools. Algebraic data types, higher-order functions everywhere, pattern matching, immutable by default, futures and promises, generators, etc. These features are many decades old in Haskell, OCaml, etc.
FP and these experts have already changed the paradigm.
No, it’s completely different. Verse is absolutely definitely not Haskell and never will be. The goal of GHC is totally different to what Epic Games is doing with Verse.
People in Epic Games have been thinking about this for decades, recently invested a bunch of time, hired a bunch of experts, made a language, decided to rewrite one of the most popular games in the world using the language they just made. So yeah, I think “Verse has a chance” at doing what it was exactly designed to do. Why not?
Verse’s current implementation compiles to Unreal Engine’s Blueprint VM, so I imagine the integration into Fortnite was simple and not a complete rewrite. The
/Fortnite.com/package contains a bunch of Fornite specific APIs that FII out to Fortnite code iiucYes, the initial integration uses existing tools, but from Tim Sweeney (just after 33 minute mark):
I really hope the Open™ implementation is implemented on top of something a lot more efficient. Blueprint is horribly wasteful. The VM and bytecode has been largely the same since the UnrealScript days - for the curious, it’s essentially a glorified tree-walk interpreter (but with compact bytecode instead of heap-allocated nodes.) It really makes me wonder why Epic decided to continue using the same inefficient tech for UE4’s Blueprint.
The guy who presented the language on stage built a lot of JavaScriptCore / WebKit internals, so they certainly have the right people to build a very fast VM.
I should’ve been clearer in what I meant, but this is definitely true! Definitely true… What I meant was any particular FP language hooking everyone into the FP paradigm (or changing the FP paradigm to a degree of making it globally attractive).
Secondly, if it’s already changed then by the right people in various ways - what the heck does Verse have to offer? I think your insight further drives home the idea that Verse is just a “fun company project”, at least to me!
Thanks for the comment :)
To be pedantic, generators are from CLU, which is where a lot of languages got them. I’d have to check my research notes but I think pattern matching is from Prolog? Definitely popularized by FP though.
Verse is a domain specific scripting language for UEFN, the Unreal Editor for Fortnite. It’s the language used to describe game logic. The goal is to allow game logic to run in a distributed fashion across many servers, where there could be a hundred thousand players in the same game.
This is a pretty extreme requirement, I don’t actually know how you can reasonably do this in a conventional language. It makes sense that they’ve chosen a logic programming language, because that allows you to express a set of requirements at a high level, without writing a ton of code to specifically take into account the fact that your logic is running in a distributed fashion. According to what I’ve read about Verse, the existing logic languages they looked at have non-deterministic evaluation, which make code unpredictable and hard to debug. So Verse is a deterministic logic language, and that is called out as a specific contribution in the research paper about the Verse calculus.
As for whether Verse will “succeed”. The only job it has to do is to describe game logic in UEFN, and the criteria for success is that it scales to allow very large game environments, in a way that Epic hasn’t succeeded in doing using C++. There’s lots of money available and they’ve hired the best people. so I think they have a good chance of succeeding.
Does Nix still have value if most of your stuff is statically linked? Memory and disk space are cheap, so more and more stuff seems to be going that way. Admittedly, it’s not as elegant to have lots of fat binaries around.
Yes! You still need to specify your build system, versions of the libraries that get statically linked in, etc. Dynamic linking is just a tiny piece of Nix.
What constitutes “your stuff”? The vast majority of what I’m using NixOS for is configuring a set of virtual machines running all sorts of software I didn’t write personally, and I have no idea whether most of it is statically linked or not - what does it even mean for, say, my random manga-reader app running in a Docker container and written in Crystal to be statically-linked anyway? Nonetheless, I need to configure it correctly or it breaks and I can’t read my manga anymore, and I’d like that configuration to be version-controlled, not break if I change something unrelated it depends on like my database machine’s address, etc.
Yeah lots of value: reproducibility, consistent build tooling, code reuse. The dynamic linking that nixpkgs configures is such a tiny part of using Nix, I really don’t care about it.
I’d like to say the Nix is ‘Hybrid Linking’ , Artefacts / outputs / Binaries (what ever you call them) , look like they are dynamically link, but nix-build’s them so they are static.
So it is neither Dynamically linked like every other Unix/Linux out there, it’s neither statically link (very popular with golang developers) …it’s both.
Some things are shared when possible, other things are not shared (like having two versions of program or dependency to keep the software ‘operational’).
Nix treats ‘the package manager’ as computer memory, it is allocated (malloc) , used, and then can be free (free)
https://www.tutorialspoint.com/c_standard_library/c_function_malloc.htm
to the effect , once a ‘piece of memory’ (a program or library) are no longer need it is simply garbage collected.
It’s really that simple!
https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)
it’s just treats the package manager (nix) and the operating system (nix/OS) as a program.
The fact that nix ‘builds’ the linux kernel, and put libc together, and then add all the other scripts and programs to make a ‘distribution’ is purely coincidental.
It could as easly build a freebsd kernel, or a solaris one, or anything in the past or the future.
It’s a build a system.
‘If you gave it a pile and bricks and some cement, it would nix-build a wall for you’ and then simple vanish.
Hence Nix/OS is a meta distribution.
https://github.com/musnix/musnix
https://discourse.nixos.org/t/snowflakeos-creating-a-gui-focused-nixos-based-distro/21856
Please feel free to fork it / overlay it and build your own.
Guix also uses the same paradigm ….. good idea’s…. get copied, and rightfully so!
Yes, in fact I think Nix solves a particular problem with static linking very well:
When you do need to upgrade a library everywhere, for example to fix a security issue, you need to rebuild everything using that library. Nix makes this really easy and automatic.
Maybe the workaround is to pay for the services you need, so you won’t have to keep moving from one investor-bilking scam to the next?
the thing is, even if you pay for them, they never seem to be satisfied. it’s also why the whole “if you don’t pay then you are the product” saying is basically false… (selling customer data is even more profitable when customers pay them to sell it)
Some people are just bad actors. But when you find a good provider, they’re much more likely to want to keep your data for you if they’re profiting from their work.
How could we know about Docker though? It’s not like the service is free, they have always had a revenue stream, and it’s the industry standard. You see that, look at the pricing model, and … what do you do when you see that their free tier happens to be enough for your use cases? Do you choose a different provider just because the free tier happens to be good enough and you don’t want to use something you’re not paying for?
Not enough of one. Docker’s lack of sustainable revenue has been discussed for years. Their business plan from the outside looked like ‘give away great tools, get bought by someone who needs them’.
No, OCI is the industry standard, Docker is a company that sells the most polished tools for dealing with the industry standard.
You look at their free tier and see that something free is going to have to go away if they aren’t bought by GitHub or similar (and then it may still go away, just be bundled with something you might already be paying for) and avoid making plans around it always being there for free.
I’m pretty sure other OCI implementations are significantly less used than Docker? It certainly was a few years ago when a lot of the decisions to use Docker were made.
“Industry standard” doesn’t mean a formal standard managed by a standards body.
Okay, and how do you do that when there’s no indication at all about what a potential future price might be? Do you put your finger in the air and guess? What if it ends up being significantly more expensive than you guessed?
When you depend on a free offering from a company with an obviously unsustainable business model, you assume it will go away at some point and have a plan B. If they move it to a paid offering that is cheaper than your plan B, then you may choose to use that instead, but you should always assume it will disappear. Possibly with no notice if the company goes bust.
It may well be; but, if your usecase has the hidden requirement of dependability/continuity of service, you need to price that in.
There ain’t no such thing as a free lunch, and never has been.
But how do you price that in when the industry-standard service is free for your use-case and they’re not giving any indication of what your use-case might cost in the future? How do you price in a completely unknown potential future price which may or may not materialise?
The price depends on what strategy you use to manage the risk
You can’t know, and nobody sensible expects you to. That’s what happens when you outsource something, whether you pay for it or not.
Paid services disappear constantly too! Company collapses, pivots, acquihires, etc.
https://ourincrediblejourney.tumblr.com/
Until they decide that they need more.
Gitlab going from $4 -> $29 per user per month for my same usage in 2 years for example.
The solution is probably a service that makes hosting free but charges for downloads. For commercial products, increasing downloads increases revenue and so it’s completely fine to increase the costs to the publisher as the number of downloads goes up, to cover the traffic. For community-developed projects, most users pay nothing.
GitHub is probably in a good position to keep this kind of thing free because they want to monetise the things that pull containers. In particular, Actions and CodeSpaces are both very large consumers of containers and benefit from making it easy for people to expose a large number of container images to their customers.
My shower thought on Nix the other day was “they’re trying to boil three different oceans, and I really only want them to boil one.”
can you elaborate on what this means?
boil the ocean
Which is the one you want?
Packaging binaries. The other oceans are cool too, but I’m sort of in Julia’s camp of just wanting to install some stuff.
The Nix programming language is a purely-functional programming language, not a configuration language. It’s used to program the Nix package manager in the same way as any other programming language would. It just happens to be good at configuration :^)
You can get Nixpkgs to run on any other Linux distro (and on Mac too), so you don’t need to boil that third ocean if you don’t want to.
Can we please please stop this
we’ve had this discussion before, it always comes full circle:
What is your solution to improve upon this shell piping ?
You can only argue that downloading it to disk and then executing can leave a trace on the disk, so you can inspect the malware later on. We’re still downloading debian ISOs straight from the host, not that big of a difference. And I bet no relevant percentage checks the signatures or checksums.
I sometimes see:
This way the website can’t detect whether it’s being piped to
sh, and you don’t get screwed over if the connection fails halfway through.Downloading it first or putting it at some reliable hosting (where you can’t change the download for specific people by random just through compromised developer hosts) is definitely one step better. But it doesn’t change much in terms of the core complaints: Running shell scripts from the internet.
Well, pretty much all free software boils down to “Running [executable code] from the internet”.
The question is whom you need to trust. The connection is HTTPS and you need to trust both, your TLS setup and determinate.systems anyway to use this installer, so not much is left.
Getting a an external signature/checksum would guard you against a compromised web server, but only if you can get that checksum out of band or from multiple, independent sources.
Agree, it’s missing a
sudo!What’s a better alternative?
Just stop…everything https://github.com/kelseyhightower/nocode
Sure—just as soon as all software is verifiable, which projects like this are an important step towards :)