Threads for nalzok

  1. 7

    This seems to be the same algorithm as Shamir’s Secret Sharing.

    1. 2

      That’s a fantastic observation! Connections like that tickle my brain in the most wonderful way, so thank you for that!

      1. 16

        Oh, hey, I wrote this.

        1. 12

          It’s literally the best article on Docker I have ever read! It explains the anatomy of Docker in great detail and gives a nice overview of its ecosystem. It took me forever to understand the relationship between docker, containerd, and nerdctl before reading the article, but everything seems so clear now. Thank you!

          Kind of an unreasonable request, but do you have any plan to write a similar article on Kubernetes (and its friends like K3s and Rancher)? Container orchestration systems are complex behemoths so I believe many people will benefit from your skillful dissection :)

          1. 4

            Thanks so much!

            Re: Kubernetes follow-up, you’re not the first person that’s requested that; I’m considering it but it’ll probably take me a while to actually put it together, since I have much less hands-on experience with that side of things.

              1. 1

                This one is informative as always (I finally understand what a Kubernetes distribution is). I just submitted a story so that crustaceans can discuss it!

            1. 1

              By the way, what in your opinion leads to Docker’s wide adoption? As you said, its predecessors already have bind mounts, control groups, and namespaces, so apparently Docker’s only novelty is that it encourages “application containers” instead of “system containers”, but I guess it’s also possible to create lightweight LXC images by removing unnecessary stuff like systemd and the package manager. Does Docker’s philosophy give it any advantage over LXC… or maybe Docker is successful simply because it’s backed by Google? :)

              1. 8

                I think it was really all about the user and developer experience. It doesn’t take much effort at all to run a Docker container and very little additional effort to create one. Docker made sure they had a library of popular applications pre-packaged and ready to go on Docker Hub. “Deploy nginx to your server with this one command without worrying about blowing up the rest of the system with the package manager, no matter what distro you’re using” and “Take that shell script you use to build and run the app and reproducibly deploy it everywhere” are very attractive stories that were enabled by the combination of application containers, Dockerfiles, and Docker Hub.

                1. 2

                  That and crossplatform. Even back when we were still on Vagrant+Virtualbox I stopped using lxc as the only Linux user in the team because I had to duplicate some effort. Then with docker it just worked on Linux and Macs.

                  1. 2

                    Cross-platform is a bit of a stretch to describe Docker. The Mac version uses a port of FreeBSD’s hypervisor to run on top of the XNU kernel and deploys a Linux VM to run the actual contain, so you’re running your app on macOS by having macOS run a Linux VM that then runs the application. That’s very different from having your app run natively on macOS.

                    The underling OCI container infrastructure supports native Windows and (almost supports) FreeBSD containers, but you can’t run these on other platforms without a VM layer.

                    1. 1

                      I think you’re talking about something else.

                      When we want to run “a service using e.g. the Debian RabbitMQ package” then a docker container with a debian image with the rabbitmq package is 100% “usable cross-platform”, from the developer standpoint of “I have this thing running locally”. It exactly does not matter what the host is, you just run that image. I don’t care about “native or not” in that scenario, just that it runs at all. Purely an end user point of view.

            1. 2

              I’m surprised to see that := cannot be rewritten in other statements. Isn’t a := ... equivalent to tmp = ...; a = tmp?

              1. 6

                := can be used in lambda’s, wheras = cannot, and as you cannot re-implement lambdas in Python, you’ll need := to achieve full feature set.

                1. 3

                  Why can’t lambdas be reimplemented as a simple “return foo” function defined right before the lambda usage? Something about different scope?

                  1. 2

                    I’d say it’s more about properties. Their __name__ is <lambda>, you can’t pickle them, etc. I’m not sure if you can re-write lambdas to functions and bring those properties back. I guess this is a case about semantics, but I think that these properties are important to keep, since there are programs that do care about it, and the author’s post was about any Python program in general.

                    1. 3

                      The OP doesn’t have async or comprehensions. It’s much harder to fake those. All you have to do for the lambda thing, if you really think it’s important that they have a meaningless name, is rewrite a few properties and make a replacement code object (which is immutable and annoyingly also has a name). Child’s play!

                      In MVPy:

                      def lit(**kw):
                          return next(iter(kw))
                      
                      def ga(x, /, **kw):
                          return getattr(x, next(iter(kw)))
                      
                      join = ga(str(), join=1)
                      
                      def concat(*a):
                          return join(a)
                      
                      lambda_with_angles = concat(chr(60), lit(lam=1), lit(bda=1), chr(62))
                      
                      def lambdafy(func):
                          code = ga(func, __code__=1)
                          new_code = ga(code, replace=1)(co_name=lambda_with_angles)
                          setattr(func, lit(__code__=1), new_code)
                          setattr(func, lit(__name__=1), lambda_with_angles)
                          dot = chr(46)
                          qualname_bits = ga(ga(func, __qualname__=1), split=1)(dot)
                          ga(qualname_bits, pop=1)()
                          ga(qualname_bits, append=1)(lambda_with_angles)
                          setattr(func, lit(__qualname__=1), ga(dot, join=1)(qualname_bits))
                      
                      x = lambda a: a
                      def y(a):
                          return a
                      lambdafy(y)
                      

                      edit: I guess if I’m using Python 3.8’s / syntax I can use CodeType.replace too

                      1. 5

                        Async is actually fairly easy. asycnio existed even before the async/await syntax was introduced. Async functions (in 3.8, before it was different) return an object with an __await__ method that returns an iterator, that the async runtime(usually asyncio, but others exist) iterate through, and get the requests for IO from, and sends back the results to the object via a send() method. await exists for connecting the iterators together conveniently, though it was (and still is) possible to do so via yield from. The parts in the language were there for a long time already, it just took a while to put it together and make it comfortable.

                        Comprehensions can usually be trivially rewritten into generators. A simple case like (<item> for <identifiers> in <source> [if <condition>]) can be rewritten into

                        def gen():
                            for <identifiers> in <source>:
                                if <condition>:
                                    yield <item>
                        

                        It of course gets a bit more complicated when you get into multiple for’s in a comprehension, but that is still not too complicated to do.

                        As for lambdas, your implementation might be workable, but that’s too much details for me to look into right now.

                        1. 1

                          Yes, it’s easy to get the important behaviour of each, but for lambdas you were asking for something that’s indistinguishable. I believe my solution for lambdas yields something that is indistinguishable from a lambda constructed the usual way; I think it would be harder to do this with async or comprehensions.

                          1. 1

                            I don’t think it is too difficult to make async to work in a way that is indistinguishable from the way it works right now. As for comprehensions, the objects are just standard generator objects, nothing fancy there to make it difficult.

                  2. 2

                    I think you can write any Python with lambdas without lambdas by repeatedly replacing a top-level lambda (i.e. one that isn’t nested inside another lambda) with a named function. The trick is to be sure to name other expressions and intermediate results you encounter along the way, so that you don’t reorder the evaluation of the lambda definitions you’re rewriting with respect to anything else; example below.

                    I think this is hard for a human to do systematically without errors, but a compiler could do this sort of thing in its sleep, and a human could probably do this for all reasonable code, because reasonable code doesn’t tend to rely on effectful lambda definitions.

                    a = [7]
                    ((lambda f: (a.append(a), f)[1])(lambda f, x: f(x)))(lambda x, _=
                     a[1].__setitem__(0, a[0] * 19): x(), lambda y=tuple(a): print(y[0])) # 133
                    
                    a = [7]
                    def fn_1(f):
                        return (a.append(a), f)[1]
                    fn_1(lambda f, x: f(x))(lambda x, _=a[1].__setitem__(0, a[0] * 19): x(),
                     lambda y=tuple(a): print(y[0])) # 133
                    
                    a = [7]
                    def fn_1(f):
                        return (a.append(a), f)[1]
                    def fn_2(f, x):
                        return f(x)
                    var_1 = fn_1(fn_2)
                    var_1(lambda x, _=a[1].__setitem__(0, a[0] * 19): x(), lambda y=tuple(a):
                     print(y[0])) # 133
                    
                    a = [7]
                    def fn_1(f):
                        return (a.append(a), f)[1]
                    def fn_2(f, x):
                        return f(x)
                    var_1 = fn_1(fn_2)
                    def fn_3(x, _=a[1].__setitem__(0, a[0] * 19)):
                        return x()
                    def fn_4(y=tuple(a)):
                        print(y[0])
                    var_1(fn_3, fn_4) # 133
                    
                1. 1

                  I wonder how it compares to other machine-readable formats like Protobuf? See also this page: https://wiki.alopex.li/BetterThanJson

                  1. 1

                    One of the features of this webserver is that chroots to a directory. Meaning that, to the web server, anything before the given directory (/var/www by default), does not exist. So if a vulnerability is found, the attacker can’t do much things, as the attacker can’t go beyond /var/www.

                    Apparently, chroot is not a security feature on Linux. Is that true on *BSD?

                    1. 1

                      As chroot(8) caveats sections states:

                      chroot should never be installed setuid root, as it would then be possible to exploit the program to gain root privileges.

                      so chroot(8) like the article states it is a hardening feature even on OpenBSD.

                    1. 10

                      Why is the databases tag insufficient in your opinion?

                      1. 1

                        I find it to be too generic, similar to the programming tag. Looking at the last 2 weeks of stories, 6 out of 13 stories under databases where about sqlite. Giving sqlite its own specific tag might make it easier to find sqlite related posts and could potentially cut the posts in databases to half, making it easier to find non-sqlite related posts for those looking for them.

                        1. 1

                          Tags are for filtering out, not for finding stuff.

                          1. 11

                            Tags are for both. That’s why you can subscribe to emails for specific tags.

                            1. 5

                              Tags are meant for filtering things out, but they surely can double as a way to filter for something you are interested in. Otherwise, why do we have helpful URLs like https://lobste.rs/t/databases?

                              1. 2

                                Tags also let you know what a story is about. If a story title is “I built a X” but it’s tagged “go” “sqlite” you know how they built it before you click. Without that, you might not care enough to click.

                                1. 1

                                  The database posts I care about are related to sqlite. With a sqlite tag, I’d filter out databases and just keep sqlite, which fits your described use case. I’d love to see this tag.

                            1. 30

                              I’m slightly against. databases covers it. mysql or postgres and so on don’t appear to need one.

                              The main thing is I haven’t seen anyone at all get annoyed at the amount of material being posted here about sqlite. The nominal main use for tags on lobsters is for filtering out submissions you don’t want to see.

                              1. 4

                                I am annoyed about all the SQLite, fwiw.

                                1. 1

                                  Being against the tag is one thing (I don’t care personally) but why annoyed about SQLite? It’s an amazing body of code.

                                  1. 9

                                    I’m not against the tag, I’m for the tag. I’d like to filter it.

                                    I’m sure it’s an amazing body of code, but I don’t enjoy the articles talking about how amazing a body of code it is. People talk about how great SQLite is as if thinking SQLite great is a fringe viewpoint, which I find tedious, and the debates about to SQLite or not are frustrating to read.

                                    I’m sure other people like the articles and the discussions they spawn, and I don’t want to feel tempted to go be grumpy at them there. A tag would just save my having to grump internally.

                                  2. 1

                                    huh okay fair enough

                                  3. 4

                                    Sure, sqlite falls under the umbrella of databases, but I cannot see how can adding the tag interfere with anyone’s use case. As I am writing, this post has received 24 upvotes, which means having it would benefit at least 24 users.

                                    From my understanding, you only think this is unnecessary but not harmful, whereas others are finding it necessary. This is not really an argument against the proposal IMO.

                                    The nominal main use for tags on lobsters is for filtering out submissions you don’t want to see.

                                    Git is nominally a distributed VCS, but most people have no problem with a central repo. I really want to see the hit statistics of URLs like https://lobste.rs/t/databases. That’s the only way we can know how users are actually using the tag system.

                                    1. 4

                                      Right – tags have other uses than filtering submissions out, regardless of the original reason the tag feature was implemented. Not only can one browse stories with a certain tag at tag pages, one can also subscribe to a tag via RSS and search for tagged stories.

                                      From my understanding, you only think this is unnecessary but not harmful, whereas others are finding it necessary. This is not really an argument against the proposal IMO.

                                      In general, adding any new tag is harmful in some way, though the benefits may outweigh the harms. The harms of adding a new tag:

                                      • It makes it harder to find other tags on https://lobste.rs/tags.
                                        • This interferes slightly with all the use cases for tags listed above, as you must learn of the existence of a tag before you can use it.
                                        • It makes it harder to understand what stories are acceptable on Lobsters, as submitters see the instruction “If no tags clearly apply to the story you are submitting, chances are it does not belong here.”
                                      • It makes it harder to tag a new submission with other tags.
                                        • The new tag will appear in the list of options, distracting some users who are submitting stories for which the tag does not apply.
                                        • The new tag may replace the tag that was previously autocompleted after typing a partial string like “s”.
                                      1. 1

                                        I did quality this with the word “slightly”.

                                        Git is nominally a…

                                        I brought it up because “are people itching for a way to filter it out?” is the official criterion here.

                                    1. 2

                                      As pragmatic as Go is, I always wonder why it insists on making unused variables an error instead of a go vet diagnostic.

                                      1. 3

                                        That sounds slightly like tuple spaces ( https://en.wikipedia.org/wiki/Linda_(coordination_language) ).

                                        1. 1

                                          Exactly! That’s it. Thanks for the pointer!

                                          This is the story I was talking about: https://lobste.rs/s/lvbxci/tuple_spaces_good_ideas_don_t_always_win

                                        1. 0

                                          Just want to mention that you can also put a text file in /usr/share/misc/http.status which can be queried with grep.

                                          1. 0

                                            I prefer scrapping MDN because they have up-to-date documentation and great examples.

                                            1. 23

                                              What is the most cursed thing you have used redbean for or seen it used for?

                                              1. 4

                                                This sounds amazingly useful. With what aspect of it are you least satisfied?

                                                1. 6

                                                  Thanks! Redbean has baked-in SSL support. I used MbedTLS and I love it, because it’s so much smaller and simpler than OpenSSL and BoringSSL which have huge dependencies like STL and Perl which would have needed to be ported beforehand. However its main drawback is it doesn’t have TLSv3 support. This makes me sad because it causes redbean to have a few extra milliseconds of connection latency if it’s used for self-contained public-facing serving. That’s something I intend to improve. However it’s not that big of an issue, since it’s very easy to use redbean in tandem with something like stunnel which can bolt on the SSL separately. But that only applies over long line internet connections. For local use cases (redbean’s primary and original intent) where CPU utilization is a greater factor than the speed of light, redbean’s SSL goes faster than any of the other stacks I’ve evaluated. For example, I haven’t posted SSL benchmarks on the website yet because wrk (Go) uses 2x as much CPU to establish ECDSA connections as redbean. So once it has the TLSv3 support, it’ll be a sight to behold.

                                                  1. 1

                                                    I thought wrk was written in C! Is the code at https://github.com/wg/wrk/tree/master/src not the correct wrk?

                                                2. 3

                                                  This is so f’in cool.

                                                  Question: I get the “serve static files” part, and “I can do dynamic stuff with Lua” part. How does the sqlite part work? Can I literally write a full-fledged web app in lua which is, say, saving new user records using sqlite, and those are just getting written to the filesystem in the folder redbean is running in?

                                                  1. 5

                                                    Indeed. Here’s the basic idea for how you’d implement a SQLite web service with redbean. https://gist.github.com/jart/f9b1a7ebe5c7ebea3abe8f98125c692c We’re in the process of adding features so that redbean can use its own .com file as the database. We have one function for just that, called StoreAsset() but right now SQLite is more stable.

                                                    1. 2

                                                      It would be cool to pair redbean with Litestream to get streaming backups of the SQLite db.

                                                  2. 2

                                                    I’m a huge fan of your work. Also fare’s, cool he invited you. Cool you are here.

                                                    1. 3

                                                      Thank you! Lobsters in a nice community I regularly visit. I was so happy to see that someone linked to redbean.

                                                    2. 2

                                                      How does it handle sqlite queries with database modification, e.g. INSERT / UPDATE / DELETE ?

                                                      1. 4

                                                        redbean simply embeds sqlite with lua bindings thereby enabling you to handle them in any manner you want. It’s expected that the user knows SQLite and is aware of its caveats. For example, redbean is a forking web server and SQLite file descriptors should never cross fork() boundaries (if the database is mutable). This isn’t a problem if you’re using Lua Server Pages. However if you’re using the faster OnHttpRequest() handler then you need to make sure you don’t call sqlite3.open() from the global scope and instead create it lazily as requests are handled. That means your database descriptor and prepared statements will persist across requests on a per-connection basis. This is nice since SQLite is significantly faster than other databases at doing this. Plus you get a nice process sandbox. So in case something goes wrong, like ASAN detects a bug, it’ll only impact one serving session. We’re also working on higher-level frameworks to abstract a lot of the handling aspects. Lastly I’ve also been focusing on creating single-file object store technology (see StoreAsset() and LoadAsset()) that should provide the fastest possible and most easily redistributable model of data storage under a forking model.

                                                        Example https://gist.github.com/jart/f9b1a7ebe5c7ebea3abe8f98125c692c

                                                        1. 1

                                                          However if you’re using the faster OnHttpRequest() handler then you need to make sure you don’t call sqlite3.open() from the global scope and instead create it lazily as requests are handled. That means your database descriptor and prepared statements will persist across requests on a per-connection basis.

                                                          Sorry, but I’m not sure what that means. If each request gets its own SQLite descriptor in OnHttpRequest(), why would “database descriptor and prepared statements persist across requests”?

                                                          1. 2

                                                            Because redbean supports HTTP keepalive pipelining and Lua variables are global unless you declare you function local. Forking happens when a TCP connection is received. That connection may send many HTTP messages. Each message for that connection is handled by the same process. If you’ve defined a hook like OnHttpRequest then that means any variables you use, unless you use local, will persist between messages so you can memoize operations on a per-session basis. When the TCP connection closes, it all gets garbage collected by the operating system when the process exits.

                                                            1. 1

                                                              I see. Does that imply we can call sqlite3.open() from the global scope in Lua, since by that time we are already in a fork()ed child process for that TCP connection? You said we shouldn’t do that and should “instead create it lazily as requests are handled” so I am a little confused.

                                                              1. 2

                                                                If it’s a foo.lua file in your redbean (i.e. a Lua Server Page) then you’re 100% OK. Those files are loaded and re-run on each request. There’s only one file that’s special and it’s .init.lua. That file is run before redbean calls bind() and no forks have been created yet. Anything you do in the global scope of that file, like import modules, will end up defining the main process “template” from which all forked processes are created. Therefore, if you create your SQLite object in the global context of that file, it needs to be strictly read-only. If it is, you can get like 800k RPS performance on read-only SQLite files. But any SQLite files you intend to mutate, it needs to happen after fork, otherwise corruption is possible. It also goes without saying, that you can have multiple SQLite file descriptors open simultaneously! So for example, if you built an Internet forum, all your years-old posts could be moved to immutable read-only SQLite files, for ultra-fast querying due to the pre-fork setup.

                                                                1. 1

                                                                  That’s awesome! I have an additional question: is it theoretically possible for redbean to support LuaJIT in the future? I’m not sure if the magic tricks you did with actually portable executables would interfere with JIT compilation/execution.

                                                                  1. 2

                                                                    It’s a work in progress. See https://github.com/ahgamut/LuaJIT-cosmo

                                                    1. 1

                                                      Interesting project! However, I find the following snippet a little confusing

                                                      a_np = np.random.randn(128, 128)
                                                      b_np = np.random.randn(128, 128)
                                                      a = lt.Tensor(a_np).to(s.m, s.k)
                                                      b = lt.Tensor(b_np).to(s.k, s.n)
                                                      
                                                      c = (a * b).sum(s.k)
                                                      

                                                      Which parameters am I tuning? Clearly I cannot change s.m and s.n because these are my input/output specification, but I cannot tune s.k either since s.m * s.k and s.k * s.n are fixed to be the number of elements in both matrices.

                                                      In addition, I assume by (a * b) you are actually matrix multiplication (it cannot be element-wise multiplication, because the number of elements in a and b may not match when, for example, you are multiplying a 2 * 3 matrix with a 3 * 5 one). In that case, I think the notation (a @ b) would make more sense to most numpy users, because (a * b) does element-wise multiplication in numpy.

                                                      1. 2

                                                        yep, the sizes of s.k, s.m, and s.n are all fixed so those can’t be tuned. However, the loops needed to execute this computation (e.g. looping in different orders) can be tuned. I wrote up a quick notebook to clarify: https://colab.research.google.com/drive/1Uf0DCdyUXLGica6vjb3rG-ntD4b_ZWUD?usp=sharing

                                                        (a * b) is actually point-wise multiplication. You’re right that the elements don’t match, so there’s implicit broadcasting (much like a.reshape(128, 128, 1) * b.reshape(1, 128, 128) in numpy). This is immediately followed by a reduction (sum) over the shared dimension s.k. In numpy this is inefficient, because the whole 128,128,128 array needs to be created. In loop tool you can order the loops to skip that big intermediate array entirely. It allows you to craft complex operations without worrying about performance ahead of time (as they can always be made fast later).

                                                        1. 4

                                                          Thanks for the clarification! Just want to add that (128, 128) * (128, 128) is probably not the best example for matrix multiplication. Maybe you can switch it to something like (200, 300) * (300, 500) to show a more general case, and more importantly, the difference in dimensions can help readers like me to identify which one is which :)

                                                          1. 3

                                                            That’s a great suggestion - updated the post and the notebook with unique sizes!

                                                      1. 2

                                                        I learned about GA144 from a submission posted a few days ago, and the chip looks really fun! Here is another great source of information about it: https://www.forthseen.de/

                                                        1. 26

                                                          I have a “no language flamewars” rule but I’d like to make an exception because this reads a lot like a similar “defense” I wrote about C++ way, way back (2007-ish?) and which I have repented in the meantime. Not due to more exposure to language theory but due to more practical exposure to large codebases, in C++ and other languages.

                                                          I think the author takes some points for granted, and they’re not universally correct. Some of these include:

                                                          1. That language complexity and application complexity are not just separable, but that more of the former means less of the latter.

                                                          To illustrate a counterpoint (“show the door before showing the key”) with a problem that I’ve been banging my head against for weeks: safely sharing data between IRQ and non-IRQ contexts in Rust is… not fun. (see https://github.com/rust-embedded/wg/issues/294 or https://github.com/rust-embedded/not-yet-awesome-embedded-rust#sharing-data-with-interrupts for some background discussion). Even the best attempts at doing it idiomatically have been absolutely horrifying so far. Debugging any code that uses them is mind-boggling, and exposes you to virtually every nuance of the underlying ownership and concurrency model of the language, many (most?) of which either don’t apply at all, or don’t quite apply the way the underlying model thinks they might. Most of this complexity stems from the fact that you’re not just building abstractions on top of the ones that the hardware provides, you’re also building on top of (or reconciling them with) the ones that the language provides, and the other abstractions built in your code. You’re effectively building on three mutually-incompatible foundations. Kind of like building a bridge on pontoons that are themselves floating in large tanks placed on other pontoons on a river.

                                                          More generally: except for the most trivial of bugs, which stem only from oversights (forgot to call an init function, forgot to update some piece of state, whatever), most bugs are a mismatch between your understanding of what the code does, and what it actually does. Understanding the language, in all of its nuances, is a pre-condition to figuring out the latter. You rarely get to reconcile the two without understanding the language, except by sheer luck.

                                                          1. That you can build incrementally more powerful languages only by incrementally adding more features

                                                          There’s a tongue-in-cheek law that I’m very fond of – Mo’s Law of Evolutionary Development – which says that you can’t get to the moon by climbing successively taller trees.

                                                          A hypothetical language like the one the author considers – exactly like Python, but without classes – would obviously suck because a lot of the code that is currently very concise would get pretty verbose. Presumably, that’s why they added classes to Python in the first place. However, that doesn’t mean a better abstraction mechanism, which could, for example, unify classes, dictionaries, and enums (currently provided through a standard library feature) wouldn’t have been possible. That would get you the same concise code, only with less complexity.

                                                          It’s probably not the best example – maybe someone who’s more familiar with Python’s growing pains could figure out a better one – but I think it carries the point across: not all language features are created equal. Some of them are really good building blocks for higher-level abstractions, and you can express many things with them. Others not so much. Having more of them doesn’t necessarily make a language better at expressing things than another one.

                                                          Edit: Or, to put it another way, there is such a thing as an “evolutionary leap”. One sufficiently powerful mechanism can make several less powerful mechanisms obsolete. Templates and template functions, for example, made a whole class of C hacks (dispatch tables, void pointer magic) completely unnecessary.

                                                          1. That language complexity is equally distributed among language users.

                                                          Many years ago, when C++ was more or less where Rust will be in a few years, you’d hear things like “oh yeah, we’re a C++ shop, but we don’t use multiple inheritance/exceptions/whatever”. Because of the sheer complexity of the language, very few people really knew, like, all of it, and most of them were either the ones writing compilers, or language evangelists who didn’t have to live with boring code for more than a consulting contract’s length. This led to the development of all sorts of “local dialects” that made integrating third-party code a nightmare. This is, I think, one of the reasons why Boost was so successful. For a long time, there were a lot of things in Boost that were much better implemented elsewhere. But getting these other implementations to work together – or, actually, just getting them to work with your compiler and compiler flags – was really unpleasant.

                                                          1. That there are sufficiently few abstractions in the world of software that you can probably handle them all at the language level

                                                          I don’t have proof for this, it’s just an opinion. IMHO there are so many abstractions out there, many of them specific to all sorts of problems, that the chances of a language ever encompassing a sufficiently large set of them in a practical manner are close to zero. More concisely: I think that hoping to solve all programming problems by providing a specific, highly expressive abstraction for each of them is about as productive as hoping to solve them by reducing them all to lambda calculus.

                                                          1. 7

                                                            Interesting response, thank you. I feel like neither this nor the article have a tone that is so absolutist or fanatical that there is a risk of a flamewar. I am glad I read both and feel I understand more about this topic now.

                                                            It seems to me after reading this that the best solution for language complexity is exactly what we are all doing already:

                                                            Design some languages, observe what patterns arise, extend those languages to simplify and codify those patterns, repeat until the language becomes too clunky and over-complicated. Then create new languages that use the lessons learned in the previous generation, but avoid the pitfalls. Patterns that are successful and widely used will slowly permeate every language removing the need to learn them when changing languages (apart from syntax differences).

                                                            As long as there is a rich enough ecosystem everyone can choose the level of complexity they need/want, and the average expressiveness to complexity ratio for all languages should increase over time.

                                                            1. 9

                                                              Design some languages, observe what patterns arise, extend those languages to simplify and codify those patterns, repeat until the language becomes too clunky and over-complicated. Then create new languages that use the lessons learned in the previous generation, but avoid the pitfalls. Patterns that are successful and widely used will slowly permeate every language removing the need to learn them when changing languages (apart from syntax differences).

                                                              I like the sound of this but it seems to rely on the unstated assumption that good design plays a large part in language adoption, which I believe is disproved by looking at … gestures at “the industry”

                                                              But you’re absolutely right that learning to identify emergent patterns is probably the most important skill in designing a language. The problem with the article IMO is that it says “being complicated is OK” which IMO is a major oversimplification. The right conclusion to draw is that you have a limited complexity budget, and you need to spend it wisely. Refusing to add any complexity at all is a mistake just as adding complexity on redundant features is a mistake. In the end it’s kind of a rehash of the “accidental vs essential complexity” discussion in Out of the Tarpit.

                                                              1. 2

                                                                seems to rely on the unstated assumption that good design plays a large part in language adoption, which I believe is disproved

                                                                I agree, and xigoi below made much the same point. But I believe that it is unproductive to try to dictate to others what language they should use. Sure if they would listen it would probably make their lives easier, but you can’t tell people they are wrong it simply does not help. What this system at least offers is that those with an open mind and a willingness to try new things at least have the opportunity to get better tools regularly. The industry does follow along eventually, after all most of the world isn’t using the languages from the 80s anymore, but if you are spending millions developing commercial software it is not practical switch languages frequently so some lag is to be expected.

                                                              2. 3

                                                                Oh, yeah, I don’t think this is inflamatory in any way :). I just… I usually prefer to stay away from discussions on the relative merits of languages and various tools. They’re generally not very productive.

                                                                1. 3

                                                                  The only problem with this is the part where nobody will use the better languages because “there’s nothing wrong with ${old language}”.

                                                                  1. 2

                                                                    It’s really hard to explain to people that a language is better because it can’t do something. When Java was introduced, people complained that it couldn’t do pointer arithmetic yet the lack of pointer operations that can violate type safety is a key benefit of languages like Java. It’s easy to explain that language X is better than language Y because X has a feature that Y doesn’t. It’s much harder to explain that language Y is better because it lacks a feature that X has and this feature makes it easier to introduce bugs / harder to reason locally about the effects of code.

                                                                    1. 2

                                                                      That reminds me of a former colleague of mine who had been freelancing for a long time with PHP and didn’t understand my fascination with other languages. He claimed PHP was the best language and would go on lyrically about how great its array functionality is. When asked what languages he knew… PHP. Oh, and Pascal from back in school.

                                                                      1. 1

                                                                        As someone who has been pitching Rust for years, I don’t fear this argument at all. That’s an “that’s all I have left” argument.

                                                                        Now there’s multiple cases: 1) there’s something wrong with the old language, they just don’t see it yet. 2) there’s something wrong with the old language, but they’ve made the calculation of how much the new language would fix 3) the new language doesn’t meet their needs 4) they just need to hold a line and are not willing to switch anyways and not give any thought to the problem.

                                                                        1), 2), 3) can change very fast - the key here is taking that answer at face value, they will probably come back to you. 4) you don’t want to work with them anyways.

                                                                    2. 3

                                                                      Some of them are really good building blocks for higher-level abstractions, and you can express many things with them. Others not so much. Having more of them doesn’t necessarily make a language better at expressing things than another one.

                                                                      This reminds me of Lua’s “mechanisms instead of policies” principle.

                                                                    1. 5

                                                                      Looks like nix-shell is a handy tool to provide project-level reproducibility, and I think it’s much more useful compared to NixOS, which seems to offer workstation-level reproducibility. Most people replace their workstations very infrequently, and when they do, chances are that there exists some migration assistant (e.g. Time Machine or perhaps dd(1)). I don’t think I need to “deploy” my workstation-level configuration anywhere; it’s only meant for me to begin with.

                                                                      Tangentially, as a research assistant, I need to share a gigantic computing cluster with everyone affiliated with the university, and I don’t think I can convince the sysadmin into installing Nix on it (especially when the installer is a scary curl -L https://nixos.org/nix/install | sh). I know a root-level /nix directory is required to make use of the binary cache since the absolute path to dynamic libraries is embedded in the cached binaries, but there must be some workaround. Like, why not just scan the relocation table of each cached binary and replace the prefix of the paths?

                                                                      1. 12

                                                                        I’m currently using NixOS as my main daily driver. The advantage for me isn’t workspace migrations, but workspace versioning. There is a lot of cruft that builds up on workspaces overtime. Packages that you download to try out, work-arounds that stick, etc. These are all versioned & documented in the gitlog. It also lets me do stupid things without having to really worry about the consequences on my machine, as I can rollback the changes easily.

                                                                        1. 1

                                                                          I tried to configure the whole workspace with Nix on macOS, but it turns out that I cannot even install Firefox. This gives me the impression that nixpkgs is currently not mature/popular enough (at least on macOS), and that at some point I will be forced to install Homebrew/MacPorts/Anaconda or run curl shiny.tool/install.sh | sh to get some niche package, and suddenly I have packages outside of version control.

                                                                          Also, nix-env -qaP some_package is already ridiculously slow, and with more packages in the repositroy, it will probably become even slower. More importantly, even a huge package repository cannot include everything, so from time to time users must write Nix expressions themselves, which I don’t think would be trivial (if it was, then Nix would have already automated that).

                                                                          I’m not complaining, but that’s the reason I’m not bold enough to use Nix as my daily driver. I guess I should donate some money to Nix to facilitate its growth.

                                                                          1. 5

                                                                            I don’t disagree with the facts that you wrote, but I thought I’d comment since I cash some of them out differently… For a little context, I first took the dive into NixOS in early 2018, when my Windows desktop’s motherboard flamed out. It was rocky (a mix of Nix, plus it being my first desktop Linux), but I started using nix and nix-darwin when I replaced my macbook air in early 2019.

                                                                            I tried to configure the whole workspace with Nix on macOS, but it turns out that I cannot even install Firefox. This gives me the impression that nixpkgs is currently not mature/popular enough (at least on macOS), and that at some point I will be forced to install Homebrew

                                                                            1. A linux-first package manager’s ability to manage all software including desktop apps on macOS is a very stringent ruler. (Don’t get me wrong–it’ll be good if it works some day–but it’s a high bar, and Nix can be very useful without clearing it.)

                                                                            2. Yes–keep homebrew. TBH, I think forcing desktop apps, many of which already want to auto-update, into the Nix paradigm is a bit weird. I, to be frank, find Nix on macOS, with nix-darwin, to be a very nice compromise over the purity of these apps on NixOS.

                                                                              I actually find it more ergonomic to let these apps update as they have new security patches, and update my Nixpkgs channel less frequently. Since early 2019, I think I’ve only twice used Homebrew to install a package–I’ve used it almost exclusively for casks, and the packages were really just to play with something that was in Homebrew to decide if it was worth porting to Nix (neither was). Once again, I’d say the freedom to do this is a really nice compromise over purity in NixOS.

                                                                              suddenly I have packages outside of version control.

                                                                              You can still version-control a .Brewfile if it is for apps. It’s obviously not the same level of reproducibility, but if I’m trying to rebuild the software I had 3 years ago I’m generally not doing it for Chrome, Firefox, Steam, etc. I added a section to my backup script to barf out a report on whether I have installed anything with brew that isn’t in my brewfile. If I really cared, I think I could script it to uninstall those packages every day to force the issue.

                                                                              If it’s for a smaller package and you care about version-controlled reproducibility this much, you’ll generally have enough motivation to port it to Nix. (In my experience it has been true, but I recognize that the practicality of this will depend on the scope of the package in question…)

                                                                            More importantly, even a huge package repository cannot include everything, so from time to time users must write Nix expressions themselves, which I don’t think would be trivial

                                                                            This is the proverbial two-edged sword. So, yes, yes, this can happen. I am personally very conservative when it comes to recommending Nix to anyone who isn’t open to learning the language. I think it can be okay, narrowly, as a tool with no knowledge of the language. Learning it can be frustrating. But:

                                                                            • Nix can be a really big lever. I’m not sure if this is of much use to people who don’t program, but I feel like my time was well spent (even if it was a bigger investment than it needed to be).
                                                                            • A lot of the difficulty of learning to write Nix packages has honestly just been my near-complete lack of experience with the processes for going from source to installed software in Unix-alikes. If you already know a lot about this, it’ll be mostly about the language.
                                                                            • Packages aren’t all hard to write. They certainly can be nightmarish, but packages for “well-behaved” software can be fairly simple. Consider something like https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/misc/smenu/default.nix which is almost entirely metadata by volume.
                                                                            • It is fairly easy to manage/integrate private packages (whether that’s from scratch, or just overrides to use PRs I took the time to submit that the upstream is ignoring). I end up writing a Nix package for most of the unreleased/private software on my system (even when I could use it without doing so).

                                                                            (if it was, then Nix would have already automated that).

                                                                            Have any other package managers automated the generation of package expressions? I’m not terribly knowledgeable on prior art, here. If so, you’ve probably got a point. IME most of the work of writing package expressions is human knowledge going into understanding the software’s own packaging assumptions and squaring them with Nix. I’d be a little surprised if this is highly automatable. I can imagine translation of existing packages from other languages being more tractable, but IDK.

                                                                            1. 2

                                                                              A lot of things that are using standard frameworks can be mostly automated. Debian/Ubuntu are doing pretty well with dh. Nix already has templates https://github.com/jonringer/nix-template

                                                                              It’s not perfect, but as long as you’re using common rolling, packaging is not terribly hard.

                                                                            2. 4

                                                                              As to -qaP, unfortunately you learn to not do it; instead I’d recommend to use Nix 2.0’s nix search, as it has some caching (and was in fact introduced primarily to solve the problem with -qaP); and then e.g. nix-env -iA nixpkgs.some_package. Or, alternatively, maybe nix profile install, though I haven’t experimented with it yet myself.

                                                                              As to getting some niche package: yes, at some point you’ll either have to do this, or write your own Nix expression to wrap it. FWIW, not every byte in the world is wrapped by Nix, and This Is Just A FOSS Project Run By Good-Willing Volunteers In Their Spare Time, and You Can (Try To) Contribute, and going into another rabbit hole trying to wrap Your Favourite Package™ in a Nix expression is probably something of a rite of passage. I like to draw a parallel between Nix and earlier days of Linux, before Ubuntu, when you had to put a lot of time into it to have some things. A.k.a. your typical day at the bleeding edge of technology (which Nix is actually already not so much bleeding as it was just a few years ago). And, actually, people say Nixpkgs are kinda at the top on https://repology.org. But you know, at least when a package is broken on Nixpkgs, it doesn’t send your whole OS crashing & burning into some glitchy state from which you’ll not recover for the next couple years… because as soon as you manage to get back to some working terminal & disk access (assuming things went really bad on NixOS), or in worst case restart to GRUB, you’re just one nix-rebuild generation away from your last known good state. And with flakes, you nearly certainly even have the source of the last known good generation in a git repo.

                                                                              1. 1

                                                                                I think the biggest problem is that for Nix to be useful, we must achieve nearly complete package coverage, i.e. almost all packages must be installable via Nix. Covering 90% of the most popular packages is still not good enough, because even a single non-reproducible package in the whole dependency graph will ruin everything. It’s an all-or-nothing deal, and assuming package usage follows a power-law distribution, we will have a very hard time covering the last few bits. This is very different from Linux, where implementing 90% of the functionality makes a pretty useful system.

                                                                                Since you mentioned Linux, I’d like to note that packages from the system repository of most distributions are outdated, and users are encouraged to install from source or download the latest release from elsewhere (e.g. on Ubuntu 20.04, you must use the project-specific “PostgreSQL Apt Repository” to install PostgreSQL 13, which was released over a year ago). I guess some people took the effort to package something they want to use, but lack the incentive to keep maintaining it. While it’s perfectly fine to sidestep apt or yum and run make && make install instead, you can never sidestep Nix because otherwise you would lose reproducibility. How can the Nix community keep nearly all packages roughly up to date? I have no clue.

                                                                                1. 5

                                                                                  What I’m trying to answer to this, is to think small and “egoistically”: instead of thinking how Nix is doomed from a “whole world” perspective, try to focus just on your own localised use case and how much reproducibility you need yourself: if you must have 100% reproducibility, it means you either have enough motivation and resources to wrap the (finite number of) specific dependencies that you need and are not yet wrapped, or it’s apparently more of a would like to than must have, i.e. you have other, higher priorities overriding that. If the latter, you can still use Nix for those packages it provides, and add a few custom scripts over that doing a bit of curl+make or whatsit that you’ve been doing all the time till now (though once you learn to write Nix expressions for packages, you may realize they’re not really much different from that). Unless you go full NixOS (which I don’t recommend for starters), your base distro stays the same as it was (you said you don’t have root access anyway, right?) and you still can do all of what you did before. If some parts of your system are reproducible and some are not (yet), is it worse than if none are? Or maybe it is actually an improvement? And with some luck and/or persistence, eventually others may start helping you with wrapping the “last mile” of their personal pet packages (ideally, when they start noticing the benefits, i.e. their Nix-wrapped colleagues’ projects always building successfully and reproducibly and not breaking, and thus them being able to “just focus on the science/whatsit-they’re-paid-for”).

                                                                                  1. 2

                                                                                    That makes sense. IMHO Nix can and should convince package authors to wrap their own stuff in Nix. It can because the Nix language is cross-platform (this is not the case for apt/yum/brew/pkg); it should because only authors can make sure the Nix derivations are always up-to-date (with a CI/CD pipeline or something) while minimizing the risk of a supply chain attack.

                                                                                    1. 4

                                                                                      That is not how the open-source movement works. You don’t get to tell people what they should do; you rather take with humbleness and gratitude what they created, try to help them by contributing back (yet humbly enough to be ready to accept and respect if they might not take your contribution for some reason - though knowing you’re also free to fork), and yes, this means to possibly also contribute back ideas, but with the same caveat of them possibly not being taken - in this even more often, given that ideas are a dime a dozen. And notably, through contributing back some high quality code, you might earn some recognition that might give you a tiny bit more attention when sharing ideas. Ah, and/or you can also try to follow up on your ideas with ownership and actions, this tends to have the highest chance of success (though still not 100% guaranteed).

                                                                                      That said, I see this thread now as veering off on a tangent from the original topic, and as such I think I will take a break and refrain from contributing to making it a digression train (however I love digressions and however tempting this is), whether I agree or not with any further replies :) thanks, cheers and wish you great Holidays! :)

                                                                                  2. 2

                                                                                    I think the biggest problem is that for Nix to be useful, we must achieve nearly complete package coverage, i.e. almost all packages must be installable via Nix.

                                                                                    Why do you think so? I’m wondering why this applies to Nix but not to Homebrew or apt or yum or the likes? One can still build a package manually by setting up the dependencies in a nix shell – that’s no different from building something that package managers of other systems still don’t have.

                                                                                    1. 3

                                                                                      From my understanding, Nix aims to provide a reproducible environment/build, so it must exhaustively know about every piece of dependency. Homebrew, apt, and yum don’t have such an ambition; they just install packages, and can thus happily co-exist with other package managers and user-installed binaries.

                                                                                      1. 6

                                                                                        Nix-build, yes; nix-shell, no. In a nix-shell env, you still see all of your pre-existing system, plus what nix-shell provides as an “overlay” (not in docker filesystem sense, just extra entries in PATH etc.). It reproducibly provides you the dependencies you asked it to provide, but it doesn’t guarantee reproducibility of what you do over that. So you could start with nix-shell (or nix develop IIRC in case of flakes).

                                                                                2. 3

                                                                                  This gives me the impression that nixpkgs is currently not mature/popular enough (at least on macOS)

                                                                                  Have no idea about Mac, but my understanding is that on Linux the amount of packaged stuff for NixOS is just ridiculously high: https://repology.org/repositories/graphs. Anecdotally, everything I need on daily basis is there.

                                                                                  Still, there are cases when you do want to try a random binary from the internet (happened to me last time when I wanted to try JetBrains Fleet first public release), and yeah, NixOS does make those cases painful.

                                                                                  1. 1

                                                                                    Nix on macOS should not even be a thing.

                                                                                3. 7

                                                                                  NixOS […] seems to offer workstation-level reproducibility.

                                                                                  Don’t forget about server reproducibility! This is especially nice when you need to spin up multiple servers for a single project that need similar configuration.

                                                                                  1. 2

                                                                                    In which case I either have docker/kurbernetes/podman or I’m using ansible to be fast and productive. Sure you may get some stuff done much more precisely in nixos, but that’s definitely not worth the hassle. That said: Best of luck to nixos, hopefully it’ll be stable enough one day.

                                                                                    1. 8

                                                                                      Wait, what hassle? And what about it isn’t “stable?” Citation needed? It’s stable enough for every NixOS user I know on server and desktop and it’s stable enough for me. Hassle is a very vague word that could mean a lot of things, so if not for the fact that exactly zero of those possible meanings make what you said a true statement, I wouldn’t know how to respond to this. What is it about NixOS you think is a hassle?

                                                                                      1. 8

                                                                                        Heh, my previous company went from NixOS to “let’s deploy using ansible on Ubuntu boxes, everyone knows those”. Productivity and velocity just went down the drain. Oh, the horrors, oh the PTSD… But everyone has different experiences, sometimes some tools work, sometimes they don’t.

                                                                                        1. 3

                                                                                          much more precisely in nixos

                                                                                          I don’t know how you could get any more precise than NixOS. It specifies everything by hash, all the way down to a linker. I’ve never seen anybody do anything like that with any other system.

                                                                                      2. 5

                                                                                        Looks like nix-shell is a handy tool to provide project-level reproducibility,

                                                                                        Definitely. Every project I use now has a shell.nix file that pins the tools I use. I switched to that workflow after I was bitten several times by brew replacing python, so virtual environments were not working, or completely forgetting what tools I need in a project (after returning to it a year later). shell.nix is acting both as a bill of materials and recipe for fetching the right tools.

                                                                                        1. 4

                                                                                          For me that work-around is ‘containers’. As in nix generates the containers (reproducible!), the clusters run the containers, and in the containers it’s /nix.

                                                                                          1. 2

                                                                                            Check if you can get the following to print “YES”:

                                                                                            $ unshare --user --pid echo YES
                                                                                            YES
                                                                                            

                                                                                            or any of the following to print CONFIG_USER_NS=y (they all check the same thing IIUC, just on various distributions some commands or paths differ):

                                                                                            $ zgrep CONFIG_USER_NS /proc/config.gz
                                                                                            CONFIG_USER_NS=y
                                                                                            $ grep CONFIG_USER_NS /boot/config-$(uname -r)
                                                                                            CONFIG_USER_NS=y
                                                                                            

                                                                                            If so, there’s reportedly a chance you might be able to install Nix without root permissions.

                                                                                            If you manage to get it, personally, I would heartily recommend trying to get into “Nix Flakes” as soon as possible. They’re theoretically still “experimental”, and even harder to find reliable documentation about than Nix itself (which is somewhat infamously not-easy already), but IMO they seem to kinda magically solve a lot of auxillary pain points I had with “classic” Nix.

                                                                                            Also, as a side note, the nix-shell stuff was apparently much earlier than NixOS. The original thesis and project was just Nix, and NixOS started later as another guy’s crazy experiment, AFAIU.

                                                                                            EDIT: If that fails, you could still try playing with @ac’s experimental https://github.com/andrewchambers/p2pkgs.

                                                                                            EDIT 2: Finally, although with much less fancy features, there are still some interesting encapsulation aspects in 0install.net: “0install also has some interesting features not often found in traditional package managers. For example, while it will share libraries whenever possible, it can always install multiple versions of a package in parallel when there are conflicting requirements. Installation is always side-effect-free (each package is unpacked to its own directory and will not touch shared directories such as /usr/bin), making it ideal for use with sandboxing technologies and virtualisation.”

                                                                                            1. 1

                                                                                              This article did sent me on search for “how do you flakes and nix-shell at the same time” and apparently there’s “nix develop” now. This link has some details for both system wide and local setups: https://www.tweag.io/blog/2020-07-31-nixos-flakes/

                                                                                              1. 1

                                                                                                Yeah, there’s also https://nixos.wiki/wiki/Flakes, and generally that’s the issue, that you need quite some google-fu to find stuff about them :)

                                                                                          1. 6

                                                                                            A bit of context: Microsoft developed PhotoDNA to identify illegal images like CSAM – NCMEC maintains a database of PhotoDNA signatures, and many companies use this service to identify and remove these images.

                                                                                            Microsoft claims:

                                                                                            A PhotoDNA hash is not reversible, and therefore cannot be used to recreate an image.

                                                                                            This project shows that this isn’t quite true: machine learning can do a pretty good job of reproducing a thumbnail-quality images from a PhotoDNA signature.

                                                                                            There has been some previous posts about PhotoDNA, reverse engineering the algorithm and claiming that it is reversible. But there was no public demonstration of this as far as I know.

                                                                                            1. 3

                                                                                              That’s interesting. Does that imply Apple’s NeuralHash is also reversible to some extent?

                                                                                              1. 7

                                                                                                I haven’t tried it, so I can’t say for sure. Perceptual hashes basically have to leak some information about their input (because they are set up so that d(x, x') small => d(h(x), h(x')) small). But NeuralHash uses a CNN and outputs a 96-bit hash, while PhotoDNA computes image statistics and outputs a 1152 bit hash, so I expect NeuralHash hashes would reveal less information and be harder to invert.

                                                                                            1. 1

                                                                                              I’m wondering how feasible it is to implement such a rotation with an FPGA. The advantage is that the hardware HDMI adapter will work for every OS and monitor, without posing any computational burden on the host :P

                                                                                              1. 0

                                                                                                I believe the project was originally named MangoDB.

                                                                                                1. 1

                                                                                                  yup it was

                                                                                                1. 31

                                                                                                  Finally, we have STRICT tables, which brings type safety to SQLite!

                                                                                                  1. 5

                                                                                                    That was my excitement source as well. I hope the datetime types get added soon - it looks like they’re not supported yet.

                                                                                                    1. 1

                                                                                                      In order to continue supporting this ability, even in STRICT tables, the new ANY datatype name is introduced. When the datatype of a column is “ANY”, that means that any kind of data - integers, floating point values, strings, or binary blobs, can be inserted into that table and its value and datatype will be preserved exactly as it is inserted. As far as we know, SQLite is the only SQL database engine that supports this advanced capability.

                                                                                                      Can’t help but think “anti-pattern” here. And find the tone of it oddly smug.

                                                                                                      Certainly a welcome feature when used responsibly nonetheless!