1. 30

    In the Hacker News thread about the new Go package manager people were angry about go, since the npm package manager was obviously superior. I can see the quality of that now.

    There’s another Lobster thread right now about how distributions like Debian are obsolete. The idea being that people use stuff like npm now, instead of apt, because apt can’t keep up with modern software development.

    Kubernetes official installer is some curl | sudo bash thing instead of providing any kind of package.

    In the meantime I will keep using only FreeBSD/OpenBSD/RHEL packages and avoid all these nightmares. Sometimes the old ways are the right ways.

    1. 7

      “In the Hacker News thread about the new Go package manager people were angry about go, since the npm package manager was obviously superior. I can see the quality of that now.”

      I think this misses the point. The relevant claim was that npm has a good general approach to packaging, not that npm is perfectly written. You can be solving the right problem, but writing terribly buggy code, and you can write bulletproof code that solves the wrong problem.

      1.  

        npm has a good general approach to packaging

        The thing is, their general approach isn’t good.

        They only relatively recently decided locking down versions is the Correct Thing to Do. They then screwed this up more than once.

        They only relatively recently decided that having a flattened module structure was a good idea (because presumably they never tested in production settings on Windows!).

        They decided that letting people do weird things with their package registry is the Correct Thing to Do.

        They took on VC funding without actually having a clear business plan (which is probably going to end in tears later, for the whole node community).

        On and on and on…

        1.  

          Go and the soon-to-be-official dep dependency managment tool manages dependencies just fine.

          The Go language has several compilers available. Traditional Linux distro packages together with gcc-go is also an acceptable solution.

          1.  

            It seems the soon-to-be-official dep tool is going to be replaced by another approach (currently named vgo).

          2. 1

            I believe there’s a high correlation between the quality of the software and the quality of the solution. Others might disagree, but that’s been pretty accurate in my experience. I can’t say why, but I suspect it has to do with the same level of care put into both the implementation and in understanding the problem in the first place. I cannot prove any of this, this is just my heuristic.

            1. 8

              You’re not even responding to their argument.

              1.  

                There’s npm registry/ecosystem and then there’s the npm cli tool. The npm registry/ecosystem can be used with other clients than the npm cli client and when discussing npm in general people usually refer to the ecosystem rather than the specific implementation of the npm cli client.

                I think npm is good but I’m also skeptical about the npm cli tool. One doesn’t exclude the other. Good thing there’s yarn.

                1.  

                  I think you’re probably right that there is a correlation. But it would have to be an extremely strong correlation to justify what you’re saying.

                  In addition, NPM isn’t the only package manager built on similar principles. Cargo takes heavy inspiration from NPM, and I haven’t heard about it having a history of show-stopping bugs. Perhaps I’ve missed the news.

              2. 8

                The thing to keep in mind is that all of these were (hopefully) done with best intentions. Pretty much all of these had a specific use case… there’s outrage, sure… but they all seem to have a reason for their trade offs.

                • People are angry about a proposed go package manager because it throws out a ton of the work that’s been done by the community over the past year… even though it’s fairly well thought out and aims to solve a lot of problems. It’s no secret that package management in go is lacking at best.
                • Distributions like Debian are outdated, at least for software dev, but their advantage is that they generally provide a rock solid base to build off of. I don’t want to have to use a version of a python library from years ago because it’s the only version provided by the operating system.
                • While I don’t trust curl | sh it is convenient… and it’s hard to argue that point. Providing packages should be better, but then you have to deal with bug reports where people didn’t install the package repositories correctly… and differences in builds between distros… and… and…

                It’s easy to look at the entire ecosystem and say “everything is terrible” but when you sit back, we’re still at a pretty good place… there are plenty of good, solid options for development and we’re moving (however slowly) towards safer, more efficient build/dev environments.

                But maybe I’m just telling myself all this so I don’t go crazy… jury’s still out on that.

                1.  

                  Distributions like Debian are outdated, at least for software dev,

                  That is the sentiment that seems to drive the programming language specific package managers. I think what is driving this is that software often has way too many unnecessary dependencies causing setup of the environment to build the software being hard or taking lots of time.

                  I don’t want to have to use a version of a python library from years ago because it’s the only version provided by the operating system.

                  Often it is possible to install libraries at another location and redirect your software to use that though.

                  It’s easy to look at the entire ecosystem and say “everything is terrible” but when you sit back, we’re still at a pretty good place…

                  I’m not so sure. I forsee an environment where actually building software is a lost art. Where people directly edit interpreted files in place inside a virtual machine image/flatpak/whatever because they no longer know how to build the software and setup the environment it needs. And then some language specific package manager for distributing these images.

                  I’m growing more disillusioned the more I read Hacker News and lobste.rs… Help me be happy. :)

                  1.  

                    So like squeak/smalltalk images then? Whats old is new again I suppose.

                    http://squeak.org

                    1.  

                      I’m not so sure. I forsee an environment where actually building software is a lost art. Where people directly edit interpreted files in place inside a virtual machine image/flatpak/whatever because they no longer know how to build the software and setup the environment it needs. And then some language specific package manager for distributing these images.

                      You could say the same thing about Docker. I think package managers and tools like Docker are a net win for the community. They make it faster for experienced practitioners to setup environments and they make it easier for inexperienced ones as well. Sure, there is a lot you’ve gotta learn to use either responsibly. But I remember having to build redis every time I needed it because it wasn’t in ubuntu’s official package manager when I started using it. And while I certainly appreciate that experience, I love that I can just install it with apt now.

                    2.  

                      I don’t want to have to use a version of a python library from years ago because it’s the only version provided by the operating system.

                      Speaking of Python specifically, it’s not a big problem there because everyone is expected to work within virtual environments and nobody runs pip install with sudo. And when libraries require building something binary, people do rely on system-provided stable toolchains (compilers and -dev packages for C libraries). And it all kinda works :-)

                      1.  

                        I think virtual environments are a best practice that unfortunately isn’t followed everywhere. You definitely shoudn’t run pip install with sudo but I know of a number of companies where part of their deployment is to build a VM image and sudo pip install the dependencies. However it’s the same thing with npm. In theory you should just run as a normal user and have everything installed to node_modules but this clearly isn’t the case, as shown by this issue.

                        1. 5

                          nobody runs pip install with sudo

                          I’m pretty sure there are quite a few devs doing just that.

                          1.  

                            Sure, I didn’t count :-) The important point is they have a viable option not to.

                          2.  

                            npm works locally by default, without even doing anything to make a virtual environment. Bundler, Cargo, Stack etc. are similar.

                            People just do sudo because Reasons™ :(

                        2.  

                          It’s worth noting that many of the “curl | bash” installers actually add a package repository and then install the software package. They contain some glue code like automatic OS/distribution detection.

                          1.  

                            I’d never known true pain in software development until I tried to make my own .debs and .rpms. Consider that some of these newer packaging systems might have been built because Linux packaging is an ongoing tirefire.

                            1.  

                              with fpm https://github.com/jordansissel/fpm it’s not that hard. But yes, using the Debian or Redhat blessed was to package stuff and getting them into the official repos is def. painful.

                              1.  

                                I used the gradle plugins with success in the past, but yeah, writing spec files by hand is something else. I am surprised nobody has invented a more user friendly DSL for that yet.

                                1.  

                                  A lot of difficulties when doing Debian packages come from policy. For your own packages (not targeted to be uploaded in Debian), it’s far easier to build packages if you don’t follow the rules. I like to pretend this is as easy as with fpm, but you get some bonus from it (building in a clean chroot, automatic dependencies, service management like the other packages). I describe this in more details here: https://vincent.bernat.im/en/blog/2016-pragmatic-debian-packaging

                                2.  

                                  It sucks that you come away from this thinking that all of these alternatives don’t provide benefits.

                                  I know there’s a huge part of the community that just wants things to work. You don’t write npm for fun, you end up writing stuff like it because you can’t get current tools to work with your workflow.

                                  I totally agree that there’s a lot of messiness in this newer stuff that people in older structures handle well. So…. we can knowledge share and actually make tools on both ends of the spectrum better! Nothing about Kubernetes requires a curl’d installer, after all.

                                1. 2

                                  Great article – it is really positive to read about somebody taking steps to take care of themselves. It is unfortunate that he had to quit his job (no sabbatical option) to do so.

                                  1. 3

                                    And sell a “large chunk” of his equity to do so. Burns himself out to make the equity worth something, sells it to recover. Painful.

                                  1. 5

                                    Thank god for Firefox’s reader mode.

                                    Being kind to your reader > leet, cyber design.

                                    1. 1

                                      I noticed that this library is performance tested, with assert statements that make sure that a given function is executed within X ms.

                                      Are these kinds of tests helpful? Does it not make a difference what machine the tests are running on? Or if the test runner instance is responsible for running other test suites too and happened to be overloaded at the time these performance tests ran?

                                      Or is there a way to isolate machine resources so that such tests yield predictable/consistent results?

                                      1. 1

                                        I’d say they are useful for preventing performance regression. They are likely fairly consistent, but it is true that there’s potential signal pollution that can occur.

                                        1. 1

                                          Exactly. I’ve done this type of assertion before and you wind up having to put such a wide margin of error in it (like order of magnitude with modern, cloud CI boxes) that it doesn’t catch anything but the most egregious of regressions.

                                          That said, you can still assert relative comparisons (assert my time < “other gem” time) with some confidence.

                                        1. 6

                                          On the one hand, it’s nice to see someone stoked about their new job. On the other hand, I definitely raised an eyebrow a few times while reading this post:

                                          To my lasting surprise, I have gone to war. There is no better way to put it. I feel like I’ve joined a literal revolutionary war, surrounded by and fighting alongside guerilla troops, and it’s win or die. […] This war is happening on two fronts: Online and offline. […] I’ve seen Grab’s hunger. I’ve felt it. I have it. This space is win or die. They will fight to the death, and I am with them.

                                          1. 20

                                            The analogy to war is pathetic. Anyone’s who’s been in an actual war would tell you it’s not something to crow about; you don’t get PTSD from a failed startup unless you are doing it very wrong.

                                          1. 3

                                            I’m not sure I get the threat posed by an app that presents a fake apple pay button on screen. You push the button. So what? What happens next?

                                            There are several other things a malicious app can do with fake UI (asking for a password is one), but I don’t see the threat posed by an ok button. The app could fake not just the button, but the tap as well if it wanted.

                                            1. 1

                                              I think it’s not a fake apple pay button, a bad app could create a fake “Do you want to?” button that maliciously maps your OK press onto a hidden Apple pay button. You are charged and don’t know it.

                                              1. 2

                                                But apps can’t hide the Apple Pay dialog. It’s always on top.

                                            1. 17

                                              If only json had allowed trailing commas in lists and maps.

                                              1. 9

                                                And /* comments! */

                                                1. 3

                                                  And 0x... hex notation…

                                                  1. 3

                                                    Please no. If you want structured configs, use yaml. JSON is not supposed to contain junk, it’s a wire format.

                                                    1. 4

                                                      But YAML is an incredibly complex and truth be told, rather surprising format. Every time I get it, I convert it to JSON and go on with my life. The tooling and support for JSON is a lot better, I think YAMLs place is on the sidelines of history.

                                                      1. 4

                                                        it’s a wire format

                                                        If it’s a wire format not designed to be easily read by humans, why use a textual representation instead of binary?

                                                        If it’s a wire format designed to be easily read by humans, why not add convenience for said humans?

                                                        1. 1

                                                          Things don’t have to be black and white, and they don’t even have to be specifically designed to be something. I can’t know what Douglas Crockford was thinking when he proposed JSON, but the fact is that since then it did become popular as a data interchange format. It means it was good enough and better than the alternatives at the time. And is still has its niche despite a wide choice of alternatives along the spectrum.

                                                          What I’m saying is that adding comments is not essential a sure-fire way to make it better. It’s a trade-off, with a glaring disadvantage of being backwards incompatible. Which warrants my “please no”.

                                                      2. 1

                                                        http://hjson.org/ is handy for human-edited config files.

                                                        1. 1
                                                        2. 5

                                                          The solutions exist!

                                                          https://github.com/json5/json5

                                                          I don’t know why it’s not more popular, especially among go people.

                                                          There is also http://json-schema.org/

                                                          1. 3

                                                            I had to do a bunch of message validation in a node.js app a while ago. Although as Tim Bray says the spec’s pretty impenetrable and the various libraries inconsistent, once I’d got my head round JSON Schema and settled on ajv as a validator, it really helped out. Super easy to dynamically generate per message-type handler functions from the schema.

                                                            1. 2

                                                              One rather serious problem with json5 is its lack of unicode.

                                                            2. 3

                                                              I think this only show that JSON has chosen tradeoff that make it more geared to be edited by software, but has the advantage of being human editable/readable for debugging. JSON as config is not appropriate. There is so many more appropriate format (toml, yaml or even ini come to mind), why would you pick the one that doesn’t allows comments and nice sugar such as trailing commas or multiline string. I like how kubernetes does use YAML as its configuration files, but seems to work internally with JSON.

                                                              1. 8

                                                                IMO YAML is not human-friendly, being whitespace-sensitive. TOML isn’t great for nesting entries.

                                                                Sad that JSON made an effort to be human-friendly but missed that last 5% that everyone wants. Now we have a dozen JSON supersets which add varying levels of complexity on top.

                                                                1. 11

                                                                  “anything whitespace sensitive is not human friendly” is a pretty dubious claim

                                                                  1. 5

                                                                    Solution: XML.

                                                                    Not even being ironic here. It has everything you’d want.

                                                                    1. 5

                                                                      And a metric ton of stuff you do not want! (Not to mention…what humans find XML friendly?)

                                                                      This endless cycle of reinvention of S-expressions with slightly different syntax depresses me. (And yeah, I did it too.)

                                                                      1. -5

                                                                        Triggered.

                                                                        1. 13

                                                                          Keep this shit off lobsters.

                                                                1. 1

                                                                  All of this pain is one big reason why Docker is so popular. It’s so much easier to distribute one Docker image with my binaries than to distribute dozens of distro- and release-specific binaries.

                                                                  1. 3

                                                                    I have a slight connection with some of the people at IBM Watson. The higher ups have no idea how it works and just pitch their dreams. The engineers ignore them and build the best system they can with their limited skillset. CIOs buy into the boondoggle, millions are wasted, projects are cancelled, life goes on.

                                                                    1. 5

                                                                      boltdb is a great example of a limited library that is “finished” from its developer’s perspective and Ben is a Go superstar.

                                                                      One caveat: an LSM tree design will be much faster if you are inserting a lot of data. I found RocksDB to be 1000x faster under heavy load.

                                                                      https://en.wikipedia.org/wiki/Log-structured_merge-tree

                                                                      1. 1

                                                                        I couldn’t find information on how you plan to handle job persistence. What happens to the jobs if the server fails?

                                                                        1. 3

                                                                          TBD. This is the public unveiling of a new project. No one ships with a perfect HA solution on day one.

                                                                          1. 2

                                                                            I am not looking for a perfect solution here, just wondering what you are planning for the project.

                                                                            An embedded database implies you want to build the HA logic in the application layer, as opposed to re-use the database capabilities. Is that the case?

                                                                            Note: I am in no way familiar with RocksDB, maybe there is something very obvious I don’t understand.

                                                                            1. 2

                                                                              “TBD” sort of covers my plans right now. I’ve heard RocksDB has something internal which can be massaged into a replication stream but I’m not deep enough to grok it yet.

                                                                              1. 1

                                                                                Okay, I’ll look into it as well, thanks.

                                                                          2. 1

                                                                            from the article itself:

                                                                            Faktory goes further and provides the same job persistence, state management and monitoring Web UI that Sidekiq does. It uses Facebook’s high-performance RocksDB embedded datastore internally to persist all job data, queues, error state, etc.

                                                                            1. 2

                                                                              I think he meant if the rocksdb server goes down, implying it has no fault tolerance.

                                                                              1. 3

                                                                                It’s no worse a problem than relying on Postgres, minus the fact that RocksDB doesn’t have an obvious replication strategy.

                                                                                So, to recap: fine for a hobby app, requires more effort for anything critcal.

                                                                                1. 1

                                                                                  rocksdb has no server, it’s an embedded database. It’s a fork of LevelDB.

                                                                                  1. 1

                                                                                    Exactly, we were talking about the server hosting Faktory and in turn RocksDb.

                                                                            1. 3

                                                                              This is cool, congrats! I’m going to play around with using it as a broker in dramatiq this weekend :D. A couple questions:

                                                                              From what I can tell, a lot of things Faktory does overlap with things RabbitMQ provides ootb. How are you planning to differentiate long term?

                                                                              Is the protocol documented anywhere? I haven’t been able to find anything. The best I could find was this file in the Ruby client.

                                                                              1. 1

                                                                                The wiki has some protocol docs. Right now, the wiki + faktory_worker_{go,ruby} are the things to look into.

                                                                              1. 2

                                                                                It blew my mind when I learned that Linux models threads as separate processes (i.e. each thread has its own PID), they just all share the same virtual memory mapping (until CoW). A really neat implementation detail, at least from 10,000 ft up.

                                                                                1. 2

                                                                                  Wait, what? I think you’re thinking of Forks, which are full heavyweight processes. They do Copy-on-write.

                                                                                  Posix threads are not forks. They operate within the same process and within the same PID.

                                                                                1. 2

                                                                                  I’ve tried to dig into this in the past, but what is the difference between the .04 and the .10 ubuntu releases? is the .10 incremental updates to the .04 but the .04 is the long term support?

                                                                                  1. 15

                                                                                    04 and 10 are the April and Oct annual releases. LTS is long term support == five years. “Even”.04 is LTS, so 16.04 is LTS and 18.04 will be LTS.

                                                                                    If you are running an Ubuntu server in production, you should be using an LTS release.

                                                                                    1. 2

                                                                                      You’ll notice that the .04 are released in April and the .10 in October ;)

                                                                                    1. 0

                                                                                      I think there’s a line to draw. There’s lots of different opinions out there we can protect. A tiny, tiny, tiny slice of them is in favor of enslaving or murdering other people. Both of those actions are illegal under U.S. law with one specifically banned by an Amendment to the Constitution. So, we start with saying service providers might censor a site promoting illegal or harmful activity. Your mention of Cloudfare sounds like this is a new thing but it’s been going on forever in other areas (esp copyright).

                                                                                      From there, we might wonder about where to draw the line for censorship of illegal activities. Many laws might be considered unjust down the line after we’ve reformed them. A certain amount of lawbreaking or civil disobedience often happens to cause those reforms. At the least, people encouraging resistance to those laws or change might be protected. We can still draw the line here between advocating illegal acts that are obviously harmful (i.e. white supremacists dominating or killing other races) or that are debatable in cost-benefit (eg healthcare policies, affirmative action). We censor the first since there’s no such thing in being neutral if their movement just needs the spread of information to grow. If you’re spreading it, you’re supporting them indirectly. I’m not for legal liability for telecoms on that or anything but certainly let them cut off those advocating hate if they choose not to spread it.

                                                                                      1. 6

                                                                                        Legality is not a good standard on which to base moral judgments like “which opinions are acceptable”. You should absolutely be able to express opinions in favor of things that are illegal, or else it would be impossible to argue against an unjust law. It’s critically important that people are free to endorse murdering people (cf Against Murderism) or whatever other crazy opinion they have that would violate the law if followed through with. Being gay used to be illegal, black people and women voting used to be illegal, etc. etc. You can’t predict a priori what people are going to think should become legal in the future.

                                                                                        1. 1

                                                                                          I said that already. I just made an exception for people promoting things that almost everyone considers harmful with no benefit. They can always run their own server on the network if the greedy ISP’s or Tier 1’s will have them. They can broadcast over the radio. They can do demonstrations. They have plenty of freedoms.

                                                                                          I just think private providers should be able to censor them if their message is only hate and harm. Ironically, directed at many of the same kinds of people working to develop or run the very devices and services they’re using to spread their hateful message. Also, the same kinds of people who would be negatively impacted by people acting on that message ranging from unemployment to slavery to death. I mean, we really expect Cloudfare workers to help people try to fire, deport, or kill them? I wouldn’t.

                                                                                        2. 4

                                                                                          Also keep in mind that we’re talking about businesses, not the government. The 1st amendment does not apply to businesses. Just because the gov’t can’t jail a person for advocating for ethnic cleansing doesn’t mean that businesses can’t refuse service to that person. Free speech doesn’t mean freedom from consequences.

                                                                                          1. 1

                                                                                            This is true, but I wish people on the left consistently applied this standard, such as when the issue was baking wedding cakes for gay weddings rather than providing access to critical public infrastructure.

                                                                                            The moral notion of free speech is also separate from the specific legal protections on free speech present in the US. I believe that businesses should abstain from political censorship, but I don’t think they should be legally required to do so.

                                                                                        1. 3

                                                                                          “codewithoutrules.com” gives us a good rule for coding. Ok.

                                                                                            1. 1

                                                                                              A circlejerk over a new release of an UNIX-gone-wild operating system.

                                                                                              Please pardon my sarcastic humor, I actually appreciate niche stuff.

                                                                                            1. 1

                                                                                              Tech docs and sample code are quite often written by interns and proofed by tech writers. In other words, no one with the skills to know the API usage is terribly wrong.

                                                                                              1. 1

                                                                                                I think the article could gain to be augmented with links to the appropriate documentation, for example save vs save! as seen here: http://api.rubyonrails.org/classes/ActiveRecord/Persistence.html#method-i-save

                                                                                                If I recall correctly, this isn’t a new behavior; I believe most of the “!”-ended methods in Ruby idiomatically do “destructive” things to the underlying object instead of working on copies (see “String.reverse” vs “String.reverse!”). Well done, otherwise.

                                                                                                1. 3

                                                                                                  The sad part of the save! name is that the bang doesn’t indicate destructive behavior, it means “this method will raise an error if the save fails”, which is different semantics from the idiomatic Ruby meaning. Another “fun” Rails quirk.

                                                                                                  1. 2

                                                                                                    I wouldn’t call it a quirk; it’s fairly consistent with the meaning of bang according to Matz : https://www.ruby-forum.com/topic/176830#773946

                                                                                                    The bang (!) does not mean “destructive” nor lack of it mean non destructive either. The bang sign means “the bang version is more dangerous than its non bang counterpart; handle with care”. Since Ruby has a lot of “destructive” methods, if bang signs follow your opinion, every Ruby program would be full of bangs, thus ugly.

                                                                                                    1. 1

                                                                                                      At my job we end up treating the bang version of save with less care, because we know it will raise an error if things go wrong and not silently propagate a bug.

                                                                                                    2. 1

                                                                                                      Agreed. I expressed myself poorly; I meant to suggest that someone familiar with Ruby would surely have some form of alarm bell ringed in his mind if he saw “!” at the end of a method name; it’s a bit sad that it potentially raises errors, though.

                                                                                                    3. 2

                                                                                                      I’ve just added the links. :)

                                                                                                    1. 1

                                                                                                      Odd that they don’t address the elephant in the room and switch their internal representation to UTF-8. UTF-16 is still limited to a single Unicode plane, right?

                                                                                                      1. 2

                                                                                                        Ah, UTF-16 can encode multiple planes. Characters are either 16- or 32-bits.

                                                                                                        https://en.wikipedia.org/wiki/UTF-16