-
Nix is one of those tools where you don’t know what you aren’t getting until you get it. There are so many things wrong with this post, but I only know that because I spent weeks wrestling with lots of those issues myself.
You basically need to read all the nix pills (https://nixos.org/nixos/nix-pills/), the nix manual, the nixpkgs manual and the nixos manual in a loop gradually filling in what is going on… which takes a long time.
Nix is very confusing at first, but enables things that you would not have thought possible once you know what you are doing. The core people don’t seem to evangelize much because it is just one of those tools that solved their problems so well, they don’t have to care about the outside world anymore.
I use nixos for my laptop, desktop and a few servers, have all my machines config under version control and can roll the machines back to any version whenever I want, remote administer them, build an install on one computer, test it in a VM and then ship it with a single command to another machine. I won’t go back to another OS despite there being room for improvement, because no other OS comes close in terms of what you can do (my path has been windows -> ubuntu -> arch linux -> freebsd -> openbsd -> nixos).
-
I use NixOS on everything and completely agree. It’s a massive investment. It was worth it for me, but it shouldn’t have to be a massive investment. Need better tooling and docs.
-
Yeah, there are lots of things I wish I could explain, but the explanations take a large investment. Take for example the complaint about making a new language instead of using something existing… It seems sensible on the surface, until you understand deeply enough to know why laziness is needed, and features like the pervasive use of interpolation to generate build scripts… Once you understand those, you know why a new language was made.
The lack of tooling IS a valid complaint, and the fact the language isn’t statically typed could also be a valid complaint, but the community is growing despite all those issues, which is a good sign.
-
I’m hoping https://github.com/haskell-nix/hnix will help with that point, and the tooling.
-
-
-
You basically need to read all the nix pills (https://nixos.org/nixos/nix-pills/), the nix manual, the nixpkgs manual and the nixos manual in a loop gradually filling in what is going on… which takes a long time.
I’ve tried reading all of this but I found it all horribly confusing and frustrating — until I read the original thesis on it, which I still think is (perhaps surprisingly) still the best resource for learning how nix works. It’s still a pretty big investment to read, but imho it’s at the very least a much less frustrating experience than bouncing from docs to docs.
(I wonder if the same is true of the NixOS paper?)
-
How do you manage secrets in configuration files? Passwords, ssh keys, tls certs and so on. If you put them into a nix-store they must be world-readable, right?
One could put a reference to files outside the store in configuration files, but then you loose a bit of the determinism of NixOS and it’s not always easily possible with third-party software to load e.g. passwords from an external file anyways.
Besides the learning curve, that was the single big problem which kept me from diving deeper into the nix ecosystem so far.
-
You are right, no passwords should ever go in the nix store.
The encryption key for my backup script is in a private root owned file I put under /secrets/ . This file is loaded in my cron job so the nix store simply references the secret but doesn’t contain it. This secret dir isn’t under version control, but is backed up with encrypted backups.
Every daemon with secret config I have seen on nixos has a “password file” option that does the same thing.
-
How do you manage secrets in configuration files?
For my desktop machine I use
passwith a hardware key. E.g. nix (home-manager) generates an .mbsyncrc withPassCmd "pass Mail/magnolia"For remote machines, I use nixop’s method for keeping keys out of the store:
-
-
Nix is one of those tools where you don’t know what you aren’t getting until you get it. There are so many things wrong with this post
I have to disagree, but not with the second sentence - I was sure as I wrote the post that it was full of misconceptions and probably outright errors. I wrote it in part to capture those in the hopes that someone can use them to improve the docs.
But to disagree with the first sentence, I was keenly aware through the learning and writing that I was missing fundamental concepts and struggling to fill the gaps with pieces from other tools that didn’t quite fit. If there is indeed a whole ‘nother level of unknown unknowns, well, that’s pretty disheartening to me.
-
I can’t speak for your experience, but that’s how it was for me anyway, on the plus side it also meant nix solved more problems I was having after I understood better. I even thought nix was over complicated to the point I started writing my own simpler package manager, only to find nix had solved problems I ran into before I knew what they were.
-
-
-
The subtext of this article is hilarious: that GPG is so hard to use that someone decided that it might be less painful to just rewrite the damn thing.
-
-
Cool (and encouraging) analysis. Does this account for putting extensions in the default-extensions in a .cabal file?
-
It doesn’t. I’d expect at least the relative frequency to be similar, but this could very well increase how frequently extensions pop up. But I think that it’s probably much more common to enable extensions on a file-by-file basis than a project basis.
It’d be cool to see what effect adding those back has, but it’s probably not doable with the GitHub API.
-
-
-
As one insignificant user of this language, please stop adding these tiny edge case syntax variations and do something about performance. But I am one small insignificant user …
-
This is exactly the attitude that leads to maintainers’ burn outs.
Do realize this:
- Adding syntax variations is not done at the expense of performance, different volunteers are working on what’s more interesting to them.
- Python is not a product, and you’re not a paying customer, you don’t get to say “do this instead of that” because none of the volunteer maintainers owes you to produce a language for you. Just walking by and telling people what to do with their project is at the very least impolite.
- If you want something to happen in an open source project, volunteer to do it.
(None of this is aimed at you personally, I don’t know who you are. I’m dissecting an attitude that you’ve voiced, it’s just all too common.)
-
Python is not a product, and you’re not a paying customer, you don’t get to say “do this instead of that” because none of the volunteer maintainers owes you to produce a language for you. Just walking by and telling people what to do with their project is at the very least impolite.
I agree with the general direction of your post, but Python is a product and it is marketed to people, through the foundation and advocacy. It’s not a commercial product (though, given the widespread industry usage, you could argue it somewhat is). It’s reasonable of users to form expectations.
Where it goes wrong is when individual users claim that this also means that they need to be consulted or their consultation will steer the project to the better. http://www.ftrain.com/wwic.html has an interesting investigation of that.
-
Where it goes wrong is when users claim that this also means that they need to be consulted or their consultation will steer the project to the better.
Wait, who is the product being built for, if not the user? You can say I am not a significant user, so my opinion is not important, as opposed to say Google which drove Python development for a while before they focused on other things, but as a collective, users’ opinions should matter. Otherwise, it’s just a hobby.
-
Sorry, I clarified the post: “individual users”. There must be a consultation process and some way of participation. RFCs or PEPs provide that.
Yet, what we regularly see is people claiming how the product would be a better place if we listened to them (that, one person we never met). Or, alternatively, people that just don’t want to accept a loss in a long-running debate.
I don’t know if that helps clarifying, it’s a topic for huge articles.
-
I often find what people end up focusing on - like this PEP - is bike shedding. It’s what folks can have an opinion on after not enough sleep and a zillion other things to do and not enough in depth knowledge. Heck I could have an opinion on it. As opposed to hard problems like performance where I would not know where to start, much less contribute any code, but which would actually help me and, I suspect, many other folks, who are with some sighing, migrating their code to Julia, or, like me, gnashing their teeth at the ugliness of Cython.
-
Yeah, it’s that kind of thing. I take a harsh, but well-structured opinion any time and those people are extremely important. What annoys me is people following a tweet-sized mantra to the end, very much showing along the path that they have not looked at what is all involved or who would benefit or not knowing when to let go off a debate.
-
-
-
-
-
Adding syntax variations is not done at the expense of performance, different volunteers are working on what’s more interesting to them.
Regrettably, a lot of languages and ecosystems suffer greatly from the incoherence that this sort of permissive attitude creates.
Software is just as much about what gets left out as what gets put in, and just because Jane Smith and John Doe have a pet feature they are excited about doesn’t mean they should automatically be embraced when there are more important things on fire.
-
the incoherence that this sort of permissive attitude creates
The Haskell community would’ve just thrown PEP 572 behind
{-# LANGUAGE Colonoscopy #-}and been done with it.Sure, this doesn’t get us out of jail free with regard to incoherence, but it kicks down the problem from the language to the projects that choose to opt-in.
-
I find it hard to see this as a good thing. For me, it mostly highlights why Haskell is a one-implementation language… er, 2 ^ 227 languages, if
ghc --supported-extensions | wc -lis to be taken literally. Of course, some of those extensions are much more popular than others, but it really slows down someone trying to learn “real world” Haskell by reading library code.-
Of course, some of those extensions are much more popular than others
Yeah, this is a pretty interesting question! I threw some plots together that might help explore it, but it’s not super conclusive. As with most things here, I think a lot of this boils down to personal preference. Have a look:
https://gist.github.com/atondwal/ee869b951b5cf9b6653f7deda0b7dbd8
-
-
-
Yes. Exactly this. One of the things I value about Python is its syntactic clarity. It is the most decidedly un-clever programming language I’ve yet to encounter.
It is that way at the expense of performance, syntactic compactness, and probably some powerful features that could make me levitate and fly through the air unaided if I learned them, but I build infrastructure and day in, day out, Python gets me there secure in the knowledge that I can pick up anyone’s code and at the VERY LEAST understand what the language is doing 99% of the time.
-
-
I find that “people working on what interests them” as opposed to taking a systematic survey of what use cases are most needed and prioritizing those is a hard problem in software projects, and I find it curious that people think this is not a problem to be solved for open source projects that are not single writer/single user hobby projects.
Python is interesting because it forms core infrastructure for many companies, so presumably they would be working on issues related to real use cases. Projects like numpy and Cython are examples of how people see an important need (performance) and go outside the official language to get something done.
“If you want something to happen in an open source project, volunteer to do it.” is also one of those hostile attitudes that I find curious. In a company with a paid product of course that attitude won’t fly, but I suspect that if an open source project had that attitude as a default, it would gradually lose users to a more responsive one.
As an example, I want to use this response from a library author as an example of a positive response that I value. This is a library I use often for a hobby. I raised an issue and the author put it in the backlog after understanding the use case. They may not get to it immediately. They may not get to it ever based on prioritization, but they listened and put it on the list.
Oddly enough, I see this kind of decent behavior more in the smaller projects (where I would not expect it) than in the larger ones. I think the larger ones with multiple vendors contributing turn into a “pay to play” situation. I don’t know if this is the ideal of open source, but it is an understandable outcome. I do wish the hostility would decrease though.
-
Performance has never been a priority for Python and this probably won’t change, because as you said, there are alternatives if you want Python’s syntax with performance. Also its interoperability with C is okeish and that means that the small niche of Python’s users that use it for performance critical operations that are not already supported by Numpy, Numba and so on, will always be free to go that extra mile to optimize their code without much trouble compared to stuff like JNI.
If you want raw performance, stick to C/C++ or Rust.
-
I also observe the same tendency of smaller projects being more responsive, but I think the issue is scale, not “pay to play”. Big projects get so much more issue reports but their “customer services” are not proportionally large, so I think big projects actually have less resource per issue.
-
-
-
-
-
Have you tried Pypy? Have you tried running your code through Cython?
Have you read any of the zillion and one articles on improving your Python’s performance?
If the answer to any of these is “no” then IMO you lose the right to kvetch about Python’s performance.
And if Python really isn’t performant enough for you, why not use a language that’s closer to the metal like Rust or Go or C/C++?
-
Yes to all of the above. But not understanding where all the personal hostility is coming from. Apparently having the opinion that “Should := be part of Python” is much less important than “Let’s put our energies towards getting rid of the GIL and creating a kickass implementation that rivals C++” raises hackles. I am amused, entertained but still puzzled at all the energy.
-
There was annoyance in my tone, and that’s because I’m a Python fan, and listening to people kvetch endlessly about how Python should be something it isn’t gets Ooooold when you’ve been listening to it for year upon year.
I’d argue that in order to achieve perf that rivals C++ Python would need to become something it’s not. I’d argue that if you need C++ perf you should use C++ or better Rust. Python operates at a very high level of abstraction which incurs some performance penalties. Full stop.
-
This is an interesting, and puzzling, attitude.
One of the fun things about Cython was watching how the C++ code generated approaches “bare metal” as you add more and more type hints. Not clear at all to me why Python can not become something like Typed Racket, or LISP with types (I forget what that is called) that elegantly sheds dynamism and gets closer to the metal the more type information it gets.
Haskell is a high level language that compiles down to very efficient code (barring laziness and thunks and so on).
Yes, I find this defense of the slowness of Python (not just you but by all commentators here) and the articulation that I, as one simple, humble user, should just shut up and go away kind of interesting.
I suspect that it is a biased sample, based on who visits this post after seeing the words “Guido van Rossum”
-
-
-
Not clear at all to me why Python can not become something like Typed Racket, or LISP with types (I forget what that is called) that elegantly sheds dynamism and gets closer to the metal the more type information it gets.
Isn’t that what mypy is attempting to do? I’ve not been following Python for years now, so really have no horse in this race. However, I will say that the number of people, and domains represented in the Python community is staggering. Evolving the language, while keeping everyone happy enough to continue investing in it is a pretty amazing endeavor.
I’ll also point out that Python has a process for suggesting improvements, and many of the core contributors are approachable. You might be better off expressing your (valid as far as I can see) concerns with them, but you might also approach this (if you care deeply about it) by taking on some of the work to improve performance yourself. There’s no better way to convince people that an idea is good, or valid than to show them results.
-
Not really. Mypy’s goal is to promote type safety as a way to increase program correctness and reduce complexity in large systems.
It doesn’t benefit performance at all near as I can tell, at least not in its current incarnation.
Cython DOES in fact do this, but the types you hint with there are C types.
-
Ah, I thought maybe MyPy actually could do some transformation of the code, based on it’s understanding, but it appears to describe itself as a “linter on steroids,” implying that it only looks at your code in a separate phase before you run it.
Typed Racket has some ability to optimize code, but it’s not nearly as sophisticated as other statically typed languages.
-
-
-
-
In no way did I imply that you should “shut up and go away”.
What I want is for people who make comments about Python’s speed to be aware of the alternatives, understand the trade-offs, and generally be mindful of what they’re asking for.
I may have made some false assumptions in your case, and for that I apologize. I should have known that this community generally attracts people who have more going on than is the norm (and the norm is unthinking end users posting WHY MY CODE SO SLOW?
-
Hey, no problem! I’m just amused at the whole tone of this set of threads set by the original response (not yours) to my comment, lecturing me on a variety of things. I had no idea that (and can’t fathom why) my brief comment regarding prioritization decisions of a project would be taken so personally and raise so much bile. What I’m saying is also not so controversial - big public projects have a tendency to veer into big arguments over little details while huge gaps in use cases remain. I saw this particular PEP argument as a hilarious illustration of this phenomenon in how Python is being run.
-
Thinking about this a little more - sometimes, when languages ‘evolve’ I feel like they forget themselves. What makes this language compelling for vast numbers of programmers? What’s the appeal?
In Python’s case, there are several, but two for sure are a super shallow learning curve, and its tendency towards ‘un-clever’ syntax.
I worry that by morphong into something else that’s more to your liking for performance reasons, those first two tenets will get lost in the shuffle, and Python will lose its appeal for the vast majority of us who are just fine with Python’s speed as is.
-
Yes, though we must also remember that as users of Python, invested in it as a user interface for our code ideas, we are resistant to any change. Languages may lose themselves, but changes are sometimes hugely for the better. And it can be hard to predict.
In Python’s 2.x period, what we now consider key features of the language, like list comprehensions and generator expressions and generators, were “evolved” over a base language that lacked those features altogether, and conservatives in the community were doubtful they’d get much use or have much positive impact on code. Likewise for the class/type system “unification” before that. Python has had a remarkable evolutionary approach over its long 3-decade life, and will continue to do so even post-GvR. That may be his true legacy.
-
-
-
-
-
-
-
-
-
-
-
Yes, that is a better description…
Except for the perception problem.
It goes something like this… “I’m an important person, if what you were saying was important, I would understand it, I haven’t clue about the intricate details of this problem you’re describing, so it can’t be important.”
ie. Humans discount whatever they don’t understand. Ever heard the phrase, “Don’t overload me with details, just give me the Big Picture”?
So called “Technical Debt” is all thousands of fine fine tiny nitty gritty messy dirty concrete details, as soon as you start describing them…. you have lost.
Eyes glaze over and refocus on nice Big Juicy Money Making Features.
-
-
The sexist behavior of those guys at Sun was just disgusting. It’s good she wrote about this.
But at the first beer bust I went to, two of the sales guys were standing around rating the women…only they called them “units.
Bob Coe once told me he did not have to interview the candidates for his Administrative Assistant position. I should simply ask each candidate to link her hands behind her head with her elbows pointing forward and walk toward the wall. If her elbows were the first part of her anatomy to touch the wall, she was eliminated from candidacy. All applicants whose breasts touched first, he would interview
-
it really is beyond the pale of what I would consider possible in a professional work environment.
-
I’m really glad she kept with it after the blowback she got!
I wonder why we don’t hear these “horror stories” as much from HR folks (who presumably have far more visibility of them than engineers!) as often, but I suppose that it’s probably a much bigger career risk to call out this stuff publicly when your job description is basically to deal with all of it and make sure it doesn’t become a legal or PR issue.
-
-
-
This is a long series, and each story is better than the last, but you have to fiddle around with blog sidebar whenever you want to find the next installment. (That’s one thing I won’t miss from the Blogger Era…). Here’s one from Part 3 (of 23):
“Hey, Nancy,” he began, “I cannot make it into work today.”
“OK,” I said, “I will let Linc know” (Linc Holland, the man who has never won an argument with me).
“Are you OK?” I asked.
“Well, yes, but I have a little problem….”
“What’s up?” I naively asked.
“ I am in jail.” Johnny replied.
“Oh no!” I exclaimed.
“What happened?” I was now thinking we had a problem with a DUI or scofflaw issue and knowing Johnny as I did, I could picture him sitting in some police holding cell refusing to post bond on principle.
“Well, I was at my mom’s house on Thanksgiving and I got into an argument with my brother-in-law and things escalated and, well, someone got stabbed in the arm with the turkey fork.”
“Johnny, do you need bail money?” I asked, trying not to laugh out loud. “Are you sitting in jail because your family will not bail you out after impaling your brother-in-law?”
“No,” Johnny said, “The judge won’t set bail, I have to serve 90 days”.
“Why?” I asked.
“Well, I do not qualify for bail because of my previous manslaughter conviction.” Johnny said…as if it was a fact that we all had at our fingertips.
Now, I have to say, and I think most folks who know me will agree, that I am rarely at a loss for words. But Johnny stumped me. I sat there, dumbfounded for what felt like a long time….then I heard Johnny put coins in a payphone and that snapped me back. “What conviction, Johny?”
“When I was 17, I was joining a gang in LA. There was an altercation with another gang…someone was shot. I never even saw the gun, but I was slow over a fence and the police caught me. I served 5 years in prison.” I needed time to figure out what we were going to do,“I have to think about this, Johnny. Can you call me back later today?”
“Yes.”
Linc and I caucused on this and decided Johnny had to be at work that month or we were going to blow the quarter.
I called the judge on Johnny’s case and begged for a work release program It was agreed that Johnny could do work release for two months and then he would have to serve a full month in jail. It was not easy, but the judge was a bit of a technofile and he was delighted to help Sun. Johnny was happy. He wanted to get back to work and spending the nights in the county jail was better than spending all day there.
Q4 was great. Johnny did a great job. We all breathed a sigh of relief.
Then Johnny needed to go to jail for a month.
Sun was pretty big by this time. It had policies and procedures and a few personnel police had been hired. I knew that I could not just let Johnny go away for a month without an explanation. He couldn’t be paid, he couldn’t take vacation…and we were not going to fire him.
I processed what I thought was the appropriate paper work and crossed my fingers.
Crawford Beveridge called me a few days later.
“Nance, talk to me about Johnny DLP” he said.
“What do you want to know?” I asked.
“ I have been told you have placed him on educational leave, is that right?”
“Yes.” I replied.
“Is he studying at an institution of of higher education?” asked Crawford.
“He is in an institution.” I replied.
“What is he learning?” asked CB.
“Uh, not to bend over for the soap?” I answered.
“OK,” said the unflappable Crawford, “so long as you think he is getting an education, I will approve this.”
I will always love Crawford Beveridge.
When I left Sun, Johnny DLP sent me an adorable note. I hope he is well and staying out of trouble.
-
“Software carpentry” is an interesting term that betrays some interesting ideological positions. The first time I heard that term is from Kevin Sullivan at UVa, who uses it to differentiate his class from JavaSchools (software “carpentry” vs software “engineering”).
He teaches CS1 (a mixed class of majors and non-majors) in Idris. I was skeptical at first (Idris? for freshmen?), but the resulting retention rate and “switching intended major to CS” rate were really impressive.
Unfortunately, I can neither remember these numbers off the top of my head, nor find where they’re posted online, but I think it raises an interesting question:
Should we be teaching undergrads carpentry, engineering, or science?
-
-
Yeah, I think this is exactly the way in which Kevin Sullivan uses it too: He used to say that when building a house, you have carpenters and you have civil engineers. Usually you have both, but they each have their roles and the engineer better be able to know carpentry when necessary, but mostly shouldn’t be a carpenter — but most of the time when you’re a carpenter you’re not building a house or any such large ambitious project that lots of people work on and depend on. Unlike civil engineers, carpenters are more likely to have their own practice and most carpenter-hours are spent (perhaps not anymore with the rise of automation, but for the metaphor…) building furniture by themselves and a few other people.
Ideally, we could answer the question I raised simply with “it depends”. Clearly if someone wants to be a carpenter, you should teach them carpentry, and if someone wants to be an engineer, you should teach them engineering, etc.
But when you have these massive almost “one-size-fits-all” CS1 classes (especially somewhere like UCSD mentioned in the OP, where every year, the department asks for a better student-to-faculty ratio, and every year, the administration sends us students faster than we can hire faculty) it’s not clear at all. Some of your students will go on to be carpenters, some engineers, some scientists, and they don’t know which yet.
And even for the sciences, it’s not always cut-and-dry. Nearly every proteomics grad student is a carpenter, but what does a physicist do? Still mostly carpentry, but there’s also these giant legacy projects in Fortran 77 that you wish involved more engineering and less carpentry. I haven’t been a physicist for a little while, but I have no doubt that there are more of them piling up every day. So we’re doing the right thing teaching scientists carpentry, but it seems that perhaps one of every few should probably still be an “engineer”.
Once I added the term “software carpentry” to my vocabulary, these sort of concerns became immediately visible! It’s a powerful concept that packs up a surprisingly large amount ideas about the structure of writing code itself.
-
-
What a garbage human being. I’m glad he’s dead–he promoted toxic culture and practices that as a sector we’re all still working to purge.
I strongly recommend Pirates of Silicon Valley if you want to see more of their personalities. Wozniak says it’s the only one that depicts them accurately. Here’s clips of jobs.
Regarding OP, the next thing he does is go on stage getting applauded for others’ work. Bill Burr did my favorite standup on Steve Jobs talking about that. It applies to a lot of other leaders in the tech industry, too.
Both the TV movie and the book it’s based on, Fire in the Valley, are quite wonderful.
“I GOT THE LOOT, STEVE!”
I love that movie.
Somehow, when it comes to Jobs, “I’m glad he’s dead” is more appropriate coming from friendlysock than from angersock.
He almost ran me over once, on his way to park his Mercedes across two handicapped spaces. I’m glad that bitch is on ice.