Position(s): Software Engineer - Confidential Computing and AI
Location: Remote / On-Site in Bochum, Germany
Description: Work on bringing confidential computing “to the world”. Our tech stack is very diverse and spans unusually far in both directions, so you’ll probably be faced with a lot of novel challenges that need pioneering work.
See the job posting for a more detailed description of what you’ll do.
Compensation: Competitive, can vouch for that, being an employee myself. Both salary- and equity-wise. Check the posting for more details.
Contact: Apply via the posting, but do mention that you come from lobste.rs!. Also feel free to email me via ms [at] edgeless.systems in case of any questions.
No, they go on the heap. Go is a bit different to most others here; taking a pointer to a stack local is fine (for instance). This essentially is initialising a heap variable.
Exactly, which is why I find this title a little misleading.Additionally, I think the aws.String will even result in stack-local for the result pointer + internal heap allocation for the actual string in there. Will the Go compiler ever give you a pointer into .text or similar?
It’s not misleading to use the terminology of the domain you’re in! This is what Go people mean when they say pointer-to-constant. (And yes, that’s exactly what aws.String does.)
IMHO this is making it more complex than it needs to be. Go the language doesn’t differentiate between the stack and the heap much like Java. The optimizer may choose to store values on the stack if it is safe to do so.
In practice this optimization is very predictable, but I don’t think it affects the promised semantics of the language.
But.. None of these actually gives you a pointer to a const?
No body cares.
I just want to pass a constant value (e.g. int) where the receiver is expecting an optional value (e.g. *int”).
I hate creating generic pointer function in every project.
If one is part of an explicit fork of a project, and you are aware of the original project containing some unpleasant vulnerability, it is easy to forget that this is not about revenge or competing with former team members which did not join you on the fork. It is ones own former users that are hurt by such an unreasonable disclosure method.
I find the title also a bit misleading. 2.24+ is way too greedy. Stating 2.24 would have been accurate, assuming version 2.25 will not have this vulnerability.
I wonder how other implementations such as Tvix did.
What I understood from the discussions around this report is nothing like that. (Disclaimer: I’m part of neither the Nix or Lix communities.)
From a distance, it looks like the Nix security team does not work very well – poor internal communication, poor communication with the rest of the Nix developers.
The Lix people have more security-conscious people so they are ahead in terms of finding vulnerabilities and handling them gracefully.
The Lix people are doing a genuine effort to communicate security issues to the Nix project, in good faith. The communication style of the reporter is very informal (“nar unpacking is fucked”), but as far as I can tell this is common among certain security communities and not a sign of arrogance or aggressivity. Other members of the project write in the more traditional “professional” style and seem very sincere in wishing the best for both projects.
If I had to make a (completely uninformed) guess, I would guess that the ruffled feathers around the Lix fork made it difficult for the Nix people to take the feedback of the Lix people seriously, and therefore increases the barrier for cooperation. Sounds like a perfectly human reaction, it takes time to rebuild shared trust. But it may be compounded by a not-so-good security response organization.
If I had to make a (completely uninformed) guess, I would guess that the ruffled feathers around the Lix fork made it difficult for the Nix people to take the feedback of the Lix people seriously, and therefore increases the barrier for cooperation. Sounds like a perfectly human reaction, it takes time to rebuild shared trust. But it may be compounded by a not-so-good security response organization.
Well, the vulnerability was acknowledged by a member of the CppNix team and a PoC was provided which made quite clear what could happen, so I don’t think that’s it.
Exchanges between Lix team members and Nix team members happened in the past without too many issues over the Nix codebase as well.
it looks like the Nix security team does not work very well – poor internal communication, poor communication with the rest of the Nix developers.
A standard problem in open source software.
If I had to make a (completely uninformed) guess, I would guess that the ruffled feathers around the Lix fork made it difficult for the Nix people to take the feedback of the Lix people seriously, and therefore increases the barrier for cooperation.
There are some people which try to take every opportunity to shoot against nix after that and it is then totally understandable that you try to avoid that.
Indeed, 2.24>=2.24.5 would be better wording here. To be honest, I just copied that from their post. I think Tvix is not susceptible to this exact vulnerability, but it has a lot of other complexity and is alpha software, so I wouldn’t be exactly sure that it’s “secure”, as of now.
The Tvix NAR parser is a completely different beast, and doesn’t have this vulnerability. It’s pretty hardened in general, and I designed it to only accept canonical-form NARs.
I believe we’re not vulnerable to the Unicode normalisation based form of this attack (‘ẽ’ and ‘ẽ’ are distinct encodings, but equivalent filenames on eg macOS), since our Nix store is (currently) always FUSE-based, so we never perform host filesystem operations with attacker-controlled filenames.
The Nix dev started the thread by accusing the vuln reporter of being irresponsible, and is now having explained to him, in detail, the myriad of ways in which he and his team failed to maintain proper security-related communications with infosec people.
I guess the lesson here is don’t start accusing people without knowing all the facts.
I’m not a security expert, but iiuc, if a vendor is communicating, working with you, and is planning to release a fix, then it’s generally best to give them a few months, in the hopes of minimizing how many people know about an exploit while it’s usable.
But if a vendor is unresponsive and has a history of being so, then the longer you wait, the more bad actors will discover or share the exploit. The rationale then becomes to let the world know ASAP, so that (1) users can decide how to mitigate their risk, and (2) going public may force the vendor to take action.
I’m not affiliated with the Nix team whatsoever, but I find a 7-day timeline for such a vulnerability for a partly (as I learned) volunteer team pretty tight.
The vulnerability report doesn’t mention it directly, but the discussion thread about it on Fedi gives some more context on that deadline: the reporter has had several previous vulnerability reports completely ignored by the Nix development team, including one open since February and still untriaged. The Nix development team received and acknowledged this new Nix 2.24 vulnerability on August 30th (so, > 9 days ago) and they seem to have mostly sat on it until today (the reporter received no further comms), to the extent that a new point release of Nix was released a few days after the vuln was reported and did not contain a fix.
I don’t think the full reasoning for a 7 days timeline was completely stated, but I’ll note that this vuln was also trivial to reverse engineer from very small amounts of public information that was already available. There are only so many places in Nix that are conductive to pre-signing-verification exploits, and looking at all changes made to those code paths since v2.24 made it trivial to find a PoC for this (I had a working PoC 4 days ago based on that info, I was probably not the only person?).
Out of curiosity, were you able to reach out to anybody at the Nix project about this? For something like this, it’s helpful to have redundant messaging attempts and I imagine everybody that uses Nix in whatever flavor would benefit from cooperation in case channels get garbled (as it looks like they did here).
Open-source shouldn’t be a zero-sum game, and giving people grace and taking proactive responsible action when needed goes a long way towards helping all of us.
There’s no proactive responsible action that can make up for the project maintainers completely ignoring you for weeks. Vuln reporters should also be given some grace and shouldn’t immediately be suspected of bad behaviour.
People tend to not understand how much effort goes into responsible disclosure and being charitable often times leads to sitting on vulns for far longer than is safe (364 days is my record). A non-responsive vendor always puts you in a rock and a hard place and almost all you have to fall back on is a disclosure policy and evidence that you tried, which is what seems like happened here based on the other surrounding context. It’s not the job the vulnerability reporter to magically understand your org structure and comms.
Vuln reporters should also be given some grace and shouldn’t immediately be suspected of bad behaviour.
In the general case, I would agree with you. But, there is context missing here, right? It wasn’t “weeks”, it was like “week”, the person was given access to the tracker from what I’ve read elsewhere, a fix was in the works, and the person has previously been pretty combative in the past in the issue tracker.
It sucks that this bug happened. It sucks that the issue didn’t get resolved on the timeline people wanted. It sucks that the behavior of the reporter months ago probably hurt their credibility. It extremely sucks that empathy and grace seem to be continually absent from parts of the lix/nix developer base for whatever reason.
I just wish we could all do better than being in a crab bucket where in order for one version of the nix ecosystem to flourish people seem to want other versions to fail.
If you read the Fedi thread–it seems like the Lix people are eager to cooperate and want Nix to succeed, while the Nix people are unavailable, or on vacation, or didn’t configure tracker access properly, and then came in hot throwing around accusations. That doesn’t build credibility. Take vacations but at least don’t immediately jump to accusations.
That doesn’t build credibility. Take vacations but at least don’t immediately jump to accusations.
Deciding after 7 days that your vulnerability report is blackholed and then dropping a 0day seems to be exhibiting a similar “coming in hot”, no? In order to get credit for being empathetic/giving grace, one must actually be empathetic and give grace.
It’s easy to say “Lix is eager to cooperate and want Nix to succeed” but their actual actions–here and elsewhere–do not support that narrative cleanly.
but their actual actions–here and elsewhere–do not support that narrative
I would believe this if the Nix developer didn’t acknowledge that the security team already knew the timeline for disclosure, immediately after (in the same fediverse thread no less) pretending like the disclosure was a surprise decision. If the Nix team needed more time, why not just ask for more time instead of acting surprised in public about timelines they had previously known about?
As a user, a project (open source or not) that can’t handle security issues in a timely manner, and blames other parties instead of owning up to and working to resolve shortcomings, is already deep in the process of failing. All you can do as an outsider is raise the flag and hope that it makes a difference, which is all I’ve seen the Lix team do here.
How long should it be until disclosure when its already potentially reached a “in-the-wild” status?
Like, the disclosure that you are not running a safe setup should be at or before that point, yes?
Further, I can’t remember which vogue list it was exactly, but as soon as any information gets out about a given vuln, they don’t allow any use of their list for coordinated disclosure (paraphrasing and loose memories of it all, but). At this point with what is known publicly and discussed publicly, It was already far past that threshold.
IMO at this point it was more benefit to the consumers of the software to be informed, than it was for only the ever-watching exploit hunters to be partially informed.
At risk of repeating what others have said in several comments, if you read fedi-thread it is clear.
The reporter even noted on the original vulnerability report that they would disclose after 7 days if there was no communication, and were willing to extend the deadline.
Nix maintainers never reached back nor asked for an extension.
Given that other vulnv reports were sitting unanswered since February while forks (Lix) had already addressed them, the conclusion is obvious.
Out of curiosity, were you able to reach out to anybody at the Nix project about this?For something like this, it’s helpful to have redundant messaging attempts and I imagine everybody that uses Nix in whatever flavor would benefit from cooperation in case channels get garbled (as it looks like they did here).
Open-source shouldn’t be a zero-sum game, and giving people grace and taking proactive responsible action when needed goes a long way towards helping all of us.
Regarding “Responsible disclosure document thingy”; I’d like to remind you that last time I used this, it took a good while for the NixOS security team to get in contact with y’all, and the end result included GHSA-wf4c-57rh-9pjg, which has been open since february 12th (with the description of the vuln being, literally, “blabla”) with zero response from anyone on the Nix team since March 8th, even after responding in the advisory multiple times, and specifically noting this in this report too.
After disclosing this vulnerability, Tom told me (September 2nd) that he created the advisory, linked it, but did not give me privileges to see it. I responded to that in 15 minutes, and got no response from him until earlier today (as he was on vacation). I’d not call this “being in touch”. At no point during that did any other member of the Nix team proactively contact me with any information, and as a point release had been made after this vulnerability was well known by the whole team, I did not feel confident that a release would happen in reasonable time. I had already said that I considered publicly disclosing this in a week after reporting it, and at no point was I told to hold off, or when a release was happening. If I was told this, I would’ve waited no problem.
At this point, my trust in the Nix team actually dealing with security issues responsibly is gone, and has been for months.
AI have found a vulnerability in Nix 2.24 that allows any untrusted user or substituter (even without trusted signing key) to potentially escalate to root on macOS and some very quirky Linux setups. Due to the severity of this bug, and my past experiences trying to disclose vulnerabilities to Nix, I am considering disclosing this vulnerability publicly in one week (seven days). Any Nix team member may message me for the details of the vulnerability.
(regarding “past experiences”; GHSA-wf4c-57rh-9pjg (a different vulnerability) was known to the Nix team on February 9th, along with an almost-fully-tested patch. The last communication I received about this vulnerability has been around March 8th, with me poking them again on May 21st, whilst pointing out that it had been patched by Lix nearly a month prior. (Amusingly, had the Nix team applied this fix, or one like it, it would have decreased the scope of the vulnerability mentioned above here.) Based on this, as well as the fact that the new vulnerability can be exploited silently, I believe a seven-day deadline is warranted.)
So anyone thinks that this is a professional way to trying to fix that?
The end users are those who are hurt the most and being irresponsible about security issues allows them to be exploited.
Also this creates a hostile and stressful environment filled with fear. Why have so many developers bad mental health and are burnt out?
This is the standard in infosec vulnerability management. Nix is a large ecosystem securing mission-critical systems, we have a responsibility to report accurately dangerous vulnerabilities in time based on the reputation of vendors.
No offense but if you believe that this will result in what you say, it may be relevant to consider sunsetting the CppNix implementation and re-allocate work in other areas.
Otherwise, I hope the CppNix project will get out of this situation with a better security process, which will also tackle the other vulnerability that has been sitting for months now.
No offense but if you believe that this will result in what you say, it may be relevant to consider sunsetting the CppNix implementation and re-allocate work in other areas.
I don’t think that follows at all from their reasonable question, again listed here for reference:
Also this creates a hostile and stressful environment filled with fear. Why have so many developers bad mental health and are burnt out?
One of the big reasons for the fork was the mental health and stress on developers and the infra team–it seems a double-standard to suddenly say “but it’s okay when we do it, because security!”
A mistake was made in this case that probably caused undue stress to the people working on the problem; it’s okay to admit that and focus on problem-solving around how to handle issues like slow communications in the future.
Pretty sure this has been said before but it’s not the job of vuln researchers to look after the mental health and burnout issues of projects. If the project team don’t respond reasonably–they will be given less leeway in the future. If they complain that it’s too short–it’s a signal that they can’t credibly handle security issues and they don’t have a viable project. Pushing back on researchers is like trying to plug a leak in a dam, it ultimately doesn’t help anybody. Vulnerable users will be in the dark longer and bad actors will have more time to exploit them.
One of the big reasons for the fork was the mental health and stress on developers and the infra team–it seems a double-standard to suddenly say “but it’s okay when we do it, because security!”
A mistake was made in this case that probably caused undue stress to the people working on the problem; it’s okay to admit that and focus on problem-solving around how to handle issues like slow communications in the future.
I believe we all did what we could on our end; we spent countless hours trying to get the respect and attention of the Nix project. I know I spent hours in meetings (Nix team, NixCon, etc.) raising awareness about these ongoing issues. Some folks unfortunately considered them unrealistic. I sympathize with the issues you are talking about, and I take them very seriously. It’s definitely not our goal to achieve this.
Nonetheless, situations sometimes impose to compromise on some objective for the sake of another one, e.g., ensuring that users are aware of an ongoing security event, a very dangerous one.
Maybe, let me write it also very clearly: while Nix is an awesome ecosystem, there’s a bunch of awesomeness everywhere, we also lack maturity in many layers that other large-scale projects figured out over years and years. Instead of taking lessons from them and speeding things up, these practices seem largely ignored or deemed unfeasible. I will say it again: running a Nix implementation is a serious responsibility (even more so when you are the historical one). If this is too hard to carry on due to a lack of resources, it’s not shameful to be honest and scale back. The problem here is one of communication and expectations. We are not demanding features here or bugfixes, for what it’s worth, just that users don’t get their shells popped trivially.
Finally, we have always been, and we are always ready to discuss moving forward from this, to the best of my knowledge, not once we have been reached out. For the sake of completeness, here’s how we are treated on the other side: https://git.lix.systems/lix-project/lix/issues/426.
From the preceding context, I expected this was to show the Nix side treating you poorly, but maybe I misunderstood? as I don’t see what would be objectionable about this:
roberth commented 2024-06-27 15:54:57 +00:00
With the Nix team we’ve posted a fix for a sandbox escape vulnerability.
Next time I’d like to coordinate this with you. Perhaps we could set up a private matrix room for this purpose?
Also I couldn’t find anything about reporting security issues just now; we provide a security policy through GitHub’s UI, so you could perhaps do something similar here, or work around a missing feature with an issue template that contains it.
roberth commented 2024-06-27 15:57:59 +00:00
In fact, let me apologize for failing to coordinate anything with you this first time.
You not only haven’t acknowledged a chance for folks to do better here, you’ve also gone the extra step (”…if it’s too hard to carry on…”) to find a way of throwing more shade at the people you’re claiming to want to cooperate with. You do this right after claiming to “take […mental health/undue stress…] very seriously”. Do you not see the connection between this behavior and injury to other people? Are you ignorant of the harm and lack of empathy here, or do you simply not care because it makes Lix look good and Nix bad?
This backhanded toxic behavior is what makes the Lix project difficult for me–and others, I would assume–to support.
It’s not straightforward. If a vendor refuses to push out fixes, what are you going to do? You can yell louder, but even just that is a form of disclosure and tells attackers where to look (this is already a thing - read the Linux commit log and you will find n-day patches that are undisclosed). Dropping the vuln/exploit makes it clear - vendors, do better.
Ah, that’s not good, then. Do you know what the Determinate installer does? I think my Ubuntu used that less than a year ago and it currently has nix 2.18.1.
We ship the latest version that has passed our validation suite. Occasionally this means we hold back, or roll back to prior versions. Yesterday we rolled back to 2.23.3 for the security issue. We’re in the process of rolling out 2.24.6 as I write this.
Precision.
“Nix” as a term is very overloaded (‘nix the complete system’, ‘nix the language’, ‘nix the tool you call to do things to/with the system’,…).
cppnix is a way to refer to the actual c++ based implementation most commonly in use.
Seems like it should be uncontroversial in a growing ecosystem with multiple implementations anyway.
Over in the world of Ruby we often refer to the canonical implementation as CRuby (formerly MRI, but Matz asked the community to move on from that name since the team is so much more than just him). It helps to disambiguate it from JRuby, Truffle Ruby, etc.
I don’t know what “Nix the complete system” is supposed to be. There is a language and a tool. The difference between a language and a tool is easily distinguishable from context. This seems to be an attempt by Lix people to portray Nix as “just another implementation of Nix”, rather than being the official implementation, which exists in contrast to reimplementations and forks.
I find it a clear disambiguating name with a well-defined prefix (‘the c++ implementation of nix”, just like a rust rewrite might be called “rustnix” (see: https://tvix.dev/) or a future Roc version might be called “rocnix”) so in that sense it doesn’t matter whether that “someone else” has a problem with it or not. You don’t get to pick your nicknames
I don’t think that is widely used. The distro is NixOS and calling things around the the package manager and language the nix ecosystem is totally reasonable.
cppnix is a way to refer to the actual c++ based implementation most commonly in use.
If it is the most common implementation, why change its name if it is already so common and widely used?
For the same reason you can have Java language as a generic term, but when speaking of behaviour of their impls you say OpenJDK or Android Runtime or OpenJ9.
Disambiguation putting precise spotlight on particular git tree maintained in github.com/nixos/nix. Otherwise it could mean the language itself, or broader ecosystem.
Just to be clear, there are several version of Nix in nixpkgs, 2.18.5 is just the current stable version e.g. pkgs.nixVersions.stable (or something like that, I’m on my phone).
FYI, right now neither Kagi nor Google know about the mentioned bug report, “GHSA-h4vv-h3jq-v493”, and the relevant link (https://github.com/advisories/GHSA-h4vv-h3jq-v493) is a 404. So presumably the report is still considered to be in quarantine?
I think you should not assume this link will be resolvable in a few years from now, considering that repository is a “playground” and could get deleted any time.
In Nix convention, a top-level flake.nix file (linked below by puffnfresh), holds the entry points. In this case, the name of the package is appliance_17_image. When building with Nix, you can target that package directly. The . stands for “the flake in the current directory” and # is the access mechanism for the target package.
Working on image-based Linux with NixOS, which I think is quite an exciting match.
NixOS can build disk images with systemd-repart for a long time, and now also has support for building UKIs with ukify. It’s just perfect for software appliances, you can get reproducible, declarative image builds with loads of available packages while securing with measured boot and FDE. Will probably try to write a blog post on the topic once I’m done.
I’m working on a similar stack with Guix, which doesn’t have systemd or even support a seperate /boot partition OOTB– measured boot will be a stretch goal, to say the least. Have a growing appreciation for all those systemd utils. I’d be really interested in seeing or hearing more about how things come together for you!
Company: Edgeless Systems
Company site: https://www.edgeless.systems/
Position(s): Software Engineer - Confidential Computing and AI
Location: Remote / On-Site in Bochum, Germany
Description: Work on bringing confidential computing “to the world”. Our tech stack is very diverse and spans unusually far in both directions, so you’ll probably be faced with a lot of novel challenges that need pioneering work. See the job posting for a more detailed description of what you’ll do.
Tech stack: Go, Nix / NixOS, Kubernetes, Rust, Terraform, AMD SEV-SNP, Intel TDX
Compensation: Competitive, can vouch for that, being an employee myself. Both salary- and equity-wise. Check the posting for more details.
Contact: Apply via the posting, but do mention that you come from lobste.rs!. Also feel free to email me via
ms [at] edgeless.systemsin case of any questions.But.. None of these actually gives you a pointer to a const?
No, they go on the heap. Go is a bit different to most others here; taking a pointer to a stack local is fine (for instance). This essentially is initialising a heap variable.
Exactly, which is why I find this title a little misleading.Additionally, I think the
aws.Stringwill even result in stack-local for the result pointer + internal heap allocation for the actual string in there. Will the Go compiler ever give you a pointer into.textor similar?It’s not misleading to use the terminology of the domain you’re in! This is what Go people mean when they say pointer-to-constant. (And yes, that’s exactly what
aws.Stringdoes.)IMHO this is making it more complex than it needs to be. Go the language doesn’t differentiate between the stack and the heap much like Java. The optimizer may choose to store values on the stack if it is safe to do so.
In practice this optimization is very predictable, but I don’t think it affects the promised semantics of the language.
No body cares. I just want to pass a constant value (e.g. int) where the receiver is expecting an optional value (e.g. *int”). I hate creating generic pointer function in every project.
If one is part of an explicit fork of a project, and you are aware of the original project containing some unpleasant vulnerability, it is easy to forget that this is not about revenge or competing with former team members which did not join you on the fork. It is ones own former users that are hurt by such an unreasonable disclosure method.
I find the title also a bit misleading. 2.24+ is way too greedy. Stating 2.24 would have been accurate, assuming version 2.25 will not have this vulnerability.
I wonder how other implementations such as Tvix did.
What I understood from the discussions around this report is nothing like that. (Disclaimer: I’m part of neither the Nix or Lix communities.)
If I had to make a (completely uninformed) guess, I would guess that the ruffled feathers around the Lix fork made it difficult for the Nix people to take the feedback of the Lix people seriously, and therefore increases the barrier for cooperation. Sounds like a perfectly human reaction, it takes time to rebuild shared trust. But it may be compounded by a not-so-good security response organization.
Well, the vulnerability was acknowledged by a member of the CppNix team and a PoC was provided which made quite clear what could happen, so I don’t think that’s it.
Exchanges between Lix team members and Nix team members happened in the past without too many issues over the Nix codebase as well.
A standard problem in open source software.
There are some people which try to take every opportunity to shoot against nix after that and it is then totally understandable that you try to avoid that.
Indeed, 2.24>=2.24.5 would be better wording here. To be honest, I just copied that from their post. I think Tvix is not susceptible to this exact vulnerability, but it has a lot of other complexity and is alpha software, so I wouldn’t be exactly sure that it’s “secure”, as of now.
The Tvix NAR parser is a completely different beast, and doesn’t have this vulnerability. It’s pretty hardened in general, and I designed it to only accept canonical-form NARs.
I believe we’re not vulnerable to the Unicode normalisation based form of this attack (‘ẽ’ and ‘ẽ’ are distinct encodings, but equivalent filenames on eg macOS), since our Nix store is (currently) always FUSE-based, so we never perform host filesystem operations with attacker-controlled filenames.
Nix developers are not happy about the way this has been disclosed it seems. Some discussion here: https://functional.cafe/@roberth/113109319874108408
The Nix dev started the thread by accusing the vuln reporter of being irresponsible, and is now having explained to him, in detail, the myriad of ways in which he and his team failed to maintain proper security-related communications with infosec people.
I guess the lesson here is don’t start accusing people without knowing all the facts.
In what world is 7 days responsible disclosure? Especially when the project requires a release and distribution.
See https://lobste.rs/s/ixb3v7/nix_2_24_is_vulnerable_remote_privilege#c_qbpdlv
I’m not a security expert, but iiuc, if a vendor is communicating, working with you, and is planning to release a fix, then it’s generally best to give them a few months, in the hopes of minimizing how many people know about an exploit while it’s usable.
But if a vendor is unresponsive and has a history of being so, then the longer you wait, the more bad actors will discover or share the exploit. The rationale then becomes to let the world know ASAP, so that (1) users can decide how to mitigate their risk, and (2) going public may force the vendor to take action.
I’m not affiliated with the Nix team whatsoever, but I find a 7-day timeline for such a vulnerability for a partly (as I learned) volunteer team pretty tight.
The vulnerability report doesn’t mention it directly, but the discussion thread about it on Fedi gives some more context on that deadline: the reporter has had several previous vulnerability reports completely ignored by the Nix development team, including one open since February and still untriaged. The Nix development team received and acknowledged this new Nix 2.24 vulnerability on August 30th (so, > 9 days ago) and they seem to have mostly sat on it until today (the reporter received no further comms), to the extent that a new point release of Nix was released a few days after the vuln was reported and did not contain a fix.
I don’t think the full reasoning for a 7 days timeline was completely stated, but I’ll note that this vuln was also trivial to reverse engineer from very small amounts of public information that was already available. There are only so many places in Nix that are conductive to pre-signing-verification exploits, and looking at all changes made to those code paths since v2.24 made it trivial to find a PoC for this (I had a working PoC 4 days ago based on that info, I was probably not the only person?).
Out of curiosity, were you able to reach out to anybody at the Nix project about this? For something like this, it’s helpful to have redundant messaging attempts and I imagine everybody that uses Nix in whatever flavor would benefit from cooperation in case channels get garbled (as it looks like they did here).
Open-source shouldn’t be a zero-sum game, and giving people grace and taking proactive responsible action when needed goes a long way towards helping all of us.
There’s no proactive responsible action that can make up for the project maintainers completely ignoring you for weeks. Vuln reporters should also be given some grace and shouldn’t immediately be suspected of bad behaviour.
People tend to not understand how much effort goes into responsible disclosure and being charitable often times leads to sitting on vulns for far longer than is safe (364 days is my record). A non-responsive vendor always puts you in a rock and a hard place and almost all you have to fall back on is a disclosure policy and evidence that you tried, which is what seems like happened here based on the other surrounding context. It’s not the job the vulnerability reporter to magically understand your org structure and comms.
In the general case, I would agree with you. But, there is context missing here, right? It wasn’t “weeks”, it was like “week”, the person was given access to the tracker from what I’ve read elsewhere, a fix was in the works, and the person has previously been pretty combative in the past in the issue tracker.
It sucks that this bug happened. It sucks that the issue didn’t get resolved on the timeline people wanted. It sucks that the behavior of the reporter months ago probably hurt their credibility. It extremely sucks that empathy and grace seem to be continually absent from parts of the lix/nix developer base for whatever reason.
I just wish we could all do better than being in a crab bucket where in order for one version of the nix ecosystem to flourish people seem to want other versions to fail.
If you read the Fedi thread–it seems like the Lix people are eager to cooperate and want Nix to succeed, while the Nix people are unavailable, or on vacation, or didn’t configure tracker access properly, and then came in hot throwing around accusations. That doesn’t build credibility. Take vacations but at least don’t immediately jump to accusations.
Deciding after 7 days that your vulnerability report is blackholed and then dropping a 0day seems to be exhibiting a similar “coming in hot”, no? In order to get credit for being empathetic/giving grace, one must actually be empathetic and give grace.
It’s easy to say “Lix is eager to cooperate and want Nix to succeed” but their actual actions–here and elsewhere–do not support that narrative cleanly.
I would believe this if the Nix developer didn’t acknowledge that the security team already knew the timeline for disclosure, immediately after (in the same fediverse thread no less) pretending like the disclosure was a surprise decision. If the Nix team needed more time, why not just ask for more time instead of acting surprised in public about timelines they had previously known about?
As a user, a project (open source or not) that can’t handle security issues in a timely manner, and blames other parties instead of owning up to and working to resolve shortcomings, is already deep in the process of failing. All you can do as an outsider is raise the flag and hope that it makes a difference, which is all I’ve seen the Lix team do here.
How long should it be until disclosure when its already potentially reached a “in-the-wild” status?
Like, the disclosure that you are not running a safe setup should be at or before that point, yes?
Further, I can’t remember which vogue list it was exactly, but as soon as any information gets out about a given vuln, they don’t allow any use of their list for coordinated disclosure (paraphrasing and loose memories of it all, but). At this point with what is known publicly and discussed publicly, It was already far past that threshold.
IMO at this point it was more benefit to the consumers of the software to be informed, than it was for only the ever-watching exploit hunters to be partially informed.
https://lobste.rs/s/ixb3v7/nix_2_24_is_vulnerable_remote_privilege#c_cxqiwb
At risk of repeating what others have said in several comments, if you read fedi-thread it is clear.
The reporter even noted on the original vulnerability report that they would disclose after 7 days if there was no communication, and were willing to extend the deadline. Nix maintainers never reached back nor asked for an extension.
Given that other vulnv reports were sitting unanswered since February while forks (Lix) had already addressed them, the conclusion is obvious.
Out of curiosity, were you able to reach out to anybody at the Nix project about this?For something like this, it’s helpful to have redundant messaging attempts and I imagine everybody that uses Nix in whatever flavor would benefit from cooperation in case channels get garbled (as it looks like they did here).
Open-source shouldn’t be a zero-sum game, and giving people grace and taking proactive responsible action when needed goes a long way towards helping all of us.
I’m not sure there is any happy way to react to somebody releasing a 0day after encountering some communication issues.
https://puckipedia.com/a3sy-g6c4/kxe3 (need to scroll down)
more spicy context:
and yes, the February issue is an entry by the nixcpp main dev, with “blablabla” as the description. https://matrix-client.matrix.org/_matrix/media/v3/download/puck.moe/d3352f2cf9d9a0e84b9d551121335a043cc9971898783d711c266638be5a2e67?allow_redirect=true
So anyone thinks that this is a professional way to trying to fix that? The end users are those who are hurt the most and being irresponsible about security issues allows them to be exploited. Also this creates a hostile and stressful environment filled with fear. Why have so many developers bad mental health and are burnt out?
This is the standard in infosec vulnerability management. Nix is a large ecosystem securing mission-critical systems, we have a responsibility to report accurately dangerous vulnerabilities in time based on the reputation of vendors.
No offense but if you believe that this will result in what you say, it may be relevant to consider sunsetting the CppNix implementation and re-allocate work in other areas.
Otherwise, I hope the CppNix project will get out of this situation with a better security process, which will also tackle the other vulnerability that has been sitting for months now.
I don’t think that follows at all from their reasonable question, again listed here for reference:
One of the big reasons for the fork was the mental health and stress on developers and the infra team–it seems a double-standard to suddenly say “but it’s okay when we do it, because security!”
A mistake was made in this case that probably caused undue stress to the people working on the problem; it’s okay to admit that and focus on problem-solving around how to handle issues like slow communications in the future.
Pretty sure this has been said before but it’s not the job of vuln researchers to look after the mental health and burnout issues of projects. If the project team don’t respond reasonably–they will be given less leeway in the future. If they complain that it’s too short–it’s a signal that they can’t credibly handle security issues and they don’t have a viable project. Pushing back on researchers is like trying to plug a leak in a dam, it ultimately doesn’t help anybody. Vulnerable users will be in the dark longer and bad actors will have more time to exploit them.
I believe we all did what we could on our end; we spent countless hours trying to get the respect and attention of the Nix project. I know I spent hours in meetings (Nix team, NixCon, etc.) raising awareness about these ongoing issues. Some folks unfortunately considered them unrealistic. I sympathize with the issues you are talking about, and I take them very seriously. It’s definitely not our goal to achieve this.
Nonetheless, situations sometimes impose to compromise on some objective for the sake of another one, e.g., ensuring that users are aware of an ongoing security event, a very dangerous one.
Maybe, let me write it also very clearly: while Nix is an awesome ecosystem, there’s a bunch of awesomeness everywhere, we also lack maturity in many layers that other large-scale projects figured out over years and years. Instead of taking lessons from them and speeding things up, these practices seem largely ignored or deemed unfeasible. I will say it again: running a Nix implementation is a serious responsibility (even more so when you are the historical one). If this is too hard to carry on due to a lack of resources, it’s not shameful to be honest and scale back. The problem here is one of communication and expectations. We are not demanding features here or bugfixes, for what it’s worth, just that users don’t get their shells popped trivially.
Finally, we have always been, and we are always ready to discuss moving forward from this, to the best of my knowledge, not once we have been reached out. For the sake of completeness, here’s how we are treated on the other side: https://git.lix.systems/lix-project/lix/issues/426.
From the preceding context, I expected this was to show the Nix side treating you poorly, but maybe I misunderstood? as I don’t see what would be objectionable about this:
Looks like dropping a vulnerability report in public with much less than 7 days warning.
You not only haven’t acknowledged a chance for folks to do better here, you’ve also gone the extra step (”…if it’s too hard to carry on…”) to find a way of throwing more shade at the people you’re claiming to want to cooperate with. You do this right after claiming to “take […mental health/undue stress…] very seriously”. Do you not see the connection between this behavior and injury to other people? Are you ignorant of the harm and lack of empathy here, or do you simply not care because it makes Lix look good and Nix bad?
This backhanded toxic behavior is what makes the Lix project difficult for me–and others, I would assume–to support.
It’s not straightforward. If a vendor refuses to push out fixes, what are you going to do? You can yell louder, but even just that is a form of disclosure and tells attackers where to look (this is already a thing - read the Linux commit log and you will find n-day patches that are undisclosed). Dropping the vuln/exploit makes it clear - vendors, do better.
Note that the Nix version packaged in nixpkgs is still 2.18.4.
2.24.0 was tagged on Aug 1, so unless you’ve specifically opted into using bleeding-edge Nix, you’re not vulnerable.
Unfortunately, the version shipped by the official installer is 2.24.5 (which will apply for almost all non-NixOS users)
Ah, that’s not good, then. Do you know what the Determinate installer does? I think my Ubuntu used that less than a year ago and it currently has nix 2.18.1.
We ship the latest version that has passed our validation suite. Occasionally this means we hold back, or roll back to prior versions. Yesterday we rolled back to 2.23.3 for the security issue. We’re in the process of rolling out 2.24.6 as I write this.
See: https://status.determinate.systems/incidents/1js0r53719f4
Update: done.
detsys’ installer bumped to cppnix 2.24 only few weeks ago https://github.com/DeterminateSystems/nix-installer/commit/6d0bd3ef6b9887ab4e4b3d5c51f721fa505b82d2 There’s been some back and forth rollbacks.
What’s the purpose of calling it “cppnix”?
Precision. “Nix” as a term is very overloaded (‘nix the complete system’, ‘nix the language’, ‘nix the tool you call to do things to/with the system’,…).
cppnix is a way to refer to the actual c++ based implementation most commonly in use.
Seems like it should be uncontroversial in a growing ecosystem with multiple implementations anyway.
Over in the world of Ruby we often refer to the canonical implementation as CRuby (formerly MRI, but Matz asked the community to move on from that name since the team is so much more than just him). It helps to disambiguate it from JRuby, Truffle Ruby, etc.
Feels normal to see the same happen here.
I don’t know what “Nix the complete system” is supposed to be. There is a language and a tool. The difference between a language and a tool is easily distinguishable from context. This seems to be an attempt by Lix people to portray Nix as “just another implementation of Nix”, rather than being the official implementation, which exists in contrast to reimplementations and forks.
The name “CppNix” predates Lix by at least several months, if not years. I think Tvix folks coined it originally? See for example this commit message from last year (just a random example I found): https://code.tvl.fyi/commit/?id=625643559100c15afe79b799c3a76ef880a07210
This is not coining a name.
I’m under impression that you’re trying to police others’ usage of language.
I’m hoping to hear convincing arguments for this apparent attempt at renaming someone else’s project.
I find it a clear disambiguating name with a well-defined prefix (‘the c++ implementation of nix”, just like a rust rewrite might be called “rustnix” (see: https://tvix.dev/) or a future Roc version might be called “rocnix”) so in that sense it doesn’t matter whether that “someone else” has a problem with it or not. You don’t get to pick your nicknames
I don’t think that is widely used. The distro is NixOS and calling things around the the package manager and language the nix ecosystem is totally reasonable.
If it is the most common implementation, why change its name if it is already so common and widely used?
For the same reason you can have Java language as a generic term, but when speaking of behaviour of their impls you say OpenJDK or Android Runtime or OpenJ9.
Same idea as “CPython”.
They named there project themselves like that https://github.com/python/cpython
Yep! And perhaps in a few years the CppNix team will rename themselves as well. Community pressure can steer a core team towards better choices.
Disambiguation putting precise spotlight on particular git tree maintained in github.com/nixos/nix. Otherwise it could mean the language itself, or broader ecosystem.
IMO I think it is silly
2.24.X is not to be considered bleeding edge. It is just the latest version. Bleeding edge is the git variant.
Just to be clear, there are several version of Nix in nixpkgs, 2.18.5 is just the current stable version e.g.
pkgs.nixVersions.stable(or something like that, I’m on my phone).Heck, even in unstable
pkgs.nixis at 2.18.5. (This might’ve been recently changed, but if so I doubt they would’ve rolled back that far!)FYI, right now neither Kagi nor Google know about the mentioned bug report, “GHSA-h4vv-h3jq-v493”, and the relevant link (https://github.com/advisories/GHSA-h4vv-h3jq-v493) is a 404. So presumably the report is still considered to be in quarantine?
it’s now open https://github.com/NixOS/nix/security/advisories/GHSA-h4vv-h3jq-v493
I assume so.
Neat, but where does the
.\#appliance_17_imagestring come from? Is the string after the hash completely arbitrary?https://github.com/blitz/sysupdate-playground/blob/1a0938cb72b58feeb7d79bc15a4c316a8eac401c/flake.nix#L55
I think you should not assume this link will be resolvable in a few years from now, considering that repository is a “playground” and could get deleted any time.
So then all the links in the blog post won’t resolve.
In Nix convention, a top-level
flake.nixfile (linked below by puffnfresh), holds the entry points. In this case, the name of the package isappliance_17_image. When building with Nix, you can target that package directly. The.stands for “the flake in the current directory” and#is the access mechanism for the target package.You don’t need to escape
#if it’s not preceded by whitespace (actual rule might a bit more complex, that’s my mental model).I image the part after the sharp to be the segment selector as is in a URL segment.
It’s a build target in the repository / flake.
Working on image-based Linux with NixOS, which I think is quite an exciting match.
NixOS can build disk images with
systemd-repartfor a long time, and now also has support for building UKIs withukify. It’s just perfect for software appliances, you can get reproducible, declarative image builds with loads of available packages while securing with measured boot and FDE. Will probably try to write a blog post on the topic once I’m done.I’m working on a similar stack with Guix, which doesn’t have systemd or even support a seperate
/bootpartition OOTB– measured boot will be a stretch goal, to say the least. Have a growing appreciation for all those systemd utils. I’d be really interested in seeing or hearing more about how things come together for you!