I’m both supportive of and we participate in the responsible disclosure process for Xen, even those times we don’t make the cut for pre-disclosure. I’m sad someone would go to the effort they have here in a criminal manner when there is more [market] demand for the skillset on display here than I have ever seen before.
I was telling a coworker about this and similar writeups and it turns out he wasn’t aware of the Hacking Team writeup from 2016. It’s detailled and very interesting. I would advise anyone to read it: https://pastebin.com/0SNSvyjJ .
The attackers did not get in through a security flaw in Matrix itself, but via an outdated Jenkins.
The nice thing about Matrix is that it is federated (much like e-mail is); there’s no reason to use the matrix.org server. Instead, you can use your own trusted home server. Another reason to use your own is that the Matrix main server tends to be pretty overloaded so everything is quite slow.
I mean, it doesn’t saying anything about the quality of the Matrix codebase as such, but some things do make you wonder about the level of understanding that the people working on it bring to it:
Attacker gains access to production infrastructure by hijacking a forwarded SSH agent logging into the compromised Jenkins worker
… and the corresponding Issues stop just short of overtly saying that that account had root access on all servers. The picture painted by that combination of a handful of facts makes me really wary…
It looks like the core parts of the protocol (including E2E encryption) could now be under takeover-in-progress by French national security agencies.
New Vector doesn’t look likely to say no to one more reasonably-looking paid change request from France, and also not likely to find security implications if there is a nation-state effort to hide them. Some of the incentives are good, though (it was promised that French goverment agencies will have an option of limited external federation with mainline installations; the current public story even claims that the agencies will run the mainline server code).
For some bugfixes in Synapse, it is worrying the the pre-fix state was widely deployed for months…
So, speaking as the project lead for Matrix, this really isn’t true. ANSSI (the french cybersecurity agency) have not made any code contributions to the E2E crypto or the protocol itself. If they did make contributions, we’d audit them ourselves incredibly carefully, if we even accepted them at all. So far we’ve minimised the number of contributors to libolm (the E2E library) to a very small set of individuals.
To be clear: we would be utterly stupid to sabotage Matrix by somehow giving any entity (whether that’s France, or New Vector, or Ericsson or whoever) a competitive advantage, no matter how much they offered to pay us. We aren’t building Matrix out of short-termist greed, but long-term altruism - to build an open global standard for comms. And we are not naive, and are very aware that some parties might try to sneak in trojan horses, and will do everything to fight them. We’ve spent a lot of time trying to codify this into the governance of the Matrix.org Foundation over at https://github.com/matrix-org/matrix-doc/blob/matthew/msc1779/proposals/1779-open-governance.md.
Now, in terms of some of Synapse’s long-lived pre-1.0 bugs being concerning (e.g. state resets; fungible event IDs)… this is true. But we were fixing these independently of the French deployment as part of our existing 1.0 roadmap. The only difference is that we knew we had to have them fixed in time for France to go live. The actual design and solution and code was written entirely by us, and France does run off the same public synapse tree as everyone else and so got them at the same time with no privileged access.
So, TL;DR: there is categorically not a takeover-in-progress, and it’s very unfortunate if it seems that way from the outside. Instead, DINSIC have been amazingly good open source citizens, and I only wish all government ministries operated like this.
I am indeed surprised that none of the bugs that looked like hotfixes (and persisted for months) were from ANSSI audit. Interesting, thanks for commenting about that.
I only consider the threat model of them managing to find and explain a real problem in a way that leads to a natural fix that has unintended complicated implications — but hopefully they don’t want to create a risk for something they will also be expected to run.
I actually hoped they already started pushing the bug reports — a fresh set of qualified reviewers who have an actual motivation to find out if the protocol works has a good chance of being useful.
Sorry for not wording some parts of my comment well, and thanks for the clarifications that they haven’t yet gave you the results of their (hopefully already ongoing) audit.
So to be clear - there have been at least 3 audits, but primarily around the infrastructure of the deployment rather than the E2EE in matrix, mainly because Matrix is obviously still developing rapidly. Once there’s a full audit on the Matrix side I’d push for it to be public, just like our US-government-funded E2EE audit from 2016.
Ugh. I’ve thought about opening up the Jenkins server on one of my open source projects to the world, but hesitate due to crap like this. Even though I update it regularly, the risk seems high just to have public build statuses (which I could probably just proxy through something else).
I also hate how you have you mount the underlying machine’s docker socket to allow docker build agents. Surely there’s got to be a better user-mode docker solution.
The person responsible seems to have taken the unusual steps of filing Github issues for the developers to tighten the security of the project:
They could have responsibly disclosed instead of being an asshat, stealing information and posting a ton of github issues from a fresh account.
stealing… information?
I’m both supportive of and we participate in the responsible disclosure process for Xen, even those times we don’t make the cut for pre-disclosure. I’m sad someone would go to the effort they have here in a criminal manner when there is more [market] demand for the skillset on display here than I have ever seen before.
Why the hell did github allow people to remove issues? This is annoying.
It appears the issues were removed by GitHub when a third party reported the user that posted the issues.
Unfortunate that GitHub was powerless to prevent nuking their account after being reported.
I was telling a coworker about this and similar writeups and it turns out he wasn’t aware of the Hacking Team writeup from 2016. It’s detailled and very interesting. I would advise anyone to read it: https://pastebin.com/0SNSvyjJ .
thanks a lot, the whole walkthrough is quite amazing and insighful with a wide variety of tools used
Did you get a copy of them? They’re deleted now :(
They’ve been reposted here: https://github.com/matrix-org/matrix.org/issues/371 (and this site has been archived here)
Thanks!
I think web archive has some of them. Maybe not every comments.
Concerning #358, what is “Flywheel” in this context?
Side-note: I hate locked threads on free software projects.
Update: I think it’s a hostname of one of their machines?
Seems like it’s the hostname of their jenkins build slave
yup, it was the hostname of the jenkins build slave.
possibly: https://getflywheel.com/
The attackers did not get in through a security flaw in Matrix itself, but via an outdated Jenkins.
The nice thing about Matrix is that it is federated (much like e-mail is); there’s no reason to use the matrix.org server. Instead, you can use your own trusted home server. Another reason to use your own is that the Matrix main server tends to be pretty overloaded so everything is quite slow.
I mean, it doesn’t saying anything about the quality of the Matrix codebase as such, but some things do make you wonder about the level of understanding that the people working on it bring to it:
… and the corresponding Issues stop just short of overtly saying that that account had root access on all servers. The picture painted by that combination of a handful of facts makes me really wary…
It looks like the core parts of the protocol (including E2E encryption) could now be under takeover-in-progress by French national security agencies.
New Vector doesn’t look likely to say no to one more reasonably-looking paid change request from France, and also not likely to find security implications if there is a nation-state effort to hide them. Some of the incentives are good, though (it was promised that French goverment agencies will have an option of limited external federation with mainline installations; the current public story even claims that the agencies will run the mainline server code).
For some bugfixes in Synapse, it is worrying the the pre-fix state was widely deployed for months…
So, speaking as the project lead for Matrix, this really isn’t true. ANSSI (the french cybersecurity agency) have not made any code contributions to the E2E crypto or the protocol itself. If they did make contributions, we’d audit them ourselves incredibly carefully, if we even accepted them at all. So far we’ve minimised the number of contributors to libolm (the E2E library) to a very small set of individuals.
To be clear: we would be utterly stupid to sabotage Matrix by somehow giving any entity (whether that’s France, or New Vector, or Ericsson or whoever) a competitive advantage, no matter how much they offered to pay us. We aren’t building Matrix out of short-termist greed, but long-term altruism - to build an open global standard for comms. And we are not naive, and are very aware that some parties might try to sneak in trojan horses, and will do everything to fight them. We’ve spent a lot of time trying to codify this into the governance of the Matrix.org Foundation over at https://github.com/matrix-org/matrix-doc/blob/matthew/msc1779/proposals/1779-open-governance.md.
Now, in terms of some of Synapse’s long-lived pre-1.0 bugs being concerning (e.g. state resets; fungible event IDs)… this is true. But we were fixing these independently of the French deployment as part of our existing 1.0 roadmap. The only difference is that we knew we had to have them fixed in time for France to go live. The actual design and solution and code was written entirely by us, and France does run off the same public synapse tree as everyone else and so got them at the same time with no privileged access.
So, TL;DR: there is categorically not a takeover-in-progress, and it’s very unfortunate if it seems that way from the outside. Instead, DINSIC have been amazingly good open source citizens, and I only wish all government ministries operated like this.
I am indeed surprised that none of the bugs that looked like hotfixes (and persisted for months) were from ANSSI audit. Interesting, thanks for commenting about that.
I only consider the threat model of them managing to find and explain a real problem in a way that leads to a natural fix that has unintended complicated implications — but hopefully they don’t want to create a risk for something they will also be expected to run.
I actually hoped they already started pushing the bug reports — a fresh set of qualified reviewers who have an actual motivation to find out if the protocol works has a good chance of being useful.
Sorry for not wording some parts of my comment well, and thanks for the clarifications that they haven’t yet gave you the results of their (hopefully already ongoing) audit.
So to be clear - there have been at least 3 audits, but primarily around the infrastructure of the deployment rather than the E2EE in matrix, mainly because Matrix is obviously still developing rapidly. Once there’s a full audit on the Matrix side I’d push for it to be public, just like our US-government-funded E2EE audit from 2016.
Reminded me of https://medium.com/@h3rm4ns3c/how-i-gained-commit-access-to-all-jenkins-projects-in-30-minutes-and-how-security-warnings-to-the-96c421dad0c3
Ugh. I’ve thought about opening up the Jenkins server on one of my open source projects to the world, but hesitate due to crap like this. Even though I update it regularly, the risk seems high just to have public build statuses (which I could probably just proxy through something else).
I also hate how you have you mount the underlying machine’s docker socket to allow docker build agents. Surely there’s got to be a better user-mode docker solution.
Yeah, that’s unfortunate. Maybe you could script it to generate some static artifacts and then drop them somewhere web accessible?