1. 19

    The elephant in the room (post?) is that the reason all these open chat protocols are failing is because of deliberate and serious damage done by attack from corporate software companies, especially Facebook and Google. Back in the day, I used XMPP to chat with people from all over the Internet, and so did a lot of my friends, precisely because it was easy to connect with people outside whatever walled garden you used primarily from a single desktop client software. Google and Facebook deliberately killed that model. That’s on them. Same thing with Slack, which had IRC and XMPP gateways for a long time.

    This is not a situation of market forces shaping markets to consumer benefit. A walled garden chat solution is by its nature a kind of monopoly - if I want to talk to all my friends and co-workers, I am required to buy into:

    • Facebook Messenger
    • SMS
    • Discord
    • Slack
    • WeeChat
    • Snapchat
    • Kik
    • Telegram

    at a minimum, not because I think any of those chat systems are good, but because that’s where my friends are, and they’re locked into them for the same reason.

    New chat systems are hard to build because there is minimal early adoption because nobody has an incentive to adopt because, in many cases, it’s impossible to build reasonable bridges - for example, such things are explicitly against the ToS of Facebook Messenger and Slack, and I’m not sure how one would go about building such a thing for Google Hangouts that didn’t require yet another level of self-hosting.

    Yes, there’s a gap in the open source market, and maybe we can do better, but it’s important not to ignore the deliberate anti-user practices of every chat company out there.

    1. 11

      I remember the early 2000s, where everyone was on multi-client messengers like Trillian. We went through the dark ages, into the light, then the big players defected again and we’re right back where we started.

      1. 9

        fwiw, Matrix is categorically not failing - our active user stats have gone up by about 40% in the last 6 weeks, and tripled over the last year. There are over 11M addressable users on the network and around 40K deployments that we’re aware of one way or another. The OP here is more than slightly frustrating because it’s based on a view of Matrix from a few years ago from when we were in beta, rather than all the work we did to get stable & more performant for the 1.0 release back in June.

        1. 3

          I’ve only recently started playing with Matrix and I have to say that I’m impressed.

          It feels a lot like what I loved about the old IRC/XMPP days (anon signup, communities based on topic, bridging), but with the modern touch of mobile apps and e2e encryption. I do hope it picks up enough adoption to be viable long term.

            1. 4

              the project dashboard you linked shows pretty clearly that almost all the privacy related issues on our radar have been fixed (the giveaway is that the issues are labelled “done” in green and greyed out). we should be releasing synapse 1.4 and riot/web 1.4 this week, which ships these to the wider world. it’s also worth noting that the vast majority of these issues are just changing the defaults in riot to not use identity & integration services by default unless the user has explicitly opted in; ie they are ideological rather than concretely impacting fixes.

              1. 1

                Let’s see when it is shipped. You were talking about it for quite a long time.

                1. 3

                  yup, as you can see from the length of the closed issue list it ended up being a pretty comprehensive project.

            2. 1

              As a Matrix user and longtime host of my own Synapse server at nora.codes, I largely agree with you, but it remains the case that it is not an attractive option for most of my nontechnical friends, not through any fault of Matrix but because of the reasons I outlined.

          1. 0

            Have they fixed the protocol yet, in particular to trusting poisoned graphs?

            1. 3

              grins wryly that an article which is all about “hey look we fixed the protocol and released a 1.0” is met with questions about whether we’ve fixed the protocol

              1. 2

                I’m not sure what you mean by that, but matrix 1.0 and synapse 1.0 include room version 4, which includes the new state resolution algorithm, which is probably the fix to what you refer to.

                https://matrix.org/docs/spec/#complete-list-of-room-versions

              1. 17

                I am thrilled to be one of the newly announced Guardians of the non-profit foundation. I’m happy to answer any questions people may have, though bearing in mind that I’m going to be a little distracted for the next while until my son goes to bed.

                1. 5

                  How were you and the other guardians for the non-profit foundation chosen? Also, what do you expect your day to day work with the foundation to be like? (I mean in this in the most prosaic sense - is there a physical office you go to, or is the work something you will mostly do over email and at occasional conferences? Is this a full-time job that they’re paying you for, or do you do something else with your time as well? These sorts of details are something I’ve wondered about for many non-profit advisory councils associated with open-source software projects, and since you’re here offering to answer questions I thought this would be a good time to do so :) )

                  1. 4

                    We were chosen by Matthew and Amandine who were, by virtue of starting the foundation, the only Guardians that existed before.

                    The Guardian position is definitely not a full time job, and it is not paid. We are spread out across multiple continents, so our communications will take place mostly virtually (over Matrix, naturally) though we may get together physically from time to time. I’m sure New Vector would be happy to loan me a desk to work out if I’m in London, but there is no physical office.

                    1. 4

                      The process we went through in selecting the Guardians was to ask folks who:

                      • Are clearly philosophically aligned with the goals of the project (i.e. radical decentralisation and liberation of communication)
                      • Are widely recognised as independent experts, trusted by the community to keep the project honest
                      • Are independent of commercial factions in Matrix
                      • Ideally use Matrix already, and represent some subset of the community (e.g. Ross on the ‘personal homeserver & legal’ side, Jutta on the ‘corporate homeserver’ side, Jon on the academic side.).
                      • Have experience and understanding of the responsibilities and requirements of being non-exec directors of a non-profit

                      This narrows it down quite a lot, and we thought very carefully about who to invite to join - and happy to say that all our first choices accepted :)

                  2. 4

                    Congrats on the release!

                    What is the official position on bridges?

                    Take sms bridge. If done right, can completely replace sms software for Android, that would be a huge win for freedom.

                    1. 4

                      I don’t know that the foundation has an official position on bridges, but if we did it would probably be something like \o/.

                      The more the merrier!

                      1. 2

                        What do you mean by position? There exist a couple of sms bridges

                        1. 2

                          If resources of the foundation (grant money or something) will be devoted to it.

                          1. 4

                            Currently the Foundation has very little financial resources, beyond a stack of t-shirts and the monthly donations arriving via Patreon & Liberapay. However, New Vector has one person working fulltime on bridges, plus a GSOC student and some support from the rest of the team. The main priority is on IRC, Slack and XMPP, but we try to help other bridge development as best we can too.

                            1. 1

                              That’s the answer I was looking for, thanks.

                        2. 1

                          You may be interested in jmp.chat – if you want to use it from Matrix IIRC there is work happening on a good XMPP bridge so it should be possible.

                          1. 2

                            jmp.chat

                            Wow, interesting, thanks.

                      1. 2

                        Using X.509 certificates to trust servers rather than perspective notaries, to simplify and improve server-side trust. This is a breaking change across Matrix, and we’ve given the community several months now to ensure their homeservers run a valid TLS certificate.

                        Nooooo, not TLS! (Looking at the specs…) Ah, HTTP. Building on existing software, then. Got it.

                        I’m more a fan of building simpler stuff from scratch, though. In this case, this would have meant using something like Noise, which is so much simpler than TLS. And maybe ditch HTTP in favour of a custom binary protocol. And probably triple time to market, so…

                        1. 11

                          You may be interested in https://matrix.org/blog/2019/03/12/breaking-the-100-bps-barrier-with-matrix-meshsim-coap-proxy/ then :) (Imagine that the video didn’t break the layout in the new blog engine…)

                          1. 1

                            really interesting! Hope this gets the attention that it deserves!

                        1. 3

                          Reminding me it’s free isn’t really a way to be endearing during a decomposition of an incident like this. Having run synapse from early on until recently I can attest that the “bad” practices, albeit being addressed over time, show up in more than just devops for the project.

                          It is altruistic, it is a project with good goals. However it’s viewed by many as being a panacea where people are not registering issues and pr(s) against the project. It needs a lot of love still before folks will be running homeservers for the families and friends that don’t become maintenance nightmares.

                          1. 13

                            hi storrgie - fwiw, from my perspective, our failure to handle your GH issues is certainly one of the biggest screwups over the last few years on synapse. your main one (https://github.com/matrix-org/synapse/issues/2419) has been been brought up time and time again; if you recall, I fixed it myself in https://github.com/matrix-org/synapse/pull/2421, only for it to get derailed by overzealous review). I then eventually fixed it again in https://github.com/matrix-org/synapse/pull/5083 a few weeks ago… which has this time been finished off properly and was merged 6 hours ago. For what it’s worth, I can’t think of any other bug in Synapse (or Matrix) which has had such a bumpy ride, but it’s finally been put to bed. It is excruciatingly embarrassing that it took so long, and doubly so that it sounds like it came too late for your use case.

                            In terms of adminability of Synapse - the thing is still not at 1.0, thanks to being t-boned by things like the security incident in the original post here. Yes, there are still some major admin challenges (lack of richness to the admin API; lack of admin GUI; memory usage and room fragmentation being the main ones), but we are still plugging away to fix them. Then, I’m hoping better servers will emerge.

                            In terms of reminding people that the matrix.org server is a best-effort free service: the intention was more to justify why we invested our ops time in building out the paid services (to try to keep the project funded) rather than trying to be endearing or to say ‘you get what you pay for’. sorry if it jarred.

                            Hopefully Matrix will eventually be something you’ll consider running again once we finally escape beta for Synapse.

                            1. 3

                              I can attest that the “bad” practices, (…) show up in more than just devops for the project.

                              This piqued my interest. Could you expand on what other areas of the project have “bad” practices?

                              1. 0

                                Until synapse is replaced with something written in a sane language that isn’t single threaded / has the dreaded GIL, it will not go anywhere.

                                Also their database schema sucks too.

                              1. 2

                                SSH should not be exposed to the general internet

                                We are rolling out a VPN as the main access to dev network

                                Isn’t that just trading one thing for another? What makes this mysterious VPN implementation they moved to more secure than a properly set up SSH server (assuming other SSH concerns in article are addressed)?

                                They are doing a lot of stuff to harden SSH access, then introducing a completely new remote access interface.. seems like an odd move given that they didn’t take steps to initially harden the original interface (SSH) until now.

                                1. 2

                                  Author here. The point is more that the VPN adds security in depth. Access to hosts is still via SSH (and production access is by SSH + jump boxes) but now also need to have VPNed in first. The VPN can then be used for accessing other intranet services (e.g. our internal matrix servers) rather than exposing them to the ’net.

                                1. 8

                                  The attackers did not get in through a security flaw in Matrix itself, but via an outdated Jenkins.

                                  The nice thing about Matrix is that it is federated (much like e-mail is); there’s no reason to use the matrix.org server. Instead, you can use your own trusted home server. Another reason to use your own is that the Matrix main server tends to be pretty overloaded so everything is quite slow.

                                  1. 3

                                    I mean, it doesn’t saying anything about the quality of the Matrix codebase as such, but some things do make you wonder about the level of understanding that the people working on it bring to it:

                                    Attacker gains access to production infrastructure by hijacking a forwarded SSH agent logging into the compromised Jenkins worker

                                    … and the corresponding Issues stop just short of overtly saying that that account had root access on all servers. The picture painted by that combination of a handful of facts makes me really wary…

                                    1. 0

                                      It looks like the core parts of the protocol (including E2E encryption) could now be under takeover-in-progress by French national security agencies.

                                      New Vector doesn’t look likely to say no to one more reasonably-looking paid change request from France, and also not likely to find security implications if there is a nation-state effort to hide them. Some of the incentives are good, though (it was promised that French goverment agencies will have an option of limited external federation with mainline installations; the current public story even claims that the agencies will run the mainline server code).

                                      For some bugfixes in Synapse, it is worrying the the pre-fix state was widely deployed for months…

                                      1. 3

                                        So, speaking as the project lead for Matrix, this really isn’t true. ANSSI (the french cybersecurity agency) have not made any code contributions to the E2E crypto or the protocol itself. If they did make contributions, we’d audit them ourselves incredibly carefully, if we even accepted them at all. So far we’ve minimised the number of contributors to libolm (the E2E library) to a very small set of individuals.

                                        To be clear: we would be utterly stupid to sabotage Matrix by somehow giving any entity (whether that’s France, or New Vector, or Ericsson or whoever) a competitive advantage, no matter how much they offered to pay us. We aren’t building Matrix out of short-termist greed, but long-term altruism - to build an open global standard for comms. And we are not naive, and are very aware that some parties might try to sneak in trojan horses, and will do everything to fight them. We’ve spent a lot of time trying to codify this into the governance of the Matrix.org Foundation over at https://github.com/matrix-org/matrix-doc/blob/matthew/msc1779/proposals/1779-open-governance.md.

                                        Now, in terms of some of Synapse’s long-lived pre-1.0 bugs being concerning (e.g. state resets; fungible event IDs)… this is true. But we were fixing these independently of the French deployment as part of our existing 1.0 roadmap. The only difference is that we knew we had to have them fixed in time for France to go live. The actual design and solution and code was written entirely by us, and France does run off the same public synapse tree as everyone else and so got them at the same time with no privileged access.

                                        So, TL;DR: there is categorically not a takeover-in-progress, and it’s very unfortunate if it seems that way from the outside. Instead, DINSIC have been amazingly good open source citizens, and I only wish all government ministries operated like this.

                                        1. 1

                                          I am indeed surprised that none of the bugs that looked like hotfixes (and persisted for months) were from ANSSI audit. Interesting, thanks for commenting about that.

                                          I only consider the threat model of them managing to find and explain a real problem in a way that leads to a natural fix that has unintended complicated implications — but hopefully they don’t want to create a risk for something they will also be expected to run.

                                          I actually hoped they already started pushing the bug reports — a fresh set of qualified reviewers who have an actual motivation to find out if the protocol works has a good chance of being useful.

                                          Sorry for not wording some parts of my comment well, and thanks for the clarifications that they haven’t yet gave you the results of their (hopefully already ongoing) audit.

                                          1. 3

                                            So to be clear - there have been at least 3 audits, but primarily around the infrastructure of the deployment rather than the E2EE in matrix, mainly because Matrix is obviously still developing rapidly. Once there’s a full audit on the Matrix side I’d push for it to be public, just like our US-government-funded E2EE audit from 2016.

                                      1. 2

                                        Ugh. I’ve thought about opening up the Jenkins server on one of my open source projects to the world, but hesitate due to crap like this. Even though I update it regularly, the risk seems high just to have public build statuses (which I could probably just proxy through something else).

                                        I also hate how you have you mount the underlying machine’s docker socket to allow docker build agents. Surely there’s got to be a better user-mode docker solution.

                                        1. 2

                                          Yeah, that’s unfortunate. Maybe you could script it to generate some static artifacts and then drop them somewhere web accessible?

                                      1. 8

                                        They could have responsibly disclosed instead of being an asshat, stealing information and posting a ton of github issues from a fresh account.

                                        1. 3

                                          stealing… information?

                                          1. 2

                                            I’m both supportive of and we participate in the responsible disclosure process for Xen, even those times we don’t make the cut for pre-disclosure. I’m sad someone would go to the effort they have here in a criminal manner when there is more [market] demand for the skillset on display here than I have ever seen before.

                                          2. 7

                                            Why the hell did github allow people to remove issues? This is annoying.

                                            1. 4

                                              It appears the issues were removed by GitHub when a third party reported the user that posted the issues.

                                              1. 2

                                                Unfortunate that GitHub was powerless to prevent nuking their account after being reported.

                                            2. 4

                                              I was telling a coworker about this and similar writeups and it turns out he wasn’t aware of the Hacking Team writeup from 2016. It’s detailled and very interesting. I would advise anyone to read it: https://pastebin.com/0SNSvyjJ .

                                              1. 1

                                                A 0day in an embedded device seemed like the easiest option, and after two weeks of work reverse engineering, I got a remote root exploit.

                                                thanks a lot, the whole walkthrough is quite amazing and insighful with a wide variety of tools used

                                              2. 3

                                                Did you get a copy of them? They’re deleted now :(

                                                1. 10

                                                  They’ve been reposted here: https://github.com/matrix-org/matrix.org/issues/371 (and this site has been archived here)

                                                  1. 2

                                                    Thanks!

                                                  2. 1

                                                    I think web archive has some of them. Maybe not every comments.

                                                  3. 1

                                                    Concerning #358, what is “Flywheel” in this context?

                                                    Side-note: I hate locked threads on free software projects.

                                                    Update: I think it’s a hostname of one of their machines?

                                                    1. 1

                                                      Seems like it’s the hostname of their jenkins build slave

                                                      1. 2

                                                        yup, it was the hostname of the jenkins build slave.

                                                      2. 1