Threads for arathorn
-
Very intersting, but I would love to hear more details on why matrix is not as scalable. They hint at the merge operations but I don’t understand why that is a problem.
-
We’d like to know too :) Matrix as protocol isn’t inherently unscalable at all. It’s true that every time you send a message in matrix you effectively are merging the state of one chatroom with the state of another one - very similar to how you push a commit in Git. Generally this is trivial, but if there’s a merge conflict, it’s heavier to resolve. The Synapse python implementation was historically terrible at this, but has been optimised a lot in the last 12 months. The Dendrite go implementation is pretty fast too.
There’s an interesting optimisation that we designed back in 2018 where you incrementally resolve state (so called ‘delta state res’), where you only resolve the state which has changed rather than considering all the room state (i.e. all key-value pairs of data associated with the room) en masse. https://matrix.org/_matrix/media/v1/download/jki.re/ubNfLtrmXZMmlGjJZYPnlHHy and https://github.com/matrix-org/synapse/pull/3122 give a bit of an idea of how that works. It would be really cool if Process One is doing something like that with ejabberd, but in practice we suspect that they’ve just done an efficiently implementation of the current state res algorithm. We’ve pinged them on Twitter to see if they want to discuss what they’re up to :) https://twitter.com/matrixdotorg/status/1580549591807975430
-
There’s an interesting optimisation that we designed back in 2018 where you incrementally resolve state
Is it really so hard to see why a protocol that cares about conversation state is more difficult to scale than a protocol that completely ignores it? Seems almost tautological to me.
-
Matrix is certainly more complex to scale (as our inefficient first gen implementations demonstrated), but i think folks are conflating together “it’s complex to write an efficient implementation” with “it doesn’t scale”. It’s like pointing out that writing an efficient Git implementation is harder than writing an efficient CVS implementation; hardly surprising given the difference in semantics.
In practice, you can definitely write a Matrix implementation where all operations (joining, sending, receiving, etc) are O(1) per destination, and don’t scale with the amount of state (i.e. key value pairs) in a room. And to be clear, Matrix never scales with the amount of history in a room; history is always lazyloaded so it doesn’t matter how much scrollback there is.
Historically, joining rooms in Matrix was O(N) with the number of the users in that room, but we’ve recently fixed this with “faster remote joins”, which allows the room state to get lazily synced in the background, thus making it O(1) with size of room, as it should be. https://github.com/matrix-org/matrix.org/blob/80b36d13c3097ffb5ba33572d9011e71940f1486/gatsby/content/blog/2022/10/2022-10-04-faster-joins.mdx is a shortly-to-be-published blog post giving more context, fwiw.
-
The post doesn’t say “Matrix doesn’t scale”, just that XMPP and MQTT scale better. This is because they’re solving dramatically simpler problems. I don’t see anything problems with that claim.
-
-
On the technical side, that’s genuinely impressive work. On the product side, I can’t help but compare with iMessage, signal, WhatsApp and discord being closer to one second.
-
-
-
-
-
We’d like to know too :) Matrix as protocol isn’t inherently unscalable at all
I suspect that this is a question of relative scale. A lot of users of eJabberd are using it as a messaging bus, rather than a chat protocol and so sending a message is likely to be on the order of a few hundred BEAM VM instructions. This is especially true of MQTT, where you don’t have the XML parsing overhead of XMPP and you can parse the packet entirely with Erlang pattern matching. If it’s a deferred message then you may write to Mnesia, but otherwise it’s very fast. In contrast, something that keeps persistent state and does server-side merging is incredibly heavy. That doesn’t mean that it isn’t the right trade off for the group collaboration scale, but it definitely means that you wouldn’t want to use Matrix as the control plane for a massively networked factory, for example.
-
I guess it will be interested to benchmark. To use the git v. cvs example again, I think it’s possible to have an efficient (but complex) merging system like git which outperforms a simple “it’s just a set of diffs” VCS. We certainly use Matrix successfully in some places as a general purpose message bus, although when we need faster throughput we typically negotiate a webrtc datachannel over Matrix (e.g. how thirdroom.io exchanges its world data).
-
The analogy isn’t really matched to this context though. SIP or XMPP or MQTT doesn’t involve diffs or storage or really even state in the basic use case, whereas Matrix is always diffs and merges.
-
Also git and CVS are programs and file formats with (roughly) one implementation, whereas MQTT and Matrix are protocols. The semantics of protocols place an upper bound on the efficiency of any potential implementation.
-
-
-
-
No one said it was unscalable, just that it was harder. If it takes a dedicated team multiple years and a full reimplementation to scale it, and even just joining a room is still slow, that says something.
I currently run Dendrite unfederated, in part (thought not solely) because I don’t want someone to accidentally bring down my small server by joining a large channel somewhere else. I still think Matrix is a good idea, but “scaling Matrix is hard” should be a pretty uncontroversial statement.
-
-
-
-
I’m a little worried about the constant focus on features when the basic experience of just text chats and such can be really rough, in terms of inconsistency and client/server performance. It needs more polish, but what attention is there to give when you’re on the next big thing?
-
-
I’ve really enjoyed using Matrix, hopefully when sync is faster I can lure more of my friends to use it.
-
-
IRC is nice but I prefer Matrix over it due to the ease of use: I got my fiancee to use Matrix with me on my homeserver, but I doubt i would’ve ever got her use IRC. :)
-
-
-
In my experience using Matrix for the last several years, most Matrix clients seem to be struggling to keep up in terms of features/functionality. Which is unfortunate, because the official clients that are web/browser based are slow and frustrating to use on older devices. That said, the Android Element.io app is not too bad :)
I think supporting multiple (unofficial…) clients is very important, since it prevents “vendor lock in”, which can totally happen even when the protocol is federated if everyone ends up depending on the official client and (hypothetically, but not totally impossible…) it is sold/acquired by some nefarious company in the future… I don’t know if the current situation is from Matrix being some quickly moving target, or if implementing the features is just… hard. In any case, it’s not great having such limited options for clients.
-
I wouldn’t say that “most” matrix clients are struggling to keep up in features/functionality - it’s more that we’re still figuring out some features (as per the OP) and everyone (including Element) is playing catchup to a fast moving target.
Totally agreed that vendor lockin is a total antigoal. Element is not the “official” Matrix client - it’s just a client, like Netscape was just a browser in the early days of the web. It happens to be written by the team who created Matrix, but the two are separate these days: matrix.org/foundation v element.io/careers.
In terms of native Desktop apps, we’re hoping matrix-rust-sdk will power a new generation of excellent native apps - ElementX iOS supports macOS too, for instance, and Fractal-next is already a GTK app based on rust-sdk.
-
and Fractal-next is already a GTK app based on rust-sdk.
Yeah… but that doesn’t even support E2EE[1]. I consider that to be a major feature of Matrix, without it Matrix is just a slower way to exchange unencrypted text online. Last time I looked, a few Matrix clients were struggling to implement E2EE.
-
If you look at the checkboxes on that bug, all the hard bits are already done (thanks to leaning on matrix-rust-sdk). You can literally use Fractal-next for E2EE today. They just need to hook up UI for the remaining edge cases (eg key backups). In terms if why that hasn’t happened yet… it’s a FOSS project; PRs welcome.
-
-
Just want to give shoutout to Nheko! I use it daily on my desktop.
-
-
-
-
-
-
-
-
While the way to go forward is interesting, and Matrix is an excellent piece of engineering, it lacks in several areas still, which I would have expected to work flawlessly by now. This is from my own experimenting with the protocol and surrounding server/clients:
- 1:1 calling is lacking, TURN appears to rarely work. Calling from mobile (Android) to Desktop and reverse works <50% of the time
- If you’re logged into both mobile (Android) and desktop, an incoming call picked up by Desktop will continue ringing on the mobile, until it drains the battery. The call pickup even is aparently not properly handled when mobile lockscreen is active.
- Recently, my mobile notifications and message history on mobile is completely off. It appears to be “receiving” messages minutes after they appear on my desktop. Comparing mobile and desktop message history reveals completely out-of-order messages, or simply not-received-on-mobile ones
It feels to me that every time the core features are stable, at least on mobile, something happens in the matrix world that triggers new features that replace the stable features with something new (often incomplete) leaving core functionality flawed. It happened with Riot -> RiotX, and it seems to happen again with Element -> Element X. This is one of the reasons I’ve stopped reporting issues with mobile clients, as there’s no point, a “new” mobile client will most likely soon appear and the cycle will start again.
-
Agreed that RiotX on Android still has some major bugs (which is frustrating, given how long it’s been around). ElementX on Android will likely not be a rewrite however: it is “just” replacing the Kotlin SDK with the Rust SDK, and replacing the calling implementation with Element Call (so we get native conferencing as well as 1:1) - which should address both issues you’ve mentioned here.
-
Will the voice/video call functionality be part of the Rust SDK? How are you implementing WebRTC on the mobile platforms, where presumably running a browser engine’s WebRTC stack inside a web view wouldn’t be a good solution? Are you using the Google WebRTC C++ library directly, do you have some kind of wrapper over it, or are you using a different WebRTC implementation?
-
currently the plan is to run webrtc inside a webview, as per the matryoshka section of the OP. the current mobile apps use libwebrtc directly, with limited success (as you can see from the original comment here), so given we’re switching to multiway native Matrix VoIP the idea is to switch to embedding the Element Call webapp, and then replace that with native impls only if performance actually requires it. So far, webrtc in a webview is actually working fine, and avoids us having to build a new native impl of the relatively complex multiway calling on each platform.
Alternatively, others are very welcome to go wild with libwebrtc or webrtc-rs on top of matrix-rust-sdk or others. After all, it’s all open…
-
-
-
The slow receiving messages on mobile will probably be fixed by the new sync. Mobile clients don’t currently keep syncing all the time in order to save battery. When a push notification arrives, the mobile client needs to sync in order to receive all the relevant data. That takes a long time and lots of data on sync v2. Exactly what sync v3 is supposed to fix.
-
These days, I am unsure what Matrix is heading for. This post explains that they want to have VoIP video conferencing and decentralised virtual reality. Then I open the lobste.rs comments and the first thing I see is a comparison to IRC.
It seems as if Matrix’ mission statement today is going far beyond the goal to open up walled text message gardens. From this post it looks as if they want to make Matrix a decentralised platform for everything. The post talks explicitely about the success of the open web and how Matrix strives to copy it, and that makes me think: don’t we already have the open web? It’s built on a protocol called HTTP. Does this mean Matrix wants to replace HTTP?
If Matrix is indeed inferior even to IRC (I cannot judge as I do not use Matrix) in the domain IRC occupies (text messaging), such a wide approach seems doomed.
-
We’ve always tried to be clear that Matrix is a general purpose comms protocol for realtime data - not just chat. For instance, right from the original launch in Sept 2014 we had VoIP signalling in there too, and did a very basic demo of 3D over Matrix on day 1 too: https://techcrunch.com/video/animatrix-presents-disrupt-sf-2014-hackathon/
The post talks explicitely about the success of the open web and how Matrix strives to copy it, and that makes me think: don’t we already have the open web? It’s built on a protocol called HTTP. Does this mean Matrix wants to replace HTTP?
Obviously we’re not trying to replace HTTP. Matrix is an API layered on top of HTTP (or other transports) to provide a communication layer for realtime data. If anything it competes with ActivityStreams as a way to link streams of activity over the open web - except with a completely different architecture. The reason for invoking the open web is that we simply want to be the comms layer for the open web: a global realtime virtual space where folks can chat, talk, interact, and publish/subscribe to realtime data of any kind.
W3C simply doesn’t provide an API for that, yet - and if they did, hopefully it might be Matrix.
-
The open web is not a federated eventually consistent database. That’s what matrix provides. https://matrix.org/ for more info. An update for people following the blog doesn’t cover the introduction.
Text chat is the first application, but matrix can be used for much more.
-
-
https://matrix.org/blog/2022/08/15/the-matrix-summer-special-2022#wysiwyg
However, given that users are now used to WYSIWYG in Teams and Slack, we’ve now decided to have another go at it
You link to a blog post that shows issues with Slack WYSIWYG, but creating richly formatted messages in MS Teams is also an incredible daily source of frustrations, with no way to opt-out and just use Markdown without that awful dynamic interpolation. I hope just writing Markdown to format messages will still be an option in Element when/if this WYSIWYG editor is introduced.
-
-
Thank you for writing this up, there’s an excellent amount of detail here. The proliferation of silo’ed chat protocols has been one of my pet peeves and has definitely (thus far) been heading the wrong direction; as bad as the proprietary protocols were back in the aughts, they were at least neutrally interoperable in their heyday – these days most companies are downright hostile in how their enforce their ToS when it comes to third-party clients etc.
I’m hoping the relatively smaller ecosystems (e.g. Discord) take note and at least loosen their ToS to allow for calling user APIs without fear of a perma-ban.
A couple of additional questions: is it known if companies might attempt to limit the exposure of their APIs to EU markets only, or does the DMA cover that explicitly? Is the DMA a pre-requisite for fully scaling out use of Matrix Bridging Services – i.e. does the interoperability climate pre-DMA preclude you from offering bridging as a commercial service?
-
Thanks for the positive feedback :)
is it known if companies might attempt to limit the exposure of their APIs to EU markets only, or does the DMA cover that explicitly?
I don’t believe the DMA covers that explicitly, but IANAL. Much like some sites decided to cut off EU traffic rather than implement GDPR, I guess it’s possible that the gatekeepers might only offer open APIs to EU IP addresses - but it feels like the negative PR of doing so (and the theatre of doing so, given how easy it is to get a EU IP) would not be worth it.
Is the DMA a pre-requisite for fully scaling out use of Matrix Bridging Services – i.e. does the interoperability climate pre-DMA preclude you from offering bridging as a commercial service?
Any kind of bridging to a closed service from Matrix (or XMPP) is pretty miserable today, given you have to do adversarial interoperability, which massively reduces the interest in building bridges or relying on them. So yes, DMA would be transformative for bridging and interop in general :)
-
-
-
Personally I’d feel a lot more excited about this if you could give a working demo. I’m glad to see you’re doing VRM support though. I really hate having to re-rig my avatar for every chat app.
-
so we’re in the middle of switching stack from matrix-js-sdk to hydrogen-sdk (hence https://thirdroom.io/ being a bit of a mess right now, although if you try hard enough it might work). There’s a video of the initial demo up at https://www.youtube.com/watch?v=e26UJRCGfGk&t=2263s however. I posted the intro post for the project today because we’re finally working on it fulltime as of this week.
-
-
I enabled beta and feel Spaces is a much polished version of Communities. On the Spaces Beta discussion on the orange site @Arathorn said that the Discord style communities is the boring obvious bit.
For what it’s worth, the thing I find most exciting about Spaces is that they provide a decentralised hierarchical namespace with decentralised access controls for every room (ie pubsub topic) in Matrix. So it’s like we’ve sprouted an openly federated global hierarchical filing system for freeform realtime data streams of all flavours - where people can go crazy defining their own trees, applying their own curation ideals; perhaps we’ll even see a single global tree emerge (although the implementation may need some more optimisation first).
It’s like a multiplayer hybrid of DMOZ and USENET and the read/write Web all rolled together. Once we start storing more interesting data streams than instant messages in it (eg forums, email, bulletin boards, DOMs, scene graphs, ticker data, IOT sensor data…) it really gets interesting :)
“wow we accidentally created the realtime read/write web”
What is the non-obvious and interesting part? His rest of comment went over my head.
-
I was trying to explain that while we wrote spaces to let users group their rooms together, in practice you can create hierarchies of spaces to group all the rooms together. For instance, I could create a space called #root:matrix.org and then a space within it called #opensource:matrix.org and then a space within that called #linux:matrix.org and then fill the space in that with all the linux chatrooms I know about. I could then give ops to other linux experts in the #linux space, and they could delegate ops onwards… until you’ve built a hierarchy that contains all the best chatrooms anyone knows about. It’s a multiplayer way to curate and categorise all the conversations of the world, including those bridged in from other networks and platforms.
Hope that makes more sense!
-
-
-
Why not jump from old and quirky IRC protocol to e.g. Matrix? Also, matrix is an open federation, so this kind of grab shouldn’t be possible.
-
-
-
IRC works well with very slow connections like dialup and archaic machines, but not unstable connections unfortunately, my main complaint is the lack of at least a small chat log without using 3rd party services/sw. Any small disruption will make me lose messages in a rural area internet.
-
-
-
My main issue with Matrix is the lack of a client that I can run easily on extremely low-powered hardware. Just about all the major, well-supported clients are built on Electron. Compare that to IRC: you can have useful IRC client in just about ~5k lines of C (yes, I’ve written my own IRC client).
-
-
i do. the irc client i use is very fast and configurable. i don’t want to run a full web browser and 100000 tons of javascript just to exchange text with people. i currently use the weechat-matrix plugin for weechat to access matrix but it is unmaintained and missing many features i guess.
-
-
Yeah ok, ‘unmaintained’ is a little strong. My point is, nothing new is being added to improve support for matrix (multiline messages, etc) and there are lots and lots of quirks, having used it daily for many months now. And the author has made it clear they have no interest in improving the existing plugin while they go off and RWIR..
It’s “good enough” for me, but a far cry from supporting everything matrix has to offer. That’s the case for almost all matrix clients though, as I’m sure you are aware.
-
-
Fwiw I heard yesterday about https://github.com/poljar/weechat-matrix-rs . When it’s cooked it might be a good way for me to try Matrix seriously.
-
-
-
-
https://github.com/ara4n/random/blob/master/bashtrix.sh as a starting point?
-
https://github.com/tulir/gomuks is roughly 15k lines of Go (+ non-trivial LOC from dependencies of course).
-
-
-
-
Everyone I’ve talked to personally who’s tried this has nothing but horror stories when it comes to running their own homeserver. The consensus I’ve heard is that it’s only practical if you have staff to look after it or if you prevent your channels from federating.
I admire the vision but they have a long way to go before actually realizing the benefits of a decentralized system.
-
I’ve had very few issues running it myself. I have Synapse, Postgres, and Nginx running along with IRC, Discord, and Slack app services on a 2 GB VPS. Other than the occasional upgrade, I’ve had minimal issues. I manage everything through my service manager, so usually it’s as simple as running an upgrade task and then restarting the service. That said, I have a lot of experience running web services, so that might contribute.
-
I’ve configured synapse by hand and using https://github.com/spantaleev/matrix-docker-ansible-deploy/ . Both work well provided you read the documentation.
-
-
-
-
w.kline.sh/
I run my own too (@sumit:battlepenguin.im). It works pretty well, and I even have bridges working. Overall I think it’s way easier to stand up than XMPP (everything is over HTTP; there is that weird federated port but you can now use a normal LetsEncrypt cert and stick it behind a Traefik or HAProxy frontend).
I will say, scaling it would be difficult. I’ve heard other people complain about larger matrix servers with a lot of users and matrix.org has had issues with theirs after multiple huge refactors that dropped CPU usage. I think Matrix would be way better if there were multiple server implementations like ActivityPub does (Mastodon, Pleroma, Peertube, etc.) but it looks like development on the Go implementation is still slow going.
-
-
-
-
-
-
-
Some interesting comments from the project lead over at HN.
And here’s the user-facing post. I created a space for Haskell here (it includes the IRC bridge to #haskell).
-
I created a space for Haskell here (it includes the IRC bridge to #haskell).
URL changed to: https://matrix.to/#/#haskell-space:matrix.org
-
-
You add an alias for the room that is the space. The UI is probably missing currently, but you can use the API. https://matrix.org/docs/spec/client_server/r0.6.1#put-matrix-client-r0-directory-room-roomalias
-
-
-
-
-
I have a hard time figuring out how to get matrix setup and working. Like what the backend and frontend are and how they work. Am I not understanding what it is?
-
TL;DR: If you want to try it out, download the Element client and let it walk you through making an account.
You’ll have to choose a Matrix homeserver (like an email provider). If you won’t use it that frequently, the free matrix.org homeserver is good but slow. For more serious use, consider a subscription to Element Matrix Services, where they host a homeserver for you. Or you can try to self-host synapse. I wouldn’t.
Other homeservers are being developed right now (Conduit is pretty cool). But none are ready for production just yet. And unfortunately the choice of homeserver is still important because your account will be tied to it. In the future, the “multi-homed accounts” feature will make this initial choice less important (hopefully).
There are two basic components to understand if you’re just getting into Matrix, and the two components are best understood as an analog to email, which is really the only popular federated protocol today.
There’s the Matrix homeserver, which is like your email provider. It’s where the messages and account information are stored. It’s what actually takes care of sending and receiving your messages. It’s where you sign up. Multiple people can have an account on the same homeserver. Synapse is the most popular homeserver right now, it’s developed by the team that founded Matrix, and it’s considered to be slow (Python) and is slated to be replaced.
Then there’s Matrix clients. Just like email, Matrix is standardized. You can use any email client to get/send your Gmail, and you can use any Matrix client to get/send Matrix messages. Element is the most popular Matrix client (again made by the team that created Matrix). It’s the most feature-complete by far. It’s written in Electron, so it’s bloated. But it works fairly well.
-
-
I should have been clearer. I meant that I don’t advise trying to self-host the homeserver at all. Self-hosting anything is a ton of work if done properly. Timely security updates, migrations, frequent (and tested) backups, and reliability all come to mind as things I personally don’t want to have to worry about. Element Matrix Services seems like a good deal for just a few people.
-
But these challanges aren’t at all Synapse-specific, are they? Updates. migrations and proper backups are something you have to do with any server that you self-host. And after running a homeserver for a few years, the only migration I ever had to do is from an older to a newer PostgreSQL version by simply dumping the whole database and reading it back in. All schema migrations are done automatically by Synapse and I never had any problems with that. Hosting a Matrix server with Synapse is so easy if you compare it e.g. to hosting your own email server. And Synapse really is battle-tested because it’s dogfooded at the huge matrix.org homeserver instance.
-
No they’re definitely not specific to Synapse. That was pretty much my point.
And I know Synapse has put a ton of work into being easy to deploy. But I still won’t ever recommend managing infrastructure to anyone. It’s awesome that Synapse makes it as easy as possible for people like us to self-host it, but $5/month is probably well worth the lack of headache for most people.
-
-
-
-
-
-
Can you please give a reason why you don’t think Synapse is ready for self-hosting? I’ve been doing it for years with enabled federation and I never had any serious problems.
-
Sure. I’ve heard again and again that if you enable federation on Synapse and someone joins a large room, the server bogs down and starts chewing through resources. Has that changed?
Also note that I’d be running it on an old laptop or a raspberry pi, just like I would run any chat server – IRC, Jabber, etc.
-
-
-
-
-
Uh oh, now I feel even dumber. The main website has information about something called Synapse and there is “element” which is a frontend I believe, but how do you install a matrix server and start using it?
-
My attempt at clarification:
- Matrix is a protocol for a federated database, currently primarily used for chat
- Synapse is the reference home server (dendrite, conduit, construct etc. are other alternatives)
- Element is the reference client (there are versions of element for the web (electron), android and ios)
- A user account is (currently) local to a home server
- A chat room is federated and not located on a specific home server. The state is shared across home servers of all users that have joined the room.
- There are P2P tests where the client and home server are bundled together on e.g. a mobile phone
- Spaces are a way to organize rooms. Spaces are just a special case of a room and can include other rooms and spaces.
-
Thank you! That clarifies a lot. I was stuck thinking Matrix is the server. So, Matrix is a protocol for a federated database, that’s very interesting and cool.
-
Is it legitimate for me, as a user rather than someone who’s interested in the infrastructure, to just think of Matrix being like a finer-grained version of IRC, where instead of (mainly) a few big networks there are more smaller networks and instead of joining e.g. #linux on freenode, I’d join e.g. #linux:somewhere …
Would I now discover ‘rooms’ by starting from a project’s website, for example, rather than just joining some set of federated servers and looking for rooms with appropriate names?
I just searched for ‘linux room matrix’ and the top hit was an Arch Linux room #archlinux:archlinux.org
(I don’t really want to join a general Linux room - just using it as an example)
-
Well, generally NO. Most all matrix home servers are all joined together via the federated protocol. So if you join #archlinux:archlinux.org on homeserver A, and your BFF uses homeserver B, you will still see each other and communicate with each other in that room like if you were both on homeserver A.
One COULD create a non-federated home server, but that’s not the typical use case, and the reasons to do so would be odd. If you are doing for example a chat server for internal chat @ $WORK, using Matrix is probably a terrible idea. Zulip, Mattermost, etc are all better solutions for that use-case.
-
-
-
I did a video to try to explain it last year (which i should update, but it’s still usable, even if it calls Element by its old name of Riot): https://www.youtube.com/watch?v=dDddKmdLEdg
-
-
Synapse is a server implementation and Element is one of the clients.
Installing Synapse: https://matrix.org/docs/guides/installing-synapse
-
-
Not trying to hate on them or anything. @ptman ‘s comment above really helped.
-
-
-
-
-
-
Could this be used to implement (seat underneath) the CapTP (Capability Transport Protocol )?
-
Pinecone is a very generic overlay - while it’s hooked up in Matrix as a way of connecting nodes as a bearer for uTP requests, you could equally well layer IPv6 over it similarly to how Yggdrasil works. CapTP is at a much higher level - aiui it’s effectively distributed RPC with a capability security model, promise pipelining and distributed GC. I see it as more similar to the higher layers of Matrix, which give you decentralised access controls and eventually consistent data replication (albeit no RPC). So: yes, you could absolutely put a CapTP implementation over Pinecone.
-
-
-
Hmm, you might be right. Perhaps, I did have the right ‘categorization’ of capabilities of these systems.
I was looking at CapTP as sort of a modern day CORBA, with the clever methodology to manage trusts across applications representing user/technical account identities (CORBA did not do that).
While Pinecone is a system that manages trust and connectivity at the level of networked devices (nodes) – which in my, perhaps, superficial understanding, CapTP did not do.
At the end of the day one need to trust network nodes, the pipes, the applications, and the user identities.
-
From an introduction to capabilities:
When a system, such as an OS or a website, is presented with a request for a service it provides, it needs to decide if it should actually do what the requestor is asking for. The way it decides is what we’re talking about when we talk about access control. If you’re like most people, the first thing you’re likely to think of is to ask the requestor “who are you?” The fundamental insight of the capabilities paradigm is to recognize that this question is the first step on the road to perdition.
We do not need to trust nodes, applications, nor identities. Capability-oriented computing arises out of an understanding that we do not need to trust remote machines in order to achieve useful computation with them.
-
-
-
-
I am cautiously optimistic that Spaces will get me something I desperately need: a way to have a set of private rooms that an allow list of accounts can enumerate and join. This is super easy with Slack and seems to be basically not possible with Matrix today, even if you set up the existing Community structure. Fingers crossed!
-
yup, that’s very much the hope. https://github.com/matrix-org/matrix-doc/blob/matthew/msc1772/proposals/1772-groups-as-rooms.md#restricting-room-membership-based-on-space-membership is the section of the spec that outlines how it should work - and yes, you can’t do it today in Matrix without a bot to manage it (a bit like how folks used eggdrop decades ago on IRC before chanservs provided access lists).
-
-
As bloat is my main annoyance, I checked out Hydrogen.
It does superficially look nice, but I quickly found there’s no sane way to move encryption keys from other clients (including importing from Element), nor does it seem to have a way for doing the needed verification (initiated by Element).
-
Hydrogen is beta, so: patches welcome. That said, sharing encryption keys with other clients works fine, if you are encrypting them and storing them in secure storage on the server. Manual import/export ironically would be much easier; just needs to be written. You’re right that it doesn’t do verification yet though.
-
-
-
Such a big fan of Matrix. Glad to see this acquisition go well!
-
-
In cased you’ve missed it - Matrix bought Gitter from GitLab.
https://blog.gitter.im/2020/09/30/gitter-element-acquisition/
-
Matrix itself is an open source project + non-profit foundation (https://matrix.org/foundation), and hasn’t acquired anything :)
Element (the startup which makes the flagship Matrix client) did acquire Gitter from Gitlab though, in order to support Gitter and its communities and fund the integration into Matrix.
-
-
It is a meaningful distinction, which is the only reason i was making it.
Matrix predates Element by 3 years, and Matrix is dedicated non-profit software foundation. There are loads of (massive) commercial companies building on the protocol; better known ones include Thales, Kudelski & Ericsson.
Separately, Element is a startup created by the team who created Matrix. It’s true that Element invests a lot of time into pushing Matrix (given if Matrix is successful, Element is successful), and Element acquired Gitter in order to push the protocol - not because it’s a competitor, but to use it a showcase for integrating Matrix into an existing chat platform, to encourage more projects to do so.
To be crystal clear, The Matrix.org Foundation (https://find-and-update.company-information.service.gov.uk/company/11648710) and Element (aka New Vector, https://find-and-update.company-information.service.gov.uk/company/10873661) are literally different companies.
But just as you wouldn’t describe Google or Mozilla acquiring Foo as “The Web acquired Foo” or “The W3C acquired Foo”, it’s very inaccurate to describe Element acquiring Gitter as “Matrix acquiring Gitter”.
-
Element is a startup created by the team who created Matrix.
This is probably where the ‘confusion’ comes from. Just because the Matrix foundation thing and Element are technically/legally separate companies doesn’t mean that the people involved are separate, motivations are separate, etc…
-
-
-
-
Considering the rising number of projects that only offer support through Discord, or other proprietary platforms, I would assume so.
-
-
-
-
-
-
-
Yeah? I’m still lukewarm on Matrix since moving over from Keybase when Zoom bought the latter.
-
-
If anyone from gitter is around, please remove the big ugly black [m] from every message, it is really distracting.
-
-
It would be nice if the core functionality of the iOS client[1] and accessibility of the ecosystem were fixed too, or at least if some visible movement was seen on those. The basic ability of two mutually encrypted users to send each other messages has been broken for months with very little responsiveness from the maintainers.
One of the key issues I have found in general with the matrix ecosystem is that it is highly inaccessible to the people who need it most – non-technical but tech-literate end users. For instance its SDKs and APIs and protocol are all very well (even beautifully) documented for the most part, but client functionality, less so – there is no documentation on the website or anywhere I could find in the repos discussing e.g. how to properly set up cross signing, or even going through the basic steps of how to verify another user’s device (not to mention why one would want to bother, or what E2EE is in layman’s terms)!
Coupling that with removing “escape hatch” functionality like the ability to manually verify someone from the mobile clients, and it becomes exceedingly difficult for non-technical users – even for more technical users like me – to understand when something isn’t working because of improper setup on the part of the user, or because the client has been broken for months as the iOS client has been.
The end result is that the entire ecosystem goes from being a genuinely useful tool for activists, journalists, and vulnerable demographics to being basically a toy whose benefits can only be properly enjoyed by more technical users for whom none of this is a matter of life and death. Blog posts like this, while interesting for someone like me who has the background to understand them, only reinforce this: very few of the blog posts on matrix.org are comprehensible to end users who aren’t also engineers.
-
So there’s a big difference between Matrix the protocol, Synapse the server implementation (subject of the OP here) and Element the client (subject of the comment here). The Matrix.org website and Synapse and the OP are unashamedly focused on developers and sysadmins.
Separately, totally agreed that matrix clients like Element need to do better at usability - the intention is of course to be as easy as WhatsApp / Telegram / Slack etc to use, and there should have some major changes in the next weeks to show visible progress there. Meanwhile, the bug you linked is a plain old bug which we’re hunting down (thank you for providing the logs).
-
I appreciate the reply. I’m aware of the differences between the protocol and the server and client implementations. That said, I’m genuinely excited to hear about those updates, and to know that you agree there’s a usability problem. Thanks for working on that bug.
-
-
Synapse was dog-slow with rooms with a large number of people for a long time, which made the experience of using Matrix frustrating and unreliable. Fixing that is as important for usability by anyone, including nontechnical people, as fixing client-side encryption.
The end result is that the entire ecosystem goes from being a genuinely useful tool for activists, journalists, and vulnerable demographics to being basically a toy whose benefits can only be properly enjoyed by more technical users for whom none of this is a matter of life and death.
People who understand encryption well enough to deal with Matrix client encryption usability issues can just as easily be activists, journalists, and people who would be harmed by having their private communications leaked, as people who don’t. These are orthogonal, not disjoint, categories.
-
Thanks for the point about performance being relevant from an accessibility perspective
With regard to your second point, the set theoretic language obscures more than it helps here since we’re talking about accessibility. These sets are fuzzy: clearly activists, journalists, and others who aren’t software engineers, security researchers, or otherwise have technical (or tech-adjacent) day jobs can and do deal with Matrix client encryption. My claim is that Element is, at the time of writing this, orders of magnitude more challenging to learn, use, and trust if you aren’t in the latter category. For some, my observed experience is that the usability concerns make it completely non-viable.
The fact that some e.g. journalists can persevere and make it work for them is not relevant to my point – that is similar to saying “even though Acme Cafe isn’t wheelchair accessible, it’s still accessible to other people”.
-
-
Bleurgh - looks like video.fosdem.org is having problems. As others have commented, https://www.youtube.com/watch?v=eUPJ9zFV5IE is the same talk (with chapters!) on youtube.