I had a very similar idea many many years ago before I’d even heard of LSP. Back in 2014 we had nREPL as the cross-editor, language-agnostic development protocol. It was basically only used for evaluation and documentation lookup, but the protocol was extensible and amenable to being used for all kinds of weird things. I proposed adding arbitrary “Code Actions” type functionality to it (along with a proof-of-concept) in a conference proposal: https://p.hagelb.org/clojurewest-2014.org.html
Unfortunately the maintainers of nREPL at the time were really just focused on making Clojure-specific REPL tooling and were not very interested in features outside that domain; even the idea that nREPL was language-agnostic was relegated to a minor footnote in the documentation.
I often think about what could have happened there if nREPL had taken off for cross-editor tooling before Microsoft’s backing caused LSP to suck all the oxygen out of the room. (We definitely would have had fewer problems with UTF-16 at least; that much is sure.)
inside magit’s UI, I want all my text editing shortcuts to work exactly the same as they work in the editor
from magit UI, I want to be able to open files in the editor. If I click on a diff line, I want to see the corresponding file opened in a split view.
The more interesting question is rather why I am not running my editor inside Emacs. Emacs is undoubtedly the closest thing to what I want. It has got most of big ideas right! But I don’t think the small things are as good as they should be.
In-process model, there’s no IPC interface to allow implementing Magit outside of Emacs (this is the missing big idea)
Poor support for asynchronous programming, the only way to do something asynchronously is to do it in an external process
Poor dependency management (admittedly, I’ve stopped using Emacs before it got package.el into core, but I think it still doesn’t do lock files, for example?)
Emacs Lisp is not a particularly great implementation of Lisp
And I am not sure Lisp is the best language for these sorts of things… Though, there isn’t a clearly better alternative. utf-8 TypeScript would be pretty rad, but utf-16 really spoils everything.
Multiple cursors not in core editing model
Ctrl+C/Ctrl+V/Ctrl+z are not part of core editing model (cua mode is not good)
I would assume tsc always could handle utf-8. It is a default encoding since before I began programming. However, JavaScript itself predates that. So JavaScript APis mandate that strings are transparently encoded as UTF-16:
Or rather WTF-16 because lone surrogates are permitted.
JavaScript predates Unicode 2.0 (which introduced UTF-8 and UTF-16) which is why JavaScript (like Java and Windows) allows malformed UTF-16. I don’t know of any systems using UTF-16 that are strict UTF-16 rather than WTF-16.
I would be interested to know anyone has come up with a good textual representation for representing and editing commit graphs (rather than linear ranges of commits):
In a certain rendering, descendant branches are indented, which makes it difficult to perform operations like move commits between branches, since you have to manage the indentation very carefully as well.
In a certain rendering, the list is flat, but it’s harder to grasp the graph topology at a glance, since it’ll involve directives like “the base for this branch is this previous commit in the list”, which potentially requires some reading and scrolling.
That would be the main thing that would be missing for me from a VCS language server design.
there’s https://github.com/9fans/acme-lsp I’ve had issues with it. though it’s unclear if my issues where related to acme-lsp, specific lsp’s or the lsp protocol, probably a bit of all three.
Well I’m guessing it works great for go and I’ve had it work great for typescript and lua, but I struggled a lot getting the deno, vuejs and php lsp’s to work with it.
Some of my problems have been fixed, but overall the whole lsp ecosystem often feels a little half baked to me ? I tend to get more value out of a repl.
you can have a “dashboard” text document, which shows the diff for the latest commit (jj) or staging area (git), a list of recent commits, a list of “branches”, and has “hyper-links” for all VCS operations
I like this idea (very Unix-y). I think a nice way to start would be to have a unified diff LSP? That seems like a subset of the Magit LSP.
And it’s a pretty well-defined format, since the patch tool parses and understands it, and git parses and understands it, etc.
So then you could do plain diff -R, git diff with arbitrary flags, hg diff, jj diff, and then have a UI that you can open in VSCode or any other LSP editor
(This feels like the kind of thing that someone would say you can do with an LLM, though personally I hit at a wall at a certain point)
If you type git diff/status or jj diff/status, then a “document” is printed to stdout.
And then if you were not tied to a terminal (as you desire, and I do too), then the document could become a user interface.
Looking at the Magit UI, that seems possible (https://magit.vc/screenshots/). Although I guess it’s more of a shell model than an editor model, although it seems clear that the two things can be unified. (because they are already are to an extent in Emacs, VS Code, and many other editors)
I guess the UI paradigm for shell is to have a “scroll” of commands and output, whereas an editor has tabs/buffers/attributed text.
But those can be combined, and there have been many attempts
I wonder if this is basically a funding problem … building new programmer-centric UI paradigms may be extremely expensive
(after all, people use Emacs and Vim, which have decades of knowledge in them, and knowledge == expense in software, at least to a first approximation)
@matklad Sooner or later you’re gonna check out the progress on CSTML as a narrow waist right? It’s just a bit funny to see you keep writing articles as if there is no real hope of achieving that goal.
I wish you luck, but I remain deeply skeptical, given my previous experience in the area. The best way to convince me that I am wrong would be to show real-world problems getting solved with the suggested technology.
To be clear I think we’ve already easily cleared the bar of solving real world problems, but we’re in a unique position because of the threat that lock-in poses to developing open standards.
We don’t actually want our infrastructure to become load bearing just yet because then we would miss our opportunity to consult capable engineers like yourself and incorporate their feedback into the version of the protocol that becomes the standard. Once a protocol is is doing real-world work it’s too late to go back and get the basics right using the benefit of experience.
I’ve still only exchanged a few messages with you on the topic. Would you be willing to give me a half an hour of your (highly valuable) time to pitch you the tech and explain why I think the ideas stand up even in the face of skepticism?
That’s fair. I am doing everything within my power to get those results. It’s just hard work. Lots of time spent debugging. Much less pleasant than talking to people.
I assume you’re referring to this proposal (which I think is yours)? If so, a link would be good in the future, as it was more than a bit difficult to find. It’s also not obvious, if that is indeed the proposal, exactly how it addresses @matklad’s points. Which is not to say it doesn’t, only that a bit more context might go a long way.
I’ll also note that solving the technical bits is only a relatively small part of the problem space here. Adoption is the really key bit, and it requires a lot more than just good technical ideas. (Perhaps that is obvious to you, but it isn’t to everyone.)
I view the adoption problem as one of reaching critical mass, since after critical mass is reached the tendency of a well designed narrow waist is to foster a virtuous cycle of further adoption through the efficiencies of interoperability. The real challenge is reaching the tipping point, since the virtues of viral adoption have also helped entrench our competitors deeply in the market despite their relative lack of technical merit
I had a very similar idea many many years ago before I’d even heard of LSP. Back in 2014 we had nREPL as the cross-editor, language-agnostic development protocol. It was basically only used for evaluation and documentation lookup, but the protocol was extensible and amenable to being used for all kinds of weird things. I proposed adding arbitrary “Code Actions” type functionality to it (along with a proof-of-concept) in a conference proposal: https://p.hagelb.org/clojurewest-2014.org.html
Unfortunately the maintainers of nREPL at the time were really just focused on making Clojure-specific REPL tooling and were not very interested in features outside that domain; even the idea that nREPL was language-agnostic was relegated to a minor footnote in the documentation.
I often think about what could have happened there if nREPL had taken off for cross-editor tooling before Microsoft’s backing caused LSP to suck all the oxygen out of the room. (We definitely would have had fewer problems with UTF-16 at least; that much is sure.)
Maybe the fanatics who like to turn every kind of interface into virtual file systems are right :)
Everything is a file! Including the difference between two tree objects :)
Why not run Emacs in the vscode terminal as your magit UI?
I need bidirectional editor integration:
The more interesting question is rather why I am not running my editor inside Emacs. Emacs is undoubtedly the closest thing to what I want. It has got most of big ideas right! But I don’t think the small things are as good as they should be.
Which small things are not as good for you as they should be?
Not intended to be well thought-out list, but
I believe straight.el solves the package management bullet point.
I’ve tried to convert to emacs myself but failed to give up Kakoune’s editing model.
Thanks for your thoughts! Re: typescript, tsc can handle utf-8 now, I think [tsconfig:docs/charset]
I would assume tsc always could handle utf-8. It is a default encoding since before I began programming. However, JavaScript itself predates that. So JavaScript APis mandate that strings are transparently encoded as UTF-16:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String#utf-16_characters_unicode_code_points_and_grapheme_clusters
Or rather WTF-16 because lone surrogates are permitted.
JavaScript predates Unicode 2.0 (which introduced UTF-8 and UTF-16) which is why JavaScript (like Java and Windows) allows malformed UTF-16. I don’t know of any systems using UTF-16 that are strict UTF-16 rather than WTF-16.
Which side of the divide is C# on? I’m assuming almost certainly WTF-16 since that’s kind of necessary for Windows?
The System.String type is Unicode 1.0 (now aka WTF-16), and more recently they’ve added separate UTF-8 strings.
I would be interested to know anyone has come up with a good textual representation for representing and editing commit graphs (rather than linear ranges of commits):
That would be the main thing that would be missing for me from a VCS language server design.
Imagine if Acme and LSP fell in love…
there’s https://github.com/9fans/acme-lsp I’ve had issues with it. though it’s unclear if my issues where related to acme-lsp, specific lsp’s or the lsp protocol, probably a bit of all three.
Wow, thanks for the link! How did you run into it?
Well I’m guessing it works great for go and I’ve had it work great for typescript and lua, but I struggled a lot getting the deno, vuejs and php lsp’s to work with it.
Some of my problems have been fixed, but overall the whole lsp ecosystem often feels a little half baked to me ? I tend to get more value out of a repl.
I like this idea (very Unix-y). I think a nice way to start would be to have a unified diff LSP? That seems like a subset of the Magit LSP.
And it’s a pretty well-defined format, since the
patchtool parses and understands it, and git parses and understands it, etc.https://en.wikipedia.org/wiki/Diff#Unified_format
So then you could do plain
diff -R,git diffwith arbitrary flags,hg diff,jj diff, and then have a UI that you can open in VSCode or any other LSP editor(This feels like the kind of thing that someone would say you can do with an LLM, though personally I hit at a wall at a certain point)
Although I also wonder if it can be integrated with a rich GUI shell, e.g. the “headless shell” I keep talking about
If you type
git diff/statusorjj diff/status, then a “document” is printed to stdout.And then if you were not tied to a terminal (as you desire, and I do too), then the document could become a user interface.
Looking at the Magit UI, that seems possible (https://magit.vc/screenshots/). Although I guess it’s more of a shell model than an editor model, although it seems clear that the two things can be unified. (because they are already are to an extent in Emacs, VS Code, and many other editors)
I guess the UI paradigm for shell is to have a “scroll” of commands and output, whereas an editor has tabs/buffers/attributed text.
But those can be combined, and there have been many attempts
https://github.com/oils-for-unix/oils/wiki/Interactive-Shell
I made that wiki page a long time ago, and I am not sure why I don’t hear about those projects, e.g.
https://domterm.org/index.html
https://hyper.is/
https://github.com/unconed/TermKit - 13 years old, but looks better than the state of the art !!!
I wonder if this is basically a funding problem … building new programmer-centric UI paradigms may be extremely expensive
(after all, people use Emacs and Vim, which have decades of knowledge in them, and knowledge == expense in software, at least to a first approximation)
@matklad Sooner or later you’re gonna check out the progress on CSTML as a narrow waist right? It’s just a bit funny to see you keep writing articles as if there is no real hope of achieving that goal.
I’ve looked at the related set of tools, summarized what I think in https://lobste.rs/s/dzk5tp/lsp_good_bad_ugly#c_1iy5l5 and my opinion on the substance hasn’t changed.
I wish you luck, but I remain deeply skeptical, given my previous experience in the area. The best way to convince me that I am wrong would be to show real-world problems getting solved with the suggested technology.
To be clear I think we’ve already easily cleared the bar of solving real world problems, but we’re in a unique position because of the threat that lock-in poses to developing open standards.
We don’t actually want our infrastructure to become load bearing just yet because then we would miss our opportunity to consult capable engineers like yourself and incorporate their feedback into the version of the protocol that becomes the standard. Once a protocol is is doing real-world work it’s too late to go back and get the basics right using the benefit of experience.
I’ve still only exchanged a few messages with you on the topic. Would you be willing to give me a half an hour of your (highly valuable) time to pitch you the tech and explain why I think the ideas stand up even in the face of skepticism?
I don’t think convincing me in particular have any bearing on whether ideas are good or bad! I could easily be wrong.
As I’ve said, at this point what would convince me are the results, not ideas.
That’s fair. I am doing everything within my power to get those results. It’s just hard work. Lots of time spent debugging. Much less pleasant than talking to people.
I assume you’re referring to this proposal (which I think is yours)? If so, a link would be good in the future, as it was more than a bit difficult to find. It’s also not obvious, if that is indeed the proposal, exactly how it addresses @matklad’s points. Which is not to say it doesn’t, only that a bit more context might go a long way.
I’ll also note that solving the technical bits is only a relatively small part of the problem space here. Adoption is the really key bit, and it requires a lot more than just good technical ideas. (Perhaps that is obvious to you, but it isn’t to everyone.)
That is an overview of my proposal written by me, yes. The definitive (executable) specification for the proposal is written in the proposed language and is still evolving for the moment: https://github.com/bablr-lang/language-en-cstml/blob/trunk/lib/grammar.macro.js
I view the adoption problem as one of reaching critical mass, since after critical mass is reached the tendency of a well designed narrow waist is to foster a virtuous cycle of further adoption through the efficiencies of interoperability. The real challenge is reaching the tipping point, since the virtues of viral adoption have also helped entrench our competitors deeply in the market despite their relative lack of technical merit
[Comment removed by author]
I am also a huge fan of Magit, and I recently had my boyfriend start using Edamagit because I roughly already know how to operate it and help him out.
That’s a really good idea. I wonder if there’s an LSP server for git.