Currently my biggest gripe with async Rust is that in terms of function colouring, it’s 3-colored for no good reason I can find.
Rust has normal functions, impl Future structs and async functions. Need to store an async function’s state in your explicit state machine for manual polling? You can’t because it’s an opaque type.
This is a great article, the gap buffer is surprisingly efficient compared to the fancier data structures.
I would have loved to see my favorite text data structure measured as well, namely the array of lines. It is briefly mentionned at the beginning but I think including it would have given interesting results. It works suprisingly well because on most text (especially code) lines are relatively small, which means you get a pretty balanced rope for free. Many text operations are very fast because it matches what people do with text:
Inserting/removing lines is O(n) in the number of lines, but very fast O(n) because you are just shifting line pointers around.
Editing is O(n) on the length of the line being edited, which is very fast for typical lines
Finding a text location is O(1) (if per byte/line pairs)
I was surprised that there’s a single buffer in the gap buffer case. I’d have thought that this approach would work better with more than one. If you start with, say, 8 gaps scattered throughout the file and you move the closest one, possibly with some bias against the least-recently used one, you’ll end up copying a lot less data. If you keep the gaps sorted, finding the next available one is cheap. The worst-time complexity is the same (assuming a fixed number of holes), but I’d expect the average latency to be better. It would also be easier to extend to allow concurrent mutation with fine-grained locking (probably unimportant for a text editor, but who knows with EMACS).
I’m also curious what happens if you use Linux’s mremap for the moves above a threshold size (e.g. 128 MiB or possibly L3 cache size). If you’re only ever shunting around data within a couple of pages, but are shuffling pages in the page table, this might be faster in the worst cases.
On FreeBSD, these buffers would almost certainly benefit from transparent superpage promotion. On Linux, you probably get a benefit from requesting superpages explicitly.
I was thinking that when a file gets big enough that it might be worth considering mmap shenanigans, instead shard the file into several smaller gap buffers. That would have a similar effect to using multiple gaps, with the restriction that they have to be spaced out a certain amount.
That will cost an extra layer of indirection to find the shard, but might be worth it. Iteration (as in the search) becomes a nested loop, but if the inner loop is over a few MiBs then that’s probably noise.
Reading your comment made me think: Are there any text editors that have two models, and the editor picks the best one when the file is opened?
Say that at file read time you quickly check whether the file looks like a multi-megabyte log file, or whether it looks like a JSON file without whitespace/newlines, or whether it looks like a standard code file. Then you pick a representation that optimizes for the kinds of operations most likely given that file (searching a big file, editing a code file, etc)
With a language like Rust it’s probably not even terribly verbose to put it behind an abstraction so the rest of the editor logic is unaware of the concrete implementation.
Reading the article made me think of a data structure in similar vain. A gap buffer of gap buffers. I’d expect an array of lines to behave poorly when inserting new lines in the middle, but a gap buffer of lines would alleviate that issue without much overhead.
I wonder how much of an advantage the contiguous memory of a single gap buffer is in practice. Is regex’s slice API required by its inner workings, or would it perform the same given an iterator?
Anyway I suppose the difference of ropey in helix and array of lines in kakoune is the reason why kakoune chugs on humongous files, but helix just stops completely.
That’s an interesting benchmark. Adding line numbers in kakoune is actually pretty good (%<a-s>ghi<c-r>#), however adding an empty line after each line (%<a-s>o) is burning a lot of CPU and still running.
Helix just eats the whole CPU at the sight of inserting a character.
Adding an empty line after each line is unfortunately not implemented in the most efficient way, we insert the first line, shift everyting down by one, insert the second line, shift everything down again… The datastructure itself would support a much more efficient implementation where we shift each lines to their final location once, but its a pretty chunky refactoring of the codebase to make this possible.
Also, when profiling here most of the time when doing that operation is actually taken by the indentation hooks, not the buffer modification.
Is regex’s slice API required by its inner workings, or would it perform the same given an iterator?
It needs to be reworked to handle the iterator or streaming case. There is some discussion about it here. The regex author was doubtful it could ever be made as performant as the slice version, but we will have to wait and see. contiguous memory is strong advantage for the search use case.
I would be surprised if ropey was the bottle neck in helix. ropes should handle large files very well.
I thought emacs used array/list of lines instead of a big gap buffer. There was a time when both emacs and vim struggled with lots of long lines (log files mainly) but that is not a common use case. nvi, which is a beast of an editor when it comes to pathological input, also uses list of lines backed by a tree of pages inside a real database (a cut-down copy of BDB).
The article mentions storing the tracking information in a tree. This could be stored in another parallel gap buffer as well. There is a neat trick, which IBM (still?) has a patent on, where the position in the run array entries after the gap are switched to refer from the end of the document. The book “Unicode Demystified” explains this in detail. The SWT StyledTextCtrl makes use of this trick as well (possibly because it was written by the patent holders).
This could be stored in another parallel gap buffer as well. There is a neat trick, which IBM (still?) has a patent on, where the position in the run array entries after the gap are switched to refer from the end of the document.
So if you moved the gap buffer from the front to the back, would that mean you need go through and update every item to point to the front?
yes, if the gap moves then the text buffer contents change so the corresponding style/linestart gap-buffer needs to be updated as well. But these are run entries so in practice, updating these entries is not expensive.
This looks very exciting to me, very close to what I’d want if I were to invest time in building my own distro. I do however wonder if the authors know of distr1.org and what they think of its mounting-based package manager. APK is the fastest “traditional” package manager that I know of, but the idea and benefits of an image-based package manager intrigues me.
Tagged with ‘api’ as this about an interface between software and hardware.
I have mixed feelings about this, mostly bad. I do think that efi was handled about as well as it possibly could have been, and that it succeeded in preserving many of the desirable features of the pc platform (which other platforms do not seem to have succeeded in replicating). But I don’t really see the value proposition, I don’t like the compatibility break in return for so little, and I fear microsoft will try the whole locked-down-secure-boot thing again.
I’m sure that if I made cpus, I would jump at the chance to rip out the legacy, though. But come on. There are far bigger problems you could solve with a ‘minor’ (admittedly still larger) compatibility break (crap encoding, strongly-ordered memory accesses, coherent i$ come to mind, plus a host of smaller ones like page size and dirty high half state).
Seems like you missed the idea completely. Removing legacy booting support has an impact on BIOS and a very small part of OS bring-up code (massively simplifying both) with no impact on userland at all. Changing the ISA or, god forbid, the memory model throws away everything IA has going for, all the legacy userland software.
My last sentence was a throwaway; no particular significance was meant to be attached to it.
Removing legacy booting support has an impact on BIOS and a very small part of OS bring-up code (massively simplifying both)
For an operating system which is already using efi, the changes required should be minimal or nil.
no impact on userland
It was pointed out to me that the removal of rings 1 and 2 and of i/o ports is likely to impact certain legacy drivers that were heretofore ‘portable’ (in that they relied on stable operating system interfaces). I would wager this is not too significant, but it is there. i/o ports can be emulated by trapping, though this is obviously pretty sucky.
Rings 1 and 2 don’t exist in Long Mode, so those legacy drivers haven’t worked on modern OSes in well over a decade if they ever existed in the first place (Linux didn’t use rings 1 and 2, and as far as I know neither did NT).
Looks like you are correct: the AMD systems programming manual chapter 2.2.3 says “Only the L (long), D (default size), and DPL (descriptor-privilege level) fields are used by the processor in 64-bit mode. All remaining attributes are ignored.” The DPL is “descriptor privilege level”, aka access control for ring 0-3, and I can’t find anything talking about their range of options being restricted in long mode.
However looking at Linux 6.3 code, comments in arch/x86 state that ring 1 and ring 2 are unused. Wikipedia handily cites this book (from 2005) stating that Windows does the same. So it seems pretty safe to say that any drivers that use ring 1 and 2 are almost certainly misbehaving anyway, not relying on stable operating system interfaces.
It turns out my understanding was wrong, and you are 100% correct. I checked the AMD system programming manual only to find that the DPL bit is not ignored for code segments (although it is for data segments; instead the DPL from CS is used). I’m not sure why I was under this impression, maybe I conflated it with paging only distinguishing between user/kernel (Rings 3 / Rings 0-2) rather than the full range of privilege levels.
Frankly my biggest gripe with Kakoune is that Helix has been stealing a lot of the attention lately. However Helix’s binary is significantly larger, yet it breaks on larger files, where Kakoune is completely fine. Kakoune also has plugin support like the Helix community is asking for, but the recent migration to Helix resulted in dropped maintenance of Kakoune plugins.
Frankly my biggest gripe with Kakoune is that Helix has been stealing a lot of the attention lately. However Helix’s binary is significantly larger, yet it breaks on larger files, where Kakoune is completely fine. Kakoune also has plugin support like the Helix community is asking for, but the recent migration to Helix resulted in dropped maintenance of Kakoune plugins.
Yeah they did a number on you there because chatGPT is always going to be too expensive. The interesting takeaway is that we are already getting to the point where we can run half decent AI on consumer hardware.
I guess ‘LLM’ scams is too nerdy but how about this: “AI is coming for your bank account” ;)
Yeah, there’s a tradeoff in giving a piece like this a title – does the average reader understand it vs. does it get at the core nugget / idea?
I think one of our main takeaways is that scam followup, not initiation, is now made way easier by conversational LLMs, the fact that they can be run on consumer hardware, and the fact that they confidently BS their way through anything is a feature not a bug.
I was talking to a scammer selling car parts (that they didn’t have) the other day, and it took me five or six emails to realise they were full of shit. My first thought was, man if this were chatGPT I bet I would have wasted a lot more time on it
I was thinking about this – I guess with LLM-based generated text we may no longer be able to distinguish a scam email than a real one. No more PayPall or silly typos. :-)
The silly typos are on purpose to groom their marks. The idiots don’t notice the typos and those are exactly the people they don’t want to target. Chesteron’s Typos.
But the typos are there only because less gullible responders waste scammers’ time. If scamming is automated, they can go after harder-to-scam victims too.
I know you meant “don’t waste”. Yeah the existence of the LLMs mean that an LLM can stalk and customize their attack to the individual prey. Romance scams will rise. They could construct bots to clone the voice of existing phone support people and their methods. World is getting cyberpunk way too quickly.
That “always” is not going to age well. It is already rapidly advancing towards running on cheaper hardware. It’s likely already cheaper for triaging spam responses and followups than manual labor, and is more scalable.
That’s the nuance in the comment and the discussion about the title. ChatGPT is the model by OpenAI. It is expensive not because it requires beefy hardware to run, but because OpenAI can charge whatever they want.
But it’s not the only LLM in town and Facebook’s leaked LLaMA can be run by anyone without paying licensing costs. That’s the cheap LLM for anyone to run, but it’s not ChatGPT.
That’s the nuance in the comment and the discussion about the title. ChatGPT is the model by OpenAI. It is expensive not because it requires beefy hardware to run, but because OpenAI can charge whatever they want.
I can’t share the exact costs, but from what I’ve heard internally, the margins on most of the OpenAI-based products in Azure are much lower than you might expect. These things really are expensive to run (especially if you factor in the amortised cost of the insane cost of training, but even without that the compute costs for each client running inference are pretty large). My cynical side says that this is why Azure is so interested in them: consumer demand for general-purpose compute is not growing that fast, if you can persuade everyone that their business absolutely depends on something that can only run on expensive accelerators then you’ve got a great business model (and, since you’re not using them all of the time, it’s much cheaper to rent time on cloud services than to buy a top-of-the-line NVIDIA GPU for every computer in your org).
Yep. My rumor mill is even more dire; scuttlebutt is that OpenAI is losing money on every query! They’re only accepting the situation because of hopes that the cost of inference (in the cloud) will continue to fall.
I’m surprised they’re already running at a profit! (Unless you mean the margins are an even lower negative than the negative I previously thought they were.)
I just finished porting shift-based DFA to Zig and I have an idea on how to achieve > 7GB/s of utf8 validation. That’s very close to vectorised algorithms without clobbering vector registers or requiring a large dataset to achieve that speed.
I want to implement and test it over the weekend. I could use some help with generating comparison graphs from criterion.rs though…
Debugger GUI that’s better than inserting prints everywhere. Maybe just a modern ddd would do.
VCS that doesn’t break code by automerging and doesn’t require manual merges for obvious situations. Think Pijul, but with semantic diffing and merging.
ABI description language. Think C headers, but with added annotations for possibly shared features (slices, (un)validated UTF{8,16,32} strings, etc.) and without C’s warts.
I’ve found both Intellij and VSCode debugging UIs to be fairly ok. Problem is they’re tied to the editor, which might be a problem if you don’t like the editors.
One thing I really want on my Linux since Unity is a searchable global menubar.
If they can get it to work in a gtk4 wayland world, I’d be extatic. Unfortunately gtk4 doesn’t seem to have a concept of a manu bar and gtk3 doesn’t expose it to dbus when running under wayland.
It’s also a shame how many GUI toolkits still haven’t figure out how to expose a buffer to Wayland and still rely on XWayland to this day (Looking at you JavaFX and Chromium/Electron)
Searchable? You want a search box for the text in the UI?
Yes, that was a thing Unity did: Ubuntu called it the HUD. I hated it myself and wanted a way to turn it off. I don’t know any other desktop that ever did that.
I’m intrigued to find another of the very few people who I ever saw say that they liked it.
A somewhat bad alternative is that you put all the strings into a single allocation, with the strings immediately after the struct which points to them.
But yeah in general the clean thing to do is for every allocation to come with its own matching free()-like function. This lets you support patterns more complicated than just a tree of pointers, like refcounted COW subfields.
Edit to add: this is one place where RAII is pretty clearly a language improvement.
RAII isn’t a requirement here, the only requirement is destructors. Whether they’re called based on lexical scope or some other notion of reachability, being able to return an object that knows how to clean itself up is very useful.
Weirdly, this is even more important for kernel code than userspace, where you often have multiple memory allocators and objects own different kinds of non-memory resources, yet kernel programmers are the ones that tend to be most resistant to using languages with native support for this kind of thing.
I don’t see how RAII is an improvement. The original call becomes the constructor, but then you still have to provide a destructor as a separate function. Yes, they’re nicely bundled together as methods on the same type, but if you decide to not provide a destructor because the return type doesn’t need one, you still can’t add one if you want to change the type in the future.
I suspect that the @0x2ba22e11 is conflating an language mechanism with one of the abstractions that you can build on top. RAII is an idiom. The underlying language mechanism is types that have destructors that are called implicitly when the object goes out of scope. RAII depends on this for automatic storage but it’s useful in this context for structure fields. If the object were defined with fields of this kind of type then it would end up with a synthesised destructor that managed all of the cleanup. A C++ version of getaddrinfo might look something like this:
struct addrinfo
{
int ai_flags;
int ai_family;
int ai_socktype;
int ai_protocol;
std::vector<sockaddr> ai_addr;
std::string ai_canonname;
};
std::list<addrinfo> getaddrinfo(const std::string_view node,
const std::string_view service,
const std::optional<addrinfo> hints);
The returned list is automatically destroyed when it goes out of scope (if it isn’t moved elsewhere). That’s the RAII bit. The list’s destructor explicitly calls the destructor on the elements. The destructor on addrinfo is synthesised and calls the destructor on std::vector, which calls the (trivial) destructor on sockaddr, and calls the destructor on on std::string for the canonical name.
This makes it easy to create APIs that return deep structures. There’s no explicit cleanup in the caller, which means that you at least maintain source compatibility if you add deeper copies.
If you think that you might want to add extra fields later then you can declare an out-of-line destructor, delete the and move constructors, and make the constructor private. This prevents any code other than yours from being able to allocate instances of this structure (so you can prevent them from existing on the stack or in globals) and means that anything that destroys them must call into your code (so if there are fields that it doesn’t know about then this remains fine). That’s a bit more work and it would be nice to have some built-in support.
In particular, it’s easy in C++ to make a class that can be allocated only on the heap and only by unique pointers by making the constructor private but making std::make_unique a friend but you can’t do that with std::shared_ptr. Instead, you need to do a horrific dance where you have a public factory method that has a private subclass of the class that has a public constructor (which can call the private constructor because it’s declared inside a method of the class) and pass that as the type for std::make_shared.
Just because you don’t write it, doesn’t mean it does not exist.
The destructor is still part of the API. As you mentioned, there is a large difference between
The first is always a function call (most likely virtual because libc is usually linked dynamically), even if the destructor itself is a no-op.
The second requires exposing more internal details, which makes compatible library upgrades more difficult and requires recompilation on library updates.
The third disallows any destructor in the future.
If your language is built to support proper deintialization, RAII adds nothing in the first and last case. In the second case, RAII allows for automatic building of the destructor, but as I stated above, I consider this a very bad choice for library interfaces.
Just because you don’t write it, doesn’t mean it does not exist. The destructor is still part of the API. As you mentioned, there is a large difference between
Hence my comment about an explicit out-of-line destructor. This will ensure that it is not inlined into the caller and so its behaviour can be changed later without breaking the ABI.
Adding a public constructor allows the class to be constructed on the stack and in globals. This means that its layout (or, at least, size and the offsets of all public fields) is now part of your ABI. See my post above for how to avoid this: you need a private constructor and factory methods.
The worst thing that C++ inherited from C is lack of awareness of the underlying memory type. I wish it were possible to write a C++ constructor with an explicit type of the memory provided so that you can have different constructors for stack, heap, global, or even guaranteed-to-be-zero memory.
You can reasonably package only object libraries. If the API (header) changed, all dependent objects have to be rebuilt. For rust, this also means invalidation on compiler update.
With that out of the way, a dependency update should only rebuild the dependency once and then relink all depending libraries/applications. And today, linking a binary is freaking cheap.
My biggest gripe with git is diff and merge. They operate on lines, but code has a syntax. Many times have I had unnecessary merge conflicts or merge downright silently breaking code.
Janestreet has shown syntaxes correlate with word boundaries and treesitter can parse many languages directly, but the line-based approach is deeply rooted in git.
Alright, this is cool, don’t get me wrong, but: Why e-ink? I too want a frontlighted panel for programming, but I feel like this usecase woul be better served by an e-paper LCD panel. Like Pebble had, but several years newer.
My old laptop had a cheaper screen which was decently readable in direct sunlight with the backlight off.
A grayscale trasflective panel would IMO be significantly better, as they don’t suffer from e-ink reponse times.
You’re referring to the Sharp memory lcd panels. I don’t recall them ever selling a large panel let alone with something close to hiDPI. You’d probably need a fairly substantial volume expectation for them to consider making such a thing.
Compare to the 13.3 eink grayscale panel IIRC has been around in some form for nearly a decade.
Go’s and Zig’s defer are rather different beasts
Go runs defered statements at the end of the function, Zig at the end of scope.
ant to lock a mutex insife a loop? Can’t use Go defer for that..
destructors can’t take arguments or return values
While most destructions only release acquired resources, passing an argument to a defered call can be very useful in many cases
hidden code
all defer code is visible in the scope. Look for all lines starting with defer in the current scope and you have all the calls.
Looking for destructors means looking how drop is implemented for all the types in the scopes.
Go’s and Zig’s defer are rather different beasts Go runs defered statements at the end of the function, Zig at the end of scope. ant to lock a mutex insife a loop? Can’t use Go defer for that..
This distinction doesn’t really matter in a language with first-class lambdas. If you want to unlock a mutex at the end of a loop iteration with Go, create and call a lambda in the loop that uses defer internally.
destructors can’t take arguments or return values
But constructors can. If you implement a Defer class to use RAII, it takes a lambda in the constructor and calls it in the destructor.
hidden code all defer code is visible in the scope
I’m not sure I buy that argument, given that the code in defer is almost always calling another function. The code inside the constructor for the object whose cleanup you are defering is also not visible in the calling function.
hidden code all defer code is visible in the scope
I’m not sure I buy that argument, given that the code in defer is almost always calling another function. The code inside the constructor for the object whose cleanup you are defering is also not visible in the calling function.
The point is that as a reader of zig, you can look at the function and see all the code which can be executed. You can see the call and breakpoint that line. As a reader of c++, it’s a bit more convoluted to breakpoint on destructors.
This can work sometimes, but other times packing pointers in a struct just so you can drop it later is wasteful. This happens a lot with for example the Vulkan API where a lot of the vkDestroy* functions take multiple arguments. I’m a big fan of RAII but it’s not strictly better.
At least in C++, most of this all goes away after inlining. First the constructor and destructor are both inlined in the enclosing scope. This turns the capture of the arguments in the constructor into local assignments in a structure in the current stack frame. Then scalar replacement of aggregates runs and splits the structure into individual allocas in the first phase and then into SSA values in the second. At this point, the ‘captured’ values are just propagated directly into the code from the destructor.
If you want to unlock a mutex at the end of a loop iteration with Go, create and call a lambda in the loop that uses defer internally.
Note that Go uses function scope for defer. So this will actually acquire locks slowly then release them all at the end of function. This is very likely not what you want and can even risk deadlocks.
Is a lambda not a function in Go? I wouldn’t expect defer in a lambda to release the lock at the end of the enclosing scope, because what happens if the lambda outlives the function?
The first point is minor, and not really changing the overall picture of leaking by default.
Destruction with arguments is sometimes useful indeed, but there are workarounds. Sometimes you can take arguments when constructing the object. In the worst case you can require an explicit function call to drop with arguments (just like defer does), but still use the default drop to either catch bugs (log or panic when the right drop has been forgotten) or provide a sensible default, e.g. delete a temporary file if temp_file.keep() hasn’t been called.
Automatic drop code is indeed implicit and can’t be grepped for, but you have to consider the trade-off: a forgotten defer is also invisible and can’t be grepped for either. This is the change in default: by default there may be drop code you may not be aware of, instead of by default there may be a leak you may not be aware of.
destructors can’t take arguments or return values. While most destructions only release acquired resources, passing an argument to a deferred call can be very useful in many cases.
Yes, more than useful:
Zero-cost abstraction in terms of state: A deferred call doesn’t artificially require objects to contain all state needed by their destructors. State is generally bad, especially references, and especially long lived objects that secretly know about each other.
Dependencies are better when they are explicit: If one function needs to run before another, letting it show (in terms of what arguments they require) is a good thing: It makes wrong code look wrong (yes, destruction order is a common problem in C++) and prevents it from compiling if you have lifetimes like Rust.
Expressiveness: In the harsh reality we live in, destructors can fail.
I think the right solution is explicit destructors: Instead of the compiler inserting invisible destructor calls, the compiler fails if you don’t. This would be a natural extension to an explicit language like C – it would only add safety. Not only that: It fits well with defer too – syntactic sugar doesn’t matter, because it just solves the «wrong default» problem. But more than anything, I think it would shine in a language with lifetimes, like Rust, where long lived references are precisely what you don’t want to mess with.
You can also achieve this with zig cc by specifying glibc version in the target
zig cc -target x86_64-linux-gnu.2.28
also works for C++ and all supported target platforms.
Extrapolating from the benchmarks game seems like a stretch too far for me.
Firstly, the quality of implementation varies greatly between languages. I see no reasonable reason for such a large difference between JS and TS.
Secondly, are the majority of cloud applications doing computations in that language? I would expect most applications to either be working a lot with databases, or to wrangle network IO. Both would see a significant reduction in language dependence, as computations are done in C/C++ DB/OS anyway.
TextMate and Transmit being “better than anything apt-get could give me” sounds rather amusing, considering that both vim and emacs have exploded in popularity in the last decade by taking in lots of those TextMate users :) And TextMate 2 is GPL, haha.
Some programmers use proprietary software, sure. Outside of game dev though they more often than not absolutely despise it.
The number one practical problem with using closed proprietary software is that you’re literally going back to basically where Stallman started – with that printer driver that he couldn’t just fix because it was closed.
doing open source because that’s what we want to do, what we’re reinforced in doing, all the while invoking generalized, thoroughly hypothetical “users” to lend importance and nobility to our hobbies, compulsions, and fancies.
I generally don’t think that much about users when working on hobby projects and don’t care about nobility. My projects are open source just because why the hell would I keep them to myself? They would just go to waste there and probably be lost forever. The public is the best backup/preservation system there is. If I helped someone by doing that, well, that’s just a nice bonus to me, not the goal.
My reference to better-than-apt referred to that earlier time, when TextMate was hot new stuff. The day I bought my license, there wasn’t any comparable substitute in open source. And certainly nothing like the filesystem-based integration with Transit.
Folks cloned the snippet system for Vim and Emacs pretty quickly, at least partway. But that wasn’t really even half the TextMate proposition. It was just the most visible bit, from the screencasts. It took a long time before anyone really went after the bundle system, the configuration flow, the UI, and the command namespace. When they did, they actually maintained TextMate bundle compatibility—direct clone. Eventually, Atom. More or less a GitHub founder’s pet project, so I’m told.
I’m back on Debian now. UltiSnips for Vim, in the terminal. But the kind of work I do has changed. And it’s been 17 years. If Allan Odgaard had kept up pace with new ideas, rather than diverting into a big, HURD-style rewrite, I wonder where editors would be today.
I think it’s fair to set games somewhat apart as its own world. It’s not really the same industry. Ditto film. But my experience doesn’t track yours beyond that. People despise bad software. Despite the hype, open ain’t always better. Mako Hill had a good bit on that.
As for what people despise, I wouldn’t take that as any indication. Twitter’s full of folks venting about open source, too. Especially when they’re made to use it. That’s not to draw any equivalence. It’s just to say samples of grousing don’t tell us much.
The Stallman printer story is canon. But I don’t think that makes it real. Most people, especially these days, don’t want to dive into an unfamiliar system for a feature add. They want to call somebody who knows the system, who’s committed to deliver. Or, failing that, drive-by some GitHub repo with a #feature issue, and miraculously see it closed by a PR on the double.
For good, closed software done right, the end user experience is actually better. You ask, someone capable responds. No one calls you a noob, tells you RTFM, or throws the work to open a pull request back on you. The work gets done by the person best positioned to do it. The experience of the software honors the value of your time.
I have more than a thousand GitHub repos, and have publicly referred to the platform as a shellmound. I’m not sure my random doodle repos counts as open source, for any meaningful sense of the term. GitHub don’t even insist on a license for free storage, as long as you’re comfortable hanging your laundry in public.
The Stallman printer story is canon. But I don’t think that makes it real. Most people, especially these days, don’t want to dive into an unfamiliar system for a feature add.
Counterpoint: For a developer or company with little money, which has been the case for the better part of my career (notably excluding my few years at Microsoft), if they want a feature, fix, or integration for an open-source dependency, they can make it happen given enough time, effort, and skill, but with a closed-source dependency, they’re stuck, unless perhaps they’re good at reverse engineering. That’s a big reason why I prefer open source for libraries, unless there just isn’t a good open-source solution for the problem at hand (e.g. speech synthesis or optical character recognition). Maybe I’m better than most at diving into an unfamiliar codebase and bending it to my will. I find it entirely plausible that Stallman was, or would have been, good at that too.
Arguably, the possible spaces of software and the way it was interacted with and written were smaller, so Stallman probably would have been good at it. He was a systems programmer, in a systems programming environment, who wanted to hack on a systems progam/driver/firmware.
That doesn’t necessarily mean that most or even all developers can or should be able to say, step out of systems programming and instantly know how to fix a React bug that is plaguing them. Software is more diverse and more specific, and programming systems are more layered and orientated towards the problem, and out of that comes subfields that are non-transferrable even if some of the basic concepts are.
I think the problem Stallman faced was that it was illegal to fix the driver. You technically don’t have to have knowledge to fix a React problem, it’s enough you can find someone who can and is allowed to (for payment, if need be).
FLOSS development doesn’t have to be a net drain on money. The FSF’s standard answer is “support and training”, and that’s fine as far as it goes, but it’s really hard to realize in an age where a project without good docs seldom gets traction and many developers choose burnout handling issues rather than asking for money.
Most people, especially these days, don’t want to dive into an unfamiliar system for a feature add
I think that is observably untrue, given the massive number of extensions available for all of the popular editors, for platforms such as GitHub, or even MS Office. The key point to remember is that being able to modify the behaviour of a program does not necessarily depend on its source being available. Somewhat counter-intuitively, it’s often easier in proprietary programs because they’re forced to maintain (and document) stable interfaces for third-party extensions, whereas open source projects can often just tell people to go and hack on the source directly. Specifically on editors, Vim, Emacs, and VS Code all have extension ecosystems that are significantly larger than the core product, which exist because people who were missing a feature decided to dive into an unfamiliar ecosystem and add it. VS Code and Emacs both did well by making that system less unfamiliar to their early userbase by building it on top of a language (Lisp, TypeScript) that this audience used.
As for what people despise, I wouldn’t take that as any indication. … The Stallman printer story is canon. But I don’t think that makes it real. Most people, especially these days, don’t want to dive into an unfamiliar system for a feature add.
I think that, for folks who agree with these points, this makes sense and contextualizes the rest of the post. But not everybody will agree with this. Personally, I have written a Free Software driver in anger, regretting my hardware purchase and petitioning the vendor for documentation. It was a choice made not just from ability, but from desperation.
For good, closed software done right, the end user experience is actually better.
And for bad closed software done wrong? It can destroy hardware and hide malware, just to pick on one particularly rude vendor. Note that I would not be able to publish this sort of code in the Free Software ecosystem and convince people to use it, because folks would complain that the software doesn’t provide any functionality to the users. And this is a foundational weakness of proprietary software vendors: they are incentivized to make harmful software experiences due to privileged legal status. (Or, to be blunt: if I published this sort of malware, I would be arrested and jailed for crimes.)
Haven’t done any real systems hacking in a long while. I did when I was young, and had more time. I definitely had to do some driver and firmware work, to get Linux working with various things. I’m not sure if I was angry going into those projects, but I remember being pretty frustrated coming out of them!
A lot of that came about from buying cheap hardware. I remember one Acer laptop in particular, the one I chose for college: great specs, great price, terrible build quality, total bodge-job, component- and firmware-wise. I eventually bought the MacBook, and came back to Linux on ThinkPads. I pay a premium for ThinkPads, but I get what I pay for, Linux hardware support very much included. It makes way more sense than spending hours brushing up and hacking patches.
As for bad proprietary software: oh yeah, it’s out there. But the idea that software vendors have some inherently privileged legal position doesn’t fly. They have copyright, and can require customers to buy licenses. But the terms of those licenses can and do vary. I have advised on several software license deals, for substantial money, on terms that basically boiled down to Apache 2.0 plus a payment clause, limited to just the paying customer.
If all you’re doing is buying licenses on take-it-or-leave-it terms, or negotiating with highly aggressive companies much bigger than yours, yeah, you’re likely to see terms that strongly favor the vendor. That’s how leverage works. But proprietary software sales get done on sane, fair terms all the time. Perpetual licenses. Modification rights. Meaningful warranties. Accountable maintenance commitments. Sometimes the less impressive product wins out, because the terms offered for it are better.
considering that both vim and emacs have exploded in popularity in the last decade by taking in lots of those TextMate users :) And TextMate 2 is GPL, haha.
Some programmers use proprietary software, sure. Outside of game dev though they more often than not absolutely despise it.
Get out of your bubble. Outside that bubble a lot of developers use JetBrains IDEs or Visual Studio. Visual Studio Code has gained a lot of traction in recent years, but initially mostly because Code was much better for web development than the competition and it is free. Not because it is open source [1].
In the most recent Stack Overflow developer survey, Visual Studio Code is used by 71.07% of developers, Visual Studio by 32.92%, IntelliJ by 29.69%. The most popular fully (?) open source editor is actually Notepad++ with 29,09%. And vim takes the next place at 24,82%, but I wouldn’t be surprised that is because people use vim when doing quick edits remote Linux machines. Emacs dangles somewhere at the bottom of the list with only 5,25%, surpassed by many proprietary applications like Xcode, PyCharm, or Sublime Text.
The number one practical problem with using closed proprietary software is that you’re literally going back to basically where Stallman started – with that printer driver that he couldn’t just fix because it was closed.
I agree that this is a major downside of closed source software. But most people want to fix the things they are working on, not their tools.
PyCharm/IntelliJ IDEA are Apache2 and most devs use the free software version
I have no reason to doubt this, but are you aware of any usage statistics? I’m curious whether “most devs” is more like “just over half” or “over 99%.”
Anecdotally, every company I’ve worked at where the devs used JetBrains IDEs has paid for the commercial version, but totally willing to believe those companies aren’t typical.
From my anecdotal experience, no company has paid for the commercial JetBrains version, even when most of the devs use it. It might very well be a cultural thing.
At the two companies where I used it, we paid for the whole toolbox subscription for every developer who wanted it. Most of us used IDEA Ultimate and one or more of CLion, AppCode or PyCharm Professional. Many of us also used DataGrip.
I still maintain a subscription for my consulting work now, too.
Can’t speak for PyCharm particularly but every other flavor of IDE based on IntelliJ, I only know of companies who have paid, be it PHPStorm or more likely Ultimate (if working on many languages).
Some programmers use proprietary software, sure. Outside of game dev though they more often than not absolutely despise it.
I mean, there’s Vivado, and then there’s Visual Studio, just like there’s, I dunno, a steaming, rotten pile of horse shit and then there’s pancakes.
There are many bubbles in the tech world and game dev is only one of them. I worked in an embedded shop where the state of Linux was that we had one colleague who tried Fedora and he thought it was like a beta or something because he couldn’t put things on the desktop and, in his own words, he expected some things not to work as well as Windows but that was years away from being useful. The thought of writing code in a text editor after finishing college, where they put you through the ritual of compiling stuff you wrote in vim by writing gcc incantations, seemed about as foreign to these guys as the idea of sailing to America on a steam ship.
There are plenty of bubbles where programmers use both kinds of tools, too. Way back when I was doing number crunching, everyone at the lab was using emacs, vim, nedit or notepad, but also Matlab, which everyone hated for various reasons but nowhere near as much as they hated, say, gdb. We worked a lot with another research group at another university where it was the other way around: someone had figured Octave was good enough and they had just one Matlab installation for things Octave really couldn’t do or to port their code, and used the money they saved on Matlab licenses to buy a bunch of Windows and Visual Studio licenses.
I don’t have much in the way of numbers here but you shouldn’t take a tweet about Vivado as the standard about what people think about closed source tools. Vivado is successful because it barely works and there are no alternatives that work (for any non-hobbyist definition of “works”) – it’s successful largely by vendor lockdown.
Well, what can I say, not every case of acute horse diarrhea deserves to be famous :-D.
But seriously, this is one of the things I love about our work. Vivado is huge. I suspect it’s already old enough that virtually everyone who finishes an EE or CompEng degree has seen it at least once (it’s about ten years old, I think, and it replaces another suite called ISE which was also all sorts of horrible in its own way). It’s very likely that it’s been involved in one way or another in dozens of stories that popped up here on lobste.rs, like stories about RISC-V or cryptography or FPGA implementations of classical systems. And it’s perfectly possible for someone to be an excellent programmer and earn a living writing code and never hear about it. No matter how good you are at anything computer related, there’s always something out there big enough that it’s got tens of thousands of people behind it that you had no idea about.
If you keep an open mind, computer engineering will surprise you in all sorts of fresh ways, all the time – not all of them good but oh well. Dogmatism is fun and it feels like you’re right all the time but it’s really fscking boring. I’ve done it and it sucks.
By reputation, another reason Vivado is successful is because all the other FPGA toolchains are reportedly even worse. Don’t get me wrong, Vivado is a tyre fire that segfaults, but the others are apparently even less reliable.
e.g. I’ve heard that at some shops, people writing FPGA code targeting competing FPGAs will actually write and debug their entire project on Xilinx FPGAs with Vivado and then port the code to the other toolchain for the originally intended target FPGA.
I’ve done too little real work on Altera’s stuff to have had first-hand experience but from the little I’ve done I can certainly say Quartus sucked at least as much as Vivado back when I last touched it (2015-ish?). Maybe it’s changed in the meantime but somehow I doubt it :-D. I heard Lattice’s stuff is tolerable but I never tried it. I did try Microsemi’s Libero thing though and it makes Vivado feel like Visual frickin’ Studio. Porting designs between toolchains is not quite like porting program code and it sounds like a really bad idea on paper but, indeed, given how bad some of these tools are, I can imagine it’s just the only way to do things productively sometimes.
But it’s really a lot more complicated than how good the toolchain is. A big – probably the biggest – reason why Vivado and Quartus are so successful is simply that Xilinx and Altera… well, Intel, are really successful. The next version of Vivado could suck ten times as bad as the current one and it would barely put a dent in their market share, just because it’s what you use to do things with Xilinx’ devices. Developer souls are a lot more fungible than silicon.
Also, being so big, they both have roots in the academic world and they run deep, and it’s simply what lots and lots of people learn in school. They’re not very good but as long as the bitstream flows out, who you gonna call?
A competitor with better development tools could, in theory, steal a chunk of this market, but that chunk would be exactly as big as how many programmable logic devices they can sell, and there are constraints – both technical (as in, performance) and non-technical – that dictate that long before anyone even considers the quality of development tools. The usual retort is that developer tools are important for productivity. And they are, but the best designs, from the most productive teams, won’t be worth crap if there are no physical devices for them, or if these devices cannot be obtained on time and in the required quantity and at the right price and so on.
I also secretely suspect that it’s just a case of the audience of these tools being a little more tolerant to weirdness and poor quality. I mean, they taught me how a transistor works in my second year of uni. I then spent the other half of my undergrad years (and I could’ve spent a whole two years of masters’ on that, too) learning about all the ways in which it doesn’t quite work exactly like that. Software segfaulting under your nose is just a tiny drop of weird shit in a very big bucket of weird shit which people way smarter than you reverse-engineered out of nature. Your entire job revolves around manipulating all sorts of weird but inevitable things. So you just learn to chalk up the fact that Vivado crashes if you have more than 64 mux instances whose names start with the letter X under “inevitable” [1], somewhere between “all P-N junctions have leakage current” and “gate capacitance varies with gate oxide thickness”. What’s one more?
[1] Note that this is something I just made up on the spot but it does sound like something Vivado would do…
A competitor with better development tools could, in theory, steal a chunk of this market, but that chunk would be exactly as big as how many programmable logic devices they can sell
You don’t exactly need to sell programmable logic devices to steal a chunk of FPGA development tool market. Symbiflow has already done that with Lattice’s FPGAs, and are slowly starting to bite into Xilinx FPGAs as well. Quicklogic just released their new FPGA product without their own software, just by giving Symbiflow their bitstream generator and FPGA definitions. There are signs that the new Renesas FPGAs are using Yosys (part of Symbiflow) too.
The reason why these closed software tools are so entrenched is that they are tied in with hardware. And as of now, open source hardware is a lot more niche thing, than open source software. With time, that will probably change, but even then, the software landscape in places dealing with hardware will progress faster. Just remember, how a while ago working on microcontrollers almost always meant dealing with vendor-specific IDEs. That is basically gone now. With time, that will happen with most FPGAs as well.
I haven’t tried Symbiflow since 2020 and it’s a project I’m really cheering for, so I’m not going to say anything bad or unenthusiastic about it. But it’s worth keeping in mind that the iCE40 line (the only one that it supports well enough to be usable for real life-ish projects) has a very straightforward, wrinkle-free architecture that lends itself easily to reverse-engineering. Even though the underlying technology is physically the same, as in, it’s based on the same technology (FPGA), the market Symbiflow can realistically target is practically different from the one where people are using Vivado and complaining about it. Its relative success in this field is important and useful, not to mention liberating to a lot of people (including yours truly) but I wouldn’t be too quick to generalize it.
This approach will probably find some success at the low-power, low-cost end of the spectrum, where not only are customers rather unwilling to pay the licensing costs, but some companies, especially fabless vendors like Quicklogic, would be reasonably happy to be rid of software development cost. But this part of the FPGA market works in entirely different ways than the part of the FPGA market that’s bringing the big bucks for Xilinx (and Altera) and where the people who associate the word Vivado with an overwhelming feeling of dread work. For one, it’s not a field where architecture and fabrication technology are important competitive advantages, so there’s not that much value in keeping it closed and keeping the software tied to it.
You’re right that systems programmers and hardware devs prefer small and lean tools, which are often released as open source, but walk into any corporate dev house and you’ll see a lot of Visual Studio and IntelliJ with Xcode and other IDEs sparkled in. The sole exception to this are web devs, whose first usable tool was VScode.
If you have the skills and time to hack on your tools, open source is better, but for most people it’s just a better proposition to pay someone else for their tools and use the saved time to do their job and make that money.
Actually most devs use proprietary software, and kemitchell even mentions this in the post. He switched to Mac, but Windows is still the most popular platform even amongst devs[1]. I suspect the Stack Overflow survey results are even skewed and that Linux likely has less market share than they found.
I hope I made the point that devs do use proprietary software. It’s not true that devs just won’t use closed code. But I don’t have data to support the claim that most devs do. I suppose you could get there by arguing Windows and OS X are closed, and the number of folks on Linux is small. I’ve enjoyed looking at the Stack Overflow survey, but I have no idea how representative that is.
For what it’s worth, when it comes to laptops and desktops, I’m back on Linux again. I did switch to Mac when I was younger. And I did my programming career that way, before going to law school.
I have a trifecta! I have a Linux laptop (which I used when I first joined the company to do development), a Mac laptop (which I also do development on) and a Windows laptop (because our Corporate Overlords are Windows only) that I don’t use, and can’t return (I’ve asked).
But how much of that is because companies require Windows? When I was hired at my current company, half the employees used Macs (mostly developers) and the other half Windows. We then got bought out by an “enterprise” company, and they use Windows exclusively. They even sent me a Windows laptop to use, even though I use a Mac. The Windows laptop sits unpowered and unused, only to be turned on when I’m reminded that I have to update the damn thing.
This does wonders on our work code. I went from several minutes for edit-compile-run to near-interactive.
It is however very trigger-happy on deadlocks. If I set more than 8 threads, it deadlocks every dingle time.
Currently my biggest gripe with async Rust is that in terms of function colouring, it’s 3-colored for no good reason I can find. Rust has normal functions, impl Future structs and async functions. Need to store an async function’s state in your explicit state machine for manual polling? You can’t because it’s an opaque type.
This is a great article, the gap buffer is surprisingly efficient compared to the fancier data structures.
I would have loved to see my favorite text data structure measured as well, namely the array of lines. It is briefly mentionned at the beginning but I think including it would have given interesting results. It works suprisingly well because on most text (especially code) lines are relatively small, which means you get a pretty balanced rope for free. Many text operations are very fast because it matches what people do with text:
I was surprised that there’s a single buffer in the gap buffer case. I’d have thought that this approach would work better with more than one. If you start with, say, 8 gaps scattered throughout the file and you move the closest one, possibly with some bias against the least-recently used one, you’ll end up copying a lot less data. If you keep the gaps sorted, finding the next available one is cheap. The worst-time complexity is the same (assuming a fixed number of holes), but I’d expect the average latency to be better. It would also be easier to extend to allow concurrent mutation with fine-grained locking (probably unimportant for a text editor, but who knows with EMACS).
I’m also curious what happens if you use Linux’s
mremapfor the moves above a threshold size (e.g. 128 MiB or possibly L3 cache size). If you’re only ever shunting around data within a couple of pages, but are shuffling pages in the page table, this might be faster in the worst cases.On FreeBSD, these buffers would almost certainly benefit from transparent superpage promotion. On Linux, you probably get a benefit from requesting superpages explicitly.
I was thinking that when a file gets big enough that it might be worth considering mmap shenanigans, instead shard the file into several smaller gap buffers. That would have a similar effect to using multiple gaps, with the restriction that they have to be spaced out a certain amount.
That will cost an extra layer of indirection to find the shard, but might be worth it. Iteration (as in the search) becomes a nested loop, but if the inner loop is over a few MiBs then that’s probably noise.
Reading your comment made me think: Are there any text editors that have two models, and the editor picks the best one when the file is opened?
Say that at file read time you quickly check whether the file looks like a multi-megabyte log file, or whether it looks like a JSON file without whitespace/newlines, or whether it looks like a standard code file. Then you pick a representation that optimizes for the kinds of operations most likely given that file (searching a big file, editing a code file, etc)
With a language like Rust it’s probably not even terribly verbose to put it behind an abstraction so the rest of the editor logic is unaware of the concrete implementation.
Reading the article made me think of a data structure in similar vain. A gap buffer of gap buffers. I’d expect an array of lines to behave poorly when inserting new lines in the middle, but a gap buffer of lines would alleviate that issue without much overhead. I wonder how much of an advantage the contiguous memory of a single gap buffer is in practice. Is regex’s slice API required by its inner workings, or would it perform the same given an iterator? Anyway I suppose the difference of ropey in helix and array of lines in kakoune is the reason why kakoune chugs on humongous files, but helix just stops completely.
list of lines works fine even for editing at the top of the sqlite3.c file (which contains almost 240k lines and is around 8mb)
inserting in the the middle (or even at the start) of a line string is also fine even for lines that span the entire screen (with wrapping)
the first version of my editor used this data structure and i could easily open and edit mb sized text files
That’s an interesting benchmark. Adding line numbers in kakoune is actually pretty good (
%<a-s>ghi<c-r>#), however adding an empty line after each line (%<a-s>o) is burning a lot of CPU and still running.Helix just eats the whole CPU at the sight of inserting a character.
Adding an empty line after each line is unfortunately not implemented in the most efficient way, we insert the first line, shift everyting down by one, insert the second line, shift everything down again… The datastructure itself would support a much more efficient implementation where we shift each lines to their final location once, but its a pretty chunky refactoring of the codebase to make this possible.
Also, when profiling here most of the time when doing that operation is actually taken by the indentation hooks, not the buffer modification.
It needs to be reworked to handle the iterator or streaming case. There is some discussion about it here. The regex author was doubtful it could ever be made as performant as the slice version, but we will have to wait and see. contiguous memory is strong advantage for the search use case.
I would be surprised if ropey was the bottle neck in helix. ropes should handle large files very well.
I thought emacs used array/list of lines instead of a big gap buffer. There was a time when both emacs and vim struggled with lots of long lines (log files mainly) but that is not a common use case. nvi, which is a beast of an editor when it comes to pathological input, also uses list of lines backed by a tree of pages inside a real database (a cut-down copy of BDB).
The article mentions storing the tracking information in a tree. This could be stored in another parallel gap buffer as well. There is a neat trick, which IBM (still?) has a patent on, where the position in the run array entries after the gap are switched to refer from the end of the document. The book “Unicode Demystified” explains this in detail. The SWT StyledTextCtrl makes use of this trick as well (possibly because it was written by the patent holders).
Looks like patents in the US last 20 years, and the “Unicode Demystified” book was published in 2002, so it’s probably expired by now!
So if you moved the gap buffer from the front to the back, would that mean you need go through and update every item to point to the front?
yes, if the gap moves then the text buffer contents change so the corresponding style/linestart gap-buffer needs to be updated as well. But these are run entries so in practice, updating these entries is not expensive.
This looks very exciting to me, very close to what I’d want if I were to invest time in building my own distro. I do however wonder if the authors know of distr1.org and what they think of its mounting-based package manager. APK is the fastest “traditional” package manager that I know of, but the idea and benefits of an image-based package manager intrigues me.
Tagged with ‘api’ as this about an interface between software and hardware.
I have mixed feelings about this, mostly bad. I do think that efi was handled about as well as it possibly could have been, and that it succeeded in preserving many of the desirable features of the pc platform (which other platforms do not seem to have succeeded in replicating). But I don’t really see the value proposition, I don’t like the compatibility break in return for so little, and I fear microsoft will try the whole locked-down-secure-boot thing again.
I’m sure that if I made cpus, I would jump at the chance to rip out the legacy, though. But come on. There are far bigger problems you could solve with a ‘minor’ (admittedly still larger) compatibility break (crap encoding, strongly-ordered memory accesses, coherent i$ come to mind, plus a host of smaller ones like page size and dirty high half state).
Seems like you missed the idea completely. Removing legacy booting support has an impact on BIOS and a very small part of OS bring-up code (massively simplifying both) with no impact on userland at all. Changing the ISA or, god forbid, the memory model throws away everything IA has going for, all the legacy userland software.
My last sentence was a throwaway; no particular significance was meant to be attached to it.
For an operating system which is already using efi, the changes required should be minimal or nil.
It was pointed out to me that the removal of rings 1 and 2 and of i/o ports is likely to impact certain legacy drivers that were heretofore ‘portable’ (in that they relied on stable operating system interfaces). I would wager this is not too significant, but it is there. i/o ports can be emulated by trapping, though this is obviously pretty sucky.
Rings 1 and 2 don’t exist in Long Mode, so those legacy drivers haven’t worked on modern OSes in well over a decade if they ever existed in the first place (Linux didn’t use rings 1 and 2, and as far as I know neither did NT).
Do you have a reference for that? It was not my understanding, and I can find no mention of it in vol 3, chap. 5 of the intel sdm.
Looks like you are correct: the AMD systems programming manual chapter 2.2.3 says “Only the L (long), D (default size), and DPL (descriptor-privilege level) fields are used by the processor in 64-bit mode. All remaining attributes are ignored.” The DPL is “descriptor privilege level”, aka access control for ring 0-3, and I can’t find anything talking about their range of options being restricted in long mode.
However looking at Linux 6.3 code, comments in
arch/x86state that ring 1 and ring 2 are unused. Wikipedia handily cites this book (from 2005) stating that Windows does the same. So it seems pretty safe to say that any drivers that use ring 1 and 2 are almost certainly misbehaving anyway, not relying on stable operating system interfaces.It turns out my understanding was wrong, and you are 100% correct. I checked the AMD system programming manual only to find that the DPL bit is not ignored for code segments (although it is for data segments; instead the DPL from CS is used). I’m not sure why I was under this impression, maybe I conflated it with paging only distinguishing between user/kernel (Rings 3 / Rings 0-2) rather than the full range of privilege levels.
Frankly my biggest gripe with Kakoune is that Helix has been stealing a lot of the attention lately. However Helix’s binary is significantly larger, yet it breaks on larger files, where Kakoune is completely fine. Kakoune also has plugin support like the Helix community is asking for, but the recent migration to Helix resulted in dropped maintenance of Kakoune plugins.
Frankly my biggest gripe with Kakoune is that Helix has been stealing a lot of the attention lately. However Helix’s binary is significantly larger, yet it breaks on larger files, where Kakoune is completely fine. Kakoune also has plugin support like the Helix community is asking for, but the recent migration to Helix resulted in dropped maintenance of Kakoune plugins.
I co-authored this (we didn’t choose the article title, though).
Yeah they did a number on you there because chatGPT is always going to be too expensive. The interesting takeaway is that we are already getting to the point where we can run half decent AI on consumer hardware.
I guess ‘LLM’ scams is too nerdy but how about this: “AI is coming for your bank account” ;)
Yeah, there’s a tradeoff in giving a piece like this a title – does the average reader understand it vs. does it get at the core nugget / idea?
I think one of our main takeaways is that scam followup, not initiation, is now made way easier by conversational LLMs, the fact that they can be run on consumer hardware, and the fact that they confidently BS their way through anything is a feature not a bug.
I was talking to a scammer selling car parts (that they didn’t have) the other day, and it took me five or six emails to realise they were full of shit. My first thought was, man if this were chatGPT I bet I would have wasted a lot more time on it
Exactly, yeah – saves them time, wastes your time, and has a higher chance of success than an unsophisticated scammer.
I was thinking about this – I guess with LLM-based generated text we may no longer be able to distinguish a scam email than a real one. No more PayPall or silly typos. :-)
The silly typos are on purpose to groom their marks. The idiots don’t notice the typos and those are exactly the people they don’t want to target. Chesteron’s Typos.
But the typos are there only because less gullible responders waste scammers’ time. If scamming is automated, they can go after harder-to-scam victims too.
I know you meant “don’t waste”. Yeah the existence of the LLMs mean that an LLM can stalk and customize their attack to the individual prey. Romance scams will rise. They could construct bots to clone the voice of existing phone support people and their methods. World is getting cyberpunk way too quickly.
That “always” is not going to age well. It is already rapidly advancing towards running on cheaper hardware. It’s likely already cheaper for triaging spam responses and followups than manual labor, and is more scalable.
That’s the nuance in the comment and the discussion about the title. ChatGPT is the model by OpenAI. It is expensive not because it requires beefy hardware to run, but because OpenAI can charge whatever they want.
But it’s not the only LLM in town and Facebook’s leaked LLaMA can be run by anyone without paying licensing costs. That’s the cheap LLM for anyone to run, but it’s not ChatGPT.
I can’t share the exact costs, but from what I’ve heard internally, the margins on most of the OpenAI-based products in Azure are much lower than you might expect. These things really are expensive to run (especially if you factor in the amortised cost of the insane cost of training, but even without that the compute costs for each client running inference are pretty large). My cynical side says that this is why Azure is so interested in them: consumer demand for general-purpose compute is not growing that fast, if you can persuade everyone that their business absolutely depends on something that can only run on expensive accelerators then you’ve got a great business model (and, since you’re not using them all of the time, it’s much cheaper to rent time on cloud services than to buy a top-of-the-line NVIDIA GPU for every computer in your org).
Yep. My rumor mill is even more dire; scuttlebutt is that OpenAI is losing money on every query! They’re only accepting the situation because of hopes that the cost of inference (in the cloud) will continue to fall.
I’m surprised they’re already running at a profit! (Unless you mean the margins are an even lower negative than the negative I previously thought they were.)
I just finished porting shift-based DFA to Zig and I have an idea on how to achieve > 7GB/s of utf8 validation. That’s very close to vectorised algorithms without clobbering vector registers or requiring a large dataset to achieve that speed. I want to implement and test it over the weekend. I could use some help with generating comparison graphs from criterion.rs though…
Debugger GUI that’s better than inserting prints everywhere. Maybe just a modern ddd would do.
VCS that doesn’t break code by automerging and doesn’t require manual merges for obvious situations. Think Pijul, but with semantic diffing and merging.
ABI description language. Think C headers, but with added annotations for possibly shared features (slices, (un)validated UTF{8,16,32} strings, etc.) and without C’s warts.
Fast PDF viewer.
I’ve found both Intellij and VSCode debugging UIs to be fairly ok. Problem is they’re tied to the editor, which might be a problem if you don’t like the editors.
One thing I really want on my Linux since Unity is a searchable global menubar. If they can get it to work in a gtk4 wayland world, I’d be extatic. Unfortunately gtk4 doesn’t seem to have a concept of a manu bar and gtk3 doesn’t expose it to dbus when running under wayland. It’s also a shame how many GUI toolkits still haven’t figure out how to expose a buffer to Wayland and still rely on XWayland to this day (Looking at you JavaFX and Chromium/Electron)
Searchable? You want a search box for the text in the UI?
Yes, that was a thing Unity did: Ubuntu called it the HUD. I hated it myself and wanted a way to turn it off. I don’t know any other desktop that ever did that.
I’m intrigued to find another of the very few people who I ever saw say that they liked it.
macOS does something similar with its global “Help” menu - selecting that (or pressing Shift+Command+?) and then typing will show matching menu items.
I’ll have to try it. This is something I’ve never even once intentionally used in >20Y on OS X.
I use it quite often in programs with sprawling menu hierarchies like Xcode, Numbers or Jetbrains.
A somewhat bad alternative is that you put all the strings into a single allocation, with the strings immediately after the struct which points to them.
But yeah in general the clean thing to do is for every allocation to come with its own matching free()-like function. This lets you support patterns more complicated than just a tree of pointers, like refcounted COW subfields.
Edit to add: this is one place where RAII is pretty clearly a language improvement.
RAII isn’t a requirement here, the only requirement is destructors. Whether they’re called based on lexical scope or some other notion of reachability, being able to return an object that knows how to clean itself up is very useful.
Weirdly, this is even more important for kernel code than userspace, where you often have multiple memory allocators and objects own different kinds of non-memory resources, yet kernel programmers are the ones that tend to be most resistant to using languages with native support for this kind of thing.
Sure. It’s just a really nice way to do destructors. IMO it’s very nearly the one and only nice thing they C++98 had.
I don’t see how RAII is an improvement. The original call becomes the constructor, but then you still have to provide a destructor as a separate function. Yes, they’re nicely bundled together as methods on the same type, but if you decide to not provide a destructor because the return type doesn’t need one, you still can’t add one if you want to change the type in the future.
I suspect that the @0x2ba22e11 is conflating an language mechanism with one of the abstractions that you can build on top. RAII is an idiom. The underlying language mechanism is types that have destructors that are called implicitly when the object goes out of scope. RAII depends on this for automatic storage but it’s useful in this context for structure fields. If the object were defined with fields of this kind of type then it would end up with a synthesised destructor that managed all of the cleanup. A C++ version of
getaddrinfomight look something like this:The returned list is automatically destroyed when it goes out of scope (if it isn’t moved elsewhere). That’s the RAII bit. The list’s destructor explicitly calls the destructor on the elements. The destructor on
addrinfois synthesised and calls the destructor onstd::vector, which calls the (trivial) destructor onsockaddr, and calls the destructor on onstd::stringfor the canonical name.This makes it easy to create APIs that return deep structures. There’s no explicit cleanup in the caller, which means that you at least maintain source compatibility if you add deeper copies.
If you think that you might want to add extra fields later then you can declare an out-of-line destructor, delete the and move constructors, and make the constructor private. This prevents any code other than yours from being able to allocate instances of this structure (so you can prevent them from existing on the stack or in globals) and means that anything that destroys them must call into your code (so if there are fields that it doesn’t know about then this remains fine). That’s a bit more work and it would be nice to have some built-in support.
In particular, it’s easy in C++ to make a class that can be allocated only on the heap and only by unique pointers by making the constructor private but making
std::make_uniquea friend but you can’t do that withstd::shared_ptr. Instead, you need to do a horrific dance where you have a public factory method that has a private subclass of the class that has a public constructor (which can call the private constructor because it’s declared inside a method of the class) and pass that as the type forstd::make_shared.Just because you don’t write it, doesn’t mean it does not exist. The destructor is still part of the API. As you mentioned, there is a large difference between
The first is always a function call (most likely virtual because libc is usually linked dynamically), even if the destructor itself is a no-op. The second requires exposing more internal details, which makes compatible library upgrades more difficult and requires recompilation on library updates. The third disallows any destructor in the future.
If your language is built to support proper deintialization, RAII adds nothing in the first and last case. In the second case, RAII allows for automatic building of the destructor, but as I stated above, I consider this a very bad choice for library interfaces.
Hence my comment about an explicit out-of-line destructor. This will ensure that it is not inlined into the caller and so its behaviour can be changed later without breaking the ABI.
? If you add a destructor and then the downstream consumer code is recompiled, it’ll automatically add calls to your destructor.
IMO constructors are boring (ordinary functions work fine), the exciting/useful part of RAII is the destructors.
ABI compatibility is just a (good) reason to always always always include a constructor.
Adding a public constructor allows the class to be constructed on the stack and in globals. This means that its layout (or, at least, size and the offsets of all public fields) is now part of your ABI. See my post above for how to avoid this: you need a private constructor and factory methods.
The worst thing that C++ inherited from C is lack of awareness of the underlying memory type. I wish it were possible to write a C++ constructor with an explicit type of the memory provided so that you can have different constructors for stack, heap, global, or even guaranteed-to-be-zero memory.
You can reasonably package only object libraries. If the API (header) changed, all dependent objects have to be rebuilt. For rust, this also means invalidation on compiler update. With that out of the way, a dependency update should only rebuild the dependency once and then relink all depending libraries/applications. And today, linking a binary is freaking cheap.
I’d like to chip in: “A Programming language” -Kenneth E. Iverson “How Do Committees Invent?” -Melvin E. Conway
The conway one is already below mythical mal month. I’ll check the other one thanks!
You’re right, my bad. Missed it.
My biggest gripe with git is diff and merge. They operate on lines, but code has a syntax. Many times have I had unnecessary merge conflicts or merge downright silently breaking code. Janestreet has shown syntaxes correlate with word boundaries and treesitter can parse many languages directly, but the line-based approach is deeply rooted in git.
You can replace git’s default merge tool with any you like, so this isn’t really a git problem.
Alright, this is cool, don’t get me wrong, but: Why e-ink? I too want a frontlighted panel for programming, but I feel like this usecase woul be better served by an e-paper LCD panel. Like Pebble had, but several years newer. My old laptop had a cheaper screen which was decently readable in direct sunlight with the backlight off. A grayscale trasflective panel would IMO be significantly better, as they don’t suffer from e-ink reponse times.
You’re referring to the Sharp memory lcd panels. I don’t recall them ever selling a large panel let alone with something close to hiDPI. You’d probably need a fairly substantial volume expectation for them to consider making such a thing.
Compare to the 13.3 eink grayscale panel IIRC has been around in some form for nearly a decade.
There are multiple points here I disagree with:
This distinction doesn’t really matter in a language with first-class lambdas. If you want to unlock a mutex at the end of a loop iteration with Go, create and call a lambda in the loop that uses
deferinternally.But constructors can. If you implement a
Deferclass to use RAII, it takes a lambda in the constructor and calls it in the destructor.I’m not sure I buy that argument, given that the code in
deferis almost always calling another function. The code inside the constructor for the object whose cleanup you aredefering is also not visible in the calling function.The point is that as a reader of zig, you can look at the function and see all the code which can be executed. You can see the call and breakpoint that line. As a reader of c++, it’s a bit more convoluted to breakpoint on destructors.
As someone that works daily with several hundred lines functions, that sounds like a con way more than a pro.
This can work sometimes, but other times packing pointers in a struct just so you can drop it later is wasteful. This happens a lot with for example the Vulkan API where a lot of the
vkDestroy*functions take multiple arguments. I’m a big fan of RAII but it’s not strictly better.At least in C++, most of this all goes away after inlining. First the constructor and destructor are both inlined in the enclosing scope. This turns the capture of the arguments in the constructor into local assignments in a structure in the current stack frame. Then scalar replacement of aggregates runs and splits the structure into individual
allocas in the first phase and then into SSA values in the second. At this point, the ‘captured’ values are just propagated directly into the code from the destructor.Note that Go uses function scope for defer. So this will actually acquire locks slowly then release them all at the end of function. This is very likely not what you want and can even risk deadlocks.
Is a lambda not a function in Go? I wouldn’t expect
deferin a lambda to release the lock at the end of the enclosing scope, because what happens if the lambda outlives the function?Sorry, I misread what you said. I was thinking
defer func() { ... }()notfunc() { defer ... }().Sorry, I should have put some code - it’s much clearer what I meant from your post.
The first point is minor, and not really changing the overall picture of leaking by default.
Destruction with arguments is sometimes useful indeed, but there are workarounds. Sometimes you can take arguments when constructing the object. In the worst case you can require an explicit function call to drop with arguments (just like
deferdoes), but still use the default drop to either catch bugs (log or panic when the right drop has been forgotten) or provide a sensible default, e.g. delete a temporary file iftemp_file.keep()hasn’t been called.Automatic drop code is indeed implicit and can’t be grepped for, but you have to consider the trade-off: a forgotten
deferis also invisible and can’t be grepped for either. This is the change in default: by default there may be drop code you may not be aware of, instead of by default there may be a leak you may not be aware of.Yes, more than useful:
I think the right solution is explicit destructors: Instead of the compiler inserting invisible destructor calls, the compiler fails if you don’t. This would be a natural extension to an explicit language like C – it would only add safety. Not only that: It fits well with
defertoo – syntactic sugar doesn’t matter, because it just solves the «wrong default» problem. But more than anything, I think it would shine in a language with lifetimes, like Rust, where long lived references are precisely what you don’t want to mess with.You could run an anonymous function within a loop in Go, just to get the per-loop defer. Returning a value in a defer is also possible.
You can also achieve this with zig cc by specifying glibc version in the target
zig cc -target x86_64-linux-gnu.2.28also works for C++ and all supported target platforms.Extrapolating from the benchmarks game seems like a stretch too far for me. Firstly, the quality of implementation varies greatly between languages. I see no reasonable reason for such a large difference between JS and TS. Secondly, are the majority of cloud applications doing computations in that language? I would expect most applications to either be working a lot with databases, or to wrangle network IO. Both would see a significant reduction in language dependence, as computations are done in C/C++ DB/OS anyway.
TextMate and Transmit being “better than anything apt-get could give me” sounds rather amusing, considering that both vim and emacs have exploded in popularity in the last decade by taking in lots of those TextMate users :) And TextMate 2 is GPL, haha.
Some programmers use proprietary software, sure. Outside of game dev though they more often than not absolutely despise it.
The number one practical problem with using closed proprietary software is that you’re literally going back to basically where Stallman started – with that printer driver that he couldn’t just fix because it was closed.
I generally don’t think that much about users when working on hobby projects and don’t care about nobility. My projects are open source just because why the hell would I keep them to myself? They would just go to waste there and probably be lost forever. The public is the best backup/preservation system there is. If I helped someone by doing that, well, that’s just a nice bonus to me, not the goal.
My reference to better-than-apt referred to that earlier time, when TextMate was hot new stuff. The day I bought my license, there wasn’t any comparable substitute in open source. And certainly nothing like the filesystem-based integration with Transit.
Folks cloned the snippet system for Vim and Emacs pretty quickly, at least partway. But that wasn’t really even half the TextMate proposition. It was just the most visible bit, from the screencasts. It took a long time before anyone really went after the bundle system, the configuration flow, the UI, and the command namespace. When they did, they actually maintained TextMate bundle compatibility—direct clone. Eventually, Atom. More or less a GitHub founder’s pet project, so I’m told.
I’m back on Debian now. UltiSnips for Vim, in the terminal. But the kind of work I do has changed. And it’s been 17 years. If Allan Odgaard had kept up pace with new ideas, rather than diverting into a big, HURD-style rewrite, I wonder where editors would be today.
I think it’s fair to set games somewhat apart as its own world. It’s not really the same industry. Ditto film. But my experience doesn’t track yours beyond that. People despise bad software. Despite the hype, open ain’t always better. Mako Hill had a good bit on that.
As for what people despise, I wouldn’t take that as any indication. Twitter’s full of folks venting about open source, too. Especially when they’re made to use it. That’s not to draw any equivalence. It’s just to say samples of grousing don’t tell us much.
The Stallman printer story is canon. But I don’t think that makes it real. Most people, especially these days, don’t want to dive into an unfamiliar system for a feature add. They want to call somebody who knows the system, who’s committed to deliver. Or, failing that, drive-by some GitHub repo with a #feature issue, and miraculously see it closed by a PR on the double.
For good, closed software done right, the end user experience is actually better. You ask, someone capable responds. No one calls you a noob, tells you RTFM, or throws the work to open a pull request back on you. The work gets done by the person best positioned to do it. The experience of the software honors the value of your time.
I have more than a thousand GitHub repos, and have publicly referred to the platform as a shellmound. I’m not sure my random doodle repos counts as open source, for any meaningful sense of the term. GitHub don’t even insist on a license for free storage, as long as you’re comfortable hanging your laundry in public.
Counterpoint: For a developer or company with little money, which has been the case for the better part of my career (notably excluding my few years at Microsoft), if they want a feature, fix, or integration for an open-source dependency, they can make it happen given enough time, effort, and skill, but with a closed-source dependency, they’re stuck, unless perhaps they’re good at reverse engineering. That’s a big reason why I prefer open source for libraries, unless there just isn’t a good open-source solution for the problem at hand (e.g. speech synthesis or optical character recognition). Maybe I’m better than most at diving into an unfamiliar codebase and bending it to my will. I find it entirely plausible that Stallman was, or would have been, good at that too.
Arguably, the possible spaces of software and the way it was interacted with and written were smaller, so Stallman probably would have been good at it. He was a systems programmer, in a systems programming environment, who wanted to hack on a systems progam/driver/firmware.
That doesn’t necessarily mean that most or even all developers can or should be able to say, step out of systems programming and instantly know how to fix a React bug that is plaguing them. Software is more diverse and more specific, and programming systems are more layered and orientated towards the problem, and out of that comes subfields that are non-transferrable even if some of the basic concepts are.
I think the problem Stallman faced was that it was illegal to fix the driver. You technically don’t have to have knowledge to fix a React problem, it’s enough you can find someone who can and is allowed to (for payment, if need be).
FLOSS development doesn’t have to be a net drain on money. The FSF’s standard answer is “support and training”, and that’s fine as far as it goes, but it’s really hard to realize in an age where a project without good docs seldom gets traction and many developers choose burnout handling issues rather than asking for money.
I think that is observably untrue, given the massive number of extensions available for all of the popular editors, for platforms such as GitHub, or even MS Office. The key point to remember is that being able to modify the behaviour of a program does not necessarily depend on its source being available. Somewhat counter-intuitively, it’s often easier in proprietary programs because they’re forced to maintain (and document) stable interfaces for third-party extensions, whereas open source projects can often just tell people to go and hack on the source directly. Specifically on editors, Vim, Emacs, and VS Code all have extension ecosystems that are significantly larger than the core product, which exist because people who were missing a feature decided to dive into an unfamiliar ecosystem and add it. VS Code and Emacs both did well by making that system less unfamiliar to their early userbase by building it on top of a language (Lisp, TypeScript) that this audience used.
I think that, for folks who agree with these points, this makes sense and contextualizes the rest of the post. But not everybody will agree with this. Personally, I have written a Free Software driver in anger, regretting my hardware purchase and petitioning the vendor for documentation. It was a choice made not just from ability, but from desperation.
And for bad closed software done wrong? It can destroy hardware and hide malware, just to pick on one particularly rude vendor. Note that I would not be able to publish this sort of code in the Free Software ecosystem and convince people to use it, because folks would complain that the software doesn’t provide any functionality to the users. And this is a foundational weakness of proprietary software vendors: they are incentivized to make harmful software experiences due to privileged legal status. (Or, to be blunt: if I published this sort of malware, I would be arrested and jailed for crimes.)
Haven’t done any real systems hacking in a long while. I did when I was young, and had more time. I definitely had to do some driver and firmware work, to get Linux working with various things. I’m not sure if I was angry going into those projects, but I remember being pretty frustrated coming out of them!
A lot of that came about from buying cheap hardware. I remember one Acer laptop in particular, the one I chose for college: great specs, great price, terrible build quality, total bodge-job, component- and firmware-wise. I eventually bought the MacBook, and came back to Linux on ThinkPads. I pay a premium for ThinkPads, but I get what I pay for, Linux hardware support very much included. It makes way more sense than spending hours brushing up and hacking patches.
As for bad proprietary software: oh yeah, it’s out there. But the idea that software vendors have some inherently privileged legal position doesn’t fly. They have copyright, and can require customers to buy licenses. But the terms of those licenses can and do vary. I have advised on several software license deals, for substantial money, on terms that basically boiled down to Apache 2.0 plus a payment clause, limited to just the paying customer.
If all you’re doing is buying licenses on take-it-or-leave-it terms, or negotiating with highly aggressive companies much bigger than yours, yeah, you’re likely to see terms that strongly favor the vendor. That’s how leverage works. But proprietary software sales get done on sane, fair terms all the time. Perpetual licenses. Modification rights. Meaningful warranties. Accountable maintenance commitments. Sometimes the less impressive product wins out, because the terms offered for it are better.
Get out of your bubble. Outside that bubble a lot of developers use JetBrains IDEs or Visual Studio. Visual Studio Code has gained a lot of traction in recent years, but initially mostly because Code was much better for web development than the competition and it is free. Not because it is open source [1].
In the most recent Stack Overflow developer survey, Visual Studio Code is used by 71.07% of developers, Visual Studio by 32.92%, IntelliJ by 29.69%. The most popular fully (?) open source editor is actually Notepad++ with 29,09%. And vim takes the next place at 24,82%, but I wouldn’t be surprised that is because people use vim when doing quick edits remote Linux machines. Emacs dangles somewhere at the bottom of the list with only 5,25%, surpassed by many proprietary applications like Xcode, PyCharm, or Sublime Text.
I agree that this is a major downside of closed source software. But most people want to fix the things they are working on, not their tools.
[1] https://underjord.io/the-best-parts-of-visual-studio-code-are-proprietary.html
PyCharm/IntelliJ IDEA are Apache2 and most devs use the free software version available at https://github.com/JetBrains/intellij-community
I have no reason to doubt this, but are you aware of any usage statistics? I’m curious whether “most devs” is more like “just over half” or “over 99%.”
Anecdotally, every company I’ve worked at where the devs used JetBrains IDEs has paid for the commercial version, but totally willing to believe those companies aren’t typical.
From my anecdotal experience, no company has paid for the commercial JetBrains version, even when most of the devs use it. It might very well be a cultural thing.
At the two companies where I used it, we paid for the whole toolbox subscription for every developer who wanted it. Most of us used IDEA Ultimate and one or more of CLion, AppCode or PyCharm Professional. Many of us also used DataGrip.
I still maintain a subscription for my consulting work now, too.
Can’t speak for PyCharm particularly but every other flavor of IDE based on IntelliJ, I only know of companies who have paid, be it PHPStorm or more likely Ultimate (if working on many languages).
I don’t really disagree with you, but your arguments seem kind of weak.
What now? It’s just another bubble.
I haven’t ever used any of those “best parts” and never seen anyone using them.
I’ve used the remote stuff. That and PlatformIO are the only things that ever cause me to use VS Code over emacs or one of the JetBrains tools.
The extension marketplace is proprietary too, and I’d call it one of the best parts of VS Code.
I mean, there’s Vivado, and then there’s Visual Studio, just like there’s, I dunno, a steaming, rotten pile of horse shit and then there’s pancakes.
There are many bubbles in the tech world and game dev is only one of them. I worked in an embedded shop where the state of Linux was that we had one colleague who tried Fedora and he thought it was like a beta or something because he couldn’t put things on the desktop and, in his own words, he expected some things not to work as well as Windows but that was years away from being useful. The thought of writing code in a text editor after finishing college, where they put you through the ritual of compiling stuff you wrote in vim by writing gcc incantations, seemed about as foreign to these guys as the idea of sailing to America on a steam ship.
There are plenty of bubbles where programmers use both kinds of tools, too. Way back when I was doing number crunching, everyone at the lab was using emacs, vim, nedit or notepad, but also Matlab, which everyone hated for various reasons but nowhere near as much as they hated, say, gdb. We worked a lot with another research group at another university where it was the other way around: someone had figured Octave was good enough and they had just one Matlab installation for things Octave really couldn’t do or to port their code, and used the money they saved on Matlab licenses to buy a bunch of Windows and Visual Studio licenses.
I don’t have much in the way of numbers here but you shouldn’t take a tweet about Vivado as the standard about what people think about closed source tools. Vivado is successful because it barely works and there are no alternatives that work (for any non-hobbyist definition of “works”) – it’s successful largely by vendor lockdown.
Speaking of bubbles, I have literally not even heard of Vivado before today.
Well, what can I say, not every case of acute horse diarrhea deserves to be famous :-D.
But seriously, this is one of the things I love about our work. Vivado is huge. I suspect it’s already old enough that virtually everyone who finishes an EE or CompEng degree has seen it at least once (it’s about ten years old, I think, and it replaces another suite called ISE which was also all sorts of horrible in its own way). It’s very likely that it’s been involved in one way or another in dozens of stories that popped up here on lobste.rs, like stories about RISC-V or cryptography or FPGA implementations of classical systems. And it’s perfectly possible for someone to be an excellent programmer and earn a living writing code and never hear about it. No matter how good you are at anything computer related, there’s always something out there big enough that it’s got tens of thousands of people behind it that you had no idea about.
If you keep an open mind, computer engineering will surprise you in all sorts of fresh ways, all the time – not all of them good but oh well. Dogmatism is fun and it feels like you’re right all the time but it’s really fscking boring. I’ve done it and it sucks.
By reputation, another reason Vivado is successful is because all the other FPGA toolchains are reportedly even worse. Don’t get me wrong, Vivado is a tyre fire that segfaults, but the others are apparently even less reliable.
e.g. I’ve heard that at some shops, people writing FPGA code targeting competing FPGAs will actually write and debug their entire project on Xilinx FPGAs with Vivado and then port the code to the other toolchain for the originally intended target FPGA.
I’ve done too little real work on Altera’s stuff to have had first-hand experience but from the little I’ve done I can certainly say Quartus sucked at least as much as Vivado back when I last touched it (2015-ish?). Maybe it’s changed in the meantime but somehow I doubt it :-D. I heard Lattice’s stuff is tolerable but I never tried it. I did try Microsemi’s Libero thing though and it makes Vivado feel like Visual frickin’ Studio. Porting designs between toolchains is not quite like porting program code and it sounds like a really bad idea on paper but, indeed, given how bad some of these tools are, I can imagine it’s just the only way to do things productively sometimes.
But it’s really a lot more complicated than how good the toolchain is. A big – probably the biggest – reason why Vivado and Quartus are so successful is simply that Xilinx and Altera… well, Intel, are really successful. The next version of Vivado could suck ten times as bad as the current one and it would barely put a dent in their market share, just because it’s what you use to do things with Xilinx’ devices. Developer souls are a lot more fungible than silicon.
Also, being so big, they both have roots in the academic world and they run deep, and it’s simply what lots and lots of people learn in school. They’re not very good but as long as the bitstream flows out, who you gonna call?
A competitor with better development tools could, in theory, steal a chunk of this market, but that chunk would be exactly as big as how many programmable logic devices they can sell, and there are constraints – both technical (as in, performance) and non-technical – that dictate that long before anyone even considers the quality of development tools. The usual retort is that developer tools are important for productivity. And they are, but the best designs, from the most productive teams, won’t be worth crap if there are no physical devices for them, or if these devices cannot be obtained on time and in the required quantity and at the right price and so on.
I also secretely suspect that it’s just a case of the audience of these tools being a little more tolerant to weirdness and poor quality. I mean, they taught me how a transistor works in my second year of uni. I then spent the other half of my undergrad years (and I could’ve spent a whole two years of masters’ on that, too) learning about all the ways in which it doesn’t quite work exactly like that. Software segfaulting under your nose is just a tiny drop of weird shit in a very big bucket of weird shit which people way smarter than you reverse-engineered out of nature. Your entire job revolves around manipulating all sorts of weird but inevitable things. So you just learn to chalk up the fact that Vivado crashes if you have more than 64 mux instances whose names start with the letter X under “inevitable” [1], somewhere between “all P-N junctions have leakage current” and “gate capacitance varies with gate oxide thickness”. What’s one more?
[1] Note that this is something I just made up on the spot but it does sound like something Vivado would do…
You don’t exactly need to sell programmable logic devices to steal a chunk of FPGA development tool market. Symbiflow has already done that with Lattice’s FPGAs, and are slowly starting to bite into Xilinx FPGAs as well. Quicklogic just released their new FPGA product without their own software, just by giving Symbiflow their bitstream generator and FPGA definitions. There are signs that the new Renesas FPGAs are using Yosys (part of Symbiflow) too.
The reason why these closed software tools are so entrenched is that they are tied in with hardware. And as of now, open source hardware is a lot more niche thing, than open source software. With time, that will probably change, but even then, the software landscape in places dealing with hardware will progress faster. Just remember, how a while ago working on microcontrollers almost always meant dealing with vendor-specific IDEs. That is basically gone now. With time, that will happen with most FPGAs as well.
I haven’t tried Symbiflow since 2020 and it’s a project I’m really cheering for, so I’m not going to say anything bad or unenthusiastic about it. But it’s worth keeping in mind that the iCE40 line (the only one that it supports well enough to be usable for real life-ish projects) has a very straightforward, wrinkle-free architecture that lends itself easily to reverse-engineering. Even though the underlying technology is physically the same, as in, it’s based on the same technology (FPGA), the market Symbiflow can realistically target is practically different from the one where people are using Vivado and complaining about it. Its relative success in this field is important and useful, not to mention liberating to a lot of people (including yours truly) but I wouldn’t be too quick to generalize it.
This approach will probably find some success at the low-power, low-cost end of the spectrum, where not only are customers rather unwilling to pay the licensing costs, but some companies, especially fabless vendors like Quicklogic, would be reasonably happy to be rid of software development cost. But this part of the FPGA market works in entirely different ways than the part of the FPGA market that’s bringing the big bucks for Xilinx (and Altera) and where the people who associate the word Vivado with an overwhelming feeling of dread work. For one, it’s not a field where architecture and fabrication technology are important competitive advantages, so there’s not that much value in keeping it closed and keeping the software tied to it.
I don’t know, I would far rather use Visual Studio than i.e. whatever editor + gdb again.
You’re right that systems programmers and hardware devs prefer small and lean tools, which are often released as open source, but walk into any corporate dev house and you’ll see a lot of Visual Studio and IntelliJ with Xcode and other IDEs sparkled in. The sole exception to this are web devs, whose first usable tool was VScode.
If you have the skills and time to hack on your tools, open source is better, but for most people it’s just a better proposition to pay someone else for their tools and use the saved time to do their job and make that money.
Actually most devs use proprietary software, and kemitchell even mentions this in the post. He switched to Mac, but Windows is still the most popular platform even amongst devs[1]. I suspect the Stack Overflow survey results are even skewed and that Linux likely has less market share than they found.
https://insights.stackoverflow.com/survey/2021#section-most-popular-technologies-operating-system
I hope I made the point that devs do use proprietary software. It’s not true that devs just won’t use closed code. But I don’t have data to support the claim that most devs do. I suppose you could get there by arguing Windows and OS X are closed, and the number of folks on Linux is small. I’ve enjoyed looking at the Stack Overflow survey, but I have no idea how representative that is.
For what it’s worth, when it comes to laptops and desktops, I’m back on Linux again. I did switch to Mac when I was younger. And I did my programming career that way, before going to law school.
I have a trifecta! I have a Linux laptop (which I used when I first joined the company to do development), a Mac laptop (which I also do development on) and a Windows laptop (because our Corporate Overlords are Windows only) that I don’t use, and can’t return (I’ve asked).
But how much of that is because companies require Windows? When I was hired at my current company, half the employees used Macs (mostly developers) and the other half Windows. We then got bought out by an “enterprise” company, and they use Windows exclusively. They even sent me a Windows laptop to use, even though I use a Mac. The Windows laptop sits unpowered and unused, only to be turned on when I’m reminded that I have to update the damn thing.