Fun! If OP is interested in taking it to the next level, I made a repository comparing different ways to implement progressively faster interpreters in Rust. At the limit, you can start to approach the performance of native code. Somewhere in there there’s a sweet spot that trades off implementation complexity against performance. closure_continuations seems like a nice place.
Unrelated to this article, but I was poking around your GitHub after following your link and was looking at your language, Tao. It has a lot of nice ideas and goals. When I see repositories like this, I’m often curious about why they stopped being developed. Does it feel like the idea that the author wanted to explore was run to ground? Did it end up leading to a dead end? Etc, etc. I’d love to hear the debrief on projects like these because it seems like there are probably a lot of hard fought lessons to be learned.
No pressure to write anything out. It was just a passing thought that I’ve had many times when looking at “finished” projects.
Does it feel like the idea that the author wanted to explore was run to ground? Did it end up leading to a dead end?
Not at all! In fact, I’m convinced that this approach to algebraic effects is very expressive, and I’m quite proud of the type-checker and what it can do. It’s mostly a matter of time: work, life, and other projects have left me with little energy left over for it. I’ve made a few attempts at making a ‘tao 2’ over the past few months, but I’ve just not had the energy to push it all the way to something that’s worth putting up on the web. It’s not a coincidence that my last public commit in the repository lines up well with the start of a new job.
Only the benchmarks that are present in the repository, although you’re welcome to run them yourself to check. There’s a fair bit of variability between machines given that each benchmark is quite tightly coupled to things like dynamic branch prediction performance.
I only recently came across an interview with Arthur Whitney about the language k and only then the array-based languages really started to draw me in. I now wish I knew more about them and am still looking for a nice intro to APL and the likes.
That said the best way to learn APL is to write it. Dyalog has a conquest that start with 10 short exercises. There is also APL quest, which you can solve en your own and then compare with Adam’s solution
I really like the information density of this design. No toolbar, less padding in the cells, and having the “formula bar” be in the status bar area. When I’m using a spreadsheet, I want to feel like I’m “in” the data so the more that I can see at the same time, the better.
Postfix syntax also feels really convenient in this scenario since the cursor in the formula area is already at the end of the selection range. Performing an operation is as simple as chaining postfix operations instead of needing to jump back to before the range and wrap it in a function call. Arrow key usage in spreadsheets gets kind of funky because it’s not always obvious if it should change the cell position in the grid or the cursor position in the formula text box so reducing the need to navigate the text box by using postfix seems helpful.
This looks really great. The goals and rationale really resonate with me.
I’m glad that docs are continuing to get more attention. The gold standard for me has always been the R ecosystem (example), but other ecosystems are getting better. I’ve found some of the doc sites a bit heavy and hard to navigate so I’m glad that the TigerBeetle team simplified during their redesign.
I’d additionally be interested to see some more exploration into the sort of side-by-side format of API docs and implementation that ratfactor was exploring here. It feels especially fitting for Zig which has a bit of a culture of encouraging users to read the implementations.
Have you used my R Markdown editor, KeenWrite? A feature that’s missing from most editors is the ability to use variables. Here’s a video showing how to use R statements in combination with TeX and variables:
Oh wow, that sounds very cool! I have recently fallen back on my parenthesis addiction and looked into Scheme languages again. I was actually wondering if/when there will be a scheme written in Rust (even though there probably already exist several, but I never bothered to actually look).
Is there a reason to prefer R6RS over R7RS? I know that not everyone is happy with the newer standard, but I was wondering if you have an opinion on that matter.
Thanks! There are a few other Scheme implementations written in Rust I think, but they are less complete and not actively worked on I believe (and of course they don’t support interoperability with async rust)
The reason R6RS is chosen is because it’s much more complete of a spec, as of right now R7RS small is the only version of R7RS released, and that spec does not include some desirable things like syntax-case. When it is fully released, the intention is to support R7RS large
Unrelated to Rust scheme implementations, but when you stated that the primary reason was async, I was reminded of lispx, which I think is a pretty cool project. It was also created to solve the ergonomics of async in its host language. Maybe there’s something there that could be of inspiration. I enjoy a scheme as much as the next parenthesis-loving person, but I’d be happy to see more kernel implementations.
By way of brief comparison, besides the async compatability, Scheme-rs has a much stronger macro system than both of these impls at the current moment, which are attempting to be compatible with R7RS small or R4RS. For example, steel cannot define a macro of the following form:
(define-syntax loop
(lambda (x)
(syntax-case x ()
[(k e ...)
(with-syntax
([break (datum->syntax #'k 'break)])
#'(call-with-current-continuation
(lambda (break)
(let f () e ... (f)))))])))
to be fair to these implementations, they have a lot of features that scheme-rs lacks. But all of the ones present in R6RS I intend to add sooner rather than later
This is very exciting to see - native compilation is something I’ve been wanting to work on for steel for a long time, I’m happy to see someone else take a stab at it for their scheme. I’ve been toying around with cranelift for years but haven’t had the dedicated time to push through the work on it
To address two of the points on syntax-case and async:
As of last night, steel can define this:
(define-syntax (loop x)
(syntax-case x ()
[(k e ...)
(with-syntax ([break #'k])
#'(call-with-current-continuation (lambda (break)
(let f ()
e
...
(f)))))]))
Just have to work on the datum->syntax and then there is parity. The syntax-case implementation was slightly broken in the mainline for a while (and also, undocumented).
For async, steel does have support for rust async functions registered against the runtime, and I have a fun way of integrating with an external executor instance via continuations, but it does not have an async public api (yet) like you do. Best of luck with this and I’m excited to see where it goes
Oh that’s sweet! Very cool to see syntax-case in steel!
I hope I wasn’t missrepresenting your project! Let me be clear here: Steel is faster and more featureful than Scheme-rs by a long shot. I brought up syntax case because that’s were the initial bulk of my effort went (the remainder went to reimplementing the project as a compiled)
It absolutely is the hardest thing to properly implement in my opinion (and I’m technically not even fully compliant, I don’t add any continuations to the expander (soooo much work)).
I’m thankful I spent so much time considering it while the eval logic was interpreted, it was a weight off my shoulder when moving to compiling. That being said, compiling it also lead to a new set of challenges, I had to somehow figure out a way to capture the environment, something that I was just throwing out after compilation.
As an aside, have you noticed how there are so many people who work with Scheme named “Matt”? It’s starting to freak me out a bit
murex claims (and I believe it, although I haven’t tested it!) to be able to pass type information through pipes in a way which allows POSIX utilities and other old-fashioned stuff to be able to treat the data as an old-fashioned stream of untyped characters.
nushell and other shells that have typed data (e.g. elvish), I believe, all make you choose in each instance whether to pass things through pipes with type information, and when you do most utilities won’t be able to use that pipe.
Yes, you can! I’m not aware of a way to see what ends up in the index for a particular row, though. I feel like the virtual column gives you the ability to see it, even if only for debugging purposes. I also think it’s pretty easy to accidentally specify your query in a way that doesn’t end up using the complex expression index – especially if you’re using a query generating library like an ORM?
Given the restrictions of the expression language - though I guess custom extensions might provide more expensive primitives - I have trouble imagining many situations in which that would be worthwhile.
The one thing I can think of is situations where you might run a filter against the generated column that requires either a full table scan or needs to consider tens of thousands of rows - but in both of those cases an expression index might be a better approach.
My understanding was that the language major mode was ideally supposed to configure eglot for optimal use for that language, just as it is supposed to configure tree sitter. It feels like cider could do quite a bit with its knowledge of the language and which other features are being used (nrepl) to smooth things over for its users here.
If the major mode has to resort to hacks, then it is a deficiency with eglot that needs to be addressed.
The third type of literal is called a quote. You can imagine a quote as an ordered list of other literals:
Is quote standard terminology for concatenative languages? Because if I “can imagine a quote as an ordered list of other literals” then I’d probably have called it list rather than quote :)
Do concatenative languages deliberately avoid the terminology that other languages use? (Another obvious example is the way they use the word “word”.) I actually think I once read that there’s some reason why they do, but I might be imagining it? If not then I vote for @jcmkk3’s explanation.
The author mentions one thing they really like about meow is that it doesn’t define any keybindings by default and lets you add the ones you want. But something I like about evil-mode is that you have evil-collection which defines a uniform set of keybindings for every popular Emacs package. I don’t have to configure everything to use modal editing because there is already code to take care of that. But on the flip side that definitely makes evil more heavy weight then something like meow.
This is the idea with the keypad mode that the post talks about. Most packages follow standard emacs keybinding practices so the keybinds should already be relatively intuitive. Keypad mode lets you use them without modifiers.
This was a nice feature for me using a colemak keyboard, I use ‘neio’ instead of ‘hjkl’ — setting up meow was a breeze.
You don’t define the key in every mode but which key you want to bind to which meow action. The different modes are bound to meow actions so it works really well.
I suspect it’s the same reasons that people create their own Smalltalk: it’s a small enough language that it’s tractable to make your own. A full, compliant and performant, Common Lisp implementation is a huge undertaking, but a small interpreted Lisp or a Blue-book-equivalent Smalltalk is a tractable single-person project.
On top of that, you may well have slightly different requirements to any of the existing ones and it’s a great project.
My surprise is the people that haven’t, at some point or other, implemented a Lisp.
How many schemes are out there that satisfy the design goals of Lone? There might be a couple, but none of the ones I’ve heard of can actually do this. Plus Lone has tables as a first class data structure with literal syntax for them; Scheme’s support for these tends to be very bad, with implicit quoting at best.
I see “first class tables” in the readme of the Lone github repo, but as far as I can tell there is no documentation, nor are there any substantial examples; the tests I examined are so trivial as to be non-illustrative.
It seems to be pretty much like clojure’s map syntax. Every two items within curly brackets is a key -> value. In the article, the output of env is:
{ "HOME" "/home/matheus" "EDITOR" "vim" … }
Table is one of those many computer terms that is so ambiguous. I think that technomancy is talking about a hash table, while I suspect that you are thinking about a database type of table structure like you have in lil.
Yeah, from the examples in the post I’m assuming “tables” is just another way to say “hash tables” like it is in Lua but it’s true the documentation doesn’t do a good job explaining it yet.
I’ll give them a break since it’s a 1-person project, and they’ve specifically explained in the post that the language isn’t ready for use yet.
Ah, disappointing. I’m always curious to see other examples of languages which have tried something similar to Lil’s tables and query language.
“Table” is not a very good choice of terminology for a simple associative structure. “Dictionary” or “map” are nearly universal, and calling it a “HashMap” or “HashTable” unnecessarily entwines the implementation details with the contract.
I love clean, well organized projects as much as the next Python dev but I have to admit that I love the idea of PEP 723, unlike the post author.
It makes it easier to write and support the category of tools that I think are one of the best use cases of Python: single file scripts. These tend to do one thing, (hopefully) in a simple and understandable way that you can hack on if you want to, without dealing with a whole project installation. They are enabled by the ubiquity of Python and its large standard library, but whenever you exceeded stdlib and had to use a dependency it was awkward to communicate “install these dependencies, then run this” compared to “just run python script.py --help”. Granted we are not exactly there yet, but this PEP is a step towards that.
I think this creates a weird activation energy hump where it’s easy to get started but then you get snagged as soon as you have more than one file, which encourages the building of too large single files. If they’re adding it, they also need a plan for how to make a smooth transition to multifile as part of the overall vision. It doesn’t need to be implemented all at once, but if there’s no vision, it seems likely that there will just be a new, weird way to get snagged as a new developer, in which everything is great until you try to add a second file and it all falls apart.
You are right, on the other hand the activation energy hump already exists to go from “script with only stdlib dependencies” to “script with external dependencies.” This simply moves the hump further. But I think it is a totally valid concern that this adds yet another way to do dependencies in Python and that it naturally breaks down at the multi-file transition. An overarching vision is indeed missing.
Yes. I agree. The impression that I get from the author’s post is that they are tunnel visioned on their own uses of python. Solving the packaging/project issues of python seems so straight forward to them and they believe an official solution that solves their use case should be put into place immediately, while other use cases like “scripting” and “data science” can be figured out later.
I am curious, why you say data science can be figured out later? Are you referring to bundling C extensions? My main problem as a day to day DS is some libraries not building wheels where I develop/deploy stuff, or not complying with latest standards (most notably, xformers by Meta).
I’m not saying that data science can be figured out later. I said that was the impression that I got from the author. They said:
My idea with that was that most people would be happy with one tool that does everything, but the scientific-Python folks might have special requirements that would work best as a second tool.
Then in regards to PEP 723 (inline script metadata) they said:
Is it super useful? I don’t think so; setting up a project with pyproject.toml would easily allow things to grow.
To me, that reads as if there are at least two common use cases of python that they either do not believe are important or that they can be figured out later.
A common thread that I see in these discussions is that folks get frustrated that there is not an official solution when it seems like there is an obvious solution that can be seen in another language ecosystem, but the reality is that those ecosystems don’t have all of the same use cases as python.
I don’t think the end use will be that. At least, it’s not explicit. The PEP speaks of other tools like pip-run and pipx, and some package managers like hatch. And it kinda makes sense because it needs installation + virtual env, I guess.
Yes, and I’d be fine with hatch or pip-run doing some magic in the background to enable that, especially if they can be assumed to be easily available for the users of the script (which they can’t be yet, IMO).
That is hilarious to put the dependencies there but I agree that it would be amazing and solve so many issues. It’s a bit of a bother to have a Pipfile just to support a single file Python script.
The proposal is for an augmentation to the select syntax so I don’t really see the connection with immutable databases. I do think that a more rich payload could be handy sometimes.
Their approach is very different. Pandas tries to operate more like a python collection, whereas polars is more like a query builder. Both styles have their challenges.
It’s not only a Pandas world now but becoming more and moreso. Skimming the PyPI stats for Pandas and Polars it looks like Pandas is growing at the rate of 1 Polars userbase per week. So interoperation is absolutely critical for Polars.
I was curious to learn of the “Python dataframe API standard”. You mention Pandas in conjunction with this, but are they adopting it? It looks very difficult to fit in with their API and I wasn’t able to find any integration efforts via google.
pandas.DataFrame.__dataframe_consortium_standard__ appears to exist in Pandas 2.1.3, so I’m guessing they will (or do?) support it. It is hard to tell, though, what support status is in general, possibly because the spec is still evolving so compliance is still a moving target.
It appears that this is a reference to a library called dataframe-api-compat which is some kind of compatibility layer between pandas (and polars) and the standard. Some kind of facade pattern like this is probably the only way to get pandas compatibility but I suspect that it must still be partial because, for example, pandas doesn’t support all the types in the dataframe standard.
Polars has become one of the reference targets as they are designing the api. They made a lot of changes in the last few months to make sure it could support the laziness and expression system that many of the newer libraries (polars, ibis, duckdb) tend to use.
I’ve been planning to do this upcoming Advent of Code in Lil. I think that it is a really cool little language. As a data analyst, a language with built in support for array programming, tables/querying, and visualization/UI is very interesting. Think numpy, pandas, matplotlib, etc. I wouldn’t use it for actual work things, but it hits closer to what I’m looking for in a language.
I really wish it was easier to try these bindings outside of specific editors. Maybe I’m in a specific situation as a kotlin developer working on a large Android project, but Android Studio has a perfectly good vim emulator built in and so does every other ide. Porting Kakoune’s behavior would be more than just mapping a list of bindings, it looks like?
It wouldn’t be a huge loss to get used to kak as a terminal editor, but then I’d feel sad all day in Android Studio, and it’s already too easy for me to feel sad there.
I think that it is probably easier to emulate kakoune bindings than vim bindings in most modern editors due to kakoune’s minimalist nature. The object-verb approach is also much closer to how a non modal editor works. I think that it is important that the host editor has multiple cursor support to build from. That makes emacs a more difficult target, for example. You can check out dance for vscode https://github.com/71/dance
as far as I’m aware, vim style editing also requires more than just mapping bindings, so I don’t see why adding a noun-first editing paradigm would take more work. Maybe plugins for this exist already in VS code and the JetBrains IDEs?
As some one in the data processing and analytics field, a fan of array languages (including R), and dataframe geek, I’m very much in agreement that this topic is worth more investment. Also, if you’re like me and want to understand the presenter’s background on the topic, this is from Graydon Hoare, the creator of Rust.
One day I hope they’ll open up to electrostatic capacitive rubber domes as an alternative to mechanical keyswitches. I’d pay premium for an ergonomic option that actually thocks.
It seems that only a very small number of people have made custom electrocapacitive keyboards, at least as far as I could find. There are a lot of things that make it finicky:
The circuitry is analogue, rather than digital, and keypress detection measures changes of a few picofarads, so the PCB design needs more expertise than usual. On the up side, like hall effect switches, you can adjust the activation point in software or use them as analogue inputs for things like mouse keys.
The rubber domes and conical springs are sandwiched between the PCB and switch housings, and loose when the keyboard is disassembled. It is tricky to get them aligned when reassembling. It is easier if the domes are on a sheet matching the size and layout of the PCB, which they won’t be in a custom layout.
After reassembly the firmware needs to be recalibrated to accommodate any changes in alignment of the domes and springs.
The plate and PCB must be firmly fixed to each other, since there is no lower switch housing to take the bottom-out force: it hits the PCB through the dome and spring, and none of it is taken directly by the upper housing or plate.
There are not many sources of switch parts. I get the impression Topre parts are only available through unofficial channels. Niz will sell switch parts to hobbyists in keyboard-sized quantities.
Yup. Which is why I want to buy one—not build one. I built a ErgoDox way back, & I’d rather have a manufacturer do the hard work. Of course back then even Cherry switches weren’t the easiest thing to come by. But the folks that have tried Topre or NiZ’s switches broadly seem to pine for the feeling of these switches despite the limited supply. With the right nudges maybe the same thing that happened for mechanical switches could happen.
NiZ is cheap & still feels great, but the firmware releases as Windows executables in a Google Drive doesn’t inspire confidence. I’ve emailed them and they don’t seem to interested in the ergo and/or split keyboard crowd.
Fun! If OP is interested in taking it to the next level, I made a repository comparing different ways to implement progressively faster interpreters in Rust. At the limit, you can start to approach the performance of native code. Somewhere in there there’s a sweet spot that trades off implementation complexity against performance.
closure_continuationsseems like a nice place.Unrelated to this article, but I was poking around your GitHub after following your link and was looking at your language, Tao. It has a lot of nice ideas and goals. When I see repositories like this, I’m often curious about why they stopped being developed. Does it feel like the idea that the author wanted to explore was run to ground? Did it end up leading to a dead end? Etc, etc. I’d love to hear the debrief on projects like these because it seems like there are probably a lot of hard fought lessons to be learned.
No pressure to write anything out. It was just a passing thought that I’ve had many times when looking at “finished” projects.
Not at all! In fact, I’m convinced that this approach to algebraic effects is very expressive, and I’m quite proud of the type-checker and what it can do. It’s mostly a matter of time: work, life, and other projects have left me with little energy left over for it. I’ve made a few attempts at making a ‘tao 2’ over the past few months, but I’ve just not had the energy to push it all the way to something that’s worth putting up on the web. It’s not a coincidence that my last public commit in the repository lines up well with the start of a new job.
Very understandable. I hope you get some time/energy to work on it again at some point.
Thank you, I appreciate it :)
That’s cool! Do you have charts or numbers on the results?
Only the benchmarks that are present in the repository, although you’re welcome to run them yourself to check. There’s a fair bit of variability between machines given that each benchmark is quite tightly coupled to things like dynamic branch prediction performance.
I only recently came across an interview with Arthur Whitney about the language k and only then the array-based languages really started to draw me in. I now wish I knew more about them and am still looking for a nice intro to APL and the likes.
The learner-friendliest I’ve found is Uiua but it’s a little different than the classic APLs.
Wow that website is actually really neat! Thank you for pointing it out, I just lost a couple of hours digging into that.
This is a nice intro to APL imho. It is short (Mastering Dyalog APL is very good, but it is massive) and introduces a few operators at a time
https://xpqz.github.io/learnapl/intro.html
That said the best way to learn APL is to write it. Dyalog has a conquest that start with 10 short exercises. There is also APL quest, which you can solve en your own and then compare with Adam’s solution
https://m.youtube.com/playlist?list=PLYKQVqyrAEj9wDIUyLDGtDAFTKY38BUMN
I think that kbook is a pretty good intro to k. There are also other resources on the k wiki.
https://xpqz.github.io/learnapl/intro.html and https://tryapl.org/ are both lovely!
However, Intro to College Math in APL is very interesting too!
I really like the information density of this design. No toolbar, less padding in the cells, and having the “formula bar” be in the status bar area. When I’m using a spreadsheet, I want to feel like I’m “in” the data so the more that I can see at the same time, the better.
Postfix syntax also feels really convenient in this scenario since the cursor in the formula area is already at the end of the selection range. Performing an operation is as simple as chaining postfix operations instead of needing to jump back to before the range and wrap it in a function call. Arrow key usage in spreadsheets gets kind of funky because it’s not always obvious if it should change the cell position in the grid or the cursor position in the formula text box so reducing the need to navigate the text box by using postfix seems helpful.
This looks really great. The goals and rationale really resonate with me.
I’m glad that docs are continuing to get more attention. The gold standard for me has always been the R ecosystem (example), but other ecosystems are getting better. I’ve found some of the doc sites a bit heavy and hard to navigate so I’m glad that the TigerBeetle team simplified during their redesign.
I’d additionally be interested to see some more exploration into the sort of side-by-side format of API docs and implementation that ratfactor was exploring here. It feels especially fitting for Zig which has a bit of a culture of encouraging users to read the implementations.
Have you used my R Markdown editor, KeenWrite? A feature that’s missing from most editors is the ability to use variables. Here’s a video showing how to use R statements in combination with TeX and variables:
https://youtu.be/XSbTF3E5p7Q?t=110
Oh wow, that sounds very cool! I have recently fallen back on my parenthesis addiction and looked into Scheme languages again. I was actually wondering if/when there will be a scheme written in Rust (even though there probably already exist several, but I never bothered to actually look).
Is there a reason to prefer R6RS over R7RS? I know that not everyone is happy with the newer standard, but I was wondering if you have an opinion on that matter.
Thanks! There are a few other Scheme implementations written in Rust I think, but they are less complete and not actively worked on I believe (and of course they don’t support interoperability with async rust)
The reason R6RS is chosen is because it’s much more complete of a spec, as of right now R7RS small is the only version of R7RS released, and that spec does not include some desirable things like syntax-case. When it is fully released, the intention is to support R7RS large
There are at least two (three now) being actively developed that I know about.
Unrelated to Rust scheme implementations, but when you stated that the primary reason was async, I was reminded of lispx, which I think is a pretty cool project. It was also created to solve the ergonomics of async in its host language. Maybe there’s something there that could be of inspiration. I enjoy a scheme as much as the next parenthesis-loving person, but I’d be happy to see more kernel implementations.
Ah I see! I think I was thinking of https://github.com/volution/vonuvoli-scheme which last saw an update two years ago.
By way of brief comparison, besides the async compatability, Scheme-rs has a much stronger macro system than both of these impls at the current moment, which are attempting to be compatible with R7RS small or R4RS. For example, steel cannot define a macro of the following form:
to be fair to these implementations, they have a lot of features that scheme-rs lacks. But all of the ones present in R6RS I intend to add sooner rather than later
Disclaimer: steel is my project
This is very exciting to see - native compilation is something I’ve been wanting to work on for steel for a long time, I’m happy to see someone else take a stab at it for their scheme. I’ve been toying around with cranelift for years but haven’t had the dedicated time to push through the work on it
To address two of the points on syntax-case and async:
As of last night, steel can define this:
Just have to work on the
datum->syntaxand then there is parity. The syntax-case implementation was slightly broken in the mainline for a while (and also, undocumented).For async, steel does have support for rust async functions registered against the runtime, and I have a fun way of integrating with an external executor instance via continuations, but it does not have an async public api (yet) like you do. Best of luck with this and I’m excited to see where it goes
Oh that’s sweet! Very cool to see syntax-case in steel!
I hope I wasn’t missrepresenting your project! Let me be clear here: Steel is faster and more featureful than Scheme-rs by a long shot. I brought up syntax case because that’s were the initial bulk of my effort went (the remainder went to reimplementing the project as a compiled)
All good - the devil is in the (lack of) documentation :)
Syntax-case was a thorn in my side for years, I only recently put in the effort to get it functioning, I found it a bit mind bendy to get right
It absolutely is the hardest thing to properly implement in my opinion (and I’m technically not even fully compliant, I don’t add any continuations to the expander (soooo much work)).
I’m thankful I spent so much time considering it while the eval logic was interpreted, it was a weight off my shoulder when moving to compiling. That being said, compiling it also lead to a new set of challenges, I had to somehow figure out a way to capture the environment, something that I was just throwing out after compilation.
As an aside, have you noticed how there are so many people who work with Scheme named “Matt”? It’s starting to freak me out a bit
My keyboard and layout are my own designs. They are focused on a 30 key count with centered inner columns. That has been the sweet spot for me.
Keyboards: rufous & berylline
Alpha Layout: bird
Full Layout: zmk-config
how is it different from nushell?
I feel like if you spend a few minutes reading the page you can get an idea about where they are different.
If your comment is that you believe they should add a comparison to other shells on the website, then that might be a bit more constructive.
murex claims (and I believe it, although I haven’t tested it!) to be able to pass type information through pipes in a way which allows POSIX utilities and other old-fashioned stuff to be able to treat the data as an old-fashioned stream of untyped characters.
nushell and other shells that have typed data (e.g. elvish), I believe, all make you choose in each instance whether to pass things through pipes with type information, and when you do most utilities won’t be able to use that pipe.
I was surprised to see this undervalue stored generated columns and not provide the syntax for using them, which is this:
Using stored columns means they don’t have to be calculated at runtime, so you can do things like run indexed lookups against them.
Without the stored feature I find generated columns to be almost indistinguishable from SQL views.
Stored columns are the closest SQLite gets to materialized views, which are a very useful performance optimization.
The example in the post added an index based on the generated column. Is the “indexed lookups” that you are talking about different to that?
I think that assertion is just not correct. From the SQLite documentation:
So a virtual (the opposite of stored) column can be added to a table after the fact, and its values can then be used in new indexes.
FYI, you can index JSON properties without needing a generated column. SQLite has long been able to create indexes on expressions, not just columns.
Generated columns do make queries a lot cleaner looking, though!
Yes, you can! I’m not aware of a way to see what ends up in the index for a particular row, though. I feel like the virtual column gives you the ability to see it, even if only for debugging purposes. I also think it’s pretty easy to accidentally specify your query in a way that doesn’t end up using the complex expression index – especially if you’re using a query generating library like an ORM?
Huh, today I learned! Yeah I was wrong, stored columns aren’t actually very useful at all then, since you can index without them.
Blogged what I learned here: https://simonwillison.net/2024/May/8/modern-sqlite-generated-columns/
FWIW, this isn’t an unusual feature of SQL databases. Many of them allow you to write indexes on arbitrary transformations of columns.
E.g. you can do
create index on num_successes / (num_successes + num_failures)and a query that uses the same expression can use that index.If the expression is expensive to compute it could still be worthwhile to store the column.
Given the restrictions of the expression language - though I guess custom extensions might provide more expensive primitives - I have trouble imagining many situations in which that would be worthwhile.
The one thing I can think of is situations where you might run a filter against the generated column that requires either a full table scan or needs to consider tens of thousands of rows - but in both of those cases an expression index might be a better approach.
My understanding was that the language major mode was ideally supposed to configure eglot for optimal use for that language, just as it is supposed to configure tree sitter. It feels like cider could do quite a bit with its knowledge of the language and which other features are being used (nrepl) to smooth things over for its users here.
If the major mode has to resort to hacks, then it is a deficiency with eglot that needs to be addressed.
Is
quotestandard terminology for concatenative languages? Because if I “can imagine aquoteas an ordered list of other literals” then I’d probably have called itlistrather thanquote:)Yeah. I think that it is the same terminology used by Joy. I’ve always assumed that it got its name from a quoted list in lisp.
It’s definitely the term used by Factor.
Do concatenative languages deliberately avoid the terminology that other languages use? (Another obvious example is the way they use the word “word”.) I actually think I once read that there’s some reason why they do, but I might be imagining it? If not then I vote for @jcmkk3’s explanation.
Word comes from forth
When my dad was teaching me Forth, he told me they were called words because they were added to the dictionary.
The author mentions one thing they really like about meow is that it doesn’t define any keybindings by default and lets you add the ones you want. But something I like about evil-mode is that you have evil-collection which defines a uniform set of keybindings for every popular Emacs package. I don’t have to configure everything to use modal editing because there is already code to take care of that. But on the flip side that definitely makes evil more heavy weight then something like meow.
This is the idea with the keypad mode that the post talks about. Most packages follow standard emacs keybinding practices so the keybinds should already be relatively intuitive. Keypad mode lets you use them without modifiers.
This was a nice feature for me using a colemak keyboard, I use ‘neio’ instead of ‘hjkl’ — setting up meow was a breeze.
You don’t define the key in every mode but which key you want to bind to which meow action. The different modes are bound to meow actions so it works really well.
You can find my config here https://github.com/green7ea/dotfiles/blob/master/.emacs.d/elisp/base-meow.el
Given the vast number of scheme implementations available, I’m always surprised that people create their own lisp.
I suspect it’s the same reasons that people create their own Smalltalk: it’s a small enough language that it’s tractable to make your own. A full, compliant and performant, Common Lisp implementation is a huge undertaking, but a small interpreted Lisp or a Blue-book-equivalent Smalltalk is a tractable single-person project.
On top of that, you may well have slightly different requirements to any of the existing ones and it’s a great project.
My surprise is the people that haven’t, at some point or other, implemented a Lisp.
But it’s so fun!
How many schemes are out there that satisfy the design goals of Lone? There might be a couple, but none of the ones I’ve heard of can actually do this. Plus Lone has tables as a first class data structure with literal syntax for them; Scheme’s support for these tends to be very bad, with implicit quoting at best.
I see “first class tables” in the readme of the Lone github repo, but as far as I can tell there is no documentation, nor are there any substantial examples; the tests I examined are so trivial as to be non-illustrative.
It seems to be pretty much like clojure’s map syntax. Every two items within curly brackets is a key -> value. In the article, the output of env is:
{ "HOME" "/home/matheus" "EDITOR" "vim" … }Table is one of those many computer terms that is so ambiguous. I think that technomancy is talking about a hash table, while I suspect that you are thinking about a database type of table structure like you have in lil.
Yeah, from the examples in the post I’m assuming “tables” is just another way to say “hash tables” like it is in Lua but it’s true the documentation doesn’t do a good job explaining it yet.
I’ll give them a break since it’s a 1-person project, and they’ve specifically explained in the post that the language isn’t ready for use yet.
Ah, disappointing. I’m always curious to see other examples of languages which have tried something similar to Lil’s tables and query language.
“Table” is not a very good choice of terminology for a simple associative structure. “Dictionary” or “map” are nearly universal, and calling it a “HashMap” or “HashTable” unnecessarily entwines the implementation details with the contract.
Map is common, but it’s even more overloaded than table IME.
I love clean, well organized projects as much as the next Python dev but I have to admit that I love the idea of PEP 723, unlike the post author.
It makes it easier to write and support the category of tools that I think are one of the best use cases of Python: single file scripts. These tend to do one thing, (hopefully) in a simple and understandable way that you can hack on if you want to, without dealing with a whole project installation. They are enabled by the ubiquity of Python and its large standard library, but whenever you exceeded stdlib and had to use a dependency it was awkward to communicate “install these dependencies, then run this” compared to “just run
python script.py --help”. Granted we are not exactly there yet, but this PEP is a step towards that.I think this creates a weird activation energy hump where it’s easy to get started but then you get snagged as soon as you have more than one file, which encourages the building of too large single files. If they’re adding it, they also need a plan for how to make a smooth transition to multifile as part of the overall vision. It doesn’t need to be implemented all at once, but if there’s no vision, it seems likely that there will just be a new, weird way to get snagged as a new developer, in which everything is great until you try to add a second file and it all falls apart.
You are right, on the other hand the activation energy hump already exists to go from “script with only stdlib dependencies” to “script with external dependencies.” This simply moves the hump further. But I think it is a totally valid concern that this adds yet another way to do dependencies in Python and that it naturally breaks down at the multi-file transition. An overarching vision is indeed missing.
Yes. I agree. The impression that I get from the author’s post is that they are tunnel visioned on their own uses of python. Solving the packaging/project issues of python seems so straight forward to them and they believe an official solution that solves their use case should be put into place immediately, while other use cases like “scripting” and “data science” can be figured out later.
I am curious, why you say data science can be figured out later? Are you referring to bundling C extensions? My main problem as a day to day DS is some libraries not building wheels where I develop/deploy stuff, or not complying with latest standards (most notably, xformers by Meta).
I’m not saying that data science can be figured out later. I said that was the impression that I got from the author. They said:
Then in regards to PEP 723 (inline script metadata) they said:
To me, that reads as if there are at least two common use cases of python that they either do not believe are important or that they can be figured out later.
A common thread that I see in these discussions is that folks get frustrated that there is not an official solution when it seems like there is an obvious solution that can be seen in another language ecosystem, but the reality is that those ecosystems don’t have all of the same use cases as python.
I don’t think the end use will be that. At least, it’s not explicit. The PEP speaks of other tools like pip-run and pipx, and some package managers like hatch. And it kinda makes sense because it needs installation + virtual env, I guess.
Yes, and I’d be fine with
hatchorpip-rundoing some magic in the background to enable that, especially if they can be assumed to be easily available for the users of the script (which they can’t be yet, IMO).That is hilarious to put the dependencies there but I agree that it would be amazing and solve so many issues. It’s a bit of a bother to have a
Pipfilejust to support a single file Python script.Did they just reinvent immutable databases?
The proposal is for an augmentation to the
selectsyntax so I don’t really see the connection with immutable databases. I do think that a more rich payload could be handy sometimes.I found myself surprised by how much clearer the Polars “SQL-like” API is than the slicing orgies I’m doing in Pandas.
Their approach is very different. Pandas tries to operate more like a python collection, whereas polars is more like a query builder. Both styles have their challenges.
It’s not only a Pandas world now but becoming more and moreso. Skimming the PyPI stats for Pandas and Polars it looks like Pandas is growing at the rate of 1 Polars userbase per week. So interoperation is absolutely critical for Polars.
I was curious to learn of the “Python dataframe API standard”. You mention Pandas in conjunction with this, but are they adopting it? It looks very difficult to fit in with their API and I wasn’t able to find any integration efforts via google.
pandas.DataFrame.__dataframe_consortium_standard__appears to exist in Pandas 2.1.3, so I’m guessing they will (or do?) support it. It is hard to tell, though, what support status is in general, possibly because the spec is still evolving so compliance is still a moving target.It appears that this is a reference to a library called dataframe-api-compat which is some kind of compatibility layer between pandas (and polars) and the standard. Some kind of facade pattern like this is probably the only way to get pandas compatibility but I suspect that it must still be partial because, for example, pandas doesn’t support all the types in the dataframe standard.
That library describes itself as being for consumers of dataframes. And it’s not one of Polars’ dependencies.
Polars has become one of the reference targets as they are designing the api. They made a lot of changes in the last few months to make sure it could support the laziness and expression system that many of the newer libraries (polars, ibis, duckdb) tend to use.
I’ve been planning to do this upcoming Advent of Code in Lil. I think that it is a really cool little language. As a data analyst, a language with built in support for array programming, tables/querying, and visualization/UI is very interesting. Think numpy, pandas, matplotlib, etc. I wouldn’t use it for actual work things, but it hits closer to what I’m looking for in a language.
I really wish it was easier to try these bindings outside of specific editors. Maybe I’m in a specific situation as a kotlin developer working on a large Android project, but Android Studio has a perfectly good vim emulator built in and so does every other ide. Porting Kakoune’s behavior would be more than just mapping a list of bindings, it looks like?
It wouldn’t be a huge loss to get used to kak as a terminal editor, but then I’d feel sad all day in Android Studio, and it’s already too easy for me to feel sad there.
I think that it is probably easier to emulate kakoune bindings than vim bindings in most modern editors due to kakoune’s minimalist nature. The object-verb approach is also much closer to how a non modal editor works. I think that it is important that the host editor has multiple cursor support to build from. That makes emacs a more difficult target, for example. You can check out dance for vscode https://github.com/71/dance
Kakuone editing in emacs which incidentally uses multiple-cursors
as far as I’m aware, vim style editing also requires more than just mapping bindings, so I don’t see why adding a noun-first editing paradigm would take more work. Maybe plugins for this exist already in VS code and the JetBrains IDEs?
As some one in the data processing and analytics field, a fan of array languages (including R), and dataframe geek, I’m very much in agreement that this topic is worth more investment. Also, if you’re like me and want to understand the presenter’s background on the topic, this is from Graydon Hoare, the creator of Rust.
One day I hope they’ll open up to electrostatic capacitive rubber domes as an alternative to mechanical keyswitches. I’d pay premium for an ergonomic option that actually thocks.
It seems that only a very small number of people have made custom electrocapacitive keyboards, at least as far as I could find. There are a lot of things that make it finicky:
The circuitry is analogue, rather than digital, and keypress detection measures changes of a few picofarads, so the PCB design needs more expertise than usual. On the up side, like hall effect switches, you can adjust the activation point in software or use them as analogue inputs for things like mouse keys.
The rubber domes and conical springs are sandwiched between the PCB and switch housings, and loose when the keyboard is disassembled. It is tricky to get them aligned when reassembling. It is easier if the domes are on a sheet matching the size and layout of the PCB, which they won’t be in a custom layout.
After reassembly the firmware needs to be recalibrated to accommodate any changes in alignment of the domes and springs.
The plate and PCB must be firmly fixed to each other, since there is no lower switch housing to take the bottom-out force: it hits the PCB through the dome and spring, and none of it is taken directly by the upper housing or plate.
There are not many sources of switch parts. I get the impression Topre parts are only available through unofficial channels. Niz will sell switch parts to hobbyists in keyboard-sized quantities.
Some links:
https://github.com/tmk/tmk_keyboard/wiki/Capacitive-Sense
https://github.com/tomsmalley/custom-topre-guide
https://www.nizkeyboard.com/products/2019-new-niz-ec-switch
https://hhkb.io/modding/Rubber_Domes/
https://www.reddit.com/r/MechanicalKeyboards/wiki/modifications_topre/
The tako is a recent one to take a look at https://github.com/ssbb/tako.
Yup. Which is why I want to buy one—not build one. I built a ErgoDox way back, & I’d rather have a manufacturer do the hard work. Of course back then even Cherry switches weren’t the easiest thing to come by. But the folks that have tried Topre or NiZ’s switches broadly seem to pine for the feeling of these switches despite the limited supply. With the right nudges maybe the same thing that happened for mechanical switches could happen.
NiZ is cheap & still feels great, but the firmware releases as Windows executables in a Google Drive doesn’t inspire confidence. I’ve emailed them and they don’t seem to interested in the ergo and/or split keyboard crowd.