Threads for mjn
-
Not obvious from the title, but this is specifically about improving parallelism in Datalog engines. They report impressive performance in their prototype implementation, SLOG. Unfortunately, it does not appear to be publicly available.
-
Typst is very exciting - it seems like the first meaningful LaTeX competitor in a long time.
And it seems they’ve just implemented footnotes, which was the main blocker for me using it for schoolwork! I’ll probably give it another shot in September.
-
Have you considered sile? https://sile-typesetter.org/what-is-sile/
-
I really like SILE and I wish I could use it for papers, but there are some blockers:
- Conferences all provide LaTeX (and, often, Word) templates. If you use these, you won’t be rejected for failing to conform to their style, if you have to recreate something like the SIGPLAN style in SILE then that’s a lot of effort.
- Last time I looked, it was still missing equivalents of math mode and bibtex, which are essential for any scientific publication.
- LaTeX has a load of packages that do everything I want. Generating tables and plots from data? Pgfplots and pgfplotstable are there for me. Typesetting code? Listings is fine there are also some newer things that wrap external parsers. Diagrams, pseudocode, whatever all have some LaTeX package.
-
SILE dev here. Some non-exhaustive notes to your three points:
-
True, the demand from conferences, universities, and journals to use X template exactly is a hard one. Note this also affects
typstjust as much assile. That being said it is possible to write classes for SILE and for the most part shouldn’t even be too hard to replicate the layout and rules for many documents to come out the same as long as the product they want is a PDF and not the LaTeX or Word sources. I’m personally in the book business and not so much academia, but I’m happy to help package devs if anybody wants to tackle setting up a class to mimic some style guide. -
It might be a while since you looked, SILE supports math and bibtex. The math support is less refined than in Typst or LaTeX, but it covers most of MathML. Typst came out with focus on math first, for SILE is was contributed later. On the flip side Typst has very rudimentary support for many other typesetting features like frames, footnotes, RTL and TTB writing directions, hyphenation and special features languages and scripts, etc., while SILE focused on those out of the gate. Your needs may vary and a different solution might be better for different projects.
-
True, there is a huge depth to the LaTeX ecosystem. There are only a few dozen core SILE packages and another handful of 3rd party ones. That being said almost all the things you mention are doable, and parsing other inputs (such as diagraming) and embedding the output is way easier in SILE than LaTeX. One major difference between SILE and Typst is that Typst strictly limits what 3rd party packages/templates/functions can do, while SILE provides a Lua interface with 100% access to extend or tinker with it’s own internals as well as reach out to the system and run other programs to generate output. SILE also has a package management solution for 3rd party packages that is a bit more robust than Typst’s current allowance of mostly copy/pasting code into your project.
-
That being said it is possible to write classes for SILE and for the most part shouldn’t even be too hard to replicate the layout and rules for many documents to come out the same as long as the product they want is a PDF and not the LaTeX or Word sources. I’m personally in the book business and not so much academia, but I’m happy to help package devs if anybody wants to tackle setting up a class to mimic some style guide.
I wonder if there’s a volunteer at the ACM who would be interested in at least making their templates work with SILE. Most conferences I care about use either an ACM or IEEE style.
It might be a while since you looked, SILE supports math and bibtex.
It is, I looked in quite a bit of detail after having breakfast with Simon at FOSDEM many years ago, but have only periodically checked for progress. I saw some progress recently on an issue that I filed a few years back about documenting the APIs. Great news. How does BibTeX support work? BibTeX can include arbitrary TeX, which makes it painful, but I tend not to do that because I also want my BibTeX to work with things like Jekyll Scholar, so more limited support is probably fine for me.
There are only a few dozen core SILE packages and another handful of 3rd party ones. That being said almost all the things you mention are doable, and parsing other inputs (such as diagraming) and embedding the output is way easier in SILE than LaTeX
Things like TikZ and pfgplots[data] are useful precisely because I don’t need an external tool. In particular, the main reason that I use TikZ is that the fonts in my diagrams match those in my document. Most of the TeX fonts are also available as OpenType, so you can do this, but then if you decide to reduce the text size by 1pt you have to go and make the matching change in all of the figures. To give another concrete example, I have made very heavy use of a LaTeX package that draws bit patterns in machine words for describing instruction encodings. All of this can be done in SILE (probably in considerably less than half the effort of doing it in TeX and with better performance), but someone else has done it in LaTeX. I really hope that the SILE ecosystem starts to grow this kind of thing.
-
If there is anybody that wants to give it a shot with me I’d be happy to spin up a class for ACM/IEEE/whatever popular style guide is out there and facilitate the process of getting everything implemented as close a match as possible. If this interests anybody pop open a new issue with a link to the style guide itself and a sample input/output document (in LaTeX/PDF presumably) and I’ll fire up the boilerplate for a new class package and get he ball rolling. Obviously the simpler/more flexible the style guide we start with the better, and the simpler the PoC input we want to start with the better.
Obviously RAW TeX in BibTeX doesn’t work great, although since “turn about is fair play” we can process RAW SIL input (or XML) in BibTeX content, although our ability to process it as RAW Markdown is probably more useful. On other words it can be reasonably adapted for any input format(s), just like SILE itself can take XML, SIL, Markdown, or other input formats.
And yes, your comments about “this is hard in LaTeX but somebody already did it” are pretty much true. I still use LaTeX some myself just because some projects are already all setup to work with it, so the momentum keeps the ball rolling. I hate having to tweak things because the internals are painful, but for the stuff that already works out of the box it’s hard to argue with nice output. I too hope over time the ease of implementing new things in SILE with access to a full fledged language ecosystem (several actually since you can easily write C, C++, Rust, and execute it as a Lua package as well as shim in/shell out Python and others with bindings too) will allow the 3rd party ecosystem for SILE to thrive too.
-
Is there a good AsciiDoc to SILE flow? I used LaTeX for my prior books but I’m looking at AsciiDoc for the next one, since it lets me embed semantic markup that can flow through to things that I can style with CSS for the ePub version (I wrote my own TeX -> HTML converter for my last couple of books because all of the LaTeX tooling I tried made a mess of it), but I’d love to be able to use SILE for PDF versions.
-
That isn’t a flow I’ve ever done in production end to end, but there are several paths you could take. AsciiDoc → DocBook via Pandoc would be one, then directly typset using SILE’s DocBook support (XML with a set of style rules for tags basically). Another path would be again using Pandoc to convert to Markdown, then use the
markdown.sileinputter. A third way might be to use CaSILE which bundles a Pandoc fork that includes a SILE writer. The extra versatility in this is it allows you to embed raw SIL commands in your input (works for Markdown, not sure how AsciiDoc would handle this but I think there is a way). Feel free to open a discussion on SILE for either of the former or a feature request issue for CaSILE if they interest you. Any of the above should take just a little bit of fiddling to work out the right commands. If you want both PDF and ePUB output from plain text sources in any format CaSILE is what I would reach for (that’s what I made it for). -
I tried AsciiDoc -> DocBook -> SILE workflow recently. My conclusion is:
- this is theoretically and practically supported workflows. SILE has first-class support for ingesting DocBook
- this seems like a “do the right thing” principled approach, rather than just a pile of hacks
- at the same time, the actual state of support for various DocBook elements is poor: https://github.com/sile-typesetter/sile/issues/1338. If you want to use it, be prepared to contribute a good half of it upstream yourself
-
Realistically if you want to typeset DocBook you probably are going to want to style all the tags yourself. About have of them have default styling and the other are pass-throughs just to get the content to the page, but would need styling for use. The default styling though is almost certainly now what you want for any book project.
-
Thanks! Does this flow properly preserve custom elements in AsciiDoc through DocBook to places where SILE can pick them up? For example, in my last book I wanted to make sure that the code examples all worked and so each one was in a file that could be compiled and tested and referenced from the LaTeX and so I would want to add some handling in SILE to grab the right lines from the file (identified by delimiters in comments), pares the file with something external, and then feed that as text with particular styling into the SILE pipeline.
-
I don’t know, but I think this should work, yeah. I think what you are asking belongs to AsciiDoc -> DocBook step (I think it would be on the asciidoctor processor to do read the source code, extract specific lines, and emit DocBook with text already “pre-processed”), and, given that AsciiDoc is essentially a different syntax for DocBook, this step should work perfectly in practice.
In any case, on SILE side you use lua to do whatever (as an aside, after wondering “why typist is a whole programming language” and looking closer at SILE, I realize that this is because typesetting itself is essentially imperative. https://sile-typesetter.org/manual/sile-0.14.9.pdf#page123 Is a good example).
Fun context: historically, I advocated for the use of AsciiDoctor. Since then, I personally switched to Djot (which isn’t yet production ready), and I was looking for a great way to convert Djot to PDF. I looked at “Djot -> DocBook -> SILE” route, and, while I was ready to do the first step from-scratch myself, I don’t have enough domain knowledge to help polish the second step to “just works” stage. So I went “Djot -> HTML -> Chromium -> Print to PDF” route, with its horrible typesetting.
-
-
-
Even if recent builds of those Lua packages are broken in Nix, you should still be able to run
sileat the last package version of SILE because the dependencies by using locked dependency versions from the time of packaging. I can’t remember the incantation for that inside NixOS but I know there is one. You can also run the Flake version which has it’s own lockfile with dependencies at known working versions. For examplenix run github:sile-typesetter/sileruns the latest flake version, but you can also run any previous version since we introduced the Flake complete with whatever dependencies they had at the time.Feel free to tag me in Lua package related issues in nixpkgs if they affect SILE.
-
-
-
-
-
-
-
I wonder if there’s a volunteer at the ACM who would be interested in at least making their templates work with SILE.
I think you would need more than that, since ACM is moving to a publication workflow that ingests article source, not author-submitted PDFs. Not every conference (yet?) requires it, but some big ones do. The end result is that LaTeX or Word are really the only two options. I have met a few people who draft articles primarily in Markdown, and then have a home-rolled converter to ACM-compliant LaTeX. But that only works because they use a small subset of Markdown features that can be one-to-one translated (like sections).
-
Yes. I caveated my suggestion/offer above noting it will only help at this point for submission systems that just need the final output product, not the source.
This is a culture problem, not just a tech one though. My hope is that eventually the tooling to build/convert is standardized just enough that folks can write in their choice of LaTeX/Typst/SILE and submit the source repository, and the other end could process any of those, say using a standardized CI workflow with options for different typesetters. The workflow and output formats could still be standardized while giving some tooling choices on how to get there.
-
-
-
Wondering if you’d mind answering some semi-tangential questions of mine regarding SILE/TeX/typesetting.
- How hard is it to get the “core” of TeX right? From the FAQ, it seems that SILE has feature parity (or better features) than TeX, but as noted is missing the ecosystem.
- How hard would it be to transpile or interpret existing TeX packages? How does this compare with the amount of effort that would be required to rewrite them?
-
That depends a lot of what you think of as “core” TeX. I’m assuming you don’t understand the actual boundaries of TeX vs. the LaTeX/XeLaTeX/LuaTeX/ConTeXt ecosystems. I assume you are talking about basic typesetting algorithms such as line breaking or leading. This doesn’t match up cleanly at all with what core TeX actually is, but we’ll go with the basic algorithms. These took a lot of thought to get right, but they are quite easy to clone. They have been ported to dozen of languages. SILE itself has almost exact ports of a few factions (including step numbers in source comments from the original). The issue really isn’t implementing any one part, the issue is coming up with a system where they can all get along. Taking a string and wrapping it at a certain length with the line breaker isn’t too hard. In a naive box, that would be the end of things, and that’s where core TeX kind of leaves you to. LaTeX for all it’s huge ecosystem just doesn’t handle some page layouts though. SILE has taken a long time to work out how to make those core bits work as expected when you start chopping the box up into variable size frames that grow when you have sidebar insertions or mix and match with other layout adjustments. It also gets complicated when you start dealing with things outside of English and a few other Latin script based languages. Suddenly some of the core TeX stuff falls apart and you realize you need much more involved interactions between different parts of the chain to get things right. Transpiling TeX is possible, but it still leaves you with all the idiosyncrasies and limitations of TeX. There is a pure Python implementation called
yexfor example, but it doesn’t get you much farther (or even as far) as TeX itself.Ground-up rethinks like Typst (Rust/custom) or XTS (Go) or SILE (Lua/Rust) are important because there is lots more wrong with the fundamental workings of TeX that just the language the typesetter is implemented in. Sure that makes it extra hard to fix these days, but as far as typesetting engines go none of the modern alternatives are really winners because of the language the engine is written in, at the end of the day they mostly stand or fall based on what they can accomplish.
-
Thanks for the detailed response, that helps clarify my understanding of the state of things. And you’re right, I don’t know jack about the ecosystem. Looking at the Typst’s description of how XeLaTeX is compiled was mind boggling to me.
To explain my questions: I’ve long desired something that acts like TeX but doesn’t look like it. For example, Typst with TeX-style math and - importantly - access to most TeX packages. I don’t use TeX for anything complicated or (that) precise. Like 80% of the documents I write are either simple prose or (often reasonably simple) math. I really would love something that can hide the complexity of the “fiddly bits” of TeX, or at least wrap it in a DSL I can actually interact with. I can see now, though, how my opinion as a user of TeX is naive to someone who knows about how TeX works. I do think that if someone can make “Typst (i.e. markdown-like front end + a way to deal with the fiddly bits that doesn’t make me so, so sad) with backward compatability with the TeX ecosystem” that would be awesome, but I suppose I understand now that this is difficult if not impossible.
-
SILE might be exactly what you want. Simon started with the original TeX papers and implemented not just the algorithms that TeX uses but also the ones that TeX wanted to use but were infeasible. For example, TeX has a really nice dynamic processing algorithm for line breaking that takes a whole paragraph, assigns badness scores to each possible breaking position, and finds a solution that minimises badness. The paper describing this points out that the same approach could be used for laying out paragraphs in a document but that, for large documents, this would need a megabyte of memory and so is infeasible. SILE implements that version, because it’s now a trivial amount of memory (even with the overhead of using a Lua table rather than a C array) is trivial.
This means that there are some important things that you can express in SILE but not in TeX. For example, most publishers have a rule that a figure must not appear before the first reference to it. TeX can’t express that because it places the figure using a greedy algorithm. SILE can assign a maximum badness value to any possible placement of the figure before it’s first reference and then have it placed nicely.
At the same time, SILE is written in Lua, whereas TeX is a horrible Turing machine that is the reason that I don’t listen to anything Knuth has to say about software engineering (while having nothing but respect for him as an expert on algorithms). This means that humans have a chance of being able to extend SILE.
Simon chose Lua specifically to expose the entire pipeline to users and allow anything to be patched (well, actually, he chose JavaScript for this reason and was persuaded to rewrite it in Lua and have the same benefits). This means that it should be easy to embed in other things and extend arbitrarily.
-
-
-
-
-
Have you looked at KeenWrite? it’s my free, open-source, cross-platform, desktop text editor that invokes ConTeXt for typesetting. The editor uses KeenType for its real-time mathematics preview.
See the video tutorials for details.
-
-
This seems like a pretty complex way of managing lifetimes and scopes. It’s not obvious to me that this is simpler or more powerful for the programmer than an ownership and lifetime model.
It seems like this was chosen because it was simpler for the compiler implementer.
What I do like is how explicit this all is.
-
Modes are a lot less powerful than explicit lifetimes, but substantially simpler. It’s a feature that covers common use cases, but cannot express a lot of things that you can express in Rust: modes give you no way to say that a value is “owned by” the data structure it resides in, for example — as a first approximation all you’re saying is that some values are allocated on the stack and some on the heap.
It’s important to remember that OCaml is a garbage collected language, and the garbage collector is still doing all of the heavy lifting for non-local allocations. This feature gives programmers a way to opt out of garbage collection (and the corresponding performance overhead) for values that can be statically proven to be short-lived — it’s not an alternative solution to memory management in OCaml.
-
This feature gives programmers a way to opt out of garbage collection (and the corresponding performance overhead) for values that can be statically proven to be short-lived
It does seem quite nice for that. Though that makes the Rust comparison feel like a red herring to me, since they are not that similar in overall design. At least from my read, this OCaml feature is a lot more similar in scope and intent to something like Common Lisp’s
dynamic-extentallocations. Though one important difference is that Common Lisp doesn’t require the implementation to verify these annotations statically (the implementation is allowed to assume that the programmer has correctly specified that the value does not escape).-
I think the static guarantee is the interesting bit (and the reason that the comparison to Rust makes sense). It’s not just allocating values on the stack, but allocating values on the stack and guaranteeing that references to those values can’t outlive the stack frame. (Speaking loosely here, since the allocation regions aren’t literally the call stack, and exclaves mean you can allocate in other regions, but you get it.)
-
-
-
-
There’s a pretty good general analysis of this as a PL design/implementation question, with discussion of what Javascript/Ruby/Python/C#/etc. each do, in a sidebar of the book Crafting Interpreters in the chapter on closures, “Design Note: Closing Over the Loop Variable”.
-
-
This is a good take, and historically accurate IMO. Rule engines were indeed marketed as a more practical way of getting the benefits of declarative rules, in a form where supposedly non-programmers would be able to maintain the rule base. And yes, the relational model (in the unattractive guise of SQL) won out in the database space.
Just like functional languages inspire people to write “more functional” code in other languages, I think Prolog can inspire us to think in relations (SQL) and rules. Rules engines grew into monstrosities, but more light-weight rule engines exist and can be a great tool.
(This is a copy of my own comment at citrus-family-colored site.)
-
I was a bit apprehensive to see something I wrote in grad school 13 years ago show up here, so I’m glad someone thinks it’s still a good take. :-) I do think logic programming hits on a bunch of good ideas, and Algorithm = Logic + Control is one of my favorite slogans in computer science. But I wrote this because I thought the explanation I had seen for Prolog’s issues (linked at the beginning), that a big Japanese government project was mismanaged and took Prolog down with it, missed the mark.
-
And yes, the relational model (in the unattractive guise of SQL) won out in the database space.
I recall a project putting Datalog atop SQL that might be interesting, even if only historically: https://lobste.rs/s/o2qvt1/project_mentat.
-
Google has something along those lines as well, intended as a front end to generate SQL for BigQuery: https://opensource.googleblog.com/2021/04/logica-organizing-your-data-queries.html
-
-
-
A lot has been written about the downfall of Prolog. As someone who didn’t live that moment but now contributes to Scryer Prolog I think there’s no one reason, but several.
One of the reasons that I do not hear in this kind of posts is the lack of standardization. You didn’t program in Prolog, you programmed in Quintus Prolog, with its dialect (Edimburgh tradition). But you had also Turbo Prolog (now Visual Prolog), Prolog II and Prolog IV from Prologia (the company from Alain), Arity/Prolog32, … and even more academic systems. When ISO Prolog got defined in 1995, it was already too late. If you didn’t hear before about some of those dialects is because they’re effectively dead. The only branch that still receives work is the Edimburgh-ISO family. I think other languages in that era suffered from the same.
It’s funny that the post, in 2010 thinks that Mercury and Oz are more promising than Prolog, because, at least from my perspective, they still have fewer users than Prolog.
-
It’s funny that the post, in 2010 thinks that Mercury and Oz are more promising than Prolog, because, at least from my perspective, they still have fewer users than Prolog.
I agree, this part has aged least well of the various speculations in there (I wrote the post). In the 2000s it felt like these multiparadigm logic-flavored languages had more excitement behind them than Prolog did, but they didn’t really take off.
One thing that did happen since 2010 is a Datalog revival of sorts, driven by the static analysis community discovering that it’s a nice way to formulate some of the analyses they’re interested in. That’s led to new, fast engines like Soufflé, and a bunch of new users who don’t come from an AI/knowledge-rep background.
-
Same as Scheme and Lisp.
This is where Java really really shined. You could take some code you found and an implementation and it would mostly just work.
My first k exposures to Scheme was to find some neat library or program, grab a Scheme I could install easily enough and then have it spew stacktraces and cryptic errors. Then reading through the code itself, I could if I was lucky, find out which Scheme they were using. They just assumed you would know which Scheme they were using. It is almost like language designers hate their users and their users hate each other.
-
One of the reasons that I do not hear in this kind of posts is the lack of standardization. You didn’t program in Prolog, you programmed in Quintus Prolog, with its dialect (Edimburgh tradition). But you had also Turbo Prolog (now Visual Prolog), Prolog II and Prolog IV from Prologia (the company from Alain), Arity/Prolog32, … and even more academic systems.
I had a play with Prolog a few years ago, and not being aware of this up-front caused me great confusion and dismay when even trivial Prolog programs weren’t working on my system …
-
I don’t know, this seems like a very inside baseball idea of why prolog didn’t take off. I think it’s fundamentally just that network effects exist and it is difficult to bootstrap language uptake when people also need to pay a large learning cost to get in. The only thing I’ve seen in the last decade that actually succeeded in this regard was rust, and that required an unbelievable level of fanatical hype & evangelism to convince people to move off the objectively terrible experience of writing C & C++. But it happened! Thousands upon thousands of people really learned how to write proofs of memory safety.
-
network effects
Balkanization prevents the network effect from occurring! If Rust pulled a D and had Phobos/Tango, boom, network effect would be destroyed. If Rust had two different backends that were 100% total, the ecosystem would split along which target they were using.
-
-
Rust helped itself by using Algol syntax instead of ML syntax.
-
I would bring up Python and Ruby but they got big more than ten years ago and now I feel old. :)
Edit: and of course they aren’t really applicable examples anyway because both have gentle, not steep, learning curves.
-
-
-
Ability to influence ranking/reputation just by praising it in plain text, without links, is a new problem. That’s really rough, because it removes the easy to identify sign of spam.
-
-
I agree, plaintext spam by paying people to expressive positive or negative sentiment seems to be in a lot of places. I notice it most often on Reddit. It’s often hard to say for sure, but some product-related subreddits (even ones I think of as useful, like /r/buyitforlife) have a substantial number of comments that I suspect are paid. Cheaply scaling up to a whole blog of that kind of content does seem new. I wonder if it’s actually effective? Seems like the Reddit spam would be more effective.
-
Reddit is already facing a deluge of ML-generated content/spam: Reddit Moderators Brace for a ChatGPT Spam Apocalypse.
Bearblog is a very small player in this space, but each entry is essentially free - why not create a bunch of free blogs and fill them with ML-generated text? The people using this are doing it across Reddit, Facebook, etc.
-
-
-
-
A lot has happened since this was written.
For political reasons, governments are spending a whole lot more money on HPC and the tools have gotten far better.
Spark has faded in importance as most of the stuff people were doing with it now fits on a single big node. But the Chevy Spark (car) became very popular which makes the Google Trends results for “spark” look positive.
Big non-HPC datacenters have adopted Infiniband and message-passing architectures.
HPC clusters can now easily run Kubernetes which opens up all the non-HPC distributed applications like databases and queues.
And MPI is still there and still extremely popular for physical simulations. Most of the simulation “codes” are relatively simple programs, that have been rigorously and thoroughly checked and re-checked over decades of work by brilliant scientists. They would be very hard to rewrite against a different API.
-
And MPI is still there and still extremely popular for physical simulations. Most of the simulation “codes” are relatively simple programs, that have been rigorously and thoroughly checked and re-checked over decades of work by brilliant scientists. They would be very hard to rewrite against a different API.
Yep, I think articles like this fail to account for the mechanisms behind how and why HPC software gets written!
I did a three-year stint writing number crunching code way back. I used a lot of MPI code, but wrote very little MPI code myself (realistically, I wrote truly non-trivial MPI code just once, and it was not the right tool for that level of abstraction so the next, better version of that code used a new, very cool library called ZeroMQ :-D).
That’s because the typical flow for what we did was:
- Study a hard engineering problem/physical phenomenon/whatever
- Devise a mathematical model for it (tl;dr turn that beautiful thing into a bunch of ugly PDEs, then figure out how to turn that into a discrete model)
- Write a program that implements the discrete mathematical model
Lots of people on the software side (and, sadly, on the journalistic side) think #3 is the hard part, which requires the most effort and the most focus.
It’s not. #1 and #2 are, and #2 is actually the fundamental one – coming up with a quantitative understanding (in the form of a model) of a phenomenon, so that you can understand it and/or put it to practical use.
#3 is the (usually trivial) part that you unfortunately have to engage in primarily in order to validate your work in #2, because lots of phenomenons just don’t result in a mathematical model that you can solve with pen and paper in practical cases. Sometimes, if the thing at #1 is really relevant on a wide enough scale that people are willing to pay for a program that solves it, it gets turned into a commercial endeavour and maybe it gets a little more attention, but that’s not extremely common.
That’s where the MPI part comes in handy. There’s lots and lots of code, some of it dating back to the 1980s, which very reliably implements all the mathematical tools that you need – code that e.g. solves huge sparse systems of linear equations through all sorts of ingeuous methods, optimised for various types of constraints.
All that was code written by a lot of smart math/CS PhDs who worked specifically on that – devising that kind of mathematical tools. Everyone else – physicists, chemists, engineers (in my case, of the electrical kind) – is forever in their debt. I did not want to spend time writing that kind of code. First of all I probably couldn’t – I was certainly not inept at math but nowhere near good enough to write the kind of code that people who’d spent years studying and advancing the field of numerical methods were. Second, even if I’d tried, the folks upstairs would’ve probably been pissed that someone who was supposed to be working on solving engineering problems was now writing abstract math code instead.
So there was a huge amount of code (“was” = almost 15 years ago, I don’t know what’s trending these days) that used MPI as in, it used e.g. MUMPS (the sparse solver, not the language). That was code written by teams like the one I was in, which tried to crack tough engineering problems, and wanted to spend as much time as possible working on cracking engineering problems and as little time as possible implementing our cracks.
You do need to write some MPI boilerplate to use those – e.g. you have to preprocess and partition simulation data. That’s kind of at the interface between #2 and #3 above, and it’s pretty nutty work. The actual programming side of it is easy, it’s the math it arises out of that’s hard, and it’s often hard to check the code against the math.
It wasn’t very uncommon for some of this software to use MPI at “higher” levels of abstraction, too, e.g. in order to share configuration data. MPI isn’t great at that but when you literally just have to send a couple of bools over the network, it’s good enough, and it beats having to import another library. It’s not something that was done out of laziness to learn another API: it was done after a few generations of PhDs learned the hard way that, while a lot of number crunching code survived for 10, 20 or even 30 years, lots of networking code didn’t, so you wanted to stick with the things that were most likely to survive the next dotcom boom lest you have to start porting the least important part of your code fifteen years from now.
Edit: OP is right to point this out:
They would be very hard to rewrite against a different API.
And lots of people on the software end of things underestimate just how hard it is. This isn’t an abstract “second system syndrome” kind of problem, where the main challenge is keeping ambitions in check, and as long as you can do that it’s a very simple project because you already have a reference implementation.
Dijkstra once pointed out that numerical code is the most trivial kind of code, and he was right in principle. What you quickly learn later, however, is that debugging parallel numerical code is insanely hard. You’re literally stuck trying to dissect the step-by-step progress of a problem that you’ve written because you can’t do that kind of math step-by-step in the first place, which is made even harder by all sorts of numerical weirdness. Even recognizing bugs is difficult, because they often arise out of new mathematical models, which may themselves be wrong in the first place. The equivalent of “it’s never the compiler” is “it’s never the solver”: if your program insists on a gate current of 43034892 A, you tend to suspect the program, not the library that does the math.
Rewriting both the libraries and the programs that use them in a new API is an extraordinarily costly endeavour. It’s not that there’s no value in it, it’s just that said value is dwarved by the cost. It looks good on paper right up until you have to do the budget, then it’s an absolutely terrible idea.
-
the tools have gotten far better … HPC clusters can now easily run Kubernetes
From the perspective of a lowly user (in academia), I feel the tools have gotten more… heterogeneous I guess? Which is overall not necessary better, though some things are easier than they used to be.
Some clusters run the classic Slurm/LSF job submission system with shared persistent NSF storage. Other clusters are Kubernetes-only, like the NSF-funded National Research Platform. And as a third option, some labs are all-in on AWS/GCP/Azure tooling and workflows, funded by research credits or CloudBank or similar. In practice I find myself spending a lot of time repackaging and porting software to run it in a way that it wasn’t expecting to be run, which is not too fun.
On the plus side, it’s nice that these resources exist in the first place. :-) The National Research Platform in particular opens up moderate HPC capability for individual researchers even at small liberal arts colleges, for no monetary cost (just the cost of figuring out how to put your stuff into Kubernetes jobs), which is pretty neat.
-
-
This is very well written and motivated :)
I was always interested in Lua because it was nice and small, but I felt the language itself was a quirky, with some footguns … Also interested in Clojure, but not the JVM.
Janet sounds interesting
-
In my experience Fennel fixes about 90% of the footguns of Lua. About the only ones left are “1 based array indexing” and “referring to a nonexistent variable/table value returns nil instead of being an error”, which are pretty hard to change without fundamentally changing the runtime.
-
-
-
Which two?
-
Fennel and Janet are both from Calvin Rose.
-
Huh I actually didn’t know that! @technomancy seems to be the defacto maintainer since 2020 or so.
-
There’s quite a bit of history to how fennel came to be what it is today. It is correct that Calvin (creator of Janet) started it, but it would have just been an experiment in their github if it weren’t for technomancy’s interest in reviving/expanding on it. I don’t know if it is written down anywhere, but Phil did a talk at FennelConf 2021 about the history of fennel, which is the most detailed background for those interested. https://conf.fennel-lang.org/2021
-
-
-
-
-
I did a survey a while back about new lisps of the past 2 decades. IIRC the only one to evolve beyond a personal project and have multiple nontrivial contributors but not use Clojure-style brackets is LFE, but LFE was released only a few months after Clojure. It’s safe to say Clojure’s influence has been enormous.
However, Janet seems to take some characteristics of Clojure out of context where they don’t make sense. For instance, Janet has if-let even tho if-let only exists in Clojure because Rich hates pattern matching. Janet also uses Clojure’s style of docstring before arglist, even tho Clojure’s reason for doing this (functions can have multiple arglists) does not apply in Janet as far as I can tell.
Although there’s also the curse of Lisp where the ecosystem becomes fragmented
The other main influence of Clojure is not syntactic at all but rather the idea that a language specifically designed to be hosted on another runtime can be an enormous strength that neatly sidesteps the fragmentation curse.
-
Ahh very interesting, what were the others? (out of idle curiosity)
I think I remember Carp uses square brackets too.
There’s also femtolisp, used to bootstrap Julia, but that actually may have existed before Clojure as a personal project. It’s more like a Scheme and uses only parens.
I agree the runtime is usually the thing I care about, and interop within a runtime is crucial.
-
Here’s the ones I found in my survey; I omitted languages which (at the time) had only single-digit contributors or double-digit commit counts, but all of these were released (but possibly not started) after Clojure:
- LFE
- Joxa
- Wisp
- Hy
- Pixie
- Lux
- Ferret
- Carp
- Fennel
- Urn
- Janet
- Maru
- MAL
All of these except Urn and LFE were created by someone who I could find documented evidence of them using Clojure, and all of them except Urn and LFE use square brackets for arglists. LFE is still going as far as I can tell but Urn has been abandoned since I made the list.
I was working on this as a talk proposal in early 2020 before Covid hit and the conference was canceled. I’d like to still give it some day at a different conference: https://p.hagelb.org/new-lisps.html
-
That link is super cool. What do you mean by “implicit quoting”?
-
Thanks!
Implicit quoting is when lisps like CL or Scheme treat certain data structure literal notation treats the data structure as if it were quoted despite there being no quote.
For example, in Racket you can have a vector
#[(+ 2 3)], without implicit quoting this is a vector containing 5 but with implicit quoting it contains the list(+ 2 3)instead where + is a symbol, not a function. Hash tables also have this problem. It’s very frustrating. Newer lisps all avoid it as far as I know.
-
-
-
-
-
-
-
Hm although now that I look, the VM doesn’t appear to be re-entrant like Lua
https://janet.guide/embedding-janet/
Python has been trying to move toward a re-entrant VM for long time, with subinterpreters, etc. – I think all the global vars are viewed as a mistake. Aside from just being cleaner, it makes the GIL baked in rather than an application policy, which limits scalability.
This kind of API looks suboptimal to me. It would be nice to take something like a
lua_State.janet_init(); /* Unlock GC */ janet_gcunlock(handle); -
-
The main reason that I use Lua is Sol3. Lua itself is just Smalltalk with weird syntax, but the integration with C++ that you get from Sol3 is fantastic.
-
-
I am trying to understand this line in the context, just out of curiosity. Maybe someone here knows more. Anyone with some background could give some general response not in a professional capacity or as an advice, but more how law and licenses work in general?
we are exercising our right to terminate and revoke any license or sublicense under Apache License v2 and the GNU AGPL v3 in accordance with the terms of those licenses.
How does this work on a legal level?
First of all Apache 2.0. What gives them the right here in general? Isn’t it that if they obtained it legally they can essentially do what they want as long as they attribute? Can that right be revoked? They break the conditions by not attributing, but can that license itself be revoked? How does that go further? Given that this settles in court and the case ends, Weka is punished and so on, given that it is revoked can Weka then still not use it anymore, even when they give attribution?
And as for GNU AGPL. How is that with enforcement? Isn’t part of it that it’s basically not them anymore holding the copyright? Doesn’t that mean it’s not theirs to revoke? I mean on a technical/legal level.
-
Claiming to revoke the Apache License v2 is odd indeed. The copyright license (section 2) explicitly states that it is perpetual and irrevocable. The patent license (section 3) does have a revocation clause, but only in the case of patent litigation instigated by a licensee.
-
Minio’s main repo is only available under the AGPL, so that can be revoked. For apache, I am not a lawyer, but I think it works like this: section 2 starts “Subject to the terms and conditions of this License,” so if you don’t follow the terms (e.g. by not attributing) then you don’t get the benefits in section 2. So the copyright grant isn’t revoked it just never occurred in the first place.
That said, I don’t see how MinIO can stop Weka using their Apache-licensed code in the future so long as Weka does follow the terms in the future.
-
-
-
Mostly spending it attending the Philosophy of Deep Learning conference at NYU. I’m not a philosopher, but I like to keep up on philosophy of AI. So far it’s interesting!
-
I actually agree that LLMs are likely to be the start of something big: they have basic ability to crap out code (like a pure tutorial jockey but faster) and they have the ability to generate bullshit, both of which are the sole skills of a negligible part of the workforce.
Unfortunately they also have significant limitations that make their promise entirely in the future:
-
They can’t take substantial new input so they can’t learn context (I assume this is basically an implementation issue - everyone will need their own model to incrementally train); which
-
means they can’t handle context; and
-
Means they can only generate fairly small snippets de novo and can’t edit existing code.
I doubt we’ll see a 5x productivity boost, but I predict within 2 years we’ll have surprisingly good refactoring tools that can also make functional changes.
I suspect one consequence of this will be that way more design and code becomes brute force based, for better or worse. Obviously better for large cloud providers.
-
Unfortunately they also have significant limitations that make their promise entirely in the future:
They can’t take substantial new input so they can’t learn context (I assume this is basically an implementation issue - everyone will need their own model to incrementally train); whichmeans they can’t handle context; andMeans they can only generate fairly small snippets de novo and can’t edit existing code.Nope to all 3.
-
I fed davinci-003 straight up copies of the Raspberry Pi Pico docs on the Programmable IO system, which is basically a novel thing which doesn’t really appear many other places (at least by that name) and which didn’t exist in the training set. The LLM did a much better explanation of how this worked than the docs do, and wrote working code based on it.
-
LLMs probably have significantly more basal context than we humans do (they’ve seen more examples of things in any arena), and when fed direct information about the current task are able to integrate the training data with the current task data. This is handling context.
-
https://github.com/joshka/Crow is an experiment I played with of one of the earlier OpenAI models editing the code that’s calling it.
-
-
How did you feed davinci the docs? 2k characters at a time?
Yeah pretty much (though it’s 4k tokens which is about 16k characters or so). Ask a question, paste the part of the summary and code that was previously generated by the same process plus a new chunk code to add relevant detail and repeat. I wasn’t dealing with the entire pico datasheet, just the part I was interested in, and the related source headers. It was good enough that I built a (simulated) working 23 port UART MIDI splitter (simulated) with its help having never written code for a pico prior. https://www.joshka.net/2022/11/pio-for-midi-with-raspberry-pi-pico
I’ve since discovered Langchain as a rather more optimized approach to the manual playground things I was doing.
Take a look at https://www.youtube.com/@chatwithdata, https://www.youtube.com/@jamesbriggs and https://www.youtube.com/@DavidShapiroAutomator for some inspiration.
I mentioned https://github.com/joshka/Crow/blob/crow-v2/crow.py ~70 SLoC + docs which was generated from 7 bootstrap lines + only English language prompts. That’s 90%+ LLM generated. More if you include the docs.
import os import openai instruction = input("Enter an instruction: ") script_name = os.path.basename(__file__) script_code = open(script_name).read() response = openai.Edit.create( model="code-davinci-edit-001", input=script_code, instruction=instruction, temperature=0) new_script_code = response["choices"][0]["text"] with open(script_name, "w") as f: f.write(new_script_code)This was using the edit model (code-davinci-edit-001), I bet that GPT-4 would probably be better at this.
-
-
-
An angle I think might address some of those limitations is to treat the LLM as a raw idea or variant generator, instead of as the top-level “AI” system. It’s pretty good at spitting out large amounts of vaguely reasonable or close-to-reasonable code. Then you can plug that in to other, existing paradigms for auto-generating code as the top-level “AI” system. Those go under names like “genetic programming”, “program synthesis”, “inductive programming”, etc. (the literature is scattered because related ideas have come out of AI, programming languages, logic, etc., under different names).
For example, this paper does genetic programming (GP), but replaces the AST-based mutation operator that a GP system would normally use with an LLM proposing code diffs. This can sometimes improve efficiency a lot, because the LLM’s training means that it has better priors on what “looks like reasonable code” than most GP systems would have. There’s a follow-up paper (disclosure: I’m 2nd author :)) that investigates also replacing the genetic “crossover” operator with LLMs.
-
That all sounds reasonable BUT the big thing about llms is that they work with imprecise natural language. Presumably if you’re using existing program synthesis techniques, a much more rigorous specification is needed.
Of course one might use an llm to help generate the specification. But again editing a specification has the same problems.
-
Yeah, that’s true. A lot of the interest is that you can kind of vaguely gesture towards a problem and get pretty decent boilerplate code out. Perhaps that is most programming!
I’m more of a researcher than a programmer, so admittedly I may have atypical problems. But for me it’s usually fairly easy to generate, if not a formal spec per se, at least a decent test that can score the quality of generated code from like 0-100. For example, one thing I’d like out of an automated programming system is auto-parallelization. You can specify that problem as: I want code that works like this existing code, but on the GPU and a lot faster. It’s not too hard to convert that “spec” into a fitness function for a GP system – score generated code quality by some weighted function of “agrees with the original on randomly generated inputs” and “is faster”. But despite it being easy to state the spec, solving it is basically impossible. Existing GP systems will just churn on that forever; the combinatorial space of program transformations is too big to actually find solutions for anything but really trivial algorithms. My hope is that LLMs might be a component of a solution, though I don’t think they are anywhere close to solving it directly.
-
-
-
-
-
I often find myself reaching for map over unordered_map not because I care about an order but because iterators that are stable in the presence of insertion/deletion are useful. Am I missing something? Nobody else seems to be talking about this use case, does it work for unordered_map too and I’m operating under the misconception that it doesn’t?
-
-
david_chisnall
3 months ago
|
link
| on:
Effortless Performance Improvements in C++: std::unordered_map
This is one of the weirdest things about the C++ stdlib. The unordered map and set were added in C++03, as I recall. The ordered ones were in the first standard. I don’t think I’ve ever had a use case where I care about stable ordering. Maybe once or twice in a couple of decades of C++. Pretty much every program I’ve written in any language wants a dictionary though, and when a standard library says ‘map’ or ‘dictionary’ (or ‘set’), I assume it’s some form of hash table, unless it’s a variant that’s specialised for very small sizes. Having the default for these be a tree is just confusing.
-
unordered_map is from C++11, so it’s relatively modern. What boggles my mind is that, despite being modern C++, standard still messes up and hard-codes a particular (and slow) implementation strategy via exposing bucket iteration. Like, I can see how
std::mapcould be a tree because, when Stepanov was designing what were to become STL, we didn’t care about cache that much. But fixing that by including a crippled hash map is beyond me.where I care about stable ordering
natural ordering is indeed rare (and often, when you have order, you want just a sorted array). However, having some stable iteration order is a much better default than non-deterministic iteration, at leas if you don’t need 100% performance. So, I’d expect more or less every GC language to do what Python and JS are doing, for the default container.
-
What boggles my mind is that, despite being modern C++, standard still messes up and hard-codes a particular (and slow) implementation strategy via exposing bucket iteration. Like, I can see how std::map could be a tree because, when Stepanov was designing what were to become STL, we didn’t care about cache that much. But fixing that by including a crippled hash map is beyond me.
See “Chaining vs Open Addressing” in the proposal for std::unordered_map for historical background about why they went with chaining in 2003.
-
In hindsight the justification was terrible: “nobody has written a good implementation yet”. And later Google has written swisstable.
-
-
unordered_map is from C++11
It was in TR1 and merged into C++ in C++03, so it’s been around for 20 years. There are C++ programmers who started programming after that. I first used it many years before C++11 was standardised.
What boggles my mind is that, despite being modern C++, standard still messes up and hard-codes a particular (and slow) implementation strategy via exposing bucket iteration
I’d have to check the spec, but I don’t believe that there’s a requirement that the size of a bucket is >1.
natural ordering is indeed rare (and often, when you have order, you want just a sorted array). However, having some stable iteration order is a much better default than non-deterministic iteration, at leas if you don’t need 100% performance.
After writing my original comment, I remembered the one case where I have recently wanted a stale (sorted) ordering. I haven’t ever wanted the Python behavior of insertion order and, for the any case where I might, it’s easy to add in a wrapper, but impossible to remove in a wrapper. I believe the main reason that languages like Python keep this is to make JSON serialisation simpler, which isn’t a consideration in most places where C++ is a sensible choice.
-
I believe the main reason that languages like Python keep this is to make JSON serialisation simpler,
Python had a json module before dict iteration order was made stable by setting it to insertion order.
A much bigger reason is that a change landed is Python 3.6 that shrunk the memory usage of Python dicts by making the hash buckets be indexes into a dense array of entries (1-8 bytes each, picking the smallest possible size for each dict at runtime during resizing), instead of the buckets being
typedef struct { Py_ssize_t me_hash; PyObject* me_key; PyObject* me_value; };(12 or 24 bytes depending on arch). Offhand I believe Python dicts typically have a load factor of about 0.5ish.dict iteration order matching insertion order fell out of this almost for free. It got added to the language documentation for python 3.7. Arguments in favour included “it already does iteration order as of python 3.6” and “it’s helpful to remove one possible source of non determinism”.
-
It was in TR1 and merged into C++ in C++03
Hm, I am 0.95 sure unordered map is TR1, but not C++03. According to https://en.cppreference.com/w/cpp/language/history TR1 came after C++03 standard?
-
Ah, you’re right. I think libstdc++ exposed it only in c++03/gnu++03 modes. I was definitely using it (possibly in the experimental namespace?) long before 2011.
-
-
-
+1 for TR1…in a previous life my gamedev buddies and I had to wrap
unordered_mapto account for some minor differences of some form or another between MSVC and GCC. I think that’s all in the past now, one hopes.
-
-
-
I personally still end up defaulting to
std::setandstd::mapunless profiling shows that it’s a bottleneck, even when I don’t care about ordering, because they Just Work for just about any reasonable type, while the coverage ofstd::hashis weirdly spotty. For example I fairly often want a tuple as dictionary key, but there is no standardstd::hash<std::tuple>specialization (or one forstd::pair, orstd::array), and I don’t want to go down a rabbit hole of rolling my own hash function or pulling in boost just to use a dictionary. -
-
Map and Set have insertion ordering in JS - it was our solution to ensuring definable behavior (and also consistency with the existing object/dictionary use cases. JS has been hit by Hyrum’s law numerous times over the decades and having a defined order meant that there was no leaky abstraction horror.
Take the “always randomize iteration order” solution people often propose - does that mean that getting two iterators of the same map will enumerate in the same order. Will the enumeration order change over time? How does mutation impact ordering, how does gc impact things, etc.
The problem with unordered_map is that it overprescribes the implementation scheme, but what it has prescribed also doesn’t provide any useful gains to end users/developers. I haven’t seen any large scale C++ projects that don’t end up using a non-std implementation of unordered maps due to the std definition due to the poor unordered_map performance mandated by the spec.
-
I’ve used unordered map a lot in C++. I often don’t have a sensible total order to use for key types, which means that map is a pain. Where performance really matters, it’s easy to drop in something like LLVM’s dense map or tsl’s robin map, but only after profiling shows me that this is necessary.
-
Oh I agree, I don’t think I’ve ever needed an ordered map (outside of code puzzles/maybe interview style quizzes). The defined ordering in JS is specifically about removing any kind of non-determinism or leaked implementation semantics.
A few committee members would produce examples of problems that can be made “better” or “smaller” with insertion order enumeration[1], but for implementor committee members it was entirely about ensuring no hyrum’s rule horrors.
[1] Though those examples either have bad memory use problems or hurt the cpu/runtime performance of maps in all other cases
-
-
-
-
It’s so weird that there’s so much pressure to maintain 4k pages - it was worth the cost to Apple to take the compatibility issues from moving from 4k to 16k pages in hardware, I’m not sure why people are so hell bent on thinking that 4k is still the best option.
-
-
-
Has Apple managed this for macOS? I spoke to some of Apple’s CoreOS team about RISC-V having an 8 KiB default page size at ASPLOS a few years ago and their belief was that it would break large amounts of code. A lot of *NIX software assumes that you can
mprotecta 4 KiB granule. For iOS, they’ve generally been willing to break a load of things and require people to put in porting effort but less so for macOS.-
The 4k assumption is actually even worse than just an “assumption” even code that tried to do the right thing: using getpagesize() didn’t work as it was a macro that expanded to 4k on intel machines (at least on Darwin), which made rosetta challenging. The M-series SoCs support some kind of semi-4k page allocation in order to deal with this problem under rosetta.
For ARM builds on iOS have been 16k for many years preceding that, so a moderate amount of Mac software (which shared code with iOS) had clear source changes to do the right thing, and getpagesize() stopped being a macro so new builds of software got the correct thing.
-
-
-
-
I use Python on an almost daily basis professionally and it has enabled me and my peers to get up and running with software remarkably quickly, but it seems that with every bigger release of Python there is syntax added to the language and I’m honestly curious as to what the amount of outcry has to be for something like structural pattern matching to be added. It must be a non-trivial amount of work to add this and maintain it for years to come, so was the amount of requests to support this non-trivial as well?
Contrasting with a language like Go, which I’ve not used all that much, it seems like the maintainers deliberately eschew many niceties people have come to expect from Python just so the language as a whole is neater with fewer ways to do things. I’m not trying to shit on Python in any respect, I guess I just haven’t come into contact with reasons why any syntactical additions are still being made. To me, just because you can add new language constructs doesn’t mean you necessarily should; that’s what the underlying motivation and rationale PEP 635 essentially comes off as.
-
I think what happens is that many Python developers are polyglots, see “good ideas” from other languages, and try to think of ways to incorporate it into Python.
Structural pattern matching has always been a bit of a “magical feature from those other languages”, but I think that Rust and Scala really paved the way for proving the utility in very imperative code. And it works alright as a way for Python to finally get a
switchstatement.And to really go to bat for the Python developers… for every feature like this we get 5 or 10 things like “Now you can write
T | Vinstead oftyping.Union[T, V]”, ornew_dictionary = {**old_dictionary, "extra_key": 1}, ormerged_dictionary = dict_1 | dict_2.There’s lots of discourse around the “harder” feature additions, because so many of the other feature additions are, to be frank, no brainers. Just clear wins for everyone involved, things that do “what you expect”. Dictionaries preserving insertion order happened only fairly recently!
The sort of summit of expressivity has not been reached in any language in my opinion, and Python is, I think, a very positive contributor to this design space.
-
I just feel like this particular addition is
A) Not applicable in a lot of codebases
B) Not super-intuitive
Which is a bad combination. 99% of Python developers will never use this, so when they are looking at piece of code that does use it, they’re going to scratch their head and ask “wait, how does this work?”
for/elseused to be the poster-child for infrequently used, often confused Python features, and probably it is still the worst offender, but I don’t think the language needs more competitors in the space.
By comparison, f-strings (which I was skeptical of) are useful more or less everywhere, so everyone who isn’t a novice will have used them and know more or less how they work (minus understanding the edge cases around brackets inside an f-string).
Here’s a quiz. What does this do?
def f(*, arg1): pass f()This feature is sort of borderline to me, but I’m curious if it’s more or less obvious than I think.
-
For me pattern matching feels Pythonic, mostly because it’s already idiomatic to use destructuring in a bunch of other places, like in assignments and looping constructs. So it felt like an odd gap that you couldn’t use destructuring in conditionals (a previous comment of mine with some examples). But there are a bunch of edge cases in this new feature so I’m not 100% sure how much I’ll use it in practice.
-
Here’s a quiz. What does this do?
def f(*, arg1): pass f()This feature is sort of borderline to me, but I’m curious if it’s more or less obvious than I think.
This blows up because parameters after
*are keyword only. You have to pass, e.g.,f(arg1="foo"). (Full disclosure, I guessed the right answer, but I definitely had to check the docs to be sure.)I’m not sure if this is your point, but for me this is a great example of Python providing (way) too many knobs and dials.
-
I feel like if you showed ~any programmer the switch statement they would be able to say what’s going on. Fancy if statements that look like how you write out case analysis for function definitions in math.
There are definitely foot guns! But “non-intuitive” feels off to me. I think it would be interesting to do some real polling on Python programmers for “normal usage”. Like the text adventure seems straightforward to me, and way better than the “first we do a length check then destructure” flow
stuff like default arguments being a function property so being a shared reference or … almost every syntactic async thing has a lot higher “this doesn’t do what I thought it does” ratio IMO. And kwarg-only parameters are pretty useful! Subjective though!
-
I dunno, seems unclear to me when using a name declares a variable vs when it describes a pattern. But maybe I’m overthinking it.
-
to be honest I was also under the impresison that the binding rules were confusing (mostly from reading examples from people who were arguing it was confusing). But going through the PEP tutorial above all the code was straightforward for me and also expressed patterns for things that I find extremely awkward in Python.
I think the biggest gotcha is around constants, since a bare name is always a capture pattern, but dotted name access is allowed. tbh I feel like there should have just been an “Escape symbol” to resolve this question…. but I think that fortunately this is a very easy lint (one that’s present in … every pattern match-included language). It’s not perfect, but I think in practice things will work out through osmosis.
-
-
-
I genuinely did not know for/else was in the language. I’m kind of glad because I’ve used better-suited data structures and programming style because of it, but, wow.
-
-
-
I’ve been writing Python as my go-to language since late 90s and just love how the language has evolved and how the rollout of new features influenced my style and way of thinking. Decorators, generators, comprehensions, context and type annotations all had a major impact on the way I design and write code. I very much like Go’s simplicity, but I find that Python and it’s community has managed to strike a good balance between simplicity (but not minimalism) and progress and keep the language evolving. Ruby’s evolution, for instance, looks a bit sad in comparison. JavaScript, for one, was a surprising success, where the new features really made the language more flowing and expressive, in particular the arrow functions and restructuring.
-
-
This is, incidentally, a thing I dislike a lot about Rust stylistically. A lot of Rust is written chaining a bunch of map()s and or_else()s and such.
It’s passable if, even as in this article, it’s a straight transform. But it rarely is. There’s often side effects (either unintentional bad ones or good ones that’d make life simpler (eg, a max/min, without using tuples on tuples))… or implicit awaiting in async version of this chaining (how many things are happening in parallel? Hope you did the right buffering call!) and… it’s just a beast to debug. A for loop would be way more obvious in a lot of cases (if for no other reason than to crack open gdb)
-
In my experience there’s a lot of cases when writing Rust code where you have a simple transform and do want to write it using chained maps and or_else (or even more concisely with the
?operator). When what you’re doing isn’t actually a simple transform, it’s definitely useful to resort to a non-functional construct like a for-loop, but that’s ideally the heavier and less common thing to do.-
Why would a for loop be “heavier” than a chain of transforms?
If anything, the for loop is easier to optimize by the compiler, and easier to understand by the layperson, right?
-
I have no idea whether iterator chains or a for-loop is easier to optimize by the compiler - I’ve seen deep-dives into rustc compiler internals that have argued that iterator chains are actually faster at least sometimes, and I think the broader problem is that it’s difficult for a programmer to actually know which of several semantically-equivalent ways of writing a program will actually result in the most performant code.
-
-
This lines up with my Java intuition as well, although there are so many different Java styles that I don’t claim it’s a universal thing other Java programmers would agree with. If something is doing explicit
forloops to transform a collection into another collection, my assumption is either: 1) it’s legacy pre-Java-8 code, written beforejava.util.streamexisted, or 2) it’s doing complex or wonky enough logic that it doesn’t map nicely onto the standard operations, so really does need to drop down to a custom loop.
-
-
A lot of Rust is written chaining a bunch of map()s and or_else()s and such.
I used to do this a lot. The APIs are there. It’s so tempting when there’s a function like
map_or_elsethat looks like it was designed to solve your exact problem. But I think it’s a trap, and as you start writing it often becomes more complicated than anticipated.These days, I am more skeptical of the ‘functional’ style in Rust. I rely more on language features like
match,?, traits likeFrom/TryFrom, and libraries likeanyhoworserde. I couldn’t tell you why, but this is how I feel after using Rust for a couple years. -
I agree. Chaining in a single expression is usually less readable than a sequence of independent expressions.
-
Yeah, we write a lot of functional heavily chained code in D, but it’s viable (IMO) because all our value types are immutable. No state, pure functions only. There’s a few keywords that are allowed to do mutation (
each,sort), but we know to look out for them.
-
-
mjn
8 months ago
|
link
| on:
Introduction to the Theory of Programming Languages (1991) now freely available
Bertrand Meyer seems to be trying to organize and freely re-release his back catalog to the extent possible. A few weeks ago, the re-release of Object-Oriented Software Construction, 2nd edition (1997) had a small discussion here.
Great article. I’m curious to better understand the term “localized responsibility” — local to what?
Personally, I believe that enterprise users of open source components are taking on responsibility for that component and that the responsibility should include a biderectional relationship with the upstream including one or more of: filing issues, fixing issues, supporting development of the component, and otherwise contributing to the project’s ongoing viability.
Of course, moderns software utilises so many components that most companies can’t reasonably become active participants in all of the projects they use. However, I think we must shift from a position of blind consumption to a default willingness and desire to have a healthy relationship with dependencies.
Good question. I went around and around on this wording. I really wanted to write:
What I was trying to get at is that once you bring in a dependency, just because you didn’t write it you must now take ownership of it and all that that entails, almost as if your company did write it. I somewhat think of it as you being handed a bunch of legacy code which no-one on your team has read and being told “this is yours now”. Unfortunately, I couldn’t think of a particularly succinct way of describing this and I think the end wording was a little clumsy. Any ideas appreciated :)
I too am tired of this. I feel like it’s born out of the pressure from being classed as a “software engineer”. Software engineers are paid to ship things on time, where “on time” almost always means “less time than they want”.
What my thinking is angling towards is that if we were able to put a real title on the work (because it is work, dependencies aren’t free even though it’s easy to think they are) then we can have “Software Dependency Engineers” saying “I was upstreaming some stuff to Project X today, which we rely on quite heavily. It’ll take some time to filter down but it is worth it.” and not the more standard software engineering “I just locally patched it to get it going because upstreaming will take ages and I need it in the release cut in 2 weeks.” (I guess another implicit role a SDE would have is helping manage local patches so software engineers get what they need with their time pressure but the SDE is actively pushing the fix upstream at the same time).
The way you describe the job here reminds me a lot of what Linux kernel developers employed by companies do. Large companies seem to accept that at least half the job of their in-house kernel developers is not just to write stuff that improves things for Intel/Nvidia/AMD/whatever products, but to also make sure it gets upstreamed, to rewrite anything that gets rejected by upstream, etc., and that it’s worth paying people to do that. I guess this is more or less forced by how massive a pain it is maintain non-upstreamed kernel patches though.