My first job out of college was with a company that heavily used k4 (although it might have been k3, it’s been a while). I’ve forgotten basically all of it, but a few impressions/fun facts remain:
It’s a nice language! Definitely weird, and focused on a particular class of problems, but the core of the language is small enough to fit in your head. Once you get past the line-noise aspect it has a certain logic to it.
k has many idiosyncrasies. I found that everything was fine “inside” the k-world , but your brain started to hurt when interfacing or thinking between k and the outside world. As an example, the k date epoch is in the future, and time is currently negative.
If you look at some of Arthur Whitney’s code, you’ll see filenames like a.c, b.c, etc. Legend has it that Arthur hates scrolling. He starts with a.c and whenever the screen fills up he splits it off into b.c and so on.
The k interpreter doesn’t rely on too many tricks. It gets its speed via small, hand-tuned array routines. A lot of k’s weirdness is rooted in bending the problem space to make the small, hand-tuned array routines happy.
k programmers’ insistence on code-golf isn’t completely baseless. Most k verbs and adverbs are one or two characters long, and roughly affect execution time within the same order of magnitude. This means that a shorter k program usually is faster.
The K binary weighs in at about 50Kb. Someone asked about the interpreter source code. A frown flickered across the face of our visitor from Microsoft: what could be interesting about that? “The source is currently 264 lines of C,” said Arthur. I thought I heard a sotto voce “that’s not possible.” Arthur showed us how he had arranged his source code in five files so that he could edit any one of them without scrolling. “Hate scrolling,” he mumbled.
Unlike OOP languages, Iversonian array languages do not need a tracing garbage collector. Reference counting is normally used for automatic storage reclamation.
My recollection is that Arthur was worried about the range of representable dates, and shifted the epoch forward to sacrifice less-useful dates in the past to be able to represent more-useful dates in the future. However, that doesn’t square with my memory of the internal date format, as a 65 year difference doesn’t change very much. I’ve updated my previous comment about 4 times, trying to track down the actual internal data format, and have come up empty handed. Would be grateful if anyone had any more information!
J is a great tragedy to me; Iverson dedicated his life to more elegant, compact tools of thought until he betrayed his life’s work…
K, on the other hand, carries no such baggage! Unfortunately, symbol overloading is quite excessive (as already APL was in a few notable cases, like geometry, due to technical limitations at the time.) The example from Wikipedia sees index, modulo, rotate (⍳, |, ⌽ in APL) all rendered with an exclamation mark: 2!!7!4! Nevertheless, true higher level functions (er, operations) are truly liberating and eye opening, does it matter what symbol’s they’re rendered as?
Am I just shallow? I’ve spilled enough ink on the importance of semantics while (minor differences in such expressive) syntax are mere aliases (though I too feel pangs of woe at every new language without s-expressions). Yet, there’s something missing… Anyway, “the killer app” is the 7 figure salary. (N.b. J’s the only one I’ve used in production.)
I’m rather curious about the various companies selling these products. Dyalog has been flourishing of late, hired more people and e.g. has a sales event in NY on April 7th to learn about earn more about “migrating from other APL platforms”. Wikipedia states, KX systems has 14 “offices” (but First Derivatives bought a controlling stake for $40mm)… Shakti has some high priced office space, who and how are the other players doing?
J is a powerful upgrade to pre-J APL, and introduced many valuable ideas that have since been taken up by other array languages, such as Dyalog APL. I’m not a J expert, but AFAIK this would include trains, leading axis theory, the rank operator, and apparently there are many new powerful operators that first appeared in J. J threw out backward compatibility with legacy APL and redesigned APL to make it more consistent and powerful.
On the downside, I find that the syntax of J is so hideous that I have no plans to learn or use it. Of course not everybody feels this way. I’ve mostly used APL and K, but BQN looks like it is worth investigating.
Innovation has also occurred outside of the J language. The “direct function” (dfn) syntax of Dyalog APL was game changing, and I wouldn’t use an APL without it. K and BQN have it, J doesn’t. (Another reason I won’t use J.)
Thank you! This wasn’t in the J language the last time I looked. Also, the APL wiki hasn’t been updated to state that J has the feature, and that’s a resource I’ve been relying on. (https://aplwiki.com/wiki/Dfn)
Dyalog direct functions have a rich feature set, which support a powerful set of idioms known from functional programming: https://www.dyalog.com/uploads/documents/Papers/dfns.pdf. (Error handling has since been added.) These idioms allow you to write terse code without boilerplate. It looks like J falls short.
Dfns are lexically scoped. If dfns are nested, then the inner dfn can capture local bindings from the outer dfn. I’m not sure if J works this way.
You can specify a default value for the left argument (α) by assigning to it. As a special case, this assignment has no effect if a left argument was given in the function call. I don’t see that in J.
A dfn may contain guards, which terminate function evaluation early if the guard condition is true. I don’t see that in the J documentation.
In Dyalog, the ∇ symbol is bound to the smallest enclosing dfn, and is used for specifying recursive functions when the dfn is anonymous (not bound to a name). Also, tail calls do not grow the stack, so you can use recursion to express iterative algorithms without blowing up the stack. In the functional programming literature, this feature is called “proper tail calls”, but some people call it “tail call optimization”. I don’t see any of this in the J documentation.
dfns are like kinda dynamically scoped idk scoping is super weird and fucked up in dyalog. i’d like straight up proper lexical closures but it doesn’t have them. (or maybe it was like. downwards but not upwards funargs?) do concede it is nicer to use in some cases though. (i had a nice proposal for lexical closures in j but henry didn’t like it and didn’t see the need :c if i ever come back to apl i’ll probably do my own j-ish thing w/ glyphs closer to apl syntax)
j $: is dyalog ∇
j was classically more focused on tacit code, so where it falls short for explicit code sometimes, it has better tacit facilities than dyalog. e.g. @. instead of guards for branching, :: for error handling, : for ambivalent verbs (a&$: : v is a straightforward idiom for a verb with a default left argument of a). these can still of course be used inside of tacit verbs (and vice versa)
J is the betrayal. Their life’s work was notation as a tool for thought. J abandons ‘notation’, ie. using custom glyphs, in hopes of making adoption easier.
I don’t think this is why @veqq thinks J is a “betrayal”, because K also is ASCII-only.
I also don’t think that’s a “betrayal”, because notational is only a tiny part of “tools of thought”, and custom glyphs is only a tiny part of notation. J’s &. operator, where f&.g x is equivalent to g⁻¹ f g x, was eye-opening to me. I don’t think J loses anything by using &. over a glyph like ⊕. A example of a non-notational APL “tool of thought” is thinking of filters as bitmasks and sorting as graded permutations. I don’t think it matters what APL you use, you still get that idea.
The more interesting question to me is “does J contribute any new tools of thought?” I can’t answer this because I only know J and a tiny bit of uiua, so I don’t know is not present in APL or K. I couldn’t find an equivalent to &. on the dyalog wiki page or in the rosetta, but I might just not know where to find it.
I think you took my explaination as a dig at J and replied to defend J, with points that although I mostly agree with have no relation with the at hand, which is about Iverson.
notational is only a tiny part of “tools of thought”
It is not. Iverson’s Turing award Lecture is titled: “Notation as a tool of thought”. It was about the importance of notation, not about “tools of thought”. Remeber that when Iverson published “A Programming Language” there wasnt any implementation of APL. The book is about using the notation to think about algorithms.
And it is not only Iverson that thought notation was important, which is why bqn uses its own notation or why Dyalog still introduces glyphs.
It is that notation that led people to useful ideas such as
filters as bitmasks and sorting as graded permutations
Yes, I think those things are useful on its own. Even outside of array languages.
Regarding the question about whether J improved on APL, my understanding is that it did and some of the improvements were adopted by Dyalog, but not all. I don’t know J so I dont know specifics.
I still don’t agree that the Iversonian notation needs to be in the form of custom glyphs. His opening example in the lecture is +/⍳5 for “sum 1 to 5”. ⍳ is new notation that’s also a glyph, but +/ for “reduce plus” is also new notation that is not a new glyph. Similarly, f&.g is new notation, but it doesn’t use new glyphs. And one of Iverson’s biggest impacts on broader mathematics, the iverson bracket, is just written [P].
To make my own preference explicit, I’ve coded maybe 40 or 50 custom glyphs into my keyboard, because I much prefer writing ∀x: □◇P to “forall x: necessarily possibly P”. But I often wonder if we’re conflating “terse, compact, powerful notation” with the specifics of Iverson’s initial choices: 1/2-arity glyphs. I like how J and K tried to explore the former without committing to the latte, and I wouldn’t consider doing so a “betrayal”.
Iverson’s Wiki page is a mess (I’ve never seen a page as disorganized as that without prominent banners), but if it is to be believed, here’s his statement on the creation of J:
When I retired from paid employment, I turned my attention back to this matter [the use of APL for teaching] and soon concluded that the essential tool required was a dialect of APL that:
Is available as “shareware”, and is inexpensive enough to be acquired by students as well as by schools
Can be printed on standard printers
Runs on a wide variety of computers
Provides the simplicity and the generality of the latest thinking in APL
The result has been J, first reported in [the APL 90 Conference Proceedings].
You can call that a betrayal, I see it as an admirable attempt in letting more people know about the power of APL.
Note that Iversen himself never finished high school as a teenager. He graduated after self-studying in the Air Force and later got a university education via Canada’s version of the GI Bill.
Of course, APL elitism has a long history. From the same source:
In one school the students became so eager that they broke into the school after hours to get more APL computer time; in another the APL enthusiasts steered newbies to BASIC so as to maximize their own APL time.
You can call that a betrayal, I see it as an admirable attempt in letting more people know about the power of APL.
Note I didnt call it a betrayl. I was merely explaining what the OP meant.
I do think J is giving up on Iverson’s life work. It did for good reasons. I acknowleged the motivation for doing so. And at the time the decision was made there it made sense. FWIU in the 80s you needed a custom terminal to be able to input APL glyphs in the computer. And it was priced at ~1000 USD iirc. In the 90s one could buy custom keyboards that would emit the keycodes for APL glyphs. Not as high a barrier but a significant one.
We now know that it didnt increase the adopting of J, but it was a good hypothesis.
Nowadays, at least on Linux, custom input methods as well as custom keymaps are well supported so Custom Glyphs don’t pose the same challlenges.
So I do agree to some extent with the OP abour being a tragedy.
[APL’s] most important use remains to be exploited: as a simple, precise, executable notation for the teaching of a wide range of subjects - Iverson
Not originally a programming language, APL was a more heroic intellectual venture; Iverson and friends published books on accounting and math, onboarded middle schools to use their curricula etc. They sought to reform mathematics itself! To @gerikson , while Iverson kept education in view, programming overtook the rest of the project with K.
It’s a veritable Graecian tragedy, a modern rendition’s recurring right now with: https://kdb.ai/
This is half tongue in cheek. J has cool features. Iverson sketched an APL 2, whose ideas became J and slowly made their way into APL. @hwayne ’s &. is ⍢, added in the 80s. But it is tragic too, making such a break with the loftier project and splitting the family.
How did you get to see Shakti’s source, given that you haven’t used K in production (ie, you weren’t a Shakti employee)?
The closest I’ve got to the Shakti source is https://github.com/kparc/ksimple, which was published by Arthur Whitney last year for educational purposes. I assume the real Shakti K interpreter is written in C in this style, but is significantly larger, since there’s a lot more functionality.
I read the whole thing, and I still don’t know when or why I would reach for k, or any of its friends. I’d love to see some task oriented documentation.
Not really. I’m still at the stage where I don’t understand and haven’t seen any tasks where this is the right tool, unlike say jq, awk, perl, sed, grep, and the like.
Terseness works well until it doesn’t, and that’s not something unique to apl-style languages.
It’s a steep curve, and I’m down at the bottom of it myself, but I have put together enough ad-hoc data plumbing in my career that I can definitely see the appeal of something so minimal and optimized. I don’t think of these APL languages as competitors to the text-oriented unix shell tools you mention, but rather to numeric-intensive ecosystems like numpy + pandas / polars or Julia. They’re obviously not for everybody, though.
Not that I’m aware of, but I wouldn’t be too surprised. All those proprietary data pipelines are supposedly incentivized to be performant, and it’s possible that a thrifty one, say DeepSeek, which is a hedge fund project, was loaned some experts from the fintech side of the business.
On the other hand, they don’t teach this stuff in data science school, it doesn’t play especially well with other technologies, and I get the impression that the big AI pipelines are mostly build out of investor cash soaked in kerosene.
I’m not saying they’re alternatives to text processing tools. I’m saying that they’re in the same category of very terse tools that presumably sit in a very narrow niche.
Cool site!
My first job out of college was with a company that heavily used
k4(although it might have beenk3, it’s been a while). I’ve forgotten basically all of it, but a few impressions/fun facts remain:khas many idiosyncrasies. I found that everything was fine “inside” thek-world , but your brain started to hurt when interfacing or thinking betweenkand the outside world. As an example, thekdate epoch is in the future, and time is currently negative.a.c,b.c, etc. Legend has it that Arthur hates scrolling. He starts witha.cand whenever the screen fills up he splits it off intob.cand so on.kinterpreter doesn’t rely on too many tricks. It gets its speed via small, hand-tuned array routines. A lot ofk’s weirdness is rooted in bending the problem space to make the small, hand-tuned array routines happy.kprogrammers’ insistence on code-golf isn’t completely baseless. Mostkverbs and adverbs are one or two characters long, and roughly affect execution time within the same order of magnitude. This means that a shorterkprogram usually is faster.Finally, the obligatory ACM interview: https://queue.acm.org/detail.cfm?id=1531242
Thank you, I had only heard about the scrolling thing from someone secondhand and am glad to see it in print!
Source seems to be https://archive.vector.org.uk/art10500700.
Great!
Unlike OOP languages, Iversonian array languages do not need a tracing garbage collector. Reference counting is normally used for automatic storage reclamation.
Is there anything I can read about why K has the epoch in 2035?
My recollection is that Arthur was worried about the range of representable dates, and shifted the epoch forward to sacrifice less-useful dates in the past to be able to represent more-useful dates in the future. However, that doesn’t square with my memory of the internal date format, as a 65 year difference doesn’t change very much. I’ve updated my previous comment about 4 times, trying to track down the actual internal data format, and have come up empty handed. Would be grateful if anyone had any more information!
Maybe they want to try out an interestingly different rollover before NTP in 2036 and unix in 2038
Maybe it’s a hack to make incorrect interest rate calculations look more obvious
Unless the epoch is based on TAI (i.e. immune to leap seconds) it sounds like a very eccentric idea.
J is a great tragedy to me; Iverson dedicated his life to more elegant, compact tools of thought until he betrayed his life’s work…
K, on the other hand, carries no such baggage! Unfortunately, symbol overloading is quite excessive (as already APL was in a few notable cases, like geometry, due to technical limitations at the time.) The example from Wikipedia sees index, modulo, rotate (⍳, |, ⌽ in APL) all rendered with an exclamation mark:
2!!7!4! Nevertheless, true higher level functions (er, operations) are truly liberating and eye opening, does it matter what symbol’s they’re rendered as?Am I just shallow? I’ve spilled enough ink on the importance of semantics while (minor differences in such expressive) syntax are mere aliases (though I too feel pangs of woe at every new language without s-expressions). Yet, there’s something missing… Anyway, “the killer app” is the 7 figure salary. (N.b. J’s the only one I’ve used in production.)
I’m rather curious about the various companies selling these products. Dyalog has been flourishing of late, hired more people and e.g. has a sales event in NY on April 7th to learn about earn more about “migrating from other APL platforms”. Wikipedia states, KX systems has 14 “offices” (but First Derivatives bought a controlling stake for $40mm)… Shakti has some high priced office space, who and how are the other players doing?
N.b. view Shakti.com’s source, it’s wildly gorgeous!
Mind if I ask for the story here? I read through his bio on Wikipedia but didn’t find anything that seems like this.
There are good things and bad things about J.
J is a powerful upgrade to pre-J APL, and introduced many valuable ideas that have since been taken up by other array languages, such as Dyalog APL. I’m not a J expert, but AFAIK this would include trains, leading axis theory, the rank operator, and apparently there are many new powerful operators that first appeared in J. J threw out backward compatibility with legacy APL and redesigned APL to make it more consistent and powerful.
On the downside, I find that the syntax of J is so hideous that I have no plans to learn or use it. Of course not everybody feels this way. I’ve mostly used APL and K, but BQN looks like it is worth investigating.
Innovation has also occurred outside of the J language. The “direct function” (dfn) syntax of Dyalog APL was game changing, and I wouldn’t use an APL without it. K and BQN have it, J doesn’t. (Another reason I won’t use J.)
Could you explain how dfn differs from J direct definitions? In J you can write
1 {{ x + y }} 2, though this was only added circa 2020.Thank you! This wasn’t in the J language the last time I looked. Also, the APL wiki hasn’t been updated to state that J has the feature, and that’s a resource I’ve been relying on. (https://aplwiki.com/wiki/Dfn)
Dyalog direct functions have a rich feature set, which support a powerful set of idioms known from functional programming: https://www.dyalog.com/uploads/documents/Papers/dfns.pdf. (Error handling has since been added.) These idioms allow you to write terse code without boilerplate. It looks like J falls short.
Dfns are lexically scoped. If dfns are nested, then the inner dfn can capture local bindings from the outer dfn. I’m not sure if J works this way.
You can specify a default value for the left argument (α) by assigning to it. As a special case, this assignment has no effect if a left argument was given in the function call. I don’t see that in J.
A dfn may contain guards, which terminate function evaluation early if the guard condition is true. I don’t see that in the J documentation.
In Dyalog, the ∇ symbol is bound to the smallest enclosing dfn, and is used for specifying recursive functions when the dfn is anonymous (not bound to a name). Also, tail calls do not grow the stack, so you can use recursion to express iterative algorithms without blowing up the stack. In the functional programming literature, this feature is called “proper tail calls”, but some people call it “tail call optimization”. I don’t see any of this in the J documentation.
dfns are like kinda dynamically scoped idk scoping is super weird and fucked up in dyalog. i’d like straight up proper lexical closures but it doesn’t have them. (or maybe it was like. downwards but not upwards funargs?) do concede it is nicer to use in some cases though. (i had a nice proposal for lexical closures in j but henry didn’t like it and didn’t see the need :c if i ever come back to apl i’ll probably do my own j-ish thing w/ glyphs closer to apl syntax)
j $: is dyalog ∇
j was classically more focused on tacit code, so where it falls short for explicit code sometimes, it has better tacit facilities than dyalog. e.g. @. instead of guards for branching, :: for error handling, : for ambivalent verbs (a&$: : v is a straightforward idiom for a verb with a default left argument of a). these can still of course be used inside of tacit verbs (and vice versa)
J is the betrayal. Their life’s work was notation as a tool for thought. J abandons ‘notation’, ie. using custom glyphs, in hopes of making adoption easier.
I don’t think this is why @veqq thinks J is a “betrayal”, because K also is ASCII-only.
I also don’t think that’s a “betrayal”, because notational is only a tiny part of “tools of thought”, and custom glyphs is only a tiny part of notation. J’s
&.operator, wheref&.g xis equivalent tog⁻¹ f g x, was eye-opening to me. I don’t think J loses anything by using&.over a glyph like ⊕. A example of a non-notational APL “tool of thought” is thinking of filters as bitmasks and sorting as graded permutations. I don’t think it matters what APL you use, you still get that idea.The more interesting question to me is “does J contribute any new tools of thought?” I can’t answer this because I only know J and a tiny bit of uiua, so I don’t know is not present in APL or K. I couldn’t find an equivalent to
&.on the dyalog wiki page or in the rosetta, but I might just not know where to find it.But you see, Iverson didn’t work on K!
I think you took my explaination as a dig at J and replied to defend J, with points that although I mostly agree with have no relation with the at hand, which is about Iverson.
It is not. Iverson’s Turing award Lecture is titled: “Notation as a tool of thought”. It was about the importance of notation, not about “tools of thought”. Remeber that when Iverson published “A Programming Language” there wasnt any implementation of APL. The book is about using the notation to think about algorithms.
And it is not only Iverson that thought notation was important, which is why bqn uses its own notation or why Dyalog still introduces glyphs.
It is that notation that led people to useful ideas such as
Yes, I think those things are useful on its own. Even outside of array languages.
Regarding the question about whether J improved on APL, my understanding is that it did and some of the improvements were adopted by Dyalog, but not all. I don’t know J so I dont know specifics.
I still don’t agree that the Iversonian notation needs to be in the form of custom glyphs. His opening example in the lecture is
+/⍳5for “sum 1 to 5”. ⍳ is new notation that’s also a glyph, but+/for “reduce plus” is also new notation that is not a new glyph. Similarly,f&.gis new notation, but it doesn’t use new glyphs. And one of Iverson’s biggest impacts on broader mathematics, the iverson bracket, is just written[P].To make my own preference explicit, I’ve coded maybe 40 or 50 custom glyphs into my keyboard, because I much prefer writing
∀x: □◇Pto “forall x: necessarily possibly P”. But I often wonder if we’re conflating “terse, compact, powerful notation” with the specifics of Iverson’s initial choices: 1/2-arity glyphs. I like how J and K tried to explore the former without committing to the latte, and I wouldn’t consider doing so a “betrayal”.Iverson’s Wiki page is a mess (I’ve never seen a page as disorganized as that without prominent banners), but if it is to be believed, here’s his statement on the creation of J:
When I retired from paid employment, I turned my attention back to this matter [the use of APL for teaching] and soon concluded that the essential tool required was a dialect of APL that:
The result has been J, first reported in [the APL 90 Conference Proceedings].
You can call that a betrayal, I see it as an admirable attempt in letting more people know about the power of APL.
Note that Iversen himself never finished high school as a teenager. He graduated after self-studying in the Air Force and later got a university education via Canada’s version of the GI Bill.
Of course, APL elitism has a long history. From the same source:
Note I didnt call it a betrayl. I was merely explaining what the OP meant.
I do think J is giving up on Iverson’s life work. It did for good reasons. I acknowleged the motivation for doing so. And at the time the decision was made there it made sense. FWIU in the 80s you needed a custom terminal to be able to input APL glyphs in the computer. And it was priced at ~1000 USD iirc. In the 90s one could buy custom keyboards that would emit the keycodes for APL glyphs. Not as high a barrier but a significant one.
We now know that it didnt increase the adopting of J, but it was a good hypothesis.
Nowadays, at least on Linux, custom input methods as well as custom keymaps are well supported so Custom Glyphs don’t pose the same challlenges.
So I do agree to some extent with the OP abour being a tragedy.
Not originally a programming language, APL was a more heroic intellectual venture; Iverson and friends published books on accounting and math, onboarded middle schools to use their curricula etc. They sought to reform mathematics itself! To @gerikson , while Iverson kept education in view, programming overtook the rest of the project with K.
It’s a veritable Graecian tragedy, a modern rendition’s recurring right now with: https://kdb.ai/
This is half tongue in cheek. J has cool features. Iverson sketched an APL 2, whose ideas became J and slowly made their way into APL. @hwayne ’s
&.is⍢, added in the 80s. But it is tragic too, making such a break with the loftier project and splitting the family.Interesting perspective: close one eye and see a future where APL is taught from grade school to everyone.
Close another, and see New Math.
How did you get to see Shakti’s source, given that you haven’t used K in production (ie, you weren’t a Shakti employee)?
The closest I’ve got to the Shakti source is https://github.com/kparc/ksimple, which was published by Arthur Whitney last year for educational purposes. I assume the real Shakti K interpreter is written in C in this style, but is significantly larger, since there’s a lot more functionality.
I meant shakti.com’s source in light of the currently popular “raw text” blog.
[Comment removed by author]
You may appreciate BQN as a noncommercial alternative.
Another nice tutorial: https://razetime.github.io/ngn-k-tutorial/
It seems that ngn/k is no longer actively maintained, but there is a fork https://codeberg.org/growler/k
I read the whole thing, and I still don’t know when or why I would reach for k, or any of its friends. I’d love to see some task oriented documentation.
Maybe view-source:https://shakti.com/ will sell you on it? (Thanks @veqq)
It’s inscrutable
then I think this one’s for you: https://lobste.rs/s/6rzmjy/stages_denial
Not really. I’m still at the stage where I don’t understand and haven’t seen any tasks where this is the right tool, unlike say jq, awk, perl, sed, grep, and the like.
Terseness works well until it doesn’t, and that’s not something unique to apl-style languages.
It’s a steep curve, and I’m down at the bottom of it myself, but I have put together enough ad-hoc data plumbing in my career that I can definitely see the appeal of something so minimal and optimized. I don’t think of these APL languages as competitors to the text-oriented unix shell tools you mention, but rather to numeric-intensive ecosystems like numpy + pandas / polars or Julia. They’re obviously not for everybody, though.
Is J/K/APL used in any great extent in developing modern LLMs?
Not that I’m aware of, but I wouldn’t be too surprised. All those proprietary data pipelines are supposedly incentivized to be performant, and it’s possible that a thrifty one, say DeepSeek, which is a hedge fund project, was loaned some experts from the fintech side of the business.
On the other hand, they don’t teach this stuff in data science school, it doesn’t play especially well with other technologies, and I get the impression that the big AI pipelines are mostly build out of investor cash soaked in kerosene.
I’m not saying they’re alternatives to text processing tools. I’m saying that they’re in the same category of very terse tools that presumably sit in a very narrow niche.