This post made me wonder - is there are serious, production level software being written in array languages these days? APL was created in different times and I understand it did have some use. But outside of gimmicks and fun does anyone now work with K, J and others? (And not just to keep legacy apps alive) What would be the reasons to do that?
Yes. K is widely used in the trading and finance world. Dyalog APL is a modern commercial vendor with at least enough paying customers to support the company – I don’t know what the numbers are. And J has at least some customers, I get the feeling fewer but don’t know for sure. These languages are definitely not just gimmicks.
For q/kdb+, there’s a huge amount of momentum and a very specific niche. Nobody is writing huge applications in q (okay, some people are, but not many). Instead, for example, they’re writing very specialized programs that analyze a billion stock trades and try to find patterns. The language is designed to make exactly that kind of thing concise and fast.
In the case of APL/J I’m not sure what the story is today. One part of the mythology was that a small team of experienced APL/J developers could be more productive than a team 5-10x the size in a mainstream language, because you are operating at a much higher level of abstraction and writing much less code. I’m sure such teams did exist, but even as a huge fan of these languages I am very skeptical of the claim generalizing to developers in general. First, because these languages are much harder to learn fluently than mainstream languages. Second, I think it’s hard to separate out the “the language” effect from the effects of “you just had very smart group of people” and “everyone is highly fluent in a single language” and “everyone is passionate about the language.”
Nevertheless, that is one of the selling points, and that central value proposition is still reflected on Dyalog’s homepage:
Dyalog delivers an APL-based development environment that allows both subject matter experts and IT specialists to efficiently convert ideas into software solutions.
these languages are much harder to learn fluently than mainstream languages.
When I was a kid, a friend showed me some APL he was writing, and also the “game of life” example. So I got hooked, and ended up writing about a thousand lines of APL. APL was easy. Not much to it. Functional programming was harder for me to learn (as a kid): the idea of recursive functions was a tough nut for me to crack. And this was before the era of modern, hyper complex programming languages, like Rust and C++, which are many orders of magnitude more complicated than APL.
C++ has float vs int, unsigned vs signed, char, short, int, long, long long vs int8_t, int16_t, int32_t, int64_t, etc, with complex conversion rules, and with different laws of arithmetic for different number types. 7/2==3 but 7.0/2.0==3.5, and so on. APL just has numbers, that’s it.
C++ has data abstraction with classes, single inheritance vs multiple inheritance, private vs protected vs public inheritance, constructors, destructors, and on and on. APL has a fixed set of primitive data types with no way to define new ones. You learn those basic data types (which are very simple) and you are done.
C++ has 14 ways to initialize a variable, APL has one. C++ has something like 23 different levels of operator precedence, which few people are ever able to memorize, APL has maybe 2 levels (verbs vs adverbs).
As a professional programmer, I’ve written hundreds of thousands of lines of C++. I still have no idea what SFINAE means, and I don’t care. I just use google, online C++ references, and copy/paste snippets like everybody else. I gave up hope years ago of understanding C++ in the deep way that I used to understand APL, or even C.
The claim that APL is “much harder to learn” is absurd, if you just compare language complexity. So this claim needs a lot of caveats and qualifications. The word “fluently” is used as a qualification. Okay, what’s different about APL is that you use different idioms to write code than you do in mainstream languages. If you have already invested years of your life to become fluent in hypercomplex mainstream idioms, then you may feel that you have to start over from scratch to learn and use APL fluently, which is a different experience than switching from one mainstream language to another. Mainstream languages tend to be mostly the same, with minor differences in syntax, but they all mostly support the same idioms.
If you have already invested years of your life to become fluent in hypercomplex mainstream idioms, then you may feel that you have to start over from scratch to learn and use APL fluently, which is a different experience than switching from one mainstream language to another. Mainstream languages tend to be mostly the same, with minor differences in syntax, but they all mostly support the same idioms.
This is basically where I am coming from. I think if you took a young person with no prior experience and taught them APL or J in high school (which has been done) you won’t have a big problem.
But for an experienced programmer “array thinking” is a huge paradigm shift. Learning J, for me, was much harder than learning Haskell, for example. There is a much steeper “unlearning” process too.
You only have to learn about 50 primitives, but many of those primitives don’t map 1-1 to standard functions in other languages (eg, J’s copy verb # and the filtering idiom #~ based on it). And to be fluent, there are many combinations and idioms you have to learn as well, not to mention hooks and forks and longer trains.
Getting to the point where you can frame a problem as an array problem, and quickly write up an idiomatic solution in the same amount of time that you could do the same in Python, say – that’s what I’m claiming is going to take much longer.
In my very much newbie view, performance (automatic SIMD, although not automatic multi-core parallelism) and succintness. (There is really something magical about the core of your program taking eg. 8 characters instead of 200 lines. The function itself is frequently is shorter than the name you’d give it.)
I’d consider Julia to be a successor of APL. It is widely used in scientific computing and seems to be starting to displace Fortran for a bunch of things.
You didn’t seem to add hostdefs and netdefs. I also had to fix jmf and pcre a bit. I have it here, https://github.com/jxy/jsource/tree/freebsd-j9 currently behind upstream 1040 commits. Some tests fail, like x:2^_53 is incorrect, but otherwise it works fine.
This post made me wonder - is there are serious, production level software being written in array languages these days? APL was created in different times and I understand it did have some use. But outside of gimmicks and fun does anyone now work with K, J and others? (And not just to keep legacy apps alive) What would be the reasons to do that?
Yes. K is widely used in the trading and finance world. Dyalog APL is a modern commercial vendor with at least enough paying customers to support the company – I don’t know what the numbers are. And J has at least some customers, I get the feeling fewer but don’t know for sure. These languages are definitely not just gimmicks.
Do you know why they’re chosen over languages with more available experience / support / better integrated IDEs?
For q/kdb+, there’s a huge amount of momentum and a very specific niche. Nobody is writing huge applications in q (okay, some people are, but not many). Instead, for example, they’re writing very specialized programs that analyze a billion stock trades and try to find patterns. The language is designed to make exactly that kind of thing concise and fast.
What lorddimwit said for K. Also this page might help: https://kx.com/resources/use-cases/.
In the case of APL/J I’m not sure what the story is today. One part of the mythology was that a small team of experienced APL/J developers could be more productive than a team 5-10x the size in a mainstream language, because you are operating at a much higher level of abstraction and writing much less code. I’m sure such teams did exist, but even as a huge fan of these languages I am very skeptical of the claim generalizing to developers in general. First, because these languages are much harder to learn fluently than mainstream languages. Second, I think it’s hard to separate out the “the language” effect from the effects of “you just had very smart group of people” and “everyone is highly fluent in a single language” and “everyone is passionate about the language.”
Nevertheless, that is one of the selling points, and that central value proposition is still reflected on Dyalog’s homepage:
When I was a kid, a friend showed me some APL he was writing, and also the “game of life” example. So I got hooked, and ended up writing about a thousand lines of APL. APL was easy. Not much to it. Functional programming was harder for me to learn (as a kid): the idea of recursive functions was a tough nut for me to crack. And this was before the era of modern, hyper complex programming languages, like Rust and C++, which are many orders of magnitude more complicated than APL.
C++ has float vs int, unsigned vs signed, char, short, int, long, long long vs int8_t, int16_t, int32_t, int64_t, etc, with complex conversion rules, and with different laws of arithmetic for different number types. 7/2==3 but 7.0/2.0==3.5, and so on. APL just has numbers, that’s it.
C++ has data abstraction with classes, single inheritance vs multiple inheritance, private vs protected vs public inheritance, constructors, destructors, and on and on. APL has a fixed set of primitive data types with no way to define new ones. You learn those basic data types (which are very simple) and you are done.
C++ has 14 ways to initialize a variable, APL has one. C++ has something like 23 different levels of operator precedence, which few people are ever able to memorize, APL has maybe 2 levels (verbs vs adverbs).
As a professional programmer, I’ve written hundreds of thousands of lines of C++. I still have no idea what SFINAE means, and I don’t care. I just use google, online C++ references, and copy/paste snippets like everybody else. I gave up hope years ago of understanding C++ in the deep way that I used to understand APL, or even C.
The claim that APL is “much harder to learn” is absurd, if you just compare language complexity. So this claim needs a lot of caveats and qualifications. The word “fluently” is used as a qualification. Okay, what’s different about APL is that you use different idioms to write code than you do in mainstream languages. If you have already invested years of your life to become fluent in hypercomplex mainstream idioms, then you may feel that you have to start over from scratch to learn and use APL fluently, which is a different experience than switching from one mainstream language to another. Mainstream languages tend to be mostly the same, with minor differences in syntax, but they all mostly support the same idioms.
Yes, C++ is a much bigger language.
This is basically where I am coming from. I think if you took a young person with no prior experience and taught them APL or J in high school (which has been done) you won’t have a big problem.
But for an experienced programmer “array thinking” is a huge paradigm shift. Learning J, for me, was much harder than learning Haskell, for example. There is a much steeper “unlearning” process too.
You only have to learn about 50 primitives, but many of those primitives don’t map 1-1 to standard functions in other languages (eg, J’s copy verb
#and the filtering idiom#~based on it). And to be fluent, there are many combinations and idioms you have to learn as well, not to mention hooks and forks and longer trains.Getting to the point where you can frame a problem as an array problem, and quickly write up an idiomatic solution in the same amount of time that you could do the same in Python, say – that’s what I’m claiming is going to take much longer.
control structures
(line) separators
guards
copulae
verbs
adverbs
stranding
parentheses, brackets, braces
And all the demons of regular precedence show up again in numeric literals. For instance: 1e¯2j3 is (1e(¯2))j3; nars and j have as many as 4 levels.
Still orders of magnitude simpler than most other languages, though.
Footnote: adverbs are accompanied by conjunctions (or, to use the traditional vocabulary, operators may be monadic or dyadic).
In my very much newbie view, performance (automatic SIMD, although not automatic multi-core parallelism) and succintness. (There is really something magical about the core of your program taking eg. 8 characters instead of 200 lines. The function itself is frequently is shorter than the name you’d give it.)
There is a list of companies using those language here.
I’d consider Julia to be a successor of APL. It is widely used in scientific computing and seems to be starting to displace Fortran for a bunch of things.
May also find my build scripts (written for freebsd, but probably compatible) of interest.
You didn’t seem to add hostdefs and netdefs. I also had to fix jmf and pcre a bit. I have it here, https://github.com/jxy/jsource/tree/freebsd-j9 currently behind upstream 1040 commits. Some tests fail, like
x:2^_53is incorrect, but otherwise it works fine.