1. 33
  1. 8

    Made it here at last! I made BQN, and I’m happy to answer any questions you have for me. You can also file issues through Github or write to me at the address in my Github profile, and I’m active on the APL Orchard. See also the previous HN thread.

    1. 6

      Oh heck yes, always pumped to see language innovations in array programming. And it looks like it’s the same person who did I, too!

      1. 5

        If you’re not an array programmer, it would probably be better to start with another language, or wait a few weeks.

        I wait with bated breath :)

        1. 2

          Same here! I keep bouncing off of APL, BQN looks like something I could finally stick to!

          1. 3

            I’ll work harder to keep my few-weeks promise now I know people are waiting for me! If you’d like something to start on now, you could take a look at my LambdaConf talk which introduces array programming for a functionally-inclined audience. BQN gets rid of the syntactic irregularity in the main subject, since it changes Outer Product from a prefix ∘. to a suffix , in addition to renaming it “Table”. Of the BQN documentation, I’d say Syntax, Blocks, and Functional programming are mostly readable without APL experience. You should also be able to understand the central ideas in the documentation for primitives, although many of the examples will just be too much for a beginner to pick apart.

        2. 3

          for example, to reverse each list in a list of lists, the programmer can use reverse under open, |. &. >. But to find the lengths of each of these lists, # &. > would yield a boxed list, which is usually not wanted, so # @ > is needed instead. BQN shows that a system that doesn’t require these distinctions is possible, as a BQN programmer would use ⌽¨ and ≠¨.

          This is really interesting and puts a fine point on some subtle inconsistencies in J that have felt not quite right to me, without my knowing exactly why.

          1. 1

            J seems more consistent to me. VERB&.> keeps the boxes, while VERB@> breaks the boxes, no matter what VERB is.

            1. 2

              As context, I love J and use it daily.

              I think the implication of the quoted piece is that the decision between &.> and @> could be inferred from the verb being used, so why force the programmer to make it manually? At the very least, the default behavior could be the one you want 95% of the time, with the ability to manually override (eg, if you did want a boxed list of lengths for some reason). Whereas in J the programmer must always make the decision manually.

              1. 2

                Quite a J-centric answer. Of course, J is a system that works and the most I can argue is that BQN is generally an improvement, but the based array answer is: what’s a box? Why do you need to add this concept to the language just so arrays make sense? In BQN terms, there’s an array with some lists in it and I want to apply a function to each of them. But if my function always returns numbers, then I have to remember this and adjust my code in order to get a numeric array instead of a boxed array. The boxes are in the way. I would say that both systems are perfectly consistent but J forces me (as a primarily J programmer from 2009 to 2020) to spend a lot more effort keeping track of details of the array model.

            2. 1

              Very exciting to me. I’ve been APL/J curious for a long time, done some in J. I vacillate between liking and disliking the APL operators; in some ways J is a lot more consistent than APL, but Dyalog’s APL dialect seems to borrow significantly from J (trains and forks and whatnot for instance). Main problem for me is I never really formally learned linear algebra and definitely do not immediately jump to matrixish solutions to problems, even though I think they often exist and leveraging that is probably key to being effective in array languages.

              1. 3

                A thing that really helped me was internalizing the correspondence between loops in imperative code and maps over ⍳ (the index generator function)

                (edit: clarity)

                1. 2

                  For what it’s worth, I don’t linear algebra will teach you to see array-oriented solutions to programming problems, only matrix solutions to linear algebra problems. BQN doesn’t even have a matrix product operator, so it takes the relatively long-winded +˝∘×⎉1‿∞⁼ to multiply matrices. On the other hand, there’s definitely a mindset of “here is my list/string/table, how can I act on the whole thing to get what I want” that you should use, and I don’t know what the best way to work yourself into it is.

                  1. 1

                    My experience with RDBMSes and the ORM impedance mismatch is that a lot of problems have whole-set solutions that are hard to get at with conventional imperative languages or even functional languages because they steer you into working item-by-item. This is one of the things that draws me to your language and other APLs. Thanks for replying!

                2. 1

                  @mlochbaum: I would like to understand “Based Array Theory”. Why is there a distinction between atoms and “scalars” (aka boxed atoms)? The array language I am most familiar with is K, which does not require this distinction, and K seems simpler to me. What is the downside of not having this distinction?

                  1. 4

                    I don’t mention it much in my docs, but K’s model is definitely simpler and this is not a bad thing (in contrast to the existing APL models which I do think are weaker than based arrays). I call K’s model the nested list model, meaning that it only natively has a one-dimensional array structure, and it represents higher-dimensional arrays by nesting them. If you have a homogeneous structure in K (meaning that for any list in it, every element of the list has the same nested structure; also, I’m ignoring dictionaries), then its “shape” is represented as a list of lengths: the length of the outer list, then the shared length of each of its elements, and so on. In BQN, the most basic structure is itself a multidimensional array, and a homogeneous structure’s “shape” is a list of lists, or equivalently a partitioned list. The outer list gives the nesting structure, and its length is the depth, while the inner lists give the shape at each level, and their lists are array ranks. The BQN array model is richer than K’s—this is not positive or negative, and in many cases K’s model is just right. In fact an array language compiler is one of these cases, as BQN’s compiler almost never uses arrays with rank other than 1.

                    On the other hand, in situations that call for multidimensional arrays, having “real” ones is pretty handy. In K, there’s no difference between a table (rank-2 array) of tables and a rank-4 array. If you want to work with it as a table of tables you’ll have to write your code for these specific ranks. In BQN, the language can keep track of this information for you. If you map over it with Each (¨), you iterate over the inner tables where K would require two 's, and more importantly it requires a different number of them depending on whether you have a list, table, or rank-3 array of tables. Join is another example of a primitive that takes advantage of multidimensional arrays, matching inner shapes to outer shapes.

                    Rank-0 arrays are just a consequence of allowing any array rank. They are needed for consistency, basically. So they’re not specifically useful—of course, the only difference between a scalar and its single element is that tiny bit of metadata—but they are part of a useful framework.

                    I wouldn’t propose to improve on K as a whole, but I think BQN’s fundamentals are better than other multidimensional array languages like APL or J. The reason I’m interested in designing with multidimensional arrays isn’t really because I think they are always the best way to program but because while the nested list model is mainstream and active (programmers doing maps and reductions in Javascript is common now!), APL-style arrays haven’t seen much theoretical development since the early days of J in the 1990s even as array programming becomes more popular with tools like NumPy and Julia.

                    1. 1

                      If I was creating a new array language, with true multi-dimensional arrays (not like K or Mathematica), then I would allow arrays with elements of mixed type. For example, you could create a 2x3 matrix where one element is a number, another element is a function, another element is a vector, another element is a matrix. This matrix would have shape 2 3 and rank 2. Numbers, functions and other scalars would have rank 0. There would be no boxes and no enclose function. In your criticism of J, you say “What’s a box? Why do you need to add this concept to the language just so arrays make sense?” I have the same question about BQN. Can you give a concrete example where the absence of boxes would create difficulties in my hypothetical language? If you modify BQN to identify atoms and boxed atoms, then what breaks?

                      1. 1

                        (BQN certainly allows the kind of array you mentioned). To give some examples of the use of scalars and Enclose < I’ll look at a “Cartesian product” program that takes a list of arrays and returns an array of lists that contains every possible choice of one element from each input array. It’s different from the Cartesian product of sets because it also retains the ordering of arguments in the result.

                        The first use of < is in the function <⊸∾, which encloses the left argument before () joining () it to the right argument. This is different from Join on its own because it treats the left argument as a single element (compare). For this purpose you could also use {⟨𝕩⟩}⊸∾, which turns the left argument into a 1-element list, but I don’t think that’s what I really mean: I’d like to add a major cell to the front of an existing list, not add a whole new list. So this is one place scalars come up, as major cells of lists. You’ll also notice I used ≍˘ (Solo Cells) to format the result in the previous example. I wanted to put each element on its own line, changing its shape to 2‿1. Cells treats its argument as a list of major cells, and applies the operand to each one. The operand Solo adds a length-1 axis before any other axes, so it turns a scalar into a 1-element list, giving an overall result of two 1-element lists, shape 2‿1. This is discussed some in the leading axis document. Without being able to use scalars you could add a length-1 axis before any axis, but not after the last one. You can probably see how this function could be useful for tasks other than formatting. Another example is to join each row of a matrix to the corresponding item of a list. Here ∾˘, or Join Cells, splits both arguments into cells. For the matrix left argument these cells are lists; for the list right argument they have to be scalars. You can see that treating them as elements wouldn’t work by using a list of strings instead of numbers. Each string will stay as one element rather than being spread out as part of a longer list.

                        The second use of < in the original example is (<⟨⟩), which is the left argument to the function <⊸∾⌜´. Let’s break that function down. <⊸∾ adds the left argument as an element to the right argument, as discussed. Applying the 1-modifier Table gives the function <⊸∾⌜, which takes two array arguments and does this for every pair of elements from them. Reduce (´) changes this from a function of two arrays to a function of any number of arrays. And <⟨⟩, the enclosed empty array, is the initial value for this reduction. Why do we need an initial value? To start with, consider applying the reduction to only one input array. With no initial value Reduce will simple return that one array, so that for example an argument of ⟨"up"‿"down"⟩ gives the result "up"‿"down" But this is only an array of elements, and not an array of lists: the elements are only incidentally lists because they’re strings. So it’s not the right result. In fact, without an initial value we’ll get the wrong result on any argument because the elements of the rightmost array will be joined to the result lists as lists, not as elements. To fix this, we need an initial value that is an array of lists. Since it shouldn’t add anything to the result, any lists it contains need to be empty. But what should its shape be? The result shape of a function with Table is the combined shapes of its arguments (example). To avoid contributing to the result shape the identity element needs to have empty shape, or rank 0! So scalars are often needed simply because an array shouldn’t have any axes. Another example uses Each (¨). Here {𝕎𝕩} applies its left argument as a function to its right; we want to apply the four functions Rank, Length, Depth, and Shape to a single array. Each normally matches up elements from its two arguments, but if one argument has a lower rank it will instead copy its elements to fill in any missing trailing axes and match the higher-rank argument’s shape. To copy a single argument for every function call, it should have no axes, so we enclose it into a scalar.

                        A final example is seen if we use other ranks for the input arrays of a Cartesian product. A scalar can be used to include a particular element in every list in the result. Because it presents no choices it shouldn’t have any axes.

                        Rank-0 arrays are a lot like zero itself in that they are so fundamental they can appear useless. But “adding” scalars to an array language is not really a complication, it’s a simplification. The shape of an array is a list of natural numbers: excluding the empty list is an extra rule while including it isn’t.

                        1. 1

                          Regarding identifying atoms and boxed atoms specifically, but not other kinds of arrays with their encloses, this is exactly the idea of “floating arrays” used by most modern APLs. It’s not so intrusive as removing scalars entirely would be, but I and many other users of these languages regard it as a wart, even if not everyone agrees with me that based arrays are the right solution. The basic issue is that atoms don’t work the same way as other arrays, and this causes (fairly minor) confusion and bugs. I gave a specific example in the based array doc.

                          1. 1

                            Okay, now I finally understand Based Array Theory. Based on reading the documentation, I was left with the impression that a “scalar” can only contain an atom, that a “scalar” is a boxed atom, and that’s not correct. In actual fact, a rank 0 array in BQN is a box that encapsulates an arbitrary value, which can be an atom, a vector, a matrix, or any other array.

                            The problem, for me, is your use of the word “scalar” to describe a rank-0 array in BQN. This contradicts the standard mathematical meaning of the word “scalar”. It also contradicts the meaning of the word “scalar” in early APL papers, where Iverson uses the word “scalar” with its mathematical meaning as a bare number, and then says that a scalar is represented by a rank 0 array in APL. It also contracts the meaning of the word “scalar” in most programming languages. For example, in NumPy, a “scalar type” is a non-array type: an integer, real, complex or boolean (type np.ScalarType for the full list).

                            If you want the documentation to be comprehensible to people who are not experts in J or modern APL, then I would suggest replacing the word “scalar” with the word “box”.

                            A rank 0 array in BQN is called a “box” for short. A box encapsulates a single value of arbitrary type: it can be an atom or an array. To construct a box from a value x, use <x (where < is pronounced “enclose”). To extract the value from a box b, use ⊑b, where is pronounced “first”).

                            1. 1

                              Thanks. I did know about these issues (even the NumPy usage!) but I hadn’t found a good replacement for the term “scalar”. I still don’t like “box” because of possible confusion and also because the concreteness of a box seems to suggest it’s in a class of its own instead of just being a kind of array; there’s also the issue that in programming “boxing” seems to be something you only do once, e.g. to primitive types in Java.

                              After some discussion it seems that “unit” meaning an atom or rank-0 array works fairly well. How do you like the following translation?

                              In BQN, an atom or array of rank 0 is a “unit”. So a “unit array” means an array of rank 0 specifically, and it contains exactly one value. For any value 𝕩, the Enclose function makes a unit <𝕩 that contains it. To get the contents of an array unit 𝕩, use First: ⊑𝕩.

                              1. 2

                                Rank 0 arrays are a sufficiently important concept that they warrant a one-word name.

                                “Box” is the correct name for this concept. The term “box” is a general PLT term meaning a trivial data structure that encapsulates a single value. You use boxes so that you can manipulate values of different types in a uniform way. J has boxes. Racket, a Scheme dialect, has boxes: https://docs.racket-lang.org/reference/boxes.html. Even Rust has boxes. A rank 0 array in BQN is a clear example of a box. The details of boxes vary in each programming language but the essential features are the same in all the examples I give.

                                • Racket: (box x); BQN: <x
                                • Racket: (unbox b); BQN: ⊑b

                                I feel that the term “box” is no more confusing than the term “array”. BQN has a unique array theory, so arrays in your language work differently than in any other language I’ve seen. But you still call them arrays.

                                I’m less enthusiastic about “unit” because it normally means a special distinguished value. In typed functional languages, Unit is a type with a single value, called the unit value. In Haskell, F#, O’Caml, etc, () is the unit value. In mathematics, the unit value of a ring is the multiplicative identity, and a unit matrix is normally the identity matrix (or a matrix filled with all ones in Mathematica).

                                1. 1

                                  BQN has a unique array theory, so arrays in your language work differently than in any other language I’ve seen.

                                  Are you sure you understand the based array model? All it does is to extend a scalar/atomic model with a dynamically-typed multidimensional collection. This looks to be identical to Julia (which I haven’t used), and it’s also the same as NumPy without the element types. I’m sure there are any number of array libraries for other languages that work similarly. It should only be strange from the perspective of APL, where the idea of data that doesn’t reside in an array is foreign.

                                  The use of “box” in Racket and Rust is reinforcing my concerns (but definitely thank you for presenting these, as well as the comments about Unit—I hadn’t seen either and what you say about them is accurate). The central feature of a box seems to be that it makes the contents easier to handle or manipulate in some way, but a rank-0 array doesn’t do this. I think it’s very easy to mistake a “box” for a special kind of data structure. Isn’t this the mistake you made?

                                  The typed functional languages use “the unit” while in BQN it’s always “a unit” or used as an adjective. Given that difference and the many meanings of “unit”, I think this might cause some annoyance but probably not the blocking confusion of “box”. And there is actually a mathematical connection: if u is a unit (or specifically the identity) in ring R with operation ×, then <u is also invertible (the identity) in the monoid of arrays of R with operation ×⌜. Rank-0 arrays are definitely distinguished, and while they aren’t unique, neither are many uses of “unit”, e.g. the unit complex numbers.