1. 27
  1.  

  2. 6

    Cutoff Multivals

    The one that that kills me a little in lua comes from this: It’s impossible to set an assert message on a function that returns multiple values.

    > function getthings() return 5, 4 end
    > assert(getthings())
    5	4
    

    That all works fine, but what if I want my assert to have more details about where I was when that function failed?

    > assert(getthings(), "no things when $FOO")
    5	no things when $FOO
    

    :/

    1. 3

      That’s frustrating, I hadn’t run into that with assert specifically! Thanks for providing that example. It’s a good instance of a case where multivals contribute to the large number of tradeoffs when determining argument order for your Lua functions.

      1. 2

        Please note that the “opposite” way of using assert is intended: somewhat resembling Go, it’s a common idiom for a failed function to return 2 values: nil, "some error message". You can then plug it straight into assert like below (as shown in “PIL”):

        local f = assert(io.open(filename))
        

        and expect assert to print the error message in case of a problem.

        As a side note, the freely available old edition of PIL is for a super ancient version of Lua, but I’d still resoundingly recommend reading it for all the tricks and ideas it communicates.

        1. 2

          Yep, for sure. I was thinking of exactly that when I mentioned adding more context.

          I’ve got the 5.3 ed of PIL and also highly reccomend it. It’s actually one of my favorite books. It’s one of the only books I’ve ever read that manages to not be overly verbose, but also doesn’t have big gaps leaving you confused.

          If anyone else is interested in picking it up Feisty Duck has a DRM free ebook version of it.

        2. 2

          What do you think of this?

          assertm = function(x, ...)
            assert(unpack(arg), x)
            return unpack(arg)
          end
          
          1. 1

            Oh, interesting, message first. Yeah, that definitely seems to work. Thanks for the tip!

            (Sidenote, that code needs a s/arg/.../g.)

          2. 2

            It’s not a problem unless you try to do something like this:

            a, b = assert(getthings(), "no things when $FOO")
            

            and then you can just do that instead:

            a, b = getthings()
            assert(a, "no things when $FOO")
            
            1. 1

              Oh, for sure, that’s why it only kills me a little. I didn’t mean that it was impossible to write it in a way that accomplished what I want, it was more just an example of the problem in the article that I had run into. (And that took me far too long to realize what was happening, before having a blinding realization that it simply couldn’t work the way I wanted it to.)

          3. 3

            Wow! Of all the complaints I’ve seen about Lua, I haven’t seen this one. “Multi-values” just aren’t a thing in Lua. This is not a data structure [1]. Lua allows a function to return multiple values, not all of which need be used. The way to use multiple return values is to assign them to variables:

            function foo()
              return 1,2,3
            end
            
            local a = foo() -- 2nd, 3rd ignored
            local b,c = foo() -- 3rd ignored
            local d,e,f = foo() -- all assigned
            local g,h,i,j = foo() -- j is nil
            

            There are two exceptions to this—if the function call is wrapped in parenthesis, only the first value is used:

            local a,b,c = (foo())
            print(a)
            

            The second is when the function is followed by a second value, only the first value is used:

            local a,b = foo(),4
            print(a,b)
            

            Lua also allows a function a variable number of parameters (like in C) and to access these, you need the select() function (it’s clunky, I admit, and it comes up from time to time on the Lua mailing list). The first parameter to select is the index of which value you want from the rest of the values given to select, and it returns values starting from that index:

            print(select(2,1,2,3,4,5))
            

            If the index is negative, it starts from the end, so this will print the last two values:

            print(select(-2,1,2,3,4,5))
            

            And if the index is the special value ‘#’, it will return the number of additional parameters given to select:

             print(select'#',1,2,3,4,5))
            

            Lua also restricts the number of parameters to 255. Again, this is not a data structure per se.

            Your example of a tail call isn’t one. A proper tail call just returns the result of a single function call. Here’s a function that is a proper example of a tail call:

            function tablen(t,n)
              n = n or 1 -- in case we don't specify n initially
              if not t[n] then
                return n-1
              else
                return tablen(t,n+1) -- TAIL CALL!
              end
            end
            

            Add an additional value to that return statement, and now it’s just a recursive call that consumes stack space. And tall calls don’t have to be in a recursive situation—any return of a single function call becomes a tall call:

            function foo()
              return 1,2,3
            end
            
            function bar()
              return foo() -- this is also a TAIL CALL!
            end
            

            Tail calls effectively become a form of GOTO.

            Returning multiple values from a function used to be an uncommon feature of programming languages, but it’s not like it’s new—one could always return multiple values in assembly; Forth has had it from the start (late 60s), so did Perl (late 80s). I don’t know enough of Smalltalk but I suspect it too, supported multiple return values (70s). And more languages today support it (Go, Ruby and Rust for example).

            [1] In the traditional sense of the word. It may look like a data structure, but it’s an annoying one to use, and I think it hampers development to even think this is a data structure.

            1. 7

              Thanks for reading the post and replying to it! While I did cover some of what you mention here, I also completely missed some stuff, like how wrapping a function call with parentheses limits its return values to 1 or how you can pass negative values to select to get values from the end of the multival. I’ll add those to the post - thanks so much for pointing them out!

              Please do note that this post is not intended to be an anti-Lua rant. I’m a frequent Lua user and my motivation is to help people get a better sense for how Lua will behave in certain edge cases, not to criticize the language as a whole. Perhaps I could make this clearer in the post.

              I do have some responses to your comment:

              “Multi-values” just aren’t a thing in Lua. This is not a data structure [1].

              While it’s convenient not to think of them as a data structure while programming Lua, that’s actually not the case. They are a datastructure in my opinion, just a second-class one with extra rules surrounding them. They have a length, they have a maximum size, and they have contents which can be addressed by index. This is the thesis of my post.

              Lua also restricts the number of parameters to 255. Again, this is not a data structure per se.

              Again - while it’s convenient to think of this as not a data structure, it absolutely is - it’s an array of items limited to 255 elements (in practice, this limit appears to actually be at 253 for some reason). Also note that this restriction applies to both arguments and return values, but only applies to literal values - this is why, for instance, you can pass a ... that contains 255 items to a function successfully, but you can’t pass the same 255 items by writing them out literally in a Lua file. See the following code for an example:

              local function range1(tab, acc, n)
                if n < 1 then return tab
                else
                  tab[acc] = acc
                  return range1(tab, acc+1, n-1)
                end
              end
              
              local function range(n)
                if n < 1 then error("n must be >= 1") end
                return range1({}, 1, n)
              end
              
              local function increment_each_value(first, ...)
                if first then
                  return first + 1, increment_each_value(...)
                end
              end
              
              local x = range(1000)
              
              local y = function(...)
                return increment_each_value(...)
              end
              
              print(y(table.unpack(x)))
              

              Your example of a tail call isn’t one. A proper tail call just returns the result of a single function call.

              This is the whole point of the section you’re referring to. Maybe I can be clearer that I don’t actually think the first example of range is a tail call - I already demonstrate it blowing the stack to show that it is not tail recursive. This is intended to demonstrate one of the places where the understanding that “multivals are not a data structure” would trip someone up - if multivals really weren’t a data structure, why would a call at the end of a multival not be a tail call? The reason it’s not a tail call is because it’s not actually in tail position - it’s inside a data structure that just happens to not have delimiters, instead being represented by multiple adjacent values separated by commas.

              You might argue that this is all obvious, but it tripped me up the first time I ran into it. Thinking about it for a bit made it make sense, but my initial instinct was that the initial implementation of range was tail recursive. Finding out that it wasn’t was part of the motivation for writing this post.

              Note that the second implementation of range in the post is properly tail recursive, and the distinction between the two implementations is what I’m trying to illustrate there.

              Imagine that the syntax to return a multival looked like <<<1, 2, 3>>> instead of 1, 2, 3. (I’m not actually proposing this! It’s just a thought experiment.) It would then be immediately obvious that calling a function at the end of the multival was not in tail position:

              local function range1(acc, n)
                if n < 1 then return
                elseif n == 1 then return acc
                else return <<<acc, range1(acc+1, n-1)>>>
                end
              end
              
              local function range(n)
                if n < 1 then error("n must be >= 1") end
                return range1(1, n)
              end
              

              You make a good point that tail calls do not require recursion, just returning a single function call. I’ll try to see if there’s a way to add this to the post that fits well.

              Returning multiple values from a function used to be an uncommon feature of programming languages, but it’s not like it’s new…

              I didn’t mean to imply that this was a recent feature in programming languages, just that it was a relatively unusual one.

              Thanks again for your detailed critique of my article!

              1. 3

                What I find unfathomable (but it may just be me) is that anyone would think to use function parameter passing as a data structure. In over 30 years of programming experience, you are the second person I’ve come across to do so [1]. Perhaps I’ve been programming for too long, or involved way too much in low level details (my second computer language I learned was 6809 Assembly, followed by 8088, then 68000 and MIPS).

                Yes, parameters can be passed around in a data structure, but these days, compiled languages on x86_64 (and some other RISC based systems—I’m also thinking of the SPARC architecture here) pass as much as possible in registers so there’s no data structure backing them. I think that’s why I find it weird to think of the “multi-values” as a data structure.

                First off, the limit I mentioned, 250? That’s the upvalues (used in closures) limit, not the paramter limit (I was mistaken there). The Lua parser might impose a limit on the number of literal parameters one can specify, but there’s no effective limit on the number of parameters a function can have. I did construct a table with 2,048 elements, and was easily able to do this:

                print(table.unpack(t))
                

                and end up with 2,048 items printed to the screen. And in am empirical test I did, I was able to construct a Lua function (in C) that returned 999,990 values (and that’s probably due to a limitation on the system than a hard-coded limit in the Lua VM).

                Also, there’s one other language I’m aware of that uses ‘…’, and uses it in the same context—C. C allows a variable number of arguments (but the restriction in C is that there has to be at least one named paramter). The declaration of printf() is (per the standard):

                extern int printf(const char *fmt, ... );
                

                And like Lua, there’s a special function you have to call to retrieve the unnamed parameters (actually, three functions that work in tandem). The function hides how the parameters are actually passed (via the stack, or maybe in X number of registers, or a mixture of the two).

                [1] The first one also used Lua, five years ago.

                1. 4

                  …data structures don’t have to live in memory. A data structure lives in the programmer’s head and the code that is an approximation of it. Something doesn’t stop being a data structure just ’cause a compiler decides to pass it in registers, or may optimize it out of a particular piece of code entirely.

                  There’s languages out there that let you very specifically treat function args as a tuple; the main one I’m familiar with is SML. Simple Lisp’s often pass them as a list. My understanding is that it’s just not common because having function args be a special case is useful. More common are languages that return a tuple to implement “multi-values”, such as Rust, pretty much any ML, sometimes Python, etc.

                  1. 1

                    I never thought of subroutine [1] parameters as a distinct “data structure”. At best, one could say that it forms an ad-hoc struct [2], and I think I might be too … something (I’m not sure what) to accept “data structures don’t have to live in memory.” There are implementations of a data structure (a stack can be an array, or a linked list, for instance), but that statement is just too “nothing there” for me to accept it.

                    So what is an array, a structure, and a tuple?

                    [1] And I’m using that word very deliberately.

                    [2] The VAX has two forms of a CALL instruction—one passes the parameters on the stack; the other in a pointer to some memory. The called subroutine doesn’t care which was used to call it, it just gets a pointer to memory containing the input. The layout will be the same in either case.

                    1. 2

                      It’s late, so, the simplest counter-argument I can think of is: does an array stop being an array if it lives entirely in registers? Sure you’re going to have a hard time indexing it with a pointer, but you could certainly write code to index it with an integer, which is just a pointer without context anyway, and I wouldn’t be surprised if some CPU somewhere even had instructions to make it easy. Structs get passed in registers all the time, in C now as well as Rust and whatever else. What about a floating point matrix that is loaded, stored, passed and manipulated entirely in SIMD registers, is that a data structures or not? Just ‘cause you can’t aim a pointer to it in main memory doesn’t mean it doesn’t exist.

                      Sure, registers are just specialized memory though. Then take it a step further and ask yourself, if the compiler optimizes the matrix math entirely at compile time down to a single output number, is the matrix still there? It’s not in the binary output, but it’s in the source code, you can go in and change the math and it gives you a different result. So, which version is “real”? Depends on what you want; if you want to know what the machine is actually doing then your “matrix” doesn’t exist, it’s been reduced to just some number sitting in the binary’s data segment somewhere. If you want to be able to logically manipulate the math, then you probably care a lot more about the functions and values in the program than what the compiler does with it, at least until you start optimizing. What about if I do the compiler’s job and do all the math myself, write the algorithm down in the program comments, and stick the constant result in the code? We start edging into the Chinese Room problem.

                      Guess I’m feeling existentialist tonight. Anyway, you probably know all this. I’ve just been messing around with Haskell too much lately, where “data structure” and “function that produces a data structure” are basically the same thing.

              2. 6

                It may look like a data structure, but it’s an annoying one to use

                I think this is part of the point of the article; multivals are troublesome because it has many properties of a data structure, (the ... expression is a compound thing that contains other things) but it’s not first-class in a way that allows it to be used in the ways you would expect to be able to use data structures.

                Edit: Ruby doesn’t support multiple values, but it does have syntactic sugar for returning an array which appears like it’s returning multiple values. Smalltalk doesn’t support it either. Forth doesn’t support returning any amount of values, only pushing values onto the stack, which is loosely analogous but a different thing.

                1. 2

                  Forth doesn’t support returning any amount of values, only pushing values onto the stack, which is loosely analogous but a different thing.

                  Not so different. Returning multiple values from a function in Lua is exactly that: pushing multiple values on the stack.

                  That’s why I’m strongly in the “not a datastructure” camp: it’s all just syntactic sugar to manipulate the stack, which is the underlying “data structure”.

                  Even if you don’t write bindings, if you write a significant amount of Lua, you should at least look at the C API calling convention to understand things like this.

                  1. 3

                    Returning multiple values from a function in Lua is exactly that: pushing multiple values on the stack.

                    Sure, but in Lua the stack is an implementation detail, whereas in Forth it’s The Whole Point; no one can use the language without thinking about the stack every step of the way.

                    you should at least look at the C API calling convention to understand things like this.

                    I agree but IMO the fact that you have to read about the C API to really understand this feature even when you have no interest in ever using C is a red flag.

              3. 2

                I’ve been highly interested in Lua after following the performance of Lua Jit, and seeing how a higher level scripting language can get perf results on par with C. Now I really want to know what all of you are using Lua for! Work, games, embedded?

                1. 3

                  I mostly use it as a host language for Fennel. I use that to write games in LÖVE. I’ve also done some experimenting with web programming using OpenResty.

                2. 1

                  Personally, having read the manual and PIL, I don’t think I ever had those problems. I seem to have understood from the start that ... etc. are “their own thing”, i.e. a special kind of syntax with specific semantics. This article’s tone makes me think it’s a result of accidentally attaching a mental model of some other construct from other language to ..., and not fact-checking it with the manual, and then being outraged at the difference. For a calmer tone I’d consider rephrasing this article to something to the effect of:

                  Lua has a special syntax related to ..., that is not commonly found in other currently popular languages. Some interestingly differentiating features of it (see also chapters X, Y and Z of the manual and PIL) are: […etc…]

                  Also, as a side note, I don’t think dynamic scoping is what the article seems to suggest - or am I missing something?

                  1. 2

                    Thanks for your comments on the tone of the article. I definitely didn’t intend for this to be a rant or to be taken as a criticism of Lua as a whole. I’ll see if I can soften the language to make it more clear that I’m trying to warn about potential gotchas rather than just criticizing the language generally.

                    Could you elaborate more on how my explanation of dynamic scoping is wrong? I definitely don’t want to mislead people there.

                    1. 1

                      You can still understand where the contents of ... is coming from by just looking at the program’s source code, so AFAIU this is still lexical scope. For an example how dynamic scoping in Lua could be implented, see e.g. here: https://leafo.net/guides/dynamic-scoping-in-lua.html

                      Truth said, I’m not really sure in what way you suggest ... results in a dynamic scope. I’m kinda guessing you’re maybe trying to convey that writing local a, b, c = ... makes a & b & have “dynamically scoped” values? But IIUC, scoping is all about resolving names, and here the “visibility” of all the names is clearly resolved purely by analyzing the text (i.e. source code) of the program; values is something different than scoping, e.g. writing local x = t[777] doesn’t make x nor t dynamically scoped either.

                      Though I personally still find dynamic scoping quite confusing, so I full well may be missing something; happy to learn if that’s so!

                      Also, I’m even less certain about it, but I think manipulating _ENV in modern Lua is kinda how you might maybe simulate dynamic scope. But maybe that is not even it either. I think Lisp is the language where you most (in)famously get dynamic scoping. I believe in #define FOO(x) in C, x could maybe be seen as dynamically scoped? But again not 100% sure :/ That’s really why I inserted the “am I missing something?” in my initial comment, I’m still struggling with the concept myself :/