1. 7

    Oddly - this sounds like the author has just discovered dependency injection? I would have thought that concept would translate pretty well to Go. I’ve written a lot of Go, but I cut my teeth largely on C, C++, and C# so dependency injection has always been on my radar. When I wrote Go, I learned it and largely applied my own lessons from C, C++, and C#.

    Due to compiler constraints and the language’s ethos, global state and func init feel weird in Rust (my current language). You can’t, without jumping through hoops, create objects that are partially constructed (e.g. using Option for uninitialized types). That said, even if you’ve got struct members that are of type Option, you are actually initializing it to something - sometimes it’s just None.

    I don’t have enough context in Go land to know why this author’s argument might be a novel conclusion. Does anyone have some context? I’d love to learn more.

    1. 9

      Many Go programmers seem to feel very comfortable with global state. When I join new organizations or projects, I often find myself needing to educate and socialize the problems that come from that. This post is just a formalization of the things I’ve been saying informally for a long time.

      I wish I knew why this was so relatively prevalent in the Go ecosystem. If I had to guess, I’d speculate that it’s because a lot of the example code in e.g. books and tutorials doesn’t really shy away from global state.

      1. 7

        It’s also related to the standard library itself having lots of globals. Which itself leads to bad things, like the cryptographic randomness source being trivially modifiable: https://github.com/golang/go/issues/24160


          The Go community has a strong culture of writing application-specific code that is “good enough”, and tends to err strongly on the side of avoiding premature abstraction. For a significant number of use cases, globals (combined with a reasonable amount of documentation, testing, and good discipline) tend to be the “good enough” solution.

          The thesis of Go’s culture is that premature abstraction often costs more than rewriting code. The argument is that you often know your domain better after writing a “good enough” first version, and that premature abstractions lock you in to specific designs more tightly that may be harder to change down the line.

          It’s definitely not a conventional position, but it’s not indefensible – it’s hard to argue that Go developers are not pragmatic (or not productive).


            Interesting. Good to know!


            Yup, this was my comment when this appeared a year ago on HN:

            In other words, use a pure dependency-injection style, and ZERO globals.


          1. 13

            The issue with double dash (--) is one “string safety” problem I’ve meant to address on my blog, but haven’t yet.

            I’m not sure I’m following the argument in this post though.

            • I don’t see the need to distinguish between paths and strings.
            • I do see the need to distinguish between args and flags, which is what the -- convention does.
            • As far as serializing arrays, it seems like the NUL convention of find -print0 and xargs -0 is (surprisingly) sufficient? I didn’t quite see that until writing the two Git Log in HTML posts.

            I think there does need to be some kind of lint check or function wrapper for --, like flagging the former but not the latter (somehow):

            grep foo $file  # oops, $file could be a flag and GNU and OS X grep accepts it
            grep -- foo $file  # this is more correct
            mygrep() {  # mygrep only accepts args
              grep -E --color -- "$@" 
            mygrep2() {  However sometimes you do want the user to be able to append flags, so this function is useful too
              grep -E --color "$@" 
            mygrep2 -v pattern *.c  # invert match

            The rule I have for Oil is to avoid “boiling the ocean” – i.e. there can’t be some new protocol that every single command has to obey, because that will never happen. Heterogeneity is a feature.

            However there should be some recommended “modern” style, and I haven’t quite figured it out for the flag/arg issue.

            I think one place Oil will help is that you can actually write a decent flags parser in it. So maybe the shell can allow for wrappers for grep/ls/dd etc. that actually parse flags and then reserialize them to “canonical form” like:

            command @flags -- @args  # in Oil syntax
            command "${flags[@]}" -- "${args[@]}"  # in bash syntax

            I addressed quoting and array types in various places, including this post:

            Thirteen Incorrect Ways and Two Awkward Ways to Use Arrays

            There are also related string safety problems here – is -a an argument or an operator to test ?

            Problems With the test Builtin: What Does -a Mean?

            1. 4

              I don’t see the need to distinguish between paths and strings.

              They’re different types: a path is a string which is or could be the pathname of a file or directory on disk. Since they’re different types, treating them differently yields all the standard benefits of strong typing.

            1. 8

              This is very cool, but note that it is a microbenchmark comparing the overhead of calling plus(int, int). This is a very specific case of FFI that is easy and simple.

              For Oil, I care more about moving strings back and forth across the VM boundary (not to mention bigger objects like records and arrays of records). There are many more design choices in that case, and I suspect the results will also look different.

              1. 2

                This was my thought as well, anything more complex than basic types or char*, you’re essentially serializing/deserializing with all the performance problems that entails.

                1. 3

                  Hm, I don’t quite see what you’re getting at, because

                  1. You don’t have to serialize complex data types to move them across the VM boundary. For example, you can move a C struct to a Python dict or a Lua table with a series of API calls, without serializing (i.e. integers and floats more or less retain their native representation; they’re just wrapped). It generally involves copying strings, but both of those languages provide various complex ways to avoid copying (buffer protocol, light user data).

                  2. I don’t think of serialization as slow. In fact sometimes I have serialized data to protobufs and moved them across the VM boundary as a single string, rather than constructing a series of complex function calls, and doing incref/decref in Python, etc.

                  I think tracking ownership can be more inefficient than serializing and copying, since it’s a global algorithm (i.e. causes contention among threads), and people invent all sorts of hacks to try to make it easier (layers like SWIG). More layers is more inefficient.

                  Anyway I don’t have numbers to back that up, which is what I would have liked to see here. But it’s pretty difficult to measure that, since these issues are situated in complex programs, with big library bindings and complex call patterns.

                  1. 1

                    You don’t have to serialize complex data types to move them across the VM boundary. For example, you can move a C struct to a Python dict or a Lua table with a series of API calls, without serializing

                    My contention is that this is effectively serialization, only without the intermediate string (or whatever) representation; each element of the data is transformed from host to c and c to host. Similar problem arises in micro kernel architectures and IPC, not being able to push/pop data direcly onto a process stack for example.

              1. 12

                I think I’ve read this paper a half dozen times now after seeing someone or other wax lyrical about it online. And I just don’t get why people like it so much. Which means either I’m too much of an insider and this is old news, or I don’t appreciate what I don’t know.

                Part of the problem is that it seems to overstate its contributions. It isn’t really “identifying” causes of complexity. Fred Brooks pointed out essential vs accidental complexity back in 1975, and there’s been a steady stream of articulation ever since. To any Haskeller the point that complexity comes from state is old news. Relational algebra is ancient. At best OP is laying things out in a new way.

                This time I figured I’d speed up my rereading time by instead chasing down past threads. And I discovered a decent summary. This was helpful, because it helped me to focus on the ‘causes of complexity’ portion of OP.

                Causes of complexity according to OP:

                • State (difficulty of enumerating states)
                • Control (difficulty of choosing an ordering)
                • Concurrency
                • Volume
                • Duplicated/dead code, unnecessary abstraction

                Compare the list I made a couple of years ago:

                • Compatibility. Forcing ourselves to continue supporting bad ideas.
                • Vestigial features. Continuing to support stuff long after it’s been obsoleted by circumstances. Simply because it’s hard to detect obsolescence.
                • People churn. Losing institutional fingerspitzengefühl about the most elegant place to make a given change. Or knowing when some kludge has become obsolete.

                Comparing these two lists, it looks like there’s a tension between the top-down and bottom-up views of software management. In the bottom-up view people seem to think about software like physics, trying to gain insight about a system by studying the atoms and forces between atoms. You tend to divide complexity into state and order, essential and accidental. Reductionism is the occupational hazard.

                In my top-down view I tend to focus on the life cycle of software. The fact that software gets more complex over time, in a very tangible way. If we could avoid monotonically adding complexity over time, life would be much better. Regardless of how many zeroes the state count has. In this worldview, I tend to focus on the stream of changes flowing into a codebase over time, alongside the stream of changes happening to its environment. This view naturally leads me to categorize complexity based on its source. Is it coming from new feature requirements, or changes to the operating environment? How can I keep my raft slender as I skim the phase transition boundary between streams?

                The blind spot of the bottom-up view is that it tends to end up at unrealistic idealizations (spherical cows as @minimax put it in this thread). The blind spot of the top-down view is that there’s a tendency to under-estimate the complexity of even small systems. Blub. The meme of the dog saying “this is fine” while surrounded by flames.

                It seems worth keeping both sides in mind. In my experience the top-down perspective doesn’t get articulated as often, and remains under-appreciated.

                1. 5

                  Here’s my take on it: https://news.ycombinator.com/item?id=15776629

                  I also don’t think it’s a great paper. It’s long on ideas but short on evidence, experience, and examples. I don’t think you’re missing anything.

                  1. 4

                    I have a similar take to you. I think it’s one of those papers that is easy to get excited about and everyone can agree that complexity is bad and all that. But I have not seen any successful application of the ideas in there. The author’s haven’t even successfully implemented the ideas beyond a little prototype, so we don’t have any idea if what they say actually pans out.

                    And to toss my unscientific hat into the ring: IME the biggest source of complexity is not programming model but people just not being disciplined about how they implement things. For example, I’m currently working in a code base where the same thing is implemented 3 times, each differently, for no obvious reason. On top of that, the same thing is some times and id, sometimes the id is a string and sometimes an int, and sometimes the string is a URL, and it’s never clear when or why. This paper is not going to help with that.

                    1. 2

                      If what you say is true, then the success of LAMP stacks with associated ecosystems for new people and veterans alike might make it high on “evidence, experience, and examples.” That architecture worked for all kinds of situations even with time and money limitations. Except that the specific implementation introduces lots of the complexities they’d want people to avoid. So, maybe instead the Haskell answer to a LAMP-style stack or something like that fitting their complexity-reduction ideas.

                      Although the others shot it down as unrealistic, your characterization seems to open doors for ways to prove or refute their ideas with mainstream stuff done in a new way. Maybe what they themselves should’ve have done or do they/others do later.

                      1. 4

                        Yes, so depending on the scope of their claims, it’s either trivial and doesn’t acknowledge the state of the art, or it’s making claims without evidence.

                        Appreciating LAMP is perhaps nontrivial. Google services traditionally used “NoSQL” for reasons of scaling, but the relatively recent development of Spanner makes your architecture look more like LAMP.

                        But either way I don’t think that LAMP can be “proven” or “refuted” using their methodology. It’s too far removed from practice.

                    2. 4

                      In my top-down view I tend to focus on the life cycle of software. The fact that software gets more complex over time, in a very tangible way. If we could avoid monotonically adding complexity over time, life would be much better.

                      Thanks for the interesting commentary. Some parts definitely resonated, particularly about the half-features and difficulty of knowing how and where to make the right change.

                      This is only the germ of an idea, but it is perhaps novel and perhaps there is an approach by analogy with forest management. Periodic and sufficiently frequent fires keep the brush under control but don’t get so big that they threaten the major trees or cause other problems.

                      Could there be a way of developing software where we don’t look at what is there and try to simplify/remove/refactor, but instead periodically open up an empty new repo and move into it the stuff we want from our working system in order to build a replacement? The big ‘trees’ of well understood core functionality are most easily moved and survive the burn, but the old crufty coupling doesn’t make the cut.

                      Some gating on what would be moved would be needed. The general idea though is that only sufficiently-well-understood code would make it across to the new repo. And perhaps sufficiently reliable/congealed black boxes. It would interplay a lot with the particularly language’s module/package and testing systems.

                      The cost would be periodic re-development of some features (with associated time costs and instability). The benefit would be the loss of those code areas which accrete complexity.

                      1. 2

                        Yes, definitely worth trying. The caveat is that it may be easy to fall into the trap of a full rewrite. There’s a lot of wisdom encoded in the dictum to avoid rewrites. So the question becomes: how would you make sure you don’t leave behind the most time consuming bugfixes you had to make in production on the old repo? Those one-liners that took a week to write?

                      2. 3

                        This paper was written in 2006, two years before Applicatives were introduced. The Haskell community’s understanding of how to best structure programs has been refined a lot in that time and I think you underestimate the insights of this paper even if it is only a refinement of Brooke’s ideas from 40 years ago.

                        1. 1

                          Thanks, I hadn’t seen that paper. What made you cite it in particular?

                          1. 1

                            It’s where Applicatives were introduced, as far as I know.

                            1. 7

                              Can you describe the connection you’re making between Applicatives and this paper?

                              1. 1

                                I got the impression that akkartik was saying that OOTP hadn’t added much new. My claim is that the fact that Applicatives were only introduced 10 years ago shows that the bar for novelty is actually quite low.

                        2. 1

                          “This was helpful, because it helped me to focus on the ‘causes of complexity’ portion of OP.”

                          That’s the part I read in detail. I skimmed the second half saving it for later since it was big. The first half I liked because it presented all those concepts you listed in one location in an approachable way. It seems like I learned that stuff in pieces from different sub-fields, paradigms, levels of formality, etc. Seeing it all in one paper published ahead of some of these things going into mainstream languages was impressive. Might have utility to introduce new programmers to these fundamental concepts if nothing else I thought. Understanding it doesn’t require a functional programming or math back ground.

                          Ok, so that’s their requirements. Minimax mentions things like business factors. You mention changes with their motivations. I’ll add social factors which includes things that are arbitrary and random. I don’t think these necessarily refute the idea of where the technical complexity comes from. It might refute their solutions for use in the real world such as business. However, each piece of the first half is already getting adoption in better forms in the business world on a small scale. There’s also always people doing their own thing in greenfield projects trying unconventional methods. So, there’s at least two ways their thinking might be useful in some way. From there, I’d have to look at the specifics which I haven’t gotten to.

                          I do thank you for the Reddit search given those are about the only discussions I’m seeing on this outside here. dylund’s comments on Forth were more interesting than this work. Seemed like they were overselling the positives while downplaying negatives, too, though.

                        1. 8

                          FWIW you can use ${a[@]+"${a[@]}"} to get around the empty array / set -u issue. In bash 4.3, arrays and set -u were incompatible. bash 4.4 fixed that, after probably a decade+ of broken behavior.

                          Recent Linux distros like Ubuntu LTS 16.04 is still on bash 4.3, so it’s hard to recommend using things that are bash 4.4 specific.

                          But I don’t recommend that trick – I mostly use strings as pseudo-arrays and I don’t use shell scripts with untrusted filenames. (e.g. my shell scripts mostly process files in my own git repos)

                          If you really need to process untrusted filenames in shell, you might want to follow these guidelines, but otherwise I think it’s too pedantic and ugly. I couldn’t recommend this to someone “with a straight face”.

                          I meant to write a followup to Thirteen Incorrect Ways and Two Awkward Ways to Use Arrays talking about the ${a[@]+"${a[@]}"} trick, but it just felt too ugly and awkward.

                          Now instead of 8 extra characters to splice an array, you need 15 !!! That’s opposed to 1 in the hypothetical Oil language: @a vs. ${a[@]+"${a[@]}"}.


                          $ set -o nounset
                          $ readonly empty=()
                          $ argv "${empty[@]}"  # DOESN'T WORK in bash 4.3 because of bug
                          -bash: empty[@]: unbound variable
                          $ argv ${empty[@]+"${empty[@]}"}  # WORKS
                          $ readonly a=('1 2' 3)  
                          $ argv ${a[@]+"${a[@]}"}  # non-empty case works too
                          ['1 2', '3']

                          EDIT: It also appears that you can just quote on the outside:

                          $ argv "${a[@]+${a[@]}}"
                          ['1 2', '3']
                          1. 1

                            I wonder what programming languages are nice for this scan/scatter style of programming. I’ve seen a lot of new GPU languages floating around, but I haven’t tried them.

                            The algorithm seems pretty elaborate. CUDA is an obvious choice but I wonder how annoying it would be to get a full JSON parser working in it (as opposed to the Dyck language). I think you would need a lot more than 8 steps, e.g. to do the backslash escape decoding.

                            Anyway it would be interesting to see a prototype of something runnable in a high-level language! APL? :)

                            (Although I don’t think APL-like languages give you the ability to express numeric types of different widths, which you probably want for GPU programming.)

                            1. 9

                              Something interested I learned while working on a shell: GNU readline was extracted from bash, while curses was at least partly extracted from vi!

                              In other words, the libraries were extracted from applications, rather than written from the ground-up as libraries. That seems to be a more fool-proof way to design a complete and usable (if not optimal) interface.


                              1. 1

                                Huh, I always thought curses was written from whole cloth for rogue?

                                1. 1

                                  Hm yeah it’s possible I misread / overstated. The wikipedia says that vi predates curses, and some code in curses was borrowed from vi. But I guess it’s also true that rogue was the first application that used curses?

                                  Sometimes it is incorrectly stated that curses was used by the vi editor. In fact the code in curses that optimizes moving the cursor from one place on the screen to another was borrowed from vi, which predated curses

                                  The first curses library was written by Ken Arnold and originally released with BSD UNIX, where it was used for several games, most notably Rogue

                              1. 5

                                Beautifully-done illustrations on top of the good info.

                                1. 4

                                  I’m thankful for the rich “metadata” structure of svg’s. Some 1, 2 are done in Visio:

                                  <!-- Generated by Microsoft Visio, SVG Export StringTable.svg Page-12 -->

                                  Others in Inkscape:

                                  <!-- Created with Inkscape (http://www.inkscape.org/) -->

                                  I’ve been experimenting https://draw.io in order to avoid Visio, their FOSS model is a little bit weird (I’m not sure how open it’s actually). At least you can use file-system export (compressed base64 encoded xml structure) from their web interface. html “export” is an embedded svg and svg’s might need additional manual rework, in case you want to publish them.

                                  1. 2

                                    Yeah I’d really like to know how he made the illustrations! They are great.

                                  1. 7

                                    I have a few things that I’ve been working on for the past 8 weeks:

                                    • data visualization project
                                    • ActivityPub implementation
                                    • working through Haskell Programming from First Principles

                                    I’m in a rut where I’ve lost conviction about all three of these projects. I’m enjoying Haskell Programming from First Principles the most but I’ve been demotivated lately by the thought that I should be focused on applying technology in useful ways more than learning new technologies. I don’t have any immediate or practical need to learn Haskell. I don’t expect to use it professionally. I just think the ideas are pretty interesting. But I can’t escape the feeling that there’s something more important I could/should be working on. So I’m in a rut where I have some things I could work on this week but little motivation to work on them, and I’m not certain how to resolve this stalemate.

                                    1. 8

                                      Would recommend to any developer who has a few months of mostly-free-time-ahead to dive deep into the Haskell world without regards to such concerns. Just a few months, then step back. Chances are high from numerous anecdotals including myself you’ll be a whole-new-developer in whatever “real-world languages” you come back to. Not in the ivory-tower over-abstracting sense either. Just in the sense of almost-deeply-instinctively circumventing the more subtle pitfalls of all non-purelyfunctional languages, and devising more-principled, less-convoluted designs. Maybe you won’t have that and maybe you’re already there anyhow — just saying, “chances are high”. In any event such time won’t be wasted and in your future real-world-work you will thank yourself for it, maybe even others (slightly less likely, nobody quantifies or detects infinite-numbers-of-troubles-avoided ;)

                                      1. 1

                                        I really appreciate the recommendation. I’m not totally new to the concepts in Haskell. For example, I know the basics of and am comfortable working with algebraic data types. I too have found that learning these concepts strengthens and clarifies my thinking in other areas of programming. That’s a big part of what I enjoy about Haskell and functional programming in general.

                                      2. 2

                                        FWIW, I don’t think I’ve ever really learned a language from a book. The examples / exercises usually feel too isolated and far-removed from the domains I work on.

                                        I do have a ton of language books, but I use them more as references while I apply the language to some problem. That focuses what you need to know. Usually what I do is to find a bunch of 500-line code snippets in the language, and choose one to add a feature to.

                                        I went through Real World OCaml a few years ago, and it was useful. But then I found an ocamlscheme project that I hacked on a bit, and that was a more useful learning experience. It wasn’t exactly practical, but I think it focused me on a few things rather than trying to ingest every concept in a book. There are still a bunch of things about OCaml I don’t know (and I think that is true for most people).

                                        The Pareto rule generally applies – 20% of the language allows you to solve 80% of problems. There is usually a long tail of features for library authors (e.g. in C++).

                                        1. 2

                                          I agree and support the principle here, which I think is that abstractions are best learned through application and experience. I like focusing on the language first and then using the language to solve a non-trivial problem. I get overwhelmed if I try to learn the syntax and core concepts while also trying to think of how to best express myself in that language. But I would never consider the language learned until the second step was taken and I had used the language to complete a real-world task.

                                      1. 3

                                        One thing I’m curious about is the overhead to using the GPU. If the structure of the JSON parser is implemented on the CPU (and I think it has to be), then presumably you have to switch between CPU and GPU a lot. What problems does that introduce?

                                        My understanding of GPU memory is fuzzy (and I’m sure there is more diversity to it than CPU architecture). I would have thought you had to transfer the string over to GPU memory, but I also remember that the GPU can access main memory (is that specific to GPGPU / CUDA ?).

                                        But still you have to coordinate the “handoff”, and I’m not sure how much overhead that introduces. JSON payloads typically have a lot of small strings, and not one large one, so this seems like it could be an issue.

                                        In other words, I think a more interesting performance metric would be MB/s in decoding a dict with a bunch of strings, rather than just a huge string.

                                        FWIW I designed an IPC protocol that used JSON, and for large strings, I “switched” to a netstring, i.e. a length-prefixed payload. That way you avoid the decoding cost. The JSON was essentially the metadata, and you could point into blocks of memory delimited by netstrings.

                                        I feel like this is a nice compromise and preserves the important property of JSON: it’s easily implemented in any programming language. netstrings are trivial to decode.

                                        I also did this so I could represent binary data accurately, which JSON isn’t good at.

                                        Also, the point about parallel copy_if is interesting. I’d be interested in examples of that technique used to speed up any problem, with the constraint that it’s used “for real”. I haven’t seen that in the C++ code I look at.

                                        All in all this is a really interesting piece of code that brings up a lot of questions :) I’ve read papers about prefix sums for parallel computing but honestly have not come across it in practice.

                                        1. 2

                                          in what way did the spam/moderation problem manifest? what suggests zulip would not have those problems? i hope these questions don’t sound too pointed. i would just like to understand the decision, lest i must make a similar one in the future.

                                          1. 2

                                            There were almost no real messages on the mailing list – and I would get messages from mailman/dreamhost every day about whether to let spam in or not. There were a couple other reasons I forgot to mention:

                                            • I’m on the toybox mailing list, which also uses mailman on Dreamhost. Rob Landley has complained that the archives go missing peridiocally. There is a hole in his mailing list history.
                                            • Also, Gmail users get auto-unsubscribed because of some spam issue. It might be Google’s fault or it might be because Dreamhost is used to send a lot of spam. Dreamhost is good at web hosting, but the mailing list hosting doesn’t appear to be well-maintained.
                                            • Some people had asked for an IRC channel, so this format might encourage more messages. There’s not enough traffic for IRC + mailing list, but there might be for a combined medium.
                                            1. 1

                                              interesting. one hitch i’ve found is that i can’t figure out how to view threads without logging in to zulip. the mailing list had web archives which could be viewed without giving my email to dropbox. i don’t browse ml archives too much in practice, but it’s useful if you want to refer someone to a thread in a ml they’re not subscribed to. i’m sure there’s a way to enable this on the zulip side, and that way you’ll get the same utility that the ml provided and more.

                                              1. 1

                                                and I would get messages from mailman/dreamhost every day about whether to let spam in or not.

                                                But how does Zulip prevent this? Is it just not popular enough to have spambots for it or do they have something specific for curation?

                                                I suspect a lot of general purpose host get blacklisted for e-mail. I hadn’t thought of “don’t send e-mail” as a potential solution to that, but it seems to work. :)

                                                1. 2

                                                  I’m not sure, but the fact that they have logins and are centralized means they can probably do a better job than a random mailing list.

                                                  Dreamhost also probably has enough data to do something good about spam, but they apparently don’t. It costs engineers and money to fight e-mail spam.

                                                  Here’s a good link about spam: Modern Anti-Spam and E2E Crypto (2014)


                                                  In my experience, Google’s spam filters have gotten significantly worse lately. I’m on the busybox mailing list and Gmail routinely marks those messages as spam. And it routinely rejects e-mails I sent from a cron job. So they’re having problems with both false positives and negatives.

                                                  1. 1

                                                    Thanks for the link and your thoughts on this! I actually didn’t know where to start looking.

                                            1. 1

                                              I can’t log in to Zulip. Logging in with GitHub just loads the login page again. I reset my password (I never set one in the first place), which appeared to succeed, but logging in with my email/password just loads the login page again, just like with the GitHub login. 🤷‍♀️

                                              1. 1

                                                Hm a wild guess: does it have anything to do with cookie settings?

                                                1. 1

                                                  FWIW, I had this same problem with Safari 11.1, and tried Chrome (65.0.something) and it worked fine.

                                                1. 1

                                                  You mention a spam/moderation problem. How does Zulip address this (and would that still work if self hosted).

                                                  Does Zulip work well for longer delays (1 day - 1 week)?

                                                  P.S., “Discuss This Post on Reddit” links to http://www.oilshell.org/blog/2018/04/TODO

                                                  1. 2

                                                    Thanks, I fixed the link. See my answer here:


                                                    As far as I know, Zulip works fine for longer delays. The threads should help with that. I think Slack might work less well because of its notion of threads.

                                                    With Zulip, I suppose spam something they deal with, rather than me getting a ton of messages and a suboptimal mailman interface to let them in or not. I was a manual spam filter.

                                                  1. 3

                                                    Zulip sounds interesting.

                                                    Can you export all the data if you want to change providers, or if Zulip goes out of business/gets purchased? Can you export the discussions to a web-based format (as you can with mailing lists) for searchability?

                                                    1. 5

                                                      Zulip has already been purchased by Dropbox. And Zulip is open source so you can run your own.

                                                      1. 4

                                                        Yes, both of those things are important to me. It looks like there is an export.py tool, although the docs aren’t the greatest:


                                                        And it looks like they’re working on truly public archives:


                                                        Right now it’s a little annoying that you have to sign in to read messages, which they’re also working on changing. But at least it doesn’t require another login – you can log in with Github.

                                                        Overall, the choice was basically between Zulip and Discourse. It’s a little sad how mailing list infrastructure hasn’t kept up with the times.

                                                        1. 3

                                                          I’m not a huge fan of the “new” style of collaboration (Slack, MSFT Teams). They try to combine the immediacy of real-time chat with some sort of structured discussion format, but there’s an implied pressure to provide an answer immediately. In a mailing list, it’s not expected for someone to reply to a message within minutes, and I believe that for most projects this is a better expectation - if nothing else than people are in different timezones.

                                                          1. 4

                                                            I’ve used instant messaging in different ways for 20 years, and I always felt that those tools are fundamentally asynchronous, that the answer is never expected immediately. Perhaps it would help if you never enable notifications for those tools? I never do, except for the tools that I use for personal communication with family.

                                                            1. 4

                                                              there’s an implied pressure to provide an answer immediately

                                                              Maybe it’s better with good threading support since the messages don’t get lost as much and it’s less an issue if you reply later.

                                                              1. 3

                                                                I feel like there needs to be adequate division between channels of communication in order for it to work successfully. A bad example is Discord: a lot of servers will have a #general channel where there are often two or three conversations going on at the same time, and replying to someone’s question more than a few minutes later seems futile.

                                                                Having used Zulip for a few months at work, I feel like the named subtopics within streams can handle slower conversations, but it might be a matter of corralling the userbase away from these catch-all channels.

                                                                1. 2

                                                                  Well, a few people had asked for an IRC channel, which I haven’t had success with. So I hope to kill 2 birds with one stone here. It’s an experiment.

                                                                  1. 2

                                                                    Please report your findings (maybe a blog post?) at some point. I’m super interested to hear how it goes, and I imagine other folks would be too.

                                                            1. 2

                                                              I’m interested! I did a tiny bit of CUDA a few years ago. Looking at the code, I don’t know what thrust is.

                                                              Unescaping doesn’t seem parallelizable because you don’t know ahead of time where \ is. Or are you saying you can compute it both ways in parallel and then assemble the right pieces later? I’m interested in details.

                                                              1. 4

                                                                Yes, it’s the latter. There’s a little state machine that’s small enough you can compute it in all initial states, then assemble the pieces together hierarchically, using a prefix sum. Sounds like there’s enough interest, I will write this up in more detail.

                                                                1. 1

                                                                  OK that roughly makes sense, but yes I’d like to hear details! Some questions:

                                                                  • Does thrust::transform_inclusive_scan have anything to do with parallelism? From your description, it sounds like the version you posted is serial, but the next step is to parallelize it with “tiles”? (I just looked it up and learned that transform_inclusive_scan is a C++17 thing)
                                                                  • Why is it faster, if you’re counting the time the round trip time to the GPU? Or does it make sense to render the result directly?

                                                                  I think the CSV use case is pretty interesting. If you assume a relatively well-formed CSV, it has similar quoting with ". Newlines are also special though.

                                                                  1. 1

                                                                    No, thrust::transform_inclusive_scan is already parallel (because it make the additional assumption that the function being scanned is associative, which is the case here). I think the reason why performance is not astonishing is that it’s spending a lot of time reading and writing global memory, which is expensive on a GPU. If it were possible to reorganize the computation so that most of it was in shared memory, I think it would be quite a bit faster still. But that’s hard.

                                                                    1. 1

                                                                      OK, that makes sense. I think I get the prefix scan thing now.

                                                                      I’ve had to implement backslash-type escaping (and unescaping) in at least 5 places in http://www.oilshell.org/ , and this style is something I hadn’t considered. I use re2c in a few places, which generates DFAs as switch/goto, rather than looping over characters by hand.

                                                                      Maybe for fun I’ll play around with SIMD vs. C loops. I don’t think a compiler would/could know to use SIMD in these cases without the STL style.

                                                                      I think the problem is that the number of DFA states explodes quickly for other string processing problems, but for backslash escaping it seems managable, and the code doesn’t seem more complicated.

                                                                      Although, for JSON, don’t you have to decode \u1234, which leads to a lot more states than just \n and \" ? Also, C/Python/shell as respect 8 hex digits like \u123456789 as far as I know (but apparently JSON doesn’t).

                                                                      1. 2

                                                                        Right, I’m only detecting backslashes here, not mapping their decoding. But all the other stuff can be done with a small finite window of input, which is pretty easy as far as GPU goes. The problem with backslashes and quotes is that, for arbitrary input, a one-byte change at one location can have an effect at another location arbitrarily later.

                                                                        JSON doesn’t respect unicode escapes larger than 4 hex digits, so uses the rather egregious hack of escaping the UTF-16 encoding.

                                                                        Agree that this technique doesn’t work well for even medium-size, much less actually large automata.

                                                                        1. 1

                                                                          OK it makes sense that you could break it up into 2 separate steps. But I wonder what the performance implications of that are.

                                                                          And it seems like there is still a serial part, because after unescaping you don’t know what position each part will end up at. The copy_if part of the algorithm calls keep, which is data-dependent and I think must be done serially.

                                                                          So I think you have two parallel passes and one serial pass to decode a string. I can imagine it’s a win in certain situations but it’s hard reason about.

                                                                          In the shell, I thought about using a lazy tree representation of strings, where you only concatenate segments if someone actually uses a string as argv – i.e. it needs to be passed to a system call. But it does introduce a lot of complexity, whereas copying small buffers is something computers are very good at. I haven’t gotten to the point where this matters yet.

                                                                          Anyway, I’m looking forward to the blog post!

                                                                          1. 2

                                                                            Thanks for the continued encouragement to write. As it turns out, copy_if is also parallel, but uses expensive global memory. If any pass were strictly serial, it would absolutely not be worth doing this on GPU.

                                                                2. 2

                                                                  FYI, here is a Thrust homepage.

                                                                1. 2

                                                                  I was thinking about this the other day as I FINALLY ditched my ancient Lincoln Stein CGI.pm book.

                                                                  So today I googled it and this came up. Kind of impressive that it survived in the perl distro until just 3 years ago! That’s incredible longevity for that kind of software IMO.

                                                                  1. 2

                                                                    Huh? But why would they remove it ? OK, it’s old. That doesn’t mean it doesn’t work, and it has precisely the problem that he’s pointing out – scripts will break on old web hosts. I assume those hosts will notice and simply backport CGI.pm.

                                                                    There are good technical reasons for this. CGI is a dying technology. In 2015, there are far better ways to write web applications in Perl. We don’t want to be seen to encourage the use of a technology which no-one should be using.

                                                                    This is not a good reason.

                                                                    EDIT: I found this in the comments: https://news.ycombinator.com/item?id=10757628

                                                                    I think that is pretty dumb, but I’m not interested in rehashing a 3-year-old argument for a language I don’t use :-)

                                                                    1. 1

                                                                      Good point! Are there any security ramifications that make CGI less desirable than any of the mod_blah methods?

                                                                      The copyright on the source is 1995. So, it took 20 years for this module to be removed. That’s a damn impressive run in technology terms :)

                                                                  1. 3

                                                                    Yup, I was surprised by that. I was watching a video in an app and the app crashed because of the auto-update.

                                                                    1. 1

                                                                      What Android version, manufacturer, and phone network do you have?

                                                                    1. 4

                                                                      So this can actually run unmodified CPython and R code, complete with extensions in C/C++/Fortran?

                                                                      Or is it like the JVM versions of Python and R, which don’t have the same native extension ecosystem?

                                                                      I think it’s the former, since they say they support LLVM-based languages. They probably don’t run Fortran, which is important for R code. But I think that is a big deal because C extensions are what is holding back PyPy.

                                                                      Interested in thoughts from anyone who has tried it.

                                                                      1. 7

                                                                        Yup, it can! Sulong indeed supports C/C++/Fortran: https://github.com/graalvm/sulong

                                                                        There’re discussions about Sulong adding support for languages like Rust, Swift, etc. as well.

                                                                        1. 3

                                                                          In theory, NVIDIA is writing Flang which compiles Fortran to LLVM. I don’t know whether Flang supports Fortran code important for R well enough.

                                                                        1. 3

                                                                          Nice blog posts on this site! This one reminds me of my 100 line “challenge” last week …


                                                                          1. 2

                                                                            Well, I think you could say there’s a top-most 100 lines of code in every compiler, somewhere. I do think some are better than others, but I’d guess familiarity has something to do with it.

                                                                            Are we counting LISP interpreters? I especially like 1) kanaka’s Make a Lisp (written in many languages, like CoffeeScript), 2) the Elm compiler, which is in Haskell, and also why not 3) the SpiderMonkey bytecode compiler, which is in C++, so it’s not going to be 99loc but give some respect.

                                                                            1. 1

                                                                              No, as I said, if it has a pyramid structure, and not a flat structure, it doesn’t count. The pyramid means you have to search around the code to see what the compiler does; flat means you can see it all at once.

                                                                              (And yes I made up the rules to this game according to my own preferences.)