1. 3

    I always wondered why Colin was involved in FreeBSD over OpenBSD (seeing how he’s into security, and is Canadian being banal reasons why he may champion the later). One reason he states right off the top of the show is FreeBSD was easy for him to install!

    1. 4

      In new OTP releases there is handle_continue/2 for that and you do not need to utilise such dirty hacks.

      1. 2

        Thank-you, @hauleth. I added a note about this on the post.

        1. 1

          Dependent of course on what I am doing, I tend to just have my handle_continue pass directly to handle_cast which lines things up for me to use handle_cast(hup, State) as a generic point to handle reloads throughout the lifetime of my processes.

          Before continue was an option, I just used to just call gen_server:cast(?MODULE, hup) before my init function returned.

          1. 1

            The problem with message to self is that it causes race condition. If anything send message after spawn but before such message then server can receive messages while in wrong state. handle_continue avoids that as it is ran just after the message is processed but before next loop call.

            1. 1

              Why would you want init to return immediately but then not actually be ready to service requests and care about this race condition?

              1. 1

                Because long running init will stop launching other processes in the supervision tree. So sometimes you want to continue launching while still prepping the process and queue all messages that will arrive in the meantime. This is where handle_continue becomes handy.

                1. 1

                  I think you need to go back and read my comments and bear in mind that the comments are about what the article is setting out to solve and not telling you what you should be doing.

                  The whole point of supervision is that it expects after your init returns your process is in a stable state. If you are using tricks (continue, gen_server:cast() from init, …) to speed up your startup time you have officially declared “I do not care about supervison” as your now out of bound long init task could still lead to a crash of that process.

                  Your point of a race condition is correct but is replacing it with an unstable supervision tree just to get a faster startup times something that is correct for everyone?

                  Either you think everyone should do this, or (more likely) you have not read or taken onboard:

                  • before continue was an option…”
                  • “Dependent of course on what I am doing…”
                  • continue is a recent OTP feature (stated by yourself)
                  • continue is not available in all generic servers (stated by the author)

                  So beating up on me to use continue when it may not be an option is not really fair, right?

                  Throwing out a stable supervision tree may be okay and manageable in your environment. The race condition you describe is correct but has zero impact on my life as either I arrange it so dropping those messages is okay (two lines in a gen_server) or alternatively expose ‘readiness’ to a load balancer (where startup times are often irrelevant) so making the race a non-issue.

                  I suspect others may have their own thoughts here, and I am interested in hearing from them. What I do not appreciate is being told to swap a race condition for a unstable supervision tree whilst being told to suck eggs.

                  1. 1

                    The obvious case, to me, is starting up a process/service that is inherently flaky, e.g. a database connection. The “stable state” of the process is disconnected, and whenever it is disconnected it is attempting to establish a connection/reconnect. While in that disconnected/trying to connect state, it can still service requests from clients, by simply returning an error. The rest of the system can proceed with startup, even if the database service was temporarily unavailable, or otherwise flaky during boot.

                    This is especially important for resilience in the supervision tree. Using the example from above, if your process tries to establish a connection during init, and it fails, it will likely fail again immediately, probably triggering the max restarts of the supervisor and bringing down large swaths of the application (or in the worst case, the entire node). The same applies later on in the nodes life, should the database connection fail, and trigger a restart, if the connection cannot be immediately re-established, then it is very likely that the process will restart in a loop, trigger the max restarts, and take down a bunch of stuff it shouldn’t have impacted.

                    The idea of doing that fragile work post-init is not to boot faster, it is to boot stable, and to ensure that restarts during the application lifecycle aren’t prone to the crash loop problem I described above.

                    1. 1

                      As database disconnects are ‘normal’ and expected (not exceptional) though this sounds like the stable process here describes a behaviour similar to a simple_one_for_one supervisor (where you do not care but if you did you would instead use one_for_one)?

                      I still need to fix my Erlang as I am still ‘broken’ in the manner that I prefer a SIGHUP-via-cast approach over process restarting as I am yet to see a simple way of handling multiple reload requests without a global lock; though I guess you could drain your message queue with a pattern for only reload requests and debounce the whole system that way before exiting?

                      1. 1

                        As database disconnects are ‘normal’ and expected (not exceptional) though this sounds like the stable process here describes a behaviour similar to a simple_one_for_one supervisor

                        The example of a database connection was illustrative, not meant as the best example, as typically they are created on demand and so would not be restarted, or would run under a simple_one_for_one supervisor. However the point it was meant to demonstrate is that some processes/services that have external dependencies are by definition unreliable; as you point out, this is part of their “normal” behavioral profile, so it is critical that the process handles the flakiness gracefully. An alternative example might be an S3/RabbitMQ/etc. consumer/producer - the same rules apply, if you require init to block on successfully connecting to the service, it will bite you eventually, and adds fragility where it isn’t necessary.

                        I still need to fix my Erlang as I am still ‘broken’ in the manner that I prefer a SIGHUP-via-cast approach over process restarting as I am yet to see a simple way of handling multiple reload requests without a global lock

                        I’m not sure I follow, they are essentially orthagonal concerns right? You can have supervision/process restarts and provide the ability to do some kind of “soft restart” via a SIGHUP-like message sent to the process. The part I think I’m missing here is what you mean by handling multiple reload requests - for a single process the requests are all serialized anyway, so if they trigger an actual process restart, then only the first message matters. If you are doing some kind of soft restart, then you could “debounce” them by dropping duplicate restart requests that come within some window of time (no need to drain the mailbox with a selective receive). In general though, I would assume the SIGHUP-like signal you are sending is something triggered by an operator, not by the system, in which case handling that edge case seems like worrying about something that is unlikely to ever happen in practice. Maybe an example of where you are using this pattern would help illustrate the problem it solves though.

        1. 1

          It may seem trivial, but the OTP team is chipping away at making the Erlang syntax a bit dev-friendlier with this major release. With R23, integers can take the syntax: 1_000_000 to represent `1000000.’ Considering how much back-lash Erlang syntax gets, the more efforts like this, the better.

          1. 2

            If the syntax is somebody’s primary concern why not use Elixir instead? For me the Erlang syntax makes more sense than Javascript for example. I think Joe really got that right. Expressing so many things so easily. Pattern matching baked into the language the the core, ease of use of recursive functions. I could go on and on. Do people get hang up on the syntax that much?

            1. 1

              Yes

          1. 2

            A link to a pdf version of Shannon’s paper can be found via here:

            https://twitter.com/snailtext/status/1254054254869770240

            1. 1

              I’m sure I’m just cross-eyed right now, but would someone that understands this explain the mechanism for a commercial company getting hit financially; if they don’t get on board with EARN IT? Is it a fee? THANKS in advance!!

              1. 8

                I’ve been thinking about state machines in the last couple days. Specifically, I think Erlang’s gen_statem pattern (from OTP) is the 8th wonder of the world. The author of this behavior tackled a huge problem and I’ll be damned, hit it out of the park, considering how muddled one can get when thinking about state machines, let alone coding them up. I did a shallow dive into it a good while back (http://blog.snailtext.com/posts/bend-it-like-gen-statem.html), which isn’t bad. OK, not Rust related, but gen_statem certainly is something to marvel at in this space.

                1. 1

                  Hey thanks to much for the link to your post! I’m an Elixir fellow myself, so I’ll happily digest what you’ve written.

                  1. 1

                    Nice blog post, and also a v. nice website design!

                    If you’d enjoy it, could you spend a few words gushing about the things that gen_statem gets right that other state machine implementations get wrong, or don’t get at all? I love reading enthusiastic praises, and also I’m curious about the design space of the interfaces of state machine libraries.

                    1. 1

                      I regularly tell Rust people to learn about FUCRS, since this blog really got me thinking a lot about techniques for combating rightward pressure when I was working more with Erlang :) gen_statem and generally all of the behaviors are particularly nice IMO because they push all of their state in your face during transitions, and they keep the reality of the system more on your mind. For a dynamic language, erlang is able to specify so many beautiful properties that can be tested in a way that kind of approaches a much more usable dependent type system in some cases, with the features of dialyzer and general amenability to model-based testing that gets bugs to fall out of a system so quickly. Working in erlang has significantly impacted how I write Rust, and especially how I try to lean into state machine-based approaches for anything distributed-systems related.

                    1. 1

                      Host my own email and websites. OpenBSD, OpenSMTP, httpd on small vultr vps’s Home fileserver running freenas on an older desktop.

                      1. 2

                        I also do this (save for Home server freenas). Gmail blocks a good deal of my outgoing mail, but honestly, this has been covered a lot recently, so I won’t rant here about is worth it or not. (Yes, it’s worth the risk of having undelivered email, but not having google handle all my email needs.)

                      1. 5

                        Cowboy is certainly the most entrenched app, and I like it quite a bit. The author is a champ, too. However, I’ve been using and quite like https://github.com/elli-lib/elli in the last few years – just pointing out the option… and I believe elli' is actually small.’ I think they have slightly different goals, but on my stuff, they’re used interchangeably (js fetch calls api endpoints).

                        1. 1

                          I wonder if :W is a typo… (vs just W).

                          1. 2

                            Beautiful naming (got, tog, …)

                            1. 4

                              Just wanted to share a nice method for converting font-types into paths (in svg) that you’d be hard-pressed to accomplish without inkscape:

                              inkscape -T -A tmp-hack.pdf orig-w-custom-fonts.svg

                              inkscape -l final.svg tmp-hack.pdf

                              1. 2

                                One issue with the free tier is Heroku idles one’s instance for a chunk of the day : https://blog.heroku.com/heroku-free-dynos

                                `free dynos are allowed 18 hours awake per 24 hour period’

                                1. 3

                                  Sad, but true, is Joe’s Law : `Frameworks grow in complexity until nobody can use them.’

                                  1. 3

                                    Similar to Zawinski’s Law:

                                    “Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.” Coined by Jamie Zawinski (who called it the “Law of Software Envelopment”) to express his belief that all truly useful programs experience pressure to evolve into toolkits and application platforms (the mailer thing, he says, is just a side effect of that). It is commonly cited, though with widely varying degrees of accuracy.

                                  1. 2

                                    I have begun using PropEr for my Erlang/OTP code, largely because I feel unit testing of routines – routines that tend to be around four lines tall – in a language that has single assignment (SSA), to be quite the Kabuki theatre. Hurdles with this new way of testing, are mostly related to my slow adoption of something quite new. So, not sure if this feedback helps in any shape or form. I can’t stress enough how little unit testing has done for me, with Erlang. So, I’ll probably stick with PropEr for a good long while. Perhaps a resource with the focus along the lines of `Property-based testing for people coming from Unit testing’ would have helped some of the growing pains.

                                    1. 2

                                      Nice [first] post, Simon! Keep them coming.

                                      1. 4

                                        For anyone thinking of trying out Erlang, this is how some of these shortcomings played out for me over the years.

                                        In no particular order:

                                        • if statements/expressions become an anti-pattern; you never use them
                                        • expression termination handling becomes second-nature (thanks vim!)
                                        • one leans on so-called strings less and less; ’tend to use atom constants and bitstrings
                                        • variables at most are reassigned once ; A' gets augmented once becomingA1’ from time to time
                                        • one falls in love with records to the point that they bring a smile to one’s face

                                        This is to say nothing of Erlang’s shortcomings, which from my vantage point, are not on this particular list.

                                        1. 3

                                          What’s Core Erlang, relative to Regular Erlang? I tried googling it, and it was not as useful as I thought it would be

                                          1. 4

                                            it’s a compiler IL. several other beam languages use it as a target, and a lot of tooling (most notably the dialyzer) works directly on it. in the context of the series, iirc it’s the form that the compiler runs a lot of transformations on for optimization.

                                            1. 2

                                              I could be mistaken, but I believe I read in some white paper at some point, that the dialyzer project needed an intermediate format (one that would make their lives easier) in order to proceed. So Core Erlang sprung out those efforts during dialyzer’s infancy at Uppsala University. Feel free to correct me where I’ve mis-remembered.

                                              1. 2

                                                I have since read a bit on Core Erlang, Dialyzer works on the intermediate format directly. The intermediate format is also what Elixir (and other BEAM languages) compile down to, which is why the Dialyzer works at all on them. Really cool.

                                              2. 1

                                                If my understanding is correct, it’s basically an intermediate compilation target between normal Erlang code and what runs on the BEAM.

                                              1. 9

                                                I found much value in this related video posted quite recently :

                                                https://www.youtube.com/watch?v=_FQJEzJ_cQw

                                                1. 1

                                                  One snafu with aliases that I never found a great work-around was using date or cal or anything that is mutable, would be sort of galvanized after that bash/zsh profile was set. One would have to source that profile file to get the current, in my case, time. So, it didn’t work for things like that (which was quite a few aliases). A quick hack to make this work is welcomed, BTW.

                                                  1. 2

                                                    I’m not sure what you mean. I tried to guess, but I ended up with dynamic date values. I’ll attempt to fix a galvanized example you give me…

                                                    hobbes@metalbaby:~/e2-scratch/s2u$ alias foo='echo $(date)'
                                                    hobbes@metalbaby:~/e2-scratch/s2u$ foo
                                                    Fri Jan 19 17:48:51 CST 2018
                                                    hobbes@metalbaby:~/e2-scratch/s2u$ foo
                                                    Fri Jan 19 17:48:53 CST 2018
                                                    hobbes@metalbaby:~/e2-scratch/s2u$ foo
                                                    Fri Jan 19 17:48:54 CST 2018
                                                    hobbes@metalbaby:~/e2-scratch/s2u$ foo
                                                    Fri Jan 19 17:48:55 CST 2018
                                                    
                                                  1. 2

                                                    I think if I were to relive my Erlang education, it would be to go against the severely pushed agenda of `no matter what, use behaviors from the get-go.’ gen_server is a great pattern, but it feels counter to everything functional programming and for that matter concurrent programming is all about. Sure, now knowing how the behaviors are implemented, it all fits, but for the first while, the behaviors felt magical (like OOP), stateful-feeling-ish (like OOP), and imperative (after all, 1. a gen_server is embodied within a single named process, despite that Erlang touts: go ahead, create millions of processes; 2. the most common thing to do with said servers is use =call= which is blocking/synchronous). In the end, of course, use the behaviors, but just blindly using them from day-one is a crappy lead-in to Erlang/OTP, and probably comes with many hidden costs as far as how fast & deep one learns the paradigms of FP and all things OTP.

                                                    1. 1

                                                      I can’t agree more with this feeling. It is, in part, the inspiration behind this blog post. The idea is to first find the problems, try to solve them, figure out they’re always the same ones and somebody before you found them too, and only then using the tools that solve most of the problem for you and write your specific code.