There are some other things, for example, the option to manage your process via web JSON API https://immortal.run/post/nginx/ if required that may be handy mainly because you have full control of your process via web, so for testing environments, the development team don’t even need to ssh the machine.
The idea of using YAML was to simplify since many operators working with ansible/salt are used to, for example, this is a naive playbook https://immortal.run/post/ansible/ no need to link/unlink files to stop start services.
And well the login option that gives you the option to chose your own logger, for example, you could pipe directly to filebeat and also write to disk if desired, all this within a single configuration file, meaning that for full automated environments (CI/CD) once you have immortaldir(8) https://immortal.run/post/immortaldir/ is just a matter of updating a file.
The current documentation maybe is not in its best shape, but any feedback/help testing suggestions are welcome.
What is the behavior if the --logger process crashes?
As I recall, daemontools handles this fairly well. It restarts the logger (./log/run) process and wires up the pipe fd again.
if the logger crashes it will restart it and attached to the writer again: https://github.com/immortal/multiwriter, without affecting the local logs in case local, and remote wanted.
I’ve recently been wanting, and I haven’t really seen any process supervisor solution that does:
Ordered and structured start/restart of unix process collections following an erlang like supervisor tree model. i.e. attempt to restart this collection of services at most N times in M minutes in a specific startup/shutdown order, if this fails revert to parent supervisor tree which is something similar. If all supervisor tree’s totally fail, the root node would be email operations and reboot machine + try again.
I think things would get interesting with this type of supervisor tree when process nodes start becoming ssh commands onto remote nodes and you add hardware watchdogs into the mix.
Hi, check the require option here https://immortal.run/post/run.yml/ basically allows defining a set of services that need to be up and running before starting the services, this allows to chain services and avoid race conditions in some cases.
I don’t see how you can use signalfd(2) for this, kqueue(2)s EVFILT_PROC is something completely different.
The CONFIG_PROC_EVENT and CONFIG_CONNECTOR stuff on linux would be comparable.
What is the idea behind EVFILT_PROC? I assumed that because kqueue is for async I/O, it delivers notice of process completion (SIGCHLD) over a file descriptor.
When you use the self-pipe trick, you’re registering a signal handler that delivers process completion notification to your event loop via a file descriptor. It’s basically unification of waiting for I/O events and waiting for signals.
OK, but I still fail to see anything new in immortal or anywhere else. All these kernel features were added explicitly for this use case – waiting simultaneously for file descriptors and process completion. In other words, more naturally integrating event loops and process completion events.
Some high level async APIs have missed this, since they were geared toward networking, like Tornado. But Twisted has had it for decades, and node.js has had it since the beginning too. libuv must do it under the hood.
What was your idea behind :-) maybe we could implemented it, currently I found very nice to use Kqueue when following the PID mainly because I don’t have to do any long pooling for checking the status of the PID, but curious to know about what more could be improved/implemented.
I’m using kqueue NOTETRACK and NOTEFORK to watch for all the children. Doesn’t look like immortal does that? If I kill a daemon, I (sometimes) want to kill all the forked processes.
Right, Immortal is only using NOTE_EXIT to check the status of a process that was not originally created by immortal, basically only used when using the -f option to follow the pid.
Is this runit for people who want yaml files? What am I missing?
There are some other things, for example, the option to manage your process via web JSON API https://immortal.run/post/nginx/ if required that may be handy mainly because you have full control of your process via web, so for testing environments, the development team don’t even need to ssh the machine.
The idea of using YAML was to simplify since many operators working with ansible/salt are used to, for example, this is a naive playbook https://immortal.run/post/ansible/ no need to link/unlink files to stop start services.
And well the login option that gives you the option to chose your own logger, for example, you could pipe directly to filebeat and also write to disk if desired, all this within a single configuration file, meaning that for full automated environments (CI/CD) once you have immortaldir(8) https://immortal.run/post/immortaldir/ is just a matter of updating a file.
The current documentation maybe is not in its best shape, but any feedback/help testing suggestions are welcome.
What is the behavior if the
--loggerprocess crashes?As I recall, daemontools handles this fairly well. It restarts the logger (
./log/run) process and wires up the pipe fd again.I would say that is was the main motivation behind creating immortal, mainly because was a pain to log and there were some bugs like within runit: http://serverfault.com/q/752839/94862 (the patch was submitted after 1 year https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207747)
if the logger crashes it will restart it and attached to the writer again: https://github.com/immortal/multiwriter, without affecting the local logs in case local, and remote wanted.
I’ve recently been wanting, and I haven’t really seen any process supervisor solution that does:
Ordered and structured start/restart of unix process collections following an erlang like supervisor tree model. i.e. attempt to restart this collection of services at most N times in M minutes in a specific startup/shutdown order, if this fails revert to parent supervisor tree which is something similar. If all supervisor tree’s totally fail, the root node would be email operations and reboot machine + try again.
I think things would get interesting with this type of supervisor tree when process nodes start becoming ssh commands onto remote nodes and you add hardware watchdogs into the mix.
Hi, check the
requireoption here https://immortal.run/post/run.yml/ basically allows defining a set of services that need to be up and running before starting the services, this allows to chain services and avoid race conditions in some cases.Somebody stole my idea! (To use kqueue to track processes.)
Where do they talk about that?
By “track processes”, do you mean to get notified of process completion?
See for instance. https://github.com/immortal/immortal/blob/master/watchpid.go
Mentioned in the manual: https://immortal.run/post/immortal/
But they don’t really do everything I was imagining, so my plan is still safe. :)
OK, but isn’t the entire point of that kqueue feature to track process completion? I don’t understand what “your idea” is.
On Linux you can do this with signalfd: http://man7.org/linux/man-pages/man2/signalfd.2.html
But as far as I know, the self-pipe trick works just fine if you only assume POSIX APIs: https://cr.yp.to/docs/selfpipe.html
I don’t see how you can use
signalfd(2)for this,kqueue(2)sEVFILT_PROCis something completely different. TheCONFIG_PROC_EVENTandCONFIG_CONNECTORstuff on linux would be comparable.What is the idea behind EVFILT_PROC? I assumed that because kqueue is for async I/O, it delivers notice of process completion (SIGCHLD) over a file descriptor.
When you use the self-pipe trick, you’re registering a signal handler that delivers process completion notification to your event loop via a file descriptor. It’s basically unification of waiting for I/O events and waiting for signals.
The proc filter returns events for processes, but there’s no file descriptor involved.
OK, but I still fail to see anything new in immortal or anywhere else. All these kernel features were added explicitly for this use case – waiting simultaneously for file descriptors and process completion. In other words, more naturally integrating event loops and process completion events.
Some high level async APIs have missed this, since they were geared toward networking, like Tornado. But Twisted has had it for decades, and node.js has had it since the beginning too. libuv must do it under the hood.
Looking at it again, it doesn’t even do supervision like its known from daemontools, calling it “supervisor” is confusing. Thats why it can’t use
SIGCHLDand why it useskqueue(2)sNOTE_EXIT, just look at the linuxWatchPidfunction which just shows the completely uselessness. https://github.com/immortal/immortal/blob/a05a0dddc688902653d092bf8d3826a3cd1d0977/watchpid_linux.gokqueue(2)sEVFILT_PROCcan be used if the service “daemonizes” which makes real supervision impossible.edit: maybe it does daemontools like supervision, https://github.com/immortal/immortal/blob/08e38b25b81de3650bfe99a2bb782beaaa40b91b/process.go#L100 its hard to follow the code.
What was your idea behind :-) maybe we could implemented it, currently I found very nice to use Kqueue when following the PID mainly because I don’t have to do any long pooling for checking the status of the PID, but curious to know about what more could be improved/implemented.
I’m using kqueue NOTETRACK and NOTEFORK to watch for all the children. Doesn’t look like immortal does that? If I kill a daemon, I (sometimes) want to kill all the forked processes.
Right, Immortal is only using NOTE_EXIT to check the status of a process that was not originally created by immortal, basically only used when using the
-foption to follow the pid.