1. 24

This is the weekly thread to discuss what you have done recently and are working on this week.

Be descriptive, and don’t hesitate to ask for help!


  2. 21

    This week my first child will be born. I expect lots of work.

    I plan to rework event loop in my experimental text editor project. I plan to have single poll in a loop and do file I/O in forks (using shared mmap as a buffer and syncing using a pipe and a POSIX semaphore). I don’t want additional dependency on heavier event loop libraries like libuv.

    I plan to explore implementing zero copy pipes for a batch image manipulation experiment. I’m thinking about doing something like this:

    name = mktemp("pipe.XXXXXX")
    fd = shm_open(name)
    buf = mmap(0, imgsize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0)
    if(!fork()) {
        dup2(1, ctl[1])
        execlp("filter", devfd_filename(fd), ...)

    On Linux shm_open could be replaced by memfd probably. The ctl pipe would be used for signaling which parts of buffer are ready.

    I don’t know if it’s sensible idea. I would like to have easy composition of filters written in whatever programming language. Also the idea of zero-copy pipes is pleasant for the thought.

    The separate thing is image manipulation idea itself. Many times when I am doing some graphics/layout work using various tools (GIMP, Inkscape, LO Writer, LO Draw, HTML+CSS) I constantly struggle. I would like to process graphics using programming language, but with ability to input parameters with mouse. It would have to be graphical environment, so that I can immediately see the results of changing parameters. Also when the work is done I would like to be able to assign same manipulations to different images.

    Simple use case: enhancing photos of presentation slides. You have N photos of slides given on a presentation. I would like to place a quad by pointing mouse at four corners of the projection screen and giving aspect ratio (where I could also select from a list: 4:3, 16:9, A4,etc.). I would like to also enhance contrast and make photos black-and-white so I could print them.

    1. 11

      Congrats on the upcoming fork/exec. And yes, it is a lot of work.

      1. 2

        Congrats! Mine is due in 2 weeks!

      2. 9

        I’m putting the finishing touches on my weekend project: F5Bot. It’s a real simple service that monitors Lobsters, Hacker News, and Reddit for keywords. It emails you when it finds something. I’m just working on making the error handling and logging a bit more robust now.

        1. 8

          writing a log-structured lock-free persistent B+ tree in rust! eventually I’ll plug it into my long-dormant rasputin linearizable db and have a 100% rust high performance linearizable distributed database / log / block store :)

          1. 7

            Today is the first day of my (willfully) being unemployed. I’ve decided that for the first two weeks I’m going to do whatever I feel like and not feel bad about switching between things as much as I want. Although today is probably going to be mostly cleaning and figuring what I need to do about health insurance.

            With the rest of the week, I’m considering continuing with my exploration of containers which I dropped when I was getting serious about quitting. I’m also considering starting learning about this whole machine learning thing. I took a class on it in school, but didn’t really learn anything, so I want go back through things and actually understand some of it. And then there’s half a dozen other projects that I’ve started that I’m considering picking up again.

            The goal with all that is to actually pick one of my projects and finish it. I’ve got a serious problem with not finishing projects and I’m pretty sure if I actually finish one of them the satisfaction of that will help me find the will finish more. We’ll see how well that works out.

            1. 4

              Oh, yeah, and I’m getting married tomorrow, I forgot that one.

              1. 2

                I’m not sure what kind of machine learning you’re interested in, but this class on computer vision is pretty good so far: http://course.fast.ai/. It is very hands-on.

                1. 1

                  I’m not sure what kind of machine learning I’m interested either, so thanks! I’ll check it out.

              2. 6

                Rebuilding my personal infrastructure (at home and hosted) using Ansible. I’m creating all of my own playbooks, roles, etc. so it’s going slowly but once it’s done my management workload should drop appreciably. I currently have far too many snowflake servers but that’ll soon be a thing of the past.

                1. 2

                  What level of home/personal severs do you have set up that Ansible has become desirable for them?

                  Also, what do you think about Ansible vs stuff like Puppet or Chef?

                  1. 6

                    I don’t have as many as my post possibly suggested - maybe ~30 servers across a mix of physical and virtual (and a mix of operating systems - mostly Debian, but also OpenBSD, NetBSD [soon to be decommissioned] and FreeBSD). I also have a few Windows Server VMs.

                    I host all of my own Internet services - DNS, mail (SMTP and IMAP), web sites for myself and family members, VPN, etc - as well as having lots of services at home (firewall, DNS, NTP, DHCP, Samba, web proxy, backups, home automation system, etc) and maintenance is becoming more and more onerous. I want to get rid of stuff like having to ssh into 10 servers to install the same package on each. Yes, there are things like Cluster SSH but I want to do things “properly” (and logging in via ssh and doing things manually doesn’t fall into that category in my book).

                    I don’t have any hands-on experience with Chef but I’ve used Puppet lightly in the past. I’m no Puppet expert so any comparison between it and Ansible wouldn’t do either of them any favours. I’ll just say that I rather like Ansible - the fact that it’s agentless is great and the relative simplicity is a plus in my book. Also, my Python skills are far better than my (rather rusty) Ruby skills. Sadly, I have no bow hunting skills.

                    1. 4

                      20-30 servers definitely seems like enough to start looking into tools to make managing them more pleasant, especially since it looks like you run enough services to host a small business.

                      As someone that has a minor ops interest (I run a server for a blog and some version control stuff, and have been doing it the hacky-manual way that one get away with when you only have a single server), Ansible does look interesting.

                      1. 4

                        There’s benefit to using a configuration management tool even if you only have a single public-facing server. Reproduceable deploys, everything in version control, etc. I still look after a few WordPress blogs for family members that are not yet managed in my own VCS repo and they’re a nightmare to look after.

                        As part of the Ansible deployment I’ll be cleaning up all of that. Also, doing things like migrating an Apache installation to nginx, deploying DNSSEC for my domains (and all that goes with that, including DANE), deploying Lets Encrypt for a few sites, etc. All in all, quite a lot of work ahead of me!

                        1. 3

                          True enough. I don’t think how I’ve done it is ideal, it’s just where I am after some years of it being a minor side project. I use it to host https://junglecoder.com and I’ve used that server as an excuse to learn Go, Bash and a few other things. If I can find the time/energy, I may put Ansible or something like that next on the list.

                          1. 3

                            Side projects are great for learning and it’s perfectly understandable that, as they grow organically, maintenance becomes more and more of an overhead. That’s what’s happened with me - I like running my own stuff but the sysadmin overhead is so high that it’s taking all of the time I should be working on proper side projects.

                            Ansible is pretty quick to get started with and the tutorials are good. Be warned that some older content online doesn’t follow current good practise - some older options are deprecated, etc. Perfectly understandable, but I’d suggest reading the “Best Practises” guide before you get started seriously.

                            1. 2

                              I suggest picking an ops / deployment / setup task that you at least kind of want to do anyway and learn Ansible by automating it as you go.

                              I still don’t sysadmin any of my own stuff, but I now have some familiarity with Ansible and have a set of playbooks for deploying a web-based RSS reader, setting up the services / accounts it depends on, and taking / restoring backups from it.

                  2. 6

                    Finishing up the move, both physically (unpacking the last boxes, getting the garage sorted, taking the giant box of computer crap to the basement) and virtually (getting Nix and python to play better in my work VM; getting Emacs to play better with Windows). I’d really like to blog again (at all?) but I find the thought of HTML and CSS really, really wearying. I used to think that my issue was yak shaving, but really, it’s that I sort of hate software in all of its myriad and grotesque forms, and the prospect of introducing more of it into my life fills me with existential … not dread, per se, more lassitude and ennui.

                    At work, I have a big, horrible data migration to shepherd through this week, which is awful, but as the problem it’s intended to solve was a fuckup on my part, it’s only fair?

                    1. 5

                      Decided I really should sort my offsite backups out, so grabbed another hetzner box. This time I’m determined to get SmartOS working with NAT in the global zone, and also start backing up some ZFS volumes from the home server.

                      Started tinkering with some twitter bots over the weekend, about halfway through a script to power them. Turns out maths with dates is harder than I remembered, lets see if I persevere this week or not.

                      1. 1

                        Never heard about SmartOS and you got me curious. Why SmartOS and not Linux+docker or VMWare ESX, which Hetzner offers too?

                        1. 3

                          I’d not come across it before I started using it at work (due to the chap who started the Operations team having used it at a previous place.) Since picking it up in the last four years I’ve really come to like it, it’s very well thought out (coming from Solaris back in the day) and has had things like containers for years. I particularly like SmartOS for servers because it runs the system from a ramdisk, so upgrading is just splat out a new platform folder (generally on a USB stick) and reboot.

                          Their Triton cloud management stuff is pretty neat too, far too much overhead for a single server (it’s intended for running a datacenter), but we’re starting to move across to that at work and it’s solving a bunch of issues we’ve got with managing zones across multiple servers.

                          Also amuses that things like KVM security issues are less of a concern, because instead of bolting KVM into SmartOS Zones as a separate zone type, it actually boots the KVM process within a SmartOS zone. So if you managed to break out of the KVM instance into the “hypervisor” layer, you’re still stuck in a native OS zone and not in the global zone of the physical server still. Neat!

                      2. 5

                        Going to port the AVI player I wrote the week before to Mezzano OS. Also, change the worn out leadscrew on my lathe.

                        1. 5

                          I got a YubiKey over the weekend. It’s a really cool device and, sadly, not as many places can use it as I hoped. So far I’ve only be able to tie it to my Gmail and my Github accounts. I have a feeling I’m not even close to making full use of it though. If anyone has any helpful advice on how to make better use of it, I’m all ears.

                          1. 3

                            I LOVE my YubiKey (got a bunch at BlackHat USA 2015). I have three - one 4 Nano in my desktop at home, one on my keychain, and one in a fireproof safe. I use Lastpass for all my emails and it’s behind my Yubikey, but I still use Google Authenticator for a bunch of two-factor sites (except Github, Fastmail, and Google). I do wish more people implemented it as a second factor, but having it on my Lastpass at least makes me feel safer.

                            1. 2

                              I’ve had a YubiKey for a while now but not really used it enough - I’m pretty much only using it for GitHub right now. I’ve been meaning to do all sorts of things (GPG key storage, system logins) but just not had the tuits. Would be curious to hear what others are doing.

                              BTW, make sure you get a second one, set it up as a backup and store it somewhere safe - losing your key and getting locked out of services would be a Bad Thing.

                            2. 4

                              Working on tutorials and documentation for Myrddin, as preparation for a first release.

                              1. 4

                                Cracked open things again on my type-level GraphQL clone in Scala.


                                I’d like to start moving to a C4 like process, so co-maintainers desired even though the library is at this point barely functional.

                                The draw is to get the same API design advantages GraphQL gives you but (a) have the types align better with Scala and, more generally, the functional/well-typed/ML-like type languages and (b) have a lot of type-level computation so that most incorrect code is impossible to write and much of the boilerplate is autogenerated.


                                Current motion is to get a features-lite first release which could be functionally used in an API before adding some very necessary features like argument parameterization of fields, result multiplicity, and a good treatment of errors. After that, I’d like to see about standardizing the wire protocol and building out a Haskell version.

                                1. 4

                                  Fleshing out the runtime for Simulscript, a new language I’m working on (https://github.com/simulscript/simulscript)

                                  The idea behind Simulscript is to run a set of commands in a script in lockstep across a set of target hosts for both configuration management and adhoc tasks. I plan on having first class support for remote command execution via bash over ssh for Unixy hosts and via WinRM for window’s hosts. I’m building it in TypeScript.

                                  1. 4

                                    Last week a power distribution unit (PDU) in one of my racks failed, leading to complete loss of power:

                                    On Tuesday February 07, 2017 at 17:02 -0800 we experienced a Power Distribution Unit (PDU) failure, leading to loss of power in one of our racks. We do not know the root cause as we have not yet found any bad equipment other than some blown fuses, though there is one piece of equipment we haven’t been able to test yet and we have not opened up the old PDU to inspect the circuit boards for overheating.

                                    We replaced the failed PDU and started bringing customers up at 19:48 and finished by 21:00. Complimentary colocation users were brought up Thursday 09 February, as we needed time to vet their equipment.

                                    While there is always something to learn from a failure of this magnitude, my favorite was having my publishing infrastructure down but my website up. This meant I hand edited HTML on production to notify users of downtime instead of pushing a blog entry.

                                    I’m hoping for a quieter week. I’d like to go to SCALE 15x in the first week of March.

                                    1. 3

                                      Support for Fuse in Elm. Working on making Elm infrastructure around Native code easier to work with.

                                      1. 3

                                        Experimenting a bit more with Red language.

                                        Initially the linking step was very slow, taking four whole minutes to link! But, after I went to file a bug on the GitHub repo I could no longer reproduce the issue. Now it compiles and links very quickly.

                                        I have an idea for a small project I’d like to implement, but Red/system’s C interface isn’t very complete. It supports pointers to cstrings, but not in-place fixed-length character arrays. I might have to write a small shim in C, then load that dll from Red/system.

                                        1. 3

                                          This week I’ve done a major new release of Peergos (It’s still alpha though) We hope that it is future-proof now from a data structure point of view. It uses IPFS for storage and P2P sharing.

                                          Any feedback is much appreciated. It’s meant to be a peer-to-peer file storage, sharing and social network, with fine grained control over who can see your stuff.

                                          There have been lots of fun problems to solve along the way. Most of our complex code is compiled to JS from Java using GWT, with some nifty tricks to use the JNI interface to call native JS libraries, and JsInterop to allow JS to call Java methods - no way we could ever write this in JS directly without taking 10X longer. Using Vue.js for the user interface has been an unexpected joy.

                                          1. 2

                                            Should I make a libfuse wrapper for Perl 6?

                                            1. 2

                                              I’m working on writing a baseball projection system for 2017! I’m also documenting the journey on my blog, where I put up the intro post yesterday if you’re interested in my approach.

                                              1. 2

                                                Studying for a trio of midterm examinations at uni: one is a deep-learning for NLP course[1], the other is one on static analysis for compiler optimizations[2], and the third is a databases course[3].