1. 22

This is the weekly thread to discuss what you have done recently and are working on this week.

Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.

  1.  

  2. 22

    Working on polishing up my new backup tool. I set out to solve a set of problems:

    • Client side encryption.
    • Deduplication of similar backups to save space.
    • Efficient server side pruning of backups.
    • Write only mode so a compromised device does not compromise your backups.
    • Work over ssh and use ssh keys for access control.
    • Trivial user interface that integrates well with unix: accept directories, arbitrary streams (e.g. pipe in pgdump) or tar streams.

    approaching something i’m happy with people reviewing and using, though there is lots of testing and stabilization that i want to do.

    1. 11

      This sounds like the tool I’ve been looking for, plus some features I didn’t know I wanted :-D

      1. 3

        Sound’s interesting! Did you, by chance, try borgbackup and could elaborate on the differences between borg and yours? I am not sure about your last point but at least the others all seem to be supported by it as far as I understand.

        1. 4

          I was unsatisfied with borg for a few reasons, which I will probably elaborate more on in a post somewhere, in general I am highlighting ease of use and I think I have a more user friendly design. I will see if anyone agrees with me once I get it out there.

        2. 3

          Write only mode

          Yes!! Thank you!

          I have been so jealous of Borg users for so long, but can’t switch because only Duplicity has this feature.

          1. 1

            Isn’t borg serve --append-only what we are talking about here?

            1. 4

              No. Borg only supports symmetric encryption, and closed the public key encryption issue as wontfix: https://github.com/borgbackup/borg/issues/672

              By implementing public key encryption, you allow data sources to operate in what @ac calls “write only mode”, because if a compromised device only has your public key, it cannot compromise your backups (there is also the issue of data destruction by overwriting, but even raw S3 can be used as an append only store if you enable object versioning).

              My use case is installing backup software liberally on every device I use (and I use more devices than I have sole control over). For example, with Borg, you could not back up your home directory on a shared server without giving the administrator of that system the ability to decrypt your entire repository.

              1. 3

                My implementation is currently not exactly as you described, but perhaps I can accommodate this with not too much difficulty. edit: I am sitting in a cafe thinking carefully about how to do it without affecting usability for less advanced users right now.

                1. 2

                  Good points, thanks for the explaination!.

                  1. 2

                    If you trust the server to not leak data a next best approach is to have a symmetric key per device and then use ssh access controls to prevent access.

                    1. 1

                      If you trust the server just use TLS or SSH tunnels to encrypt in motion. If that’s really your threat model there is no need for additional complexity.

                      1. 2

                        For example, with Borg, you could not back up your home directory on a shared server without giving the administrator of that system the ability to decrypt your entire repository.

                        You have to backup to a different machine with a different administrator, it is true the first admin can decrypt your data, but he cannot fetch it because the ssh key can be granted write only access, even with borg via append only. a random key that is encrypted with a public key then discarded by the client is probably better though, still thinking how to do it well.

              2. 2

                Looking forward to test this! Much struggling with actual backup solutions!

                1. 1

                  Awesome! Can’t wait to hear more about this.

                  1. 1

                    I’d really like to know about how you tackle the intersection of client-side encryption and de-duplication.

                    1. 2

                      Its relatively straight forward using a https://en.wikipedia.org/wiki/Rolling_hash function. The ‘shape’ or ‘fingerprint’ of data guides you in finding split points, and each split chunk is encrypted independently. There is potential that size of chunks may give some clues about potential contents, but there are a few mitigations you can do such as random padding, keeping your hash function secret and a few others.

                      Another sticking point is allowing the server to do garbage collection of chunks that are no longer needed while at the same time not being able to read the user data. I came up with a solution I hope to get reviewed around layering trust.

                      1. 1

                        I know about splitting a file into chunks, but how do you derive a repeatable IV/key for a given chunk without leaking the contents of it, or opening yourself up to some form of chosen-plaintext attack?

                        1. 3

                          I use a random IV, and random encryption key, but the content address (i.e. dedup key) generated is repeatable by the client as HMAC(DATA, CLIENT_SECRET). AFAIK the attacker cannot recover the secret or decryption key even if he has a chosen plaintext, and has no way to derive the data without the secret. An attacker also cannot forge chunks because the HMAC address will be obviously wrong to a client.

                          There is also a write cache that prevents the same data from being uploaded twice with the same content address but different IV. Though that is more a performance thing than security, I could be wrong. I hope people can shoot down any flaws in my design which is why I need to get it finalized a bit.

                    2. 1

                      That sounds like a really useful tool! I’d seen a reference to Convergent Encryption today/yesterday, which “deduplication of similar backups to save space” sounded like. https://en.wikipedia.org/wiki/Convergent_encryption sounds like there are fundamental security implications to using it btw; deduping sounds pretty orthogonal to the rest of what it does, and I’d be excited to see a Show and Tell post :)

                      1. 3

                        Yeah, I rejected that specific approach for the reasons described. My keys are random, but some of the ideas are similar from a distance.

                      2. 1

                        This sounds similar to something I’ve had a couple of stabs at (one such stab “currently”). What language are you writing in?

                        My approach is built around two basic “processes”.

                        A “collection” phase during which a series of scripts do source specific “dumps” (eg dump mysql, ldap, etc, identify user generated file system data, etc) into service backup directories

                        A “store” process which compares a hash of each raw file (created by the collection phase) to an existing hash (if found). If no match is made, the hash is stored and the file is gpg encrypted using the specified public key. Once this process finishes, the hash files and gpg files are stored remotely via rsync, using the --link-dest option to create “time machine style” snapshots.

                        The heavy lifting is obviously all “written already” in terms of shashum, gpg and rsync. The glue layer is just shell.

                        I’d be keen to see how our approaches differ and if we can take any ideas from each other’s solutions.

                        1. 1

                          Mine is written in go currently. A big difference is It doesn’t sound like your approach deduplicates within files or across similar but not identical files, a tool such as mine could easily be hooked into your store phase to deal with cross file deduplication.

                          There are similar tools currently out there such as ‘bup’, ‘borg’ and ‘restic’ you should look into. I feel like mine is superior, but those all work and are ready today.

                          1. 1

                            No, it doesn’t attempt any kind of de-dupe except for not storing dupes of the same file if it hasn’t changed.

                            That’s part of why I’m not using those other tools - I want pubkey encryption (as mentioned elsewhere here, it means eg two+ devices can share backup space without leaking data they don’t already possess to the other) and I’d prefer if, when all else fails, I/someone can restore data from a backup by just running regular shell commands.

                            This part can of course be built into a companion tool, but being able to do ssh backup-space -- cat backups/20180210-2300/database/users.sql | gpg | mysql prod-restore-copy is a huge bonus to me. No need for the remote end to support anything beyond basic file access, no worrying about recombining files. No worrying about whether I have the same version of the backup tool installed, and/or if the format/layout has changed.

                            So possibly we have not as many overlapping goals as I originally thought, but it’s always nice to hear about activity in the same space.

                            1. 1

                              Yeah, tbh I wouldn’t release a backup tool as without fully documenting the formats and having them re-implementable in a simple way, e.g. as a python script. You need a complexity cap to protect you from yourself. I agree using public/private key pairs is a good idea.

                              Your system seems decent, though you don’t really have access controls protecting the machine from deleting its own backups (perhaps a worm that spreads via ssh). Do you deal with backup rotation?

                              1. 2

                                So, the original version of this was built to store on an internally controlled file server, and the “store” process finished by touching a trigger file, which (via inotify) caused a root-perms having daemon to run on the storage host and remove write accesss to the last backup from the ssh user rsync connected as

                                The same daemon also handled pruning of old backups.

                                The new version is designed to work with offsite storage like rsync.net/similar so for now it relies on remote end functionality to protect previous versions (eg zfs snapshots).

                      3. 12

                        Applying to Georgia Tech’s online CS master’s degree program and looking for cheap apartments.

                        1. 3

                          I hope both go well!

                          1. 1

                            Thank you! I’m sure I’ll be posting updates here in the future!

                          2. 2

                            I’ve considered applying for that (and there’s a similar program at UIUC). if you’re not a recent student, how have you handled the application’s requirement of letters of recommendation?

                            1. 4

                              I haven’t directly worked in admissions for these kinds of programs, but been in the vicinity of people doing so, and from what I’ve gathered, letters from current or past supervisors or colleagues are fine, doesn’t need to be faculty. If a colleague, ideally someone w/ a reasonably senior and obviously technical title.

                              If you have a solid technical CV, the letters to be honest are probably not that important, just so long as they’re there. They partly just attest that you’re a reasonable person, motivated and capable of doing honest work, etc. They’re more important if it isn’t immediately obvious from the CV that you have a background to do a CS masters. For example if you have a non-CS bachelor’s and recent work experience is in a non-technical role, admissions might look to the letters for some verification that you still do know enough to make taking masters-level CS courses not a waste of everyone’s time. A surprising number of people apply to CS masters programs without either knowing CS-101-level stuff or being able to really program…

                              1. 3

                                I’m asking coworkers and managers. It’s not ideal, but it’s been over a decade since I got my undergrad, so it’s really my only option.

                            2. 11

                              This breaks the usual format for these, but it’s been on my mind anyway, and it’s indeed something I’m working on, so:

                              Last week, when the new Go versioning proposal came out, my initial reaction was, “this is completely ludicrous.” Then, over the week, as I read through Russ Cox’s write-up of how it would work and why he was thinking the way he was, my response changed to, “this is honestly really well-thought-out,” and I felt a bit ashamed and disappointed that I’d leapt to a conclusion without bothering to actually sit down and think.

                              I realized that I’ve done that quite a bit, both generically, and with this group in particular. E.g., while I know Go acceptably, I’ve generally ignored it as being a step backwards from other languages and environments I use. Who needs Go when I’ve got C#, Rust, and Erlang? 9p is neat, but inefficient compared to SMB/NFS, and less supported. Upspin’s a thing, but it’s silly that it’s doing its own thing instead of building off Perkeep, IPFS, kbfs, or any of the other systems that were there earlier and have similar operation (and are overwhelmingly already written in Go, no less). Acme has some neat ideas, but needlessly ignores lots of UI advancements and modern development tools. And so on.

                              But I’ve never really tried actually living that world and trying to view the collection of output as a whole. So, for this week, just to force myself to be more open and think harder before having a reaction, I’m going to try to focus on getting outside my comfort zone a bit:

                              1. Most generally, whenever I see a technical opinion this week that I disagree with, I’m not going to post a response disagreeing (unless the advice is genuinely actively dangerous and I know that for a fact, anyway). Instead, I’m going to try to explore the other point of view, downloading software if applicable, trying out the solution if possible, and leaving it there.
                              2. More specifically to the Go team and its predecessors, I’m going to try to use the Acme editor for all my work for one week. Maybe the lack of syntax highlighting won’t actually bother me, and maybe having an editor that is, in its own way, as extensible as Emacs, but radically simpler, will be a good fit for how my brain works. And maybe not. But then, the next time that an Acme discussion comes up, my opinion will be at least marginally more based in experience than random opinions I’ve read online.
                              3. I’m going to use Go (specifically the vgo fork) for everything this week, to the extent I can get away with it. I’m going to try to pay attention to how much time I spend in total coming up with a solution, including debugging the results, writing the initial code, and shipping.

                              The goal here’s not to use Go for everything indefinitely (that’s really unlikely for personal stuff and not practical for work), and I really doubt I’ll throw out a decade of Vim and JetBrains (although who knows), but rather just to force myself to think and try before having an opinion.

                              1. 1

                                I was with you up until “Acme” ;-)

                                I’m what you’d call a “follower” of Go. I don’t want to use it, because I don’t much care for the noise in dealing with errors and I’m a big fan of generics, but I’m keeping an eye on it and think it would make a great compilation target.

                                I definitely got some value from the posts about versioning, it really resonated with the various pains that npm has inflicted on me over the last few years.

                              2. 10

                                I’m trying to make DragonFly work on GCE. It seems that GCE requires vioscsi / virtio_scsi, which is in DragonFly, and is nearly identical to the version in FreeBSD (which does officially work on GCE), and DFly’s vtscsi does work locally in KVM if using vtscsi as a secondary disc as per the NetBSD instructions for QEMU (which are, like, total magic), but appears to fail to attach the da child in GCE.

                                I’ve been adding a bunch of printf’s this past week, partially because it appears that DragonFly doesn’t appear to support reading input from the GCE serial port, so, you cannot do any live debugging; and doing the same changes to FreeBSD to see how the things differ. This included removing some support from FreeBSD’s vtscsi to see if that’ll make it stop working when matched for some framework features that are missing from DragonFly, but it appears that the error is likely outside of virtio_scsi.c, as FreeBSD still seemed to work even when downgrading vtscsi. Already tried CAMDEBUG w/ all the CAM_DEBUG_FLAGS, but CAM itself is quite different between FreeBSD and DragonFly to see the issue in the clear.

                                My plan now is to get outside of virtio_scsi.c, both upstream into cam(4) and downstream into da(4), to understand how the whole thing works in FreeBSD and DragonFly locally, and how it doesn’t work on DragonFly in GCE. I’ll start by adding a few print_backtrace(int) calls, which is what panic() calls to get the backtrace onto the terminal, to get a better understanding of the paths that get da to attach via vtscsi over cam.

                                P.S. The fact that someone in FreeBSD decided to name the driver virtio_scsi, attach it as vtscsi and have the actual physical discs attached as da (without documenting any such thing whatsoever in virtio_scsi(4)) doesn’t help much, but I’m way past that misunderstanding; which is even worse in the case of virtio_blk(4), which shows up as vtblk in dmesg, yet attaches the discs as vbd — seriously?! Where’s any sort of mention of /dev/vbd%d in the man-page?!

                                1. 9

                                  Home:

                                  • Learning Rust… because, shiny! [1]
                                  • Continue learning Elixir.
                                  • More exercise, spring is coming!

                                  Work:

                                  • Project still circling the drain as corporate dithers, but…
                                  • My team (and I) hit the cover off the ball with annual reviews and feedback.

                                  [1] Actually, years ago I set myself an annual goal: learn one technology-related thing outside work and one personal thing (skill, habit, etc.). It’s a pretty low bar but makes me feel good about myself.

                                  1. 3

                                    years ago I set myself an annual goal

                                    I love this. Simple, but keeps you learning.

                                  2. 8

                                    I have been hacking on a Snobol3 interpreter from Research Unix V10. I find it an interesting challenge to replace the numerous gotos with structured constructs.

                                    1. 6

                                      I’ve been working on a pure C11 implementation of a PNG steganography tool that I prototyped originally in Python and Perl (so I know it works). Current issue I’m running into is that the bytes of the PNG chunks are represented in my library as unsigned char *, and the CRC32 function I’m using gives me a checksum of that data as an unsigned long. I’m not experienced enough with C to know how to append that checksum to my unsigned char * bytes variable, so I’m trying to figure that out. The fun part of the project is that I’m also testing the library using the C testing framework that I built a little while back.

                                      I had to take a break from contributing to OSH (https://github.com/oilshell/oil) because of travel/work so I’m hoping to get back into that as well.

                                      1. 1

                                        You may want to use memcpy to get those bytes in there, or you can shift and mask the correct number of times. I don’t know enough about how the checksum has to be written.

                                      2. 6

                                        Finishing up working out my second week of my two week notice at my current gig (being required to work 7 days a week is not something I want to do…and also working later on weekdays). While I enjoyed all the people I’ve worked with the lack of structure, no room for growth, among other things is why I decided to take this new opportunity. Since I am the last full time non-contract developer here I’m curious to see how this will go.

                                        I start my new job next week working remote, so I am beyond excited for that.

                                        1. 5

                                          Working on web cache proxy server nuster, implementing feature to update cache TTL at run time by API

                                          1. 5

                                            Work:

                                            • Soft release! So now a small amount of clients have access to the product.

                                            Other:

                                            1. 5

                                              This week I’m mostly working on getting the 44CON CFP launched tomorrow for 44CON 2018. If anyone here is interested in speaking, we cover flights and accommodation for speakers to try and keep the event as cost neutral as possible.

                                              I’m also working on a separate blockchain-related project this week, and if I have time am going to start working on a reverse engineering 101 course.

                                              1. 5

                                                Pro:

                                                • Off for few days to enjoy personal life
                                                • Reduce deployment time by re-thinking the whole process. (What’s needed to Ben deployed instead of how to deploy.)

                                                Perso:

                                                • Starting 4th week of CIS194, really enjoying it and hope I’ll be able to do some useful projects soon!
                                                • Help friends to improve their deployment pipelines. One of them shared that he was manually doing git pulls and npm installs on production servers. Now that I convinced him that this is not good practice, I’m helping him to work on it!
                                                1. 5

                                                  Past week:

                                                  • Ran a fundraiser for Unicode to sponsor the Lobster emoji. GoFundMe made this very easy to run rather than a major project.
                                                  • Gave my blog a new theme. It’s a little nicer and responsive. Still need to design the footer, but otherwise it’s down to minor tweaks. Had fun learning the checkbox hack for tucking away sidebar content on mobile.
                                                  • Spent lots of good time with an old friend who was in town for a couple weeks.
                                                  • Taxes and other adult chores.
                                                  • Gave up on trying Backblaze B2 after a bug of their locked my account on any use and I couldn’t get past tier one support, which sounded undertrained or undertooled.

                                                  This week:

                                                  • A while ago I started a site for my articles and talks on code but the distinction doesn’t make sense to anyone else (most common reaction: “wait, you have two blogs?”) and I don’t like the history and current practices of the .io domain. It’s in talks recorded on YouTube so I need to keep the registration to 301 redirect people over, but I can fold all the content into my blog and otherwise shutter the site.
                                                  • Write a new blog post or two. All this time tinkering on my blog has me itching to write.
                                                  1. 4

                                                    Work: We’re going from having each team working on a thing to having a thing that is worked on by each of the teams, and part of that change is to change our release process. We do CD but with some manual oversight related to giving regulated financial advice, that works currently but won’t grow with our projections for this year. That’s the thing that stands out for me this week, and will have the biggest impact on the rest of our work.

                                                    Non-work: I’m speaking at NSLondon tonight, to a collection of Swift developers. Other than that I haven’t had a whole lot of energy for computering outside of work these last few weeks so am loathe to say “I’m going to do this” when the likelihood is I’m going to go home, have a nap, play Arcade Pool on the Amiga then go to bed.

                                                    1. 4

                                                      job: Learning AWS lambda to see if it can solve a problem at work.

                                                      hobby: building custom raspi image for a digital picture frame project

                                                      1. 4

                                                        Trying to do some more cleanups to my little dependent type system thingy, now called Pikelet - getting all the tests to pass, cleaning up the pretty printer so it doesn’t spit out ugly generated variables from the bowels of the pretty printer. Feel free to come over and chat on Gitter!

                                                        1. 4

                                                          Working on my fantasy baseball projection scripts in anticipation of draft season. On list of things to do this week:

                                                          • Update the scrapers to pull position eligibility data from every site I play on
                                                          • Update the auction value calculator to adjust for different formats
                                                          1. 4

                                                            Bug triage and fighting an utterly lethargic gdb.

                                                            1. 4

                                                              I worked on my directories project.

                                                              It is a tiny that provides the platform-specific, user-accessible locations for storing configuration, cache and other data on Linux, Windows (≥ Vista) and macOS.

                                                              The library provides the location of these directories by leveraging the mechanisms defined by

                                                              • the XDG base directory and the XDG user directory specifications on Linux,
                                                              • the Known Folder system on Windows, and
                                                              • the Standard Directories on macOS.

                                                              I made some good progress on the Rust version, and now plan to backport these improvements to the original Java version again.

                                                              1. 4

                                                                Hi, it appears we are tackling the same problem. Do you have a link to your source code?

                                                                Edit: found it.

                                                                Are you aware that XDG_DATA_DIR (and similar) is not part of the XDG standard, nor does it appear to be part of the xdg-user-dirs program? I believe the correct environmental variables are XDG_DATA_HOME and XDG_DATA_DIRS, the former referring to the location a program should store data, while the latter is a list of directories to search for it (including XDG_DATA_HOME). On my system XDG_DATA_HOME is undefined (and defaults to ~/.local/share), and XDG_DATA_DIRS="/usr/local/share:/usr/share". Do you plan to support different semantics for storing and searching for data?

                                                                Additionally, why do you distinguish between normal and roaming application data directories? There is only a difference on Windows, and I believe the intention is that all applications should use the roaming data directory if it is present for the current user. Shouldn’t you just read from %APPDATA% and let the user change that if they want different behavior? It seems to me the most likely scenario is that some application might want to always write to the local directory (perhaps it is storing screen resolution settings or something). In that case, it might want a “default” data directory and a “local” data directory. I can’t think of a reason to need a value which always refers to a roaming directory, especially when on Mac and Linux these are the same value (which would defeat any benefit of differing behavior).

                                                                I don’t see the need to expose a runtime directory. It is linux-specific (and you don’t support values for other operating systems) and holds linux/unix-specific files like sockets and named pipes. Any application which would use this directory would still need platform-specific code.

                                                                In a similar vein, why expose a value for “executable directory”? Installers will be platform-specific, and PATH should be used to find executable to run.

                                                                For project directories, why separate this information from base directories? The correct behavior for most programs is to only access files within their project directory (when there is an appropriate project directory). Do you plan to support the Windows practice of placing all program data in a subdirectory named after the author (e.g. %APPDATA%\Mozilla\Firefox instead of %APPDATA%\Firefox)? You could always default to using the application name, but that defeats the semantics anyway.

                                                                1. 2

                                                                  Hi, thanks for your review!

                                                                  Are you aware that XDG_DATA_DIR (and similar) is not part of the XDG standard, nor does it appear to be part of the xdg-user-dirs program?

                                                                  Where did you see this variable? There were some mistakes in the readme, but I fixed them a few days ago: https://github.com/soc/directories-rs/commit/93d5e32dd56c29852946172a2feba530ee59b64b

                                                                  Did you find any other places where the env name is incorrect?

                                                                  Do you plan to support different semantics for storing and searching for data?

                                                                  That’s what my blurb of “user-accessible locations for storing configuration, cache and other data” is kind of alluding to: I’m trying to focus on the locations that can safely be considered to be readable and writable to applications without elevated privileges.

                                                                  You make a good point about providing some lookup mechanism for existing files though! I would need to think about whether and how I could support that.

                                                                  I can’t think of a reason to need a value which always refers to a roaming directory, especially when on Mac and Linux these are the same value (which would defeat any benefit of differing behavior).

                                                                  The core idea behind this is that the library always returns a workable value wherever possible, avoiding the need to have platform-specific checks if possible. For instance, if data_roaming_dir only had a value on Windows, what would be the next thing devs would do that want to support Windows, macOS and Linux? They would fallback to data_dir!

                                                                  That’s why data_roaming_dir just gives the value of data_dir on unsupported platforms: Developers often forget to test and handle such cases.

                                                                  I don’t see the need to expose a runtime directory.

                                                                  It’s a useful directory and it’s in the XDG standard.

                                                                  Because the platform differences cannot me ignored in this case, runtime_dir is an option, and there are no values on macOS and Windows.

                                                                  In a similar vein, why expose a value for “executable directory”? Installers will be platform-specific, and PATH should be used to find executable to run.

                                                                  That’s a part I’m not happy with. I’ll probably make it Linux-only, because the semantics differ too much. The .local/bin directory is on the $PATH in many distributions already, so things should work as expected on Linux.

                                                                  For project directories, why separate this information from base directories?

                                                                  This is because base directories are more or less information that is globally valid, while project directories are specific to the individual application.

                                                                  In general, most applications should be fine with only using project directories, and I want to discourage people from spinning their own logic on how they place their data in e. g. .config. That’s why the stuff in base directories is intentionally kept separate.

                                                                  Do you plan to support the Windows practice of placing all program data in a subdirectory named after the author …

                                                                  This is a very good point that I didn’t consider! Thanks for bringing this up!

                                                                  In general I feel that many Rust libraries try to act too much like some all-encompassing framework (“describe your application to the framework, and the framework decides on what you need”). I try to stay away from that with my library, and want it to be more of a straight-forward “tell me want you want, and the libraries gives it to you”.

                                                                  That’s why there are factory methods with really long, descriptive names that try to make what happens as transparent as possible.

                                                                  … (e.g. %APPDATA%\Mozilla\Firefox instead of %APPDATA%\Firefox)?

                                                                  I think I will need to address this, but I need to consider the options. I could probably get away with a few more factory methods similar to the existing ones that just take the organization as another parameter. Not sure if this is the best approach though Thanks for bringing this up!

                                                                  1. 1

                                                                    Where did you see this variable?

                                                                    It’s still in the java readme.

                                                                    The core idea behind this is that the library always returns a workable value wherever possible, avoiding the need to have platform-specific checks if possible. For instance, if data_roaming_dir only had a value on Windows, what would be the next thing devs would do that want to support Windows, macOS and Linux? They would fallback to data_dir!

                                                                    My suggestion is to rework the api so that there is a data_dir and a data_local_dir (replacing the existing data_roaming_dir and data_dir, respectively). That way, you always get a reasonable value (i.e. data_dir should always return the roaming dir if it exists, because the default value should be to store data in the roaming directory).

                                                                    The core idea behind this is that the library always returns a workable value wherever possible, avoiding the need to have platform-specific checks if possible. Because the platform differences cannot me ignored in this case, runtime_dir is an option, and there are no values on macOS and Windows.

                                                                    hmm…. :P

                                                                    I’d just remove this api entirely. If you need a linux-specific feature, don’t include it in the api.

                                                                    In general, most applications should be fine with only using project directories, and I want to discourage people from spinning their own logic on how they place their data in e. g. .config. That’s why the stuff in base directories is intentionally kept separate.

                                                                    In my view, just don’t expose it in the first place.

                                                                    That’s why there are factory methods with really long, descriptive names that try to make what happens as transparent as possible.

                                                                    But I hate typing >.>

                                                                    I could probably get away with a few more factory methods similar to the existing ones that just take the organization as another parameter. Not sure if this is the best approach though Thanks for bringing this up!

                                                                    Personally, I don’t think the api design of appdirs is awful.

                                                                    1. 1

                                                                      Thanks for your help!

                                                                      I made the changes you suggested to local/roaming data dirs and removed the executable dir on macOS: https://github.com/soc/directories-rs/commit/4c4b4106186f25d4a39f940fcbc1def118bcfb36

                                                                      I still need to update the Java library though, and add some handling of organization names when creating project paths.

                                                                      Personally, I don’t think the api design of appdirs is awful.

                                                                      True, although I probably leave out the version parameter, because that is not something that needs to be abstracted across platforms.

                                                              2. 4
                                                                Not work:

                                                                I added Markdown support to my wiki. I started an unofficial fan wiki for Heat Signature, going to be adding that in fits and starts.

                                                                I did a little bit of fiddling around in Godot 3.0 this weekend, related to Alakajam 2.0. I didn’t end up taking part in the jam at any real level, but Godot continues to prove to be pleasant to work with.

                                                                Work:

                                                                Work is currently mostly the same stuff, being on maintenance and trying to make sure that recurring issues are dealt with.

                                                                1. 3

                                                                  Off to Edinburgh for the week with work. Not expecting to progress much of anything outwith socialising and prototyping, which should be acres of fun.

                                                                  At home I’ve been toying some more with smart bits & home automation. I noticed Home Assistant got native Homekit support and started playing with that as a thing to both hold the current state of the house (which switch is on), and expose the devices to Homekit without having to run node.

                                                                  Seems to work well. MQTT integration is pretty nice, I’m now reworking go-envsensor to output presence & config topics to MQTT so it dynamically configures home assistant when you start a sensor. (Still waiting on hardware to tinker with these more sensibly on ESP8266 boards rather than RPi’s.)

                                                                  1. 3

                                                                    I finally finished reading “The Science of Debugging”, after some interruptions the past couple of weeks. I was going to write a review of it with another book (“Why Programs Fail”), but since it took so long to read I may do a review considering it by itself. Ignoring interruptions, it took so long to read because I think it’s a muddled mess and it wasn’t until the final 10 pages that I finally found the angle for reviewing it.

                                                                    I’ve also got to start some course on machine learning for some reviews I have to do of other work. If anyone has some recommendations on a good introduction or online course, I’m interested.

                                                                    1. 3

                                                                      At work, I’ll mostly be fixing some issues from pen tests (last words: there is no way this can take up an entire sprint).

                                                                      At home, I’ve been working on a personal project for organizing ideas. It’s a weird project in that straddles being a static site generator, and read it later service. I’m wrote it in Haskell, which has meant learning the language at the same time. That has slowed me down a lot, but I wanted to use pandoc because I write in org-mode.

                                                                      I also started sketching out a datetime library for Haskell based on the Java 8 java.time.* libraries. I’m not aware of anything for Haskell that works well in that area.

                                                                      1. 3

                                                                        Home:

                                                                        • Working on a virtual-machine and compiler for the toy language I’m developing.

                                                                        School:

                                                                        • An emulator for the MOS 6502 microcontroller.

                                                                        Work:

                                                                        • Fighting with our Microsoft Exchange server.
                                                                        1. 3

                                                                          Work: All the Magento 2 research possible; creating an easy way for our existing magento site to manage GA/GTM snippets; Consulting on another team’s project kickoff for their stack (SPA and a lot of AWS Lambdas).

                                                                          Non-Work: Recording a comedy show!

                                                                          1. 3
                                                                            • Evaluated enough libs to move into active experimentation on my main research project
                                                                            • Now working on the networking test-spikes I sketched out awhile ago(various combinations of ZMQ, 9P, CBOR, etc.)
                                                                            • Looking more at K/Kona to see if it’d be useful for the actual 3D math and data storage nodes.
                                                                            1. 2

                                                                              Finally launched my Wordpress theme studio: https://morethemes.baby

                                                                              Still fixing IE11 and Chrome … khmmm … bugs for the main site but the Log Lolla theme is done, you can freely download it and use it. Enjoy :)

                                                                              1. 2

                                                                                Continuing work on c-appdir. I’m working on the Windows port, but I’ve running into a problem where the tests have different behavior under wine, wine+gdb, and on a real system (likely because the environmental variables are moving around). Should I be looking at docker to test this?

                                                                                I’m also applying for several internships, so hopefully one pans out.

                                                                                1. 2

                                                                                  Work:

                                                                                  • Keep adding documentation and examples to my Reflex SVG library
                                                                                  • Finish off and release our newly designed website.

                                                                                  Personal:

                                                                                  • I’ve just started learning Q, so keep at it..
                                                                                  • Unpack and start coding for my LeapMotion.
                                                                                  • Decide if I want to declare text editor bankruptcy and start again with a new one. :<
                                                                                  1. 2

                                                                                    Taking really crappy v1.0 code and adding a couple of unit tests, fixing the options processing to make sense, give it a valid README.md etc.

                                                                                    There is absolutely nothing exciting about this project unless you’re ripping your DVD collection and like me decided that having to rename a couple thousand clip extras files by hand to match Plex’s format was not a good use of your time :)

                                                                                    https://github.com/feoh/plex_tv_extras_fixer

                                                                                    1. 1

                                                                                      I’ve had a week “off” from regular client work (albeit with ~12 hours of prod-support aka “what the fuck just happened” in the mix) so its back to regular work this week.

                                                                                      This will probably cover planning next steps for the gradual improvement of said clients infra. Load Balanced/HA is a key goal still, as is local full-page caching to reduce app server load.

                                                                                      At home I’ve finally got all the floodlights installed around the house, so probably move onto getting more irrigation setup before the fuck-you weather really starts in March and nobody wants to water plants by hand.