1. 5

    I’ve recently been investigating deduplicating backup software for my personal use, and ended up deciding on borg, but one major annoyance of borg is that it cannot easily use a cloud storage API (like aws s3, wasabi, backblaze, google cloud, etc.) as a backing store. Some other backup tools do allow this (restic for instance), but lack other features I care about.

    I also note that you didn’t mention “compression” as a desideradum. As long as you care about deduplicating data to save backup storage case, however, you probably also care about compressing that same stored data as much as possible.

    Also, looking at you’re asmcrypt repo, I have to ask, why are you writing new software in C in 2018? Especially cryptography software?

    1. 4

      As another note, I’m currently working on a product that may be close to what you would like: https://backupbox.io/ (And there is no C in that code base, all Go currently ) Supporting borg is a specific goal, unfortunately it isn’t quite ready for public use.

      1. 3

        Just FYI, there’s already a project called Box Backup.

        1. 1

          thanks for letting me know.

      2. 4

        If you haven’t looked at https://www.tarsnap.com/ you may find it interesting. The backend is closed source, which I realize may be a problem, but the frontend if open source and free for perusal.

        1. 4

          “The Tarsnap client source code is available so that you can see exactly how your data is protected; however, it is not distributed under an Open Source license.” - https://www.tarsnap.com/open-source.html

        2. 2

          https://www.google.com/search?q=desideradum

          Please note your typo, otherwise thanks for the new word.

          1. 1

            Bah I thought the word didn’t feel right when I typed it!

          2. 2

            edit: Added note to post. Thank you.

            With regard to compression, this will come in a post about deduplication.

          1. 4

            I’ve been self-hosting since around 2002.

            I’m using FreeBSD + Exim + Dovecot and pepperfish.net (run by some friends) as my inbound MX for spam filtering as I got fed up of trying to keep up with making spam filtering effective.

            I like having full control over data and storage, and having mail going back this far would be troublesome to migrate to another service. I mostly use Mutt locally on the mail server, but occasionally use an IMAP client. My Mutt setup has barely changed since ~2002 and my workflow in it is fairly well ingrained now.

            It used to be a pain running things myself when I received mail directly (not via Pepperfish) as my spam filtering wasn’t up to scratch. Since I fronted with Pepperfish it’s taken all the pain away.

            I have Gmail as a secondary account.

              1. 7

                The author can’t have worked in an environment that’s heavily audited and subject to compensating controls.

                Standards like PCI-DSS require bastion hosts for accessing nodes which process customer data and you won’t get a pass without them. (Admittedly there are many things in PCI-DSS which are just box-ticking, but you might be losing customers without that certification.)

                1. 2

                  Find Manufacturer of Device

                  First get the IEEE MAC assignment list.

                  Huh, I never thought of seeing how to get that data. That’s cool. I wrote a quick thing: https://github.com/jamesog/whatthemac

                  1. 6

                    I found this article particularly interesting because I’ve been helping a colleague to write his first program (in Python). I caught him creating reports in a very manual way so offered to show him how to automate it. Someone who was previously uninterested in programming is now really enjoying it and realising what it can do for him. The problem before had always been having a need or specific thing to do rather than just following tutorials.

                    1. 10

                      From the readme:

                      You probably noticed the peculiar default line length. Black defaults to 88 characters per line, which happens to be 10% over 80. This number was found to produce significantly shorter files than sticking with 80 (the most popular), or even 79 (used by the standard library). In general, 90-ish seems like the wise choice.

                      This is a table stakes deal breaker for me. I know, I know, I’m likely old fashioned. I prefer old school. :-)

                      1. 6

                        It is a default though, you can pass --line-length to it.

                        1. [Comment removed by author]

                          1. 4

                            Honestly though, is your terminal window really 80 columns wide? And should outdated defaults matter?

                            1. 3

                              Yes, my terminal window is really 80 columns wide.

                              I also have a source file where a single line of code is 250 characters (and no, it really can’t be broken up due to semantic constraints).

                              So, what should be the minimum width of a terminal window?

                              1. 1

                                I actually prefer to code with a 80-wide terminal most of the time, because it tends to remind me to simplify my code more than I would otherwise :o

                              2. 1

                                I think 79 is better than 80, because 79 allows for a single-column ruler on the side of the window and stuff

                                1. 1

                                  This is about the size of your code “viewport”, not of your terminal.

                                  3 columns are already used by my line length indicator in vim, but that number is really arbitrary too.

                                2. 1

                                  departing from established standards because you feel like it is a pretty bad sign in general. as are --long-gnu-style-options, but that’s a different issue.

                                3. 2

                                  I just counted the length of lines over 266 Lua files [1], calculating the 95th percentile [2] of line length. Only 4% had a 95 percentile of 89 or higher; 11% had a 95th percentile of 80 or higher. And just because, 4% had 95th percentiles of 80 and 81. For maximum line lengths:

                                  • 42% with longest line of 79 characters or less
                                  • 46% with longest line of 80 characters or less
                                  • 56% with longest line of 88 characters or less

                                  Longest line found: 204 characters

                                  [1] I don’t use Python, so I’m using what I have. And what I have are multiple Lua modules I’ve downloaded.

                                  [2] That is, out of all the lines of code, 95% of line lengths are less than this length.

                                  1. 1

                                    https://en.wikipedia.org/wiki/88_(number)

                                    I can’t help it but read it as a fascist code, or at least I start thinking about whether it could be on every occasion I see this number (not totally unfounded, because where I live it is used this way every now and then). I don’t think they meant to use it this way, so I think it’s fine (more than that, good, because it devalues the code by not treating the numebr as taboo).

                                    1. 1

                                      Personally I don’t like line break at 80 or 90 with python, as the 4-spaces indent quickly uses up a lot of horizontal space. For example, if you write unittest-style unit tests, before you write any assignment or call, you have already lost 8 spaces.

                                      class TestMyFunctionality(unittest.TestCase):
                                          def setUp(self):
                                              # ....
                                          def test_foo(self):
                                              x = SomeModule.SomeClass(self.initial_x, get_dummy_data(), self.database_handle)
                                      

                                      Of course you could start introducing line breaks, but that quickly leads to debates on “hanging indent”, lisp-style-indent, etc. or you end up with a lot of vanity variables for local variables.

                                      With lisp-style indent I mean the following snippet, that (if it was the way the autoformatter would do it would convince me to accept a 80 character line length limit)

                                      class TestMyFunctionality(unittest.TestCase):
                                          def setUp(self):
                                              # ....
                                          def test_foo(self):
                                              x = SomeModule.SomeClass(self.initial_x,
                                                                       get_dummy_data(),
                                                                       self.database_handle)
                                      

                                      Whereas I find the “hanging indent” style makes understanding the structure of the syntax tree so much more difficult.

                                      class TestMyFunctionality(unittest.TestCase):
                                          def setUp(self):
                                              # ....
                                          def test_foo(self):
                                              x = SomeModule.SomeClass(
                                                      self.initial_x,
                                                      get_dummy_data(),
                                                      self.database_handle)
                                      
                                    1. 6

                                      I’ve installed the vim plugin and so far I’m liking it. I’ve been rather spoiled by vim-go so it’s nice to see similar things for other languages.

                                      The Black plugin for vim doesn’t include format-on-write but someone opened a pull request to demonstrate how to configure vim to do it.

                                      I’ve added this to my .vimrc:

                                      if has("autocmd")
                                          augroup filetype_python
                                              au!
                                              au BufNewFile,BufRead *.py setlocal sw=4 sts=4 et
                                              au BufWritePost *.py execute ':Black'
                                              au FileType python setlocal sw=4 sts=4 et
                                          augroup END
                                      endif
                                      
                                      1. 3

                                        At work we use YYYY.MM.NN for internal software (NN being a 0-indexed release number for that month).

                                        I like this for knowing when something was last updated, but it’s not helpful for identifying major changes vs. bugfixes. Perhaps that’s not such a big deal for software that’s on a rapid release cycle.

                                        1. 2

                                          It’s also not a big deal for software that’s too big or complex for “major change” to be meaningful. If a tiny, rarely used Debian package removes support for a command-line flag, that’s a major change (in the SemVer sense) and since Debian includes that package it’s therefore technically a major change in Debian. But if Debian followed SemVer that strictly, its version number would soon leave Chrome and Firefox in the dust, and version numbers would cease being a useful indicator of change.

                                          1. 7

                                            Isn’t Debian’s solution to this to not include major-version changes in updates to an existing release? So it does wait for the next major version of Debian to be included, usually

                                            1. 1

                                              Yep, and this is where the -backports or vendor repos are really useful - newer packages built against the stable release.

                                            2. 2

                                              It’s why we have to make “stable” releases. Otherwise everyone goes crazy. If someone is updating their SemVer too often, they have bad design or do not care for their developers.

                                            3. 2

                                              There’s a discussion of CalVer and breaking changes here: https://github.com/mahmoud/calver/issues/4

                                              Short version, “public” API is a bit of a pipe dream and there’s no replacement for reading (and writing) the docs :)

                                              1. 2

                                                The concept of breaking changes in a public API isn’t really related to ‘read the docs’, except when it comes to compiled in/statically linked libraries.

                                                If you have dependency management at the user end (i.e. via apt/dpkg dependencies that are resolved at install time), you can’t just say “well, install a version that will work, having read the docs and understood what changes when”.

                                                You instead say “I require X major version of package Foo”, because no matter what the developer does, Foo version X.. will always be backwards compatible - new features might be added in a X.Y release, but thats not a problem if theyre added in a backwards compatible manner (either theyre new functionality that has to be opted in for, or e.g. they don’t require extra options to work).

                                                Yes, I know that things like Composer and NPM have a concept of a ‘lock’ file to fix the version to a specific one, but that’s not a solution for anything but internal projects. If you’re installing tools that you aren’t directly developing on yourself using NPM or Composer, you’re doing it wrong.

                                                1. 1

                                                  I really don’t see what that has to do with the linked thread. In the very first line, you mention a “public” API. The point is that there’s much less consensus on what constitutes a public API than developers assume. So, you end up having to write/read the docs about what would constitute a “semantic” version change. (Not that docs are a silver bullet, they’re just a necessary part of healthy software development.)

                                                  1. 1

                                                    The point is that there’s much less consensus on what constitutes a public API than developers assume.

                                                    A comment by you making that same claim on GitHub isn’t really evidence of a lack of consensus. What possible definition is there for “public API” besides, “something that will be consumed my someone outside the project”.

                                                    So, you end up having to write/read the docs about what would constitute a “semantic” version change.

                                                    The decision tree for SemVer is two questions, with 3 possible outcomes. And you’ve still ignored my point. Adherence to semver means you can automatically update dependencies independently of the developer.

                                                    So, for instance, if the developer depended on a shared library, that happens to have a security vulnerability, when the library author/project releases a new patch version, end-users can get the fix, regardless of what the tool/app developer is doing that week.

                                                    1. 1

                                                      The automatic updates work until they don’t. Here is a recent example where Python’s pip broke with many assumptions about public APIs. Your point has not been ignored, I’ve written about it extensively, in the linked thread and in the linked site (and links therein).

                                                      As for your closing comment, I’m noticing an important assumption I’m working to uproot: current date is not the only possible date in the release version. manylinux2010 came out in 2018, and is named as much because it’s backwards compatible to 2010.

                                                      The Teradata example on the CalVer site also highlights maintaining multiple releases, named after their initial release date. At the consumer level, Windows 98 got updates for years after 2000 came out.

                                                      1. 1

                                                        That isn’t a failing of semver, it’s a failing of the developers who didn’t properly identify they had a breaking change.

                                                        The same thing would have happened under calver, they would have marked it as a patch release with compatibility to the previous version, regardless of the date component.

                                                        Expecting people to just forget about the possibility of automatic dependency updates is like suggesting people forget that coffee exists after they’ve had it daily for 10 years.

                                            1. 5

                                              Similar to others, however:

                                              • Personal mail (Exim, Dovecot)
                                              • Various websites (mostly Nginx)
                                              • GitLab
                                              • Buildbot for CI on various platforms as most hosted things are Linux-only
                                              • Plex media server

                                              When travelling I spin up an Algo VPN on GCE or DigitalOcean.

                                              I used to run my own colo server but $DAYJOB has 6000+ physical servers and I got bored of having to do maintenance of yet another physical box :-)

                                              Now I run most of my things from a VPS at Mythic Beasts running FreeBSD with misc. other bits running on dedicated boxes from Hetzner which currently run SmartOS, although I’m considering moving to FreeBSD with bhyve sometime.

                                              I’m using Prometheus and Grafana for monitoring all of it.

                                              1. 5

                                                Whitelisting and automatic VPN connection can be achieved with Apple profiles. See for example how algo does it

                                                1. 1

                                                  Yep, I use Algo to set up an IPSec VPN. It generates macOS and iOS configs which can be configured to automatically connect, with SSID exclusions e.g. for home and work.

                                                  Before Algo I used Streisand, and had to go through similar trickery as the OP to make it auto reconnect. When I saw the Algo way I realised that it’s both simple and pretty smart.

                                                  (Caveat: I don’t know if Algo generates similar auto connect setups for non-Apple OSes.)

                                                1. 3

                                                  Neat idea. How about making your .bash_history file a named pipe and have the script read from that, instead of using PROMPT_COMMAND?

                                                  1. 4

                                                    I did this at a hackathon. I found it kept having weird sync issues with a lot of shells open, but didn’t have time to fully investigate.

                                                    1. 1

                                                      Won’t this block your shell after a while if the reading process somehow dies?

                                                      1. 1

                                                        Yep. Equally PROMPT_COMMAND could cause delays/hangs depending on connectivity to the remote system.

                                                      2. 1

                                                        That’s a very creative idea that I hadn’t thought of. I did want to avoid interfering with the client system’s history mechanism though.

                                                        1. 1

                                                          Ah, good point!

                                                      1. 2

                                                        Aaand I missed the other post when submitting this: https://lobste.rs/s/mimoad/red_hat_acquire_coreos_expanding_its

                                                        1. 4

                                                          This is a more technical take on the news, compared to that link’s business take.

                                                        1. 4
                                                          vnoremap < <gv
                                                          vnoremap > >gv
                                                          

                                                          gv is pretty nifty.

                                                          1. 1

                                                            That is a top tip indeed!

                                                            1. 1

                                                              I’ve been 20% happier since I discovered that. Wish VSCode’s Vim emulation allowed me to do the same.

                                                          1. 1

                                                            I also make use of many of Tim Pope’s plugins, notably vim-sensible, vim-commentary and vim-fugitive.

                                                            These three are incredibly useful.

                                                            1. 2

                                                              I run a Buildbot cluster for an open source project I host for pretty much this reason, and needing to build on multiple platforms which most of the hosted solutions don’t do.

                                                              I’d be really interested in hosted solutions which can do multiple platforms.

                                                              1. 3

                                                                @jamesog@mastodon.social although I’ve yet to post there. Honestly I keep forgetting I have it as I don’t have an equivalent of Tweetbot for desktop use.

                                                                1. 1

                                                                  WASD 105-key ISO with Cherry MX Clear switches.

                                                                  This is the same company that makes the CODE keyboard, but I have no need for backlit keys and prefer to have the number pad on the right, which the CODE lacks.