1. 8
  1. 59

    I don’t think this one is the fault of the tool, where he manually and incorrectly edited its metadata without adequate backups. Syncthing’s documentation is explicit about it not being a backup tool because of the potential for exactly this scenario of destructive changes being propagated.

    In the alternate universe version of this post, Syncthing has the confirmation he wants for large, sweeping changes or deletions. (It’s telling that he doesn’t specify what that threshold would be, because any value would be obviously wrong for some use.) He ends up equally frustrated because something he did renamed/touched a lot of files and he didn’t see the confirmation dialog, so his files stopped syncing for weeks before a data loss incident, and again he’d inaccurately blame Syncthing.

    1. 1

      I went looking for what syncthing actually uses the .stfolder for, and it appears to just be a marker that “this folder is healthy” and not e.g. unmounted: https://docs.syncthing.net/intro/gui.html?highlight=stfolder. That also says it’s allowed to be a file. So I don’t think that’s the issue.

    2. 29

      “Do not use Syncthing” seems like a strange takeaway from this post. Syncthing explicitly does not claim to be a backup program, and using it without backups seems like a recipe for disaster, as is using any computer program without backups.

      I am curious if there is a Syncthing bug here, and would like to see a postmortem of this from the Syncthing folks — it’s hard to tell what exact setting were used in this case. But this story definitely doesn’t rise to the level of “Do not use Syncthing” for me, it’s more “be wary that Syncthing may behave in unexpected ways and have backup plans in place”.

      1. 1

        How do you think Dropbox stands in comparison[1]? Do people not comfortably use Dropbox as an implicit backup for their files, especially as it supports file history?

        [1] aside from it being centralized and proprietory, that is; asked from the standpoint of reliability.

        1. 3

          Syncthing also has file history, but you must enable it per-share.

          I think you were very cavalier about your files. You assumed you understood how Syncthing was going to react to the manual changes you made, but you didn’t have a backup plan, really, for if it did not. I’m sorry you got burned, but now you know better, and it doesn’t have to happen again. Think about how much expense this incurred for you, and buy a backup disk. Start doing regular backups, the slow and boring way, because it will save you from the next misunderstanding.

          1. 3

            I personally would not trust Dropbox as a backup solution, especially given that you can only go back 180 days without buying a addon for which the price is “Contact Sales”.

            I’m sure many people do rely on Dropbox for backups, I think that’s probably a bad idea that will cause a reasonable fraction of those people undue suffering down the line.

            1. 2

              Plus you are only one account compromise away from losing access to your files.

            2. 2

              I think Dropbox represents a fine backup strategy provided it’s used as one factor in a multi-factor backup plan.

              So, for example. I use Backblaze for my off site backups. You could use Dropbox instead. When paired with the onsite backups I have (A regularly scheduled HyperBackup job from my Synology NAS to a 5TB USB drive) I feel pretty confident that my data is safe.

            3. 1

              There clearly isn’t a bug, and you make a great point about SyncThing not being a backup tool, it explicitly requires a carefully thought out and very rigorous backup regime to ensure no data is lost.

              It’s a great tool, but with great power comes great responsibility, and it’s not clear to me that a lot of people will do the right thing.

              1. 1

                There’s quite the leap from “not claiming to be a backup program” to active destruction of data due to negligence in the program’s design.

                A synchronization tool isn’t excused from destroying data due to design faults.

                1. 2

                  A synchronization tool isn’t excused from destroying data due to design faults.

                  No, but is there a verified design fault at play here?

                  1. 2

                    Yes. If I read the article correctly, it does handle corner cases - such as disk full - poorly.

              2. 9

                Without commentary as to whether syncthing is better or worse than competing options, I would very stubbornly argue that the takeaways are mis-ordered. The major takeaway should be that you need to have a rock solid, tested backup and recovery system before you even sniff around this kind of automation. If you have one copy, you’ve got none. If you have two you’ve got one. And they’re all meaningless unless you test a recovery. The particular bit of nastiness with syncthing is really secondary to that lesson.

                It really sucks that there is a path that could lead syncthing to treat something the user marked “read-only” as not read-only, though. Yikes.

                1. 15

                  Do not use hard drives ! I used an HDD to store my data, it failed me and now all my data is gone !

                  This is the exact same use case here. Software may fail just as much as hardware, and we must prepare for it. This blog post should only serve as a loud reminder to check your backups (because you have backups, right?).

                  1. 5

                    The other takeaway is that, backup, and backup often. If I was already backing up every week, I wouldn’t have lost what I lost.

                    I guess they discovered that file synchronization is not a backup. Though, they do not seem to have learned a lot from this. Losing a week of work is also insane. Do hourly incremental backups to multiple locations. restic and Arq are your friends.

                    1. 1

                      Isn’t backup some kind of file synchronization? Would you define it as one way synchronization? If it is just one way synchronization, I think Synchthing supports this kind of mode, doesn’t it?

                    2. 3

                      While a lot of commentors are right about the Syncthing is not a backuptool thing, i think the problem OP encountered is a little broader then to trust Syncthing for backups.

                      At the beginning OP wrote:

                      My folders were configured so that the copy on the computer was read only, i.e. modifications made on the phone would not propagate.

                      So even though the changes on OPs phone should not matter because they would sync against a read only folder on OPs PC, Syncthing decided to ignore that flag and delete everything in that folder. Sure one could argue that creating a bunch of empty files instead of a folder might have been a little careless. but Syncthing ignoring read only flags is quite disturbing as well.

                      1. 3

                        I agree that the author is guilty of using a sharp tool in a very sloppy way and bringing data loss down on her own head, BUT…

                        When I hear people talking about this tool, the very first thing that runs through my head is “Hrrm. Allowing for arbitrary storage of data anywhere there’s a free block. What could possibly go wrong?”

                        I say that NOT from a technical perspective - I’m quite sure the tool works as advertised given the recommendations it gets from very clueful people, but from a human psychology perspective.

                        When I use something like a NAS for my file storage, I am putting all my eggs in that one basket. I know exactly where all my data is going, and if I back it up religiously (automation!) using industry best practice (1 offsite, 1 onsite) and test restores, I can be reasonably sure my data is safe.

                        With SyncThing, it allows for arbitrarily complex data storage flows, which on the one hand is awesome, but on the other super dangerous, because the complexity of my backup strategy potentially scales with the number of nodes being synced.

                        I know I don’t personally have the discipline to use such a system safely for my mission critical data. I’m sure others feel differently, but I do think this article is a cautionary tale worthy of some attention for SyncThing users.

                        1. 3

                          My condolences to the author. Data loss sucks.

                          In hindsight its always easier to see flaws in your plans.

                          1. 2

                            I keep hearing (reading) troubling stories about Syncthing. I have my list of pain points about it as well (no data loss so far), but honestly I could not find something as flexible and decentralized, and at least as trustworthy.

                            I’d be willing to pay up to 100$ (one time, at least 3 devices, with at least security updates for a few years) for a nice p2p file sync solution, that is audited, has Android and Windows support at least, and has selective sync and on-demand access as well (to be able to access stuff on phone which is not synced by default, this one Syncthing does not support).

                            1. 2

                              I haven’t had any problems with it, and have been using it for a few years. My only gripe is that when I create a new share, I have to do an ssh port forward trick to set it up on my remote host.

                              1. 1

                                At my last company we used Resilio that supports android and windows, has selective sync and on-demand access. Not sure about auditing, but it seems to tick off most of your boxes, and was very solid for the whole time. We had a monthly subscription but there might be different pricing.

                              2. 2

                                Interesting. An unsynchronized un-backed-up file store is an SPOF. A synchronized un-backed-up file store is a multiple SPOF. Backing up gives you safety, automatic synchronization removes it.

                                1. 1

                                  I use Resilio Sync myself which have worked out great. Syncthing looked like a great FOSS alternative to me but I never took the plunge to migrate over to it. A friend of mine did experience issues with it consuming 100% CPU on his FreeNAS setup which is why i have been hesitant.

                                  Are there any other good alternatives for syncing between Android and Linux/BSD?

                                  EDIT: ZFS snapshot integration would be a killer feature.

                                  1. 2

                                    About that last thing, I have a vague plan for using SyncThing at work with a setup like this:

                                    • a server that for each shared folder:
                                      • is a SyncThing device
                                      • exposes it to SMB for access to files without needing to install syncthing
                                      • does hourly, daily, monthly, yearly snapshots
                                    • syncthing client installed on everyone’s computers (and possibly phones)
                                    1. 1

                                      I use Resilio Sync for sharing files within our family. But I would strongly discourage anyone to switch it now. It is a great product, but updates have been very slow, with known issues going unaddressed for months. It seems like they have lost interest in their non-enterprise product.

                                    2. 1

                                      The obvious takeaway is: “Don’t forget to make backups”. Syncing things with multiple devices is not a replacement for backups…

                                      1. 1

                                        Backup your important data. SyncThing is a synchronization tool, not a backup tool. Syncing is no substitute for backups, as you have seen.