1. 8

    Hey folks this is my post. Thanks for the comments! I’m not making A Statement, I just am trying to learn a thing and keep notes in public about it. But I will editorialize here; I found the Hugo docs super confusing. There’s a lot of docs but somehow I read them twice and didn’t come away understanding what Hugo actually does. Hoping my notes help someone else in the same boat.

    1. 4

      I had a similar experience to you: I wanted to build a simple static website, I decided to use Hugo, and the official docs didn’t help me get started. I tried to use some existing themes but they were all opaque to me; I struggled to customise them how I wanted.

      I ended up reading Build Websites With Hugo, which takes the same approach as your article: create a custom theme, and build up the layouts etc. from there. That made it a lot easier.

      The book’s available on O’Reilly Online, which I access via my ACM membership but is also often available through your local library. It’s definitely worth a look.

      1. 2

        I had a similar experience. It has documentation, and it’s better than the alternative, but for reasons I’ve always had trouble putting my finger on, it takes a lot of work beyond the docs to understand.

        1. 6

          I’ve had the same trouble. IMO it’s because it looks like comprehensive documentation, but it’s all just reference docs, and poorly organized references at that.

          1. 2

            Yeah, I think there are a couple of intersecting problems:

            1. Hugo has grown enormously over the years. I started using it around 2016, at which point it didn’t support e.g. base templates because Go didn’t support base templates. The layering of new features is good if you’re continuously using it (“hey, cool, I can rewrite X to be much simpler!”) but bad for new users. Which is related to

            2. It uses Go’s templates as its core. (It actually used to support a HAML-like templating language, but that was dropped at some point.) Go templates are idiosyncratic (less politely, weird), and to understand them you have to read https://pkg.go.dev/text/template very carefully. I had done that already when I started using Hugo because I like Go and wanted to use Go as a web server anyway, but the average website theme creator just knows HTML and CSS and doesn’t want to learn anything Go specific.

            3. The documentation is as mentioned, all reference docs. It’s not that bad as a reference, but there are areas where old systems haven’t had their docs updated to explain how they work with new systems (e.g. asset pipelines for pages vs. the site). However, it definitely needs tutorials and cookbooks to actually round it out.

            A different problem from my point of view as a Go developer is that the code base is too gnarly to extract anything from it. I would love to be able to use, e.g., Hugo’s template functions in my Go templates, but god help anyone who wants to figure out how they’re exported. (I think there is some sort of namespace struct that does something??? I really could not follow it in my attempts to figure it out.)

            I’d love to make a new, simpler SSG based on the learnings of Hugo and the development of Go (second system syndrome), but I don’t think it will happen because Hugo already exists and is hard to challenge.

      1. 1

        Thank you for this post. Truly insidious and scary stuff. Looking at the bullet point list of issues caused by mercurial cores at Google…wow I would not enjoy debugging those.

        I have worked in several different parts of Amazon and AWS. When it comes to storage most teams shrug and say “Just use DynamoDB or S3”. But having worked in DynamoDB, and seen the durability techniques that S3 use, durability is not trivial or given and it must be designed into distributed systems.

        But something I’ve never considered, and the scary part of these papers, is the integrity of calculations. Is it common for distributed data processing frameworks like Apache Spark to have an option “Hey do this thing twice on separate physical machines”? I’ve never heard of it. I’ve seen teams internal to AWS do this, but I haven’t heard this as “a thing”.

        I loosely recall an apocryphal story from a mentor over a decade ago that the Typhoon fighter’s CPU hardware actually runs on 7 separates physical instances built by 2 separate vendors, and they all voted and the majority decisions won. I never did find a citation for this, not sure if it’s true.

        1. 2

          My personal nightmare scenario was this one:

          A deterministic AES mis-computation, which was “self-inverting”: encrypting and decrypting on the same core yielded the identity function, but decryption elsewhere yielded gibberish.

          I’ve not seen systems with a setting to reproduce computation across different machines either. I’d guess it’s an uncommon option as it would halve the capacity of the system. (You could reduce the impact by limiting that to “important” computations – but another lesson from the article is “It’s hard to predict which computations are important until they start getting corrupted”.)

        1. 3

          I’m working on Piccle, a static site generator for photographers. It uses EXIF metadata embedded in your photos to build an explorable web portfolio. (You can see an example gallery on my website, though some pictures are NSFW.)

          As far as I know I’m the only user so far (hence its prerelease tag!) but I’ve been working on it for a while and it’s stable/complete enough for my main uses. Over the past month I’ve focused more on editing photos/tidying my metadata, but I still want to add some extra features soon. I’m also planning to experiment with client-side rendering support – not least because the current rendering pipeline can be drastically simplified if that turns out to be a bad idea.

          1. 3

            I really liked your pictures, great work!

            1. 1

              Thanks! I’m glad you like them. :)

            2. 1

              This is relevant to my interests - I’ve been searching for something I can selfhost as opposed to Flickr, and I like the idea of static generation.

              Many moons ago I used something similar that was supposed to generate a portfolio but it lacked the metadata integration.

            1. 11

              I like Apple hardware a lot, and I know all of the standard this-is-why-it-is-that-way reasoning. But it’s wild that the new MacBook Pros only have two USB-C ports and can’t be upgraded past 16GB of RAM.

              1. 18

                Worse yet, they have “secure boot”, where secure means they’ll only boot an OS signed by Apple.

                These aren’t computers. They are Appleances.

                Prepare for DRM-enforced planned obsolence.

                1. 9

                  I would be very surprised if that turned out to be the case. In recent years Apple has been advertising the MacBook Pro to developers, and I find it unlikely they would choose not to support things like Boot Camp or running Linux based OSs. Like most security features, secure boot is likely to annoy a small segment of users who could probably just disable it. A relevant precedent is the addition of System Integrity Protection, which can be disabled with minor difficulty. Most UEFI PCs (to my knowledge) have secure boot enabled by default already.

                  Personally, I’ve needed to disable SIP once or twice but I can never bring myself to leave it disabled, even though I lived without it for years. I hope my experience with Secure Boot will be similar if I ever get one of these new computers.

                  1. 12

                    Boot Camp

                    Probably a tangent, but I’m not sure how Boot Camp would fit into the picture here. ARM-based Windows is not freely available to buy, to my knowledge.

                    1. 7

                      Disclaimer: I work for Microsoft, but this is not based on any insider knowledge and is entirely speculation on my part.

                      Back in the distant past, before Microsoft bought Connectix, there was a product called VirtualPC for Mac, an x86 emulator for PowerPC Macs (some of the code for this ended up in the x86 on Arm emulator on Windows and, I believe, on the Xbox 360 compatibility mode for Xbox One). Connectix bought OEM versions of Windows and sold a bundle of VirtualPC and a Windows version. I can see a few possible paths to something similar:

                      • Apple releases a Boot Camp thing that can load *NIX, Microsoft releases a Windows for Macs version that is supported only on specific Boot Camp platforms. This seems fairly plausible if the number of Windows installs on Macs is high enough to justify the investment.
                      • Apple becomes a Windows OEM and ships a Boot Camp + Windows bundle that is officially supported. I think Apple did this with the original Boot Camp because it was a way of de-risking Mac purchases for people: if they didn’t like OS X, they had a clean migration path away. This seems much less likely now.
                      • Apple’s new Macs conform to one of the new Arm platform specifications that, like PREP and CHRP for PowerPC, standardise enough of the base platform that it’s possible to release a single OS image that can run on any machine. Microsoft could then release a version of Windows that runs on any such Arm machine.

                      The likelihood of any of these depends a bit on the economics. In the past, Apple has made a lot of money on Macs and doesn’t actually care if you run *NIX or Windows on them because anyone running Windows on a Mac is still a large profit-making sale. This is far less true with iOS devices, where a big chunk of their revenue comes from other services (And their 30% cut on all App Store sales). If the new Macs are tied more closely to other Apple services, they may wish to discourage people from running another OS. Supporting other operating systems is not free: it increases their testing burden and means that they’ll have to handle support calls from people who managed to screw up their system with some other OS.

                      1. 2

                        Apple’s new Macs conform to one of the new Arm platform specifications

                        We already definitely know they use their own device trees, no ACPI sadly.

                        Supporting other operating systems is not free

                        Yeah, this is why they really won’t help with running other OS on bare metal, their answer to “I want other OS” is virtualization.

                        They showed a demo (on the previous presentation) of virtualizing amd64 Windows. I suppose a native aarch64 Windows VM would run too.

                      2. 2

                        ARM-based Windows is available for free as .vhdx VM images if you sign up for the Windows Insider Program, at least

                      3. 9

                        In the previous Apple Silicon presentation, they showed virtualization (with of-course-not-native Windows and who-knows-what-arch Debian, but I suspect both native aarch64 and emulated amd64 VMs would be available). That is their offer to developers. Of course nothing about running alternative OS on bare metal was shown.

                        Even if secure boot can be disabled (likely – “reduced security” mode is already mentioned in the docs), the support in Linux would require lots of effort. Seems like the iPhone 7 port actually managed to get storage, display, touch, Wi-Fi and Bluetooth working. But of course no GPU because there’s still no open PowerVR driver. And there’s not going to be an Apple GPU driver for a loooong time for sure.

                        1. 2

                          I think dual-booting has always been a less-than-desireable “misfeature” from Apple’s POV. Their whole raisin de et is to offer an integrated experience where the OS, hardware, and (locked-down) app ecosystem all work together closely. Rip out any one of those and the whole edifice starts to tumble.

                          So now they have a brand-new hardware platform with an expanded trusted base, so why not use it to protect their customers from “bad ideas” like disabling secure boot or side-loading apps? Again, from their perspective they’re not doing anything wrong, or hostile to users; they’re just deciding what is and isn’t a “safe” use of the product.

                          I for one would be completely unsurprised to discover that the new Apple Silicon boxes were effectively just as locked down as their iOS cousins. You know, for safety.

                          1. 3

                            They’re definitely not blocking downloading apps. Federighi even mentioned universal binaries “downloaded from the web”. Of course you can compile and run any programs. In fact we know you can load unsigned kexts.

                            Reboot your Mac with Apple silicon into Recovery mode. Set the security level to Reduced security.

                            Remains to be seen whether that setting allows it to boot any unsigned kernel, but I wouldn’t just assume it doesn’t.

                            1. 4

                              They also went into some detail at WWDC about this, saying that the new Macs will be able to run code in the same contexts existing ones can. The message they want to give is “don’t be afraid of your existing workflow breaking when we change CPU”, so tightening the gatekeeper screws alongside the architecture shift is off the cards.

                            2. 2

                              I think dual-booting has always been a less-than-desireable “misfeature” from Apple’s POV. Their whole raisin de et is to offer an integrated experience where the OS, hardware, and (locked-down) app ecosystem all work together closely. Rip out any one of those and the whole edifice starts to tumble.

                              For most consumers, buying their first Mac is a high-risk endeavour. It’s a very expensive machine and it doesn’t run any of their existing binaries (especially since they broke Wine with Catalina). Supporting dual boot is Apple’s way of reducing that risk. If you aren’t 100% sure that you’ll like macOS, there’s a migration path away from it that doesn’t involve throwing away the machine: just install Windows and use it like your old machine. Apple doesn’t want you to do that, but by giving you the option of doing it they overcome some of the initial resistance of people switching.

                              1. 7

                                The context has switched, though.

                                Before, many prospective buyers of Macs used Windows, or needed Windows apps for their jobs.

                                Now, many more prospective buyers of Macs use iPhones and other iOS devices.

                                The value proposition of “this Mac runs iOS apps” is now much larger than the value proposition of “you can run Windows on this Mac”.

                                1. 2

                                  There’s certainly some truth to that but I would imagine that most iOS users who buy Macs are doing so because iOS doesn’t do everything that they need. For example, the iPad version of PowerPoint is fine for presenting slides but is pretty useless for serious editing. There are probably a lot of other apps where the iOS version is quite cut down and is fine for a small device but is not sufficient for all purposes.

                                  In terms of functionality, there isn’t much difference between macOS and Windows these days, but the UIs are pretty different and both are very different from iOS. There’s still some risk for someone who is happy with iOS on the phone and Windows on the laptop buying a Mac, even if it can run all of their iOS apps. There’s a much bigger psychological barrier for someone who is not particularly computer literate moving to something new, even if it’s quite like similar to something they’re more-or-less used to. There are still vastly more Windows users than iOS users, though it’s not clear how many of those are thinking about buying Macs.

                                  1. 2

                                    There are still vastly more Windows users than iOS users, though it’s not clear how many of those are thinking about buying Macs.

                                    Not really arguing here, I’m sure you’re right, but how many of those Windows users choose to use Windows, as opposed to having to use it for work?

                                    1. 1

                                      I don’t think it matters very much. I remember trying to convince people to switch from MS Office ‘97 to OpenOffice around 2002 and the two were incredibly similar back then but people were very nervous about the switch. Novell did some experiments just replacing the Office shortcuts with OpenOffice and found most people didn’t notice at all but the same people were very resistant to switching if you offered them the choice.

                            3. 1

                              That “developer” might means Apple developers.

                            4. 3

                              Here is the source of truth from WWDC 2020 about the new boot architecture.

                              1. 2

                                People claimed the same thing about T2 equipped intel Macs.

                                On the T2 intels at least, the OS verification can be disabled. The main reason you can’t just install eg Linux on a T2 Mac is the lack of support for the ssd (which is managed by the T2 itself). Even stuff like ESXi can be used on T2 Macs - you just can’t use the built in SSD.

                                That’s not to say that it’s impossible they’ve added more strict boot requirements but I’d wager that like with other security enhancements in Macs which cause some to clutch for their pearls, this too can probably be disabled.

                              2. 10

                                … This is the Intel model it replaces: https://support.apple.com/kb/SP818?viewlocale=en_US&locale=en_US

                                Two TB3/USB-C ports; Max 16GB RAM;

                                It’s essentially the same laptop, but with a non-intel CPU/iGPU, and with USB4 as a bonus.

                                1. 1

                                  Fair point! Toggling between “M1” and “Intel” on the product page flips between 2 ports/4 ports and 16GB RAM/max 32GB RAM, and it’s not clear this is a base model/higher tier toggle. I still think this is pretty stingy, but you’re right – it’s not a new change.

                                2. 5

                                  These seem like replacements for the base model 13” MBP, which had similar limitations. Of course, it becomes awkward that the base model now has a much, much better CPU/IGP than the higher-end models.

                                  1. 2

                                    I assume this is just a “phase 1” type thing. They will probably roll out additional options when their A15 (or whatever their next cpu model is named) ships down the road. Apple has a tendency to be a bit miserly (or conservative, depending on your take) at first, and then the next version looks that much better when it rolls around.

                                    1. 2

                                      Yeah, they said the transition would take ~2 years, so I assume they’ll slowly go up the stack. I expect the iMacs and 13-16” MacBook Pros to be refreshed next.

                                      1. 3

                                        Indeed. Could be they wanted to make the new models a bit “developer puny” to keep from cannabalizing the more expensive units (higher end mac pros, imacs) until they have the next rev of cpu ready or something. Who knows the amount of marketing/portfolio wrangling that goes behind the scenes to suss out timings for stuff like this (billion dollar industries), in order to try to hit projected quarterly earnings for a few quarters out down the road.

                                        1. 5

                                          I think this is exactly right. Developers have never been a core demographic for Apple to sell to - it’s almost accidental that OS X being a great Unix desktop, coupled with software developer’s higher income made Macs so popular with developers (iOS being an income gold mine helped too, of course).

                                          But if you’re launching a new product, you look at what you’re selling best of (iPads and Macbook Air’s) and you iterate on that.

                                          Plus, what developer in their right mind would trust their livelihood to a 1.0 release?!

                                          1. 9

                                            I think part of the strategy is that they’d rather launch a series of increasingly powerful chips, instead of starting with the most powerful and working their way down - makes for far better presentations. “50% faster!” looks better than “$100 cheaper! (oh, and 30% slower)”.

                                            1. 2

                                              It also means that they can buy more time for some sort of form-factor update while having competent, if not ideal, machines for developers in-market. I was somewhat surprised at the immediate availability given that these are transition machines. This is likely due to the huge opportunity for lower-priced machines during the pandemic. It is prudent for Apple to get something out for this market right now since an end might be on the horizon.

                                              I’ve seen comments about the Mini being released for this reason, but it’s much more likely that the Air is the product that this demographic will adopt. Desktop computers, even if we are more confined to our homes, have many downsides. Geeks are not always able to understand these, but drive the online conversations. Fans in the Mini and MBP increase the thermal envelope, so they’ll likely be somewhat more favourable for devs and enthusiasts. It’s going to be really interesting to see what exists a year from now. It will be disappointing, if at least some broader changes to the form factor and design aren’t introduced.

                                            2. 1

                                              Developers have never been a core demographic for Apple to sell to

                                              While this may have been true once, it certainly isn’t anymore. The entire iPhone and iPad ecosystem is underpinned by developers who pretty much need a Mac and Xcode to get anything done. Apple knows that.

                                              1. 2

                                                Not only that, developers were key to switching throughout the 00s. That Unix shell convinced a lot of us, and we convinced a lot of friends.

                                                1. 1

                                                  In the 00s, Apple was still an underdog. Now they rule the mobile space, their laptops are probably the only ones that make any money in the market, and “Wintel” is basically toast. Apple can afford to piss off most developers (the ones who like the Mac because it’s a nice Unix machine) if it believes doing so will make a better consumer product.

                                                  1. 2

                                                    I’ll give you this; developers are not top priority for them. Casual users are still number one by a large margin.

                                                2. 1

                                                  Some points

                                                  • Developers for iOS need Apple way more than Apple needs them
                                                  • You don’t need an ARM Mac to develop for ARM i-Devices
                                                  • For that tiny minority of developers who develop native macOS apps, Apple provided a transition hardware platform - not free, by the way.

                                                  As seen by this submission, Apple does the bare minimum to accommodate developers. They are certainly not prioritized.

                                                  1. 1

                                                    I don’t really think it’s so one-sided towards developers - sure, developers do need to cater for iOS if they want good product outreach, but remember that Apple are also taking a 30% cut on everything in the iOS ecosystem and the margins on their cut will be excellent.

                                              2. 2

                                                higher end mac pros

                                                Honestly trepidatiously excited to see what kind of replacement apple silicon has for the 28 core xeon mac pro. It will either be a horrific nerfing or an incredible boon for high performance computing.

                                        2. 4

                                          and can’t be upgraded past 16GB of RAM.

                                          Note that RAM is part of the SoC. You can’t upgrade this afterwards. You must choose the correct amount at checkout.

                                          1. 2

                                            This is not new to the ARM models. Memory in Mac laptops, and often desktops, has not been expandable for some time.

                                          2. 2

                                            I really believe that most people (including me) don’t need more than two Thunderbolt 3 ports nowadays. You can get a WiFi or Bluetooth version of pretty much anything nowadays and USB hubs solve the issue when you are at home with many peripherals.

                                            Also, some Thunderbolt 3 displays can charge your laptop and act like a USB hub. They are usually quite expensive but really convenient (that’s what I used at work before COVID-19).

                                            1. 4

                                              it’s still pretty convenient to have the option of plugging in on the left or right based on where you are sitting so disappointing for that reason

                                              1. 4

                                                I’m not convinced. A power adapter and a monitor will use up both ports, and AFAIK monitors that will also charge the device over Thunderbolt are pretty uncommon. Add an external hard drive for Time Machine backups, and now you’re juggling connections regularly rather than just leaving everything plugged in.

                                                On my 4-port MacBook Pro, the power adapter, monitor, and hard drive account for 3 ports. My 4th is taken up with a wireless dongle for my keyboard. Whenever I want to connect my microphone for audio calls or a card reader for photos I have to disconnect something, and my experiences with USB-C hubs have shown them to be unreliable. I’m sure I could spend a hundred dollars and get a better hub – but if I’m spending $1500 on a laptop, I don’t think I should need to.

                                                1. 2

                                                  and AFAIK monitors that will also charge the device over Thunderbolt are pretty uncommon

                                                  Also, many adapters that pass through power and have USB + a video connector of some sort only allow 4k@30Hz (such as Apple’s own USB-C adapters). Often the only way to get 4k@60Hz with a non-Thunderbolt screen is by using a dedicated USB-C DisplayPort Alt Mode adapter, which leaves only one USB-C port for everything else (power, any extra USB devices).

                                              2. 1

                                                I’ve been trying to get a Mac laptop with 32GB for years. It still doesn’t exist. But that’s not an ARM problem.

                                                Update: Correction, 32GB is supported in Intel MBPs as of this past May. Another update: see the reply! I must have been ignoring the larger sizes.

                                                1. 3

                                                  I think that link says that’s the first 13 inch MacBook Pro with 32GB RAM. I have a 15 inch MBP from mid-2018 with 32GB, so they’ve been around for a couple of years at least.

                                                  1. 1

                                                    You can get 64GB on the 2020 MBP 16” and I think on the 2019, too.

                                                1. 5

                                                  I like this article a lot. A related guideline I often use at work is: will this be harder to add in the future? When one person says “You ain’t gonna need it” a common response is “Of course you’re going to need it”, and “You don’t need it yet” is met with “but we’re already adding X, so we might as well add Y simultaneously.” These discussions aren’t really about predicting the future, but they look like it. It’s easy for a team to be split down the middle, and for people to hold no strong opinions either way.

                                                  But often people will agree that adding X doesn’t close any doors to adding Y, and it doesn’t make it harder or slower to add Y in future. The aggregate effort for Y hasn’t changed, plus we’ll know more if we defer it. I’ve found this is often an easier path to consensus, even if it’s an indirect path to YDNIY.

                                                  1. 15

                                                    That is beyond weird to me.

                                                    Perhaps I’m missing some context, but if I were instructed by my boss to help create a fake but top secret version of my team’s consumer product, I wouldn’t be anywhere near as comfortable as this guy seemed to be.

                                                    Maybe it’s that this was all happening pre-Snowden, or maybe it’s an American vs. Australian thing. But I wouldn’t even trust that the gentlemen involved were actually from (or perhaps solely from) the agency they claimed to be, let alone that their work was even halfway legal or constitutional.

                                                    1. 4

                                                      Perhaps the biggest thing that was lost with Snowden’s revelations is trust. Undoubtedly intelligence services are doing all sorts of important work, and some amount of secrecy is required so we need to trust them to make the right calls. But many people feel – quite rightfully – that this trust has been betrayed, which among other things makes the legitimate work harder.

                                                      Generally speaking though, if they’re willing to spend this much effort on something then it’s probably targetted at a very specific and narrow use case, which is probably a legitimate one.

                                                      1. 2

                                                        But many people feel – quite rightfully – that this trust has been betrayed, which among other things makes the legitimate work harder.

                                                        Exactly. Government does not have the resources nor capability to do everything it legitimately must. No single organization can. So, it goes out to other companies who do have the required knowledge and ability. That’s just how it would all fit together. This worked great so long as it was expected that our government was beholden to the ideals behind our nation’s founding.

                                                      2. 3

                                                        The tone of this post doesn’t surprise me. I’ve read a few stories like this from the early 2000s, and heard a couple more in person—all with fairly similar attitude towards helping the government.

                                                        I do think pre/post-Snowden explains the disconnect. We’ve been living in a post-Snowden world for 7 years now. At this point it’s hard to remember, but I don’t think many people seriously believed in government mass surveillance before. Not at the scope or scale Snowden revealed.

                                                        1. 4

                                                          Edward Snowden’s revelations were important, but there was lots of information about government mass surveillance before him. Echelon was written up in 1988 (PDF), and investigated by the EU parliament in 2000. There was coverage of post-9/11 warrantless surveillance in 2005; the NSA’s data collection facilities were widely covered in 2006.

                                                          This list is the “greatest hits” – high profile reports that had governmental confirmation and lots of media coverage. Hackers, engineers, and the “alternative scene” have shared their knowledge of surveillance possibilities and probabilities for years before that. It might not have been in the general public’s consciousness, but it was certainly in programmer consciousness.

                                                          1. 5

                                                            I think you overestimate how many programmers are woke Stallman-esque hackers. They’re greatly outnumbered by hackers who just want to do neat stuff, who in turn are greatly outnumbered by people who just code because it’s their job. I doubt the average PHP web dev of the early 2000s was any more knowledgeable about these possibilities and probabilities than the general public. I’m not confident most programmers at companies like Apple even knew or cared much about the subject. And even for those who suspected, possibilities are nothing compared to the staggering volume of hard evidence leaked by Snowden.

                                                            1. 2

                                                              Indeed. I must say I was a bit surprised that Snowden’s revelations generated such chock. Perhaps it’s because as someone living outside the USA, I’ve known that the various TLAs have been hoovering up data since forever.

                                                              Back in the day one could run M-x spook in Emacs to generate some “bad words” to trigger Echelon in your email or Usenet sig. Running it now shows the layers of history geeks have imagined the NSA will get hot and bothered by:

                                                              Riot South Africa Reflection Jyllandsposten IACIS AHPCRC MI6 Red Cross Nike Cartel de Golfo Crash Arellano-Felix E. Coli Southwest afsatcom

                                                            2. 3

                                                              This isn’t really related to mass surveillance though.

                                                              1. 2

                                                                No, but the revelations about mass surveillance significantly undermined public trust in government intelligence agencies—as you pointed out in your other comment.

                                                          1. 10

                                                            This has been discussed before, but I was wondering if anything has changed.

                                                            Several difficulties and problems were raised … have they been addressed?

                                                            Does anyone use it?

                                                            1. 11

                                                              I’ve been there for close to 2 years, and have tried to build my own SSB server from scratch (in a non-JS language). Feel free to ask any questions. For starters:

                                                              • The low level protocol (transport encryption, RPC and discovery) is very well documented.

                                                              • The application level protocol has almost no documentation, and what’s there is outdated. You really have to resort to reverse engineer behaviour from existing applications, or reading other’s code.

                                                              • Replication/gossip mechanism is very inefficient, which leads to clients (especially mobile ones) spending a lot of time during the initial sync. There’s a newer gossip protocol which fixes some of these problems, but has zero documentation, and there’s only one implementation (in JS). There are no plans to port it to other languages since there’s a lot of tricky edge cases in there.

                                                              • Yes, the JSON encoding was a mistake. There’s a new format using CBOR, but it’s still a few ways off in terms of mainstream usage in the network.

                                                              • There are questionable decisions at the application level. For example, anyone can assign you new profile pictures or visible names, which can–and has–lead to bullying/name-calling.

                                                              In terms of community, it’s mostly tech-centric, most discussions are either about SSB itself, or related protocols. The overall spirit is positive, focusing on sustainable living, gardening, off-grid, etc.

                                                              However, the community is very small. This can get tiring, considering that most clients will show any replies to threads you follow at the top of your timeline (you will see the same 3 to 5 people all the time).

                                                              1. 4

                                                                I’ve also built a partial SSB implementation, in C++. I found a C port of the CLI tool (in an obscure Git repo hosted in SSB), which helped immeasurably with the secure handshake and packet codec. I used Couchbase Lite [of which I’m the architect] as the database. This proved a lot faster than the JS data store, but I still found that pulling all the subscribed content from one pub (on which I’m following a handful of accounts) resulted in a 600MB database. It would have helped if the protocol had an option to pull only messages back to a certain date.

                                                                I’d love to know more about the new protocol, but not if the only way is to decipher a JS codebase.

                                                                It’s a shame they’re so JS-centric. That’s a serious problem for iOS, which has security restrictions that disallow JITs outside of a web browser. Not to mention embedded systems. (And on a personal level I dislike doing serious programming in JS; it feels like sculpting in Jell-O.)

                                                                1. 3

                                                                  There are two C implementations of the low level protocol: one for the secret handshake and one for the boxstream (transport encryption). There’s also some integration tests that you can run against your implementation to validate that everything works.

                                                                  As for the new replication protocol: the new CBOR format includes “off-chain” contents, which means that the actual log only contains a hash of the post content. This should make initial sync much faster, since clients only fetch the chain of hashes, without downloading anything else.

                                                                  Messages can also be downloaded out of order, so you only download what you want, if you have the hash for it. As most things, though, the only implementation is in JS.

                                                                  As for the database, I planned to use SQLite, but never got far enough to test that. I’m unconvinced that the log is a good abstraction for the kind of apps that SSB is used right now (social media). There are future plans to plug more applications on top of the log replication, but that’s in the long term, while the current use-cases are suffering from it.

                                                                  Edit: wanted to say, though, that for me the biggest block when developing an SSB implementation is the lack of documentation w.r.t. the application-level protocol, and forcing you to develop everything on top of the log abstraction. The JSON formatting can be painful, but solved via forking some json library and doing some changes (hacky, but it works).

                                                                2. 1

                                                                  Yes, the JSON encoding was a mistake.

                                                                  Was JSON not fast enough?

                                                                  1. 4

                                                                    It’s not about JSON per-se, but more about how message signing works. In SSB, every post is represented as a JSON blob signed with your publick key, and it expects other clients to validate this, as well as produce valid JSON messages.

                                                                    The spec goes over the requirements of a valid message, as well as the steps to compute a valid signature. Unfortunately, it assumes things like key order (which the official JSON spec doesn’t say anything about), indentiation, spacing, etc. (This all goes back to how the V8 engine implements JSON.stringify()). This adds a lot of complexity when implementing SSB in another language, as most JSON libraries won’t care about specific formatting when printing, and specifically the key order requirement makes it quite complicated.

                                                                    All in all, it’s not the end of the world, but it adds enough friction to make SSB pretty dependend on the “blessed” javascript implementation.

                                                                3. 5

                                                                  I use it regularly. Why..? After being a heavy social media user on the usual platforms, I’ve pretty much removed myself and don’t participate, but Scuttlebutt is the exception because it’s a fun place to be. Nothing more, nothing less.

                                                                  1. 3

                                                                    What difficulties and problems were raised?

                                                                    1. 12

                                                                      There’s two big obstacles I’ve seen that seem specific to Scuttlebutt:

                                                                      • There’s no easy way to share identities across devices. If you want to use Scuttlebutt on your computer and your phone, they’ll be separate accounts.

                                                                      • The protocol is built around an append-only log, which I’m not convinced is a good principle for any social network. Inadvertent mistakes are forever (eg. I post an unboxing photo that has my unredacted invoice visible; I paste an embarrassing link by accident).

                                                                        It also seems like you could grief pub servers (the Scuttlebutt “hub nodes” that federate content more widely). What happens if someone posts a bunch of illegal content to the pub? As I understand it, all the pub users will pull that content down. You might be able to blacklist certain posts in your client, and you can block users, but their content is still on your device. (Bitcoin has faced similar problems.)

                                                                      1. 2

                                                                        Your objection to log storage is valid, but there are ways around it. The data format could have ways to redact posts out of the log while leaving the overall integrity intact; in fact all revisions of a post other than the latest one could be redacted.

                                                                        Of course the redactions need to be propagated, and there’s no guarantee every copy will be redacted, but that’s an intrinsic problem with most P2P protocols, since distributed caching/replication is so important for availability.

                                                                        1. 1

                                                                          Good points.

                                                                          Also ironically where Facebook could have a chance to differentiate themselves, but chose to go in almost the exact different direction:

                                                                          • “with federated networking you are trusting each and every host that your host ever federated with to delete a post, with us, once you click delete it is gone. Worldwide. Same with sharing: if you share something with your close friends it stays there. With a federated network it depends on every host around the globe sticking implementing the rules correctly and sticking to the rules.”

                                                                          • fortunately IMO Facebook messed up massively early on and now everyone in tech now they are fundamentally untrustworthy.

                                                                        2. 8

                                                                          The main problem I saw when I looked into it was that it was a single program rather than a well-defined protocol that anyone in the ecosystem could implement.

                                                                          This might have changed by now, but (for instance) there were aspects baked into the protocol that fundamentally prevented you from building a compatible client unless you used a JSON serializer with the exact same behavior as node.js, because the cryptographic identity of a post was based on a checksum of the output of that particular serializer rather than some inherent property of the post. An easy mistake to make, but one with far-reaching consequences.

                                                                          1. 6

                                                                            That’s my issue as well. It relies heavily on the interaction of a few dozen NodeJS repos. Different frontends all rely on the same backend code, making it not-that-diverse.

                                                                            Also, while the protocol itself is well documented and designed, there are some obvious shortcomings. The protocol relies on hashes of pretty-printed JSON. The last time I checked for documentation on HOW to pretty-print that json, it was documented as “like v8 does it”. Tell you - it’s REALLY hard to format JSON like V8 JSON.format(x, true) does. Especially floating point numbers.

                                                                            Now this could easily be fixed by changing the protocol from hash-of-pretty-printed-json-subobject to hash-of-blob. ({"data": "{\"foo\" 42}", "hash": ...} vs. {"data": {"foo": 42}, hash: ...} But you can’t do that without breaking compatibility. Worse, relying on hash-chains, you need to implement the old behaviour to be able to verify old hash chains.

                                                                            1. 5

                                                                              That’s my top issue too. Which is sad, because there’s so much prior art on canonicalizing JSON, going back to 2010 or so.

                                                                              The spec is fine, but limited. It doesn’t cover all of the interactions between peers; there are plenty of messages, and properties in the schema, that aren’t documented.

                                                                              A lot of the discussion and information about the protocol takes place on Scuttlebutt itself, meaning it’s (AFAICT) invisible to search engines, and accessible over HTTP only through some flaky gateways that often time out.

                                                                              The main client is (like everything else) written in JS, so it’s a big Electron app, and in my experience very slow and resource hungry. I only joined two pubs and followed a handful of people and topics, but every time I fire up the app, it starts a frenzy of downloading and database indexing that lasts a long time. (They use a custom log-based DB engine written in JS.)

                                                                              Part of the slowness must be because when you join a pub you’re implicitly following every user of that pub, and I believe all of the IDs they’re following. So there’s kind of an explosion of data that it pulls in indiscriminately to replicate the social graph and content.

                                                                            2. 4

                                                                              Reading the previous discussions shows a degree of frustration and scepticism. I’m sure it’s changed, and many of the questions will have been addressed, but the previous discussions are unconvincing.

                                                                              Here are some links to some of them … it’s worth reading them in context:





                                                                            3. 2

                                                                              I’m one of the core developers. Happy to answer any questions.

                                                                              1. 1

                                                                                Despite the fact it’s in a ‘junkyard’, I believe this issue remains unresolved, which I believe effectively means that:

                                                                                1. scuttlebutt is not cross platform and only works on x86
                                                                                2. It’s difficult to implement scuttlebutt libraries and clients in other languages

                                                                                Limiting development to people who like nodejs, and limiting usage to x86 devices (when it seems like the sort of concept that should work well with mobile devices) massively reduces its appeal.

                                                                                I would be happy to find that I’m wrong.

                                                                                1. 3

                                                                                  You’re off on this one. I’m running SSB on ARM64 just fine, also many pubs are actually just raspberry pis on some closet.

                                                                                  I is still difficult to implement SSB in other languages mostly because of the amount of work than technical challenges. The mistakes of the past are well understood at this point even if not fixed. At the moment there are 2 Rust based implementations and one based in Go. IIRC there is also implementations in Elixir and Haskell but I am not sure how far they are. I’ve toyed with a client mixing C and Lua (just a toy prototype but it worked).

                                                                                  1. 2

                                                                                    It definitely didn’t work on arm last time I tried, so it’s good to hear they’re making some progress. It was the prototype Haskell implementation which pointed to that issue as a blocker: it looks like it hasn’t been updated since, so probably doesn’t work.

                                                                                    1. 2

                                                                                      I know it is not what you’re looking for but the JS implementation works fine under Linux/ARM64, which is also how the mobile apps are running.

                                                                                      My daily driver is a Surface Pro X with an ARM64 CPU. I’ve run go-ssb as native ARM32 binary and the Electron based client apps under win32 x86-32 bits emulation. The reason for it is just that for the love of all that is sacred I can’t find out how to build the native nodejs modules as ARM64 binaries.

                                                                                    2. 1

                                                                                      Could you link to the Haskell implementation? I’d be interested in working on that!

                                                                                1. 1

                                                                                  Personally, I approach storage and publishing as two separate solutions. For storage I basically use the filesystem; directories by year, subdirectories for each shoot/event (along with some generic folders like “202005 - misc Toronto” for one-off snaps through the month). I also use macOS’ filesystem tags to help me find things. (I wrote a blogpost about this a few months ago, if you want more detail.)

                                                                                  I also have a macOS Automator script set up that watches my “finished” directory for new images, and copies it to my “publishing queue” directories if it matches a regex. This is a nice little timesaver; Automator is one of those things that I use rarely but always think “I should do more with this.” I’ve got another Automator script that proxies through to a little scp wrapper so I can right-click a photo and upload it to the web, leaving the URL in my clipboard.

                                                                                  I don’t currently have a publishing process that I’m happy with, though my current side project is aimed at tackling exactly this. I use some static gallery generators to generate preview galleries, and I have a self-built gallery on my website, but I don’t update it much. The new version will be much better, and is almost ready. It’s just that last 80% that’s the challenge. :)

                                                                                  1. 7

                                                                                    I’ve been a Feedbin subscriber since 2013 and love it (and it’s open source too). I subscribe to a lot of feeds (and some Twitter accounts) for my curation of Read Rust. In fact Read Rust integrates with the Feedbin API to make sharing posts from easier.

                                                                                    I use its web UI on the desktop and Reeder on my phone. Recently Feedbin support was merged into NewsFlash an in development RSS reader using GTK and Rust. Eventually I’d like to use that on the desktop. I miss Reeder for Mac since moving to Linux.

                                                                                    1. 3

                                                                                      My setup’s similar to yours – I’m a long-time Feedbin user, and really like it. I use the web interface on the desktop, and an old version of Reeder on my iPad/iPhone.

                                                                                      Feedbin has a neat email newsletter tool, too – every user gets a custom email address for subscribing to newsletters. These show up in Feedbin just like a regular feed. I find I don’t read newsletters if they land in my email inbox, but they’re a natural fit in an RSS reader.