1. 6

    I use Hugo, and I’m surprised how content I’m with it. I switched from a Jekyll clone, and despite its quirks (mostly originating from its Go ancestry) it is pretty usable and also pretty fast.

    What really interest me is why would I not use S3 and Cloudfront? Not that my blog written in a language spoken by a tiny population could possibly get overwhelmed with traffic, but my monthly fees are below 1$, and the site could handle practically unlimited load, should it face it. Also no hassle with hosting, security upgrades, SSL certificates. I have hosted it on a simple DO instance, and it was totally OK, yet AWS is superior in every possible respect for serving purely static pages.

    Instead of rsync the aws cli can be used to sync the bucket, it is incremental, and also pretty fast.

    1. 4

      I think it would be simple to use this tool in conjunction with S3/Cloudfront. I use Jekyll to build a local version of my site and then s3_website to push it to AWS. I like that the tool I use for building the site doesn’t tie me to a particular hosting strategy. (AFAICT the linked script only uses rsync to build the site locally in a target directory, not to deploy it to some remote host.)

      1. 2

        Sure it would be simple, it is just as simple as using rsync, I’m curious why would anybody run httpd and self-host given the drawbacks. Maybe I have a different usecase in mind, and that doesn’t let me see, or simply a matter of different preferences…

        1. 4

          I don’t think httpd is for self-hosting. I think it’s for previewing things locally (similar to jekyll’s serve feature).

          1. 2

            Thank you for your comments.

            Yes, rsync(1) is just to copy source files (html, css, md, etc) to ssg working directory. Yes, I use httpd(8) in debug mode (not like a daemon) locally, just for previewing. Why httpd -d? It’s already installed on OpenBSD by default. On macOS you can use python -m SimpleHTTPServer for the same purpose.

            1. 2

              I ran into problems with SimpleHTTPServer because it has no concurrency: a single client can block everything. You can work around this with the threading mixin, something like: https://kdecherf.com/blog/2012/07/29/multithreaded-python-simple-http-server/

          2. 1

            I’m curious why would anybody run httpd and self-host given the drawbacks.

            We are talking about serving static files, for a personal blog (out if a disk cache)…what are the drawbacks again?

            1. 1

              Even if you have only http facing the public internet you need to track the security reports. I found having to track CVEs for the few services I had on my machine too burdensome. Also you may need TLS, which also has its overhead, and hosting costs more than on S3 imho. If you need the machine for other purposes that may make the equation a bit different though.

              1. 3

                All true. But of course, some people do this kind of thing as a hobby, or as part of their jobs. Others might find it a fun learning exercise, and even rewarding.

                1. 2

                  Oh, it totally fell out of my sight. My bad.

                  I abandoned the pet server approach to have more of my limitd freetime devoted to my blog, creating content, as I already did enough ops at work.

        2. 2

          +1 to Hugo, I’d say “pretty fast” is underselling how ridiculously fast it is (at least compared to other popular static site generators).

          Re: why not S3+Cloudfront?

          I started with this a while ago. The problem is you end up using something like Route53 to get your custom domain and TLS, which ultimately ends up costing you $2-3/mo per domain altogether, which adds up pretty quickly when you have a bunch of domains. Not to mention the ordeal of managing AWS and their massive dashboards and weird config syntax.

          These days I use Github pages + Cloudflare for DNS/TLS in their free tier. If I were up for migrating again, I’d consider using Netlify which is great by all accounts and supports some basic dynamic forms that are handy for static sites (contact form, etc).

          1. 2

            I agree that Route53 costs can add up, but if your DNS provider can serve “APEX entries” you can get away without that, if I recall correctly (or maybe you can use Cloudfare then?). My single domain site takes <1€/mo, (Route53 + S3 + Cloudfront).

            Regarding Netlify: recently I have seen some useful/impressive tools they have open-sourced, so I’d also consider their services.

        1. 1

          Why httpd? It’s only available for OpenBSD (and maybe other BSDs), and while I haven’t (yet) looked at the script, it would seem that it offeres no specific feature, which a more portable and lightweight server like darkhttpd would offer.

          But otherwise, very interesting, I will consider maybe adopting and tweaking it to use on my site.

          Also if the “pps” value is important, consider using cmark, since it’s known for it’s speed, that is beside it being commonmark compliant.

          Edit: The choice makes sense from a OpenBSD perspective, and taking into consideration that it’s written for personal use, and not as a general replacment tool.

          1. 7

            Why httpd? It’s only available for OpenBSD (and maybe other BSDs), and while I haven’t (yet) looked at the script, it would seem that it offeres no specific feature, which a more portable and lightweight server like darkhttpd would offer.

            I think the author is an OpenBSD user, and is probably trying to do as much with the base system (which includes httpd) and a minimum of additional packages as possible. darkhttpd is an additional package, and doesn’t come with a man page.

            Also if the “pps” value is important, consider using cmark, since it’s known for it’s speed, that is beside it being commonmark compliant.

            cmark hasn’t been ported to OpenBSD yet. Granted, the author could probably build it from source or do their own port, but that would defeat the point of the exercise.

            1. 2

              Yeah I guess I should write a manpage for darkhttpd at some point.

              1. 1

                is probably trying to do as much with the base system (which includes httpd) and a minimum of additional packages as possible

                I understand that, but seeing that neither rsync, lowdown or entr are in the base system, it doesn’t seem too problematic. Especially since, this is my main point, httpd locks out all other systems from using it. To be fair, it’s a shell script and each to change, but it just seems like a weird design choice.

                1. 5

                  I see what you’re talking about, but I don’t think it’s weird. I think the author is scratching their own itch, and documenting how they did it on the off chance that it’s useful or interesting to others. This isn’t an other-focused project with widespread adoption as a goal. If it were, then it would make more sense to just dump the output in a directory that could be transferred to a Apache2 or nginx host.

                  Personally, I’m more surprised that the author chose to build HTML and XML files directly from the shell script instead of creating templates and populating them using sed (and maybe awk). Now that would be badass.

                  1. 3

                    Yes, I’m an OpenBSD user. httpd(8) can be replaced with anything, it’s just for previewing. lowdown, entr, rsync are small and easy to install software:

                    # pkg_add lowdown entr rsync
                    

                    My HTML template is so tiny, I’ve decided to place it right into the shell script. Also it’s a bit faster than my version of my awk script.

                    Thank you!

                    1. 2

                      This is good stuff, and I had thought of extending it for my use case, but since I’m already getting what I want from Pelican I should stick with what works for me. Though I do use rsync to deploy; that’s baked into the Makefile. :)

                      1. 2

                        Cool! :)

                        One more point:
                        78.5MB ruby + jekyll + its dependences (25 gems)
                        0.82MB lowdown + rsync + entr

              2. 6

                The author mentions to use OpenBSD for servers and workstations, so why not? Portability isn’t so important for personal tools. And httpd is only used for the local preview feature and could easily replaced by any other http server, including python -m http.server. The 6 line httpd config is already included in the <160 lines of shell scripts.

                The script is very short, reading such a thing before commenting generally improves sites like lobste,rs ;)

                1. 2

                  I just finished reading it, since I couldn’t open it on my phone (the scripts content type is application/octet-stream making it un-openable on a phone). And as I guessed, it is easy to change, but I would still maintain that

                  cd "$DOCS" || exit
                  doas httpd -d -f "$conf"
                  ;;
                  

                  meaning that it requires root to run, as compared to darkhttpd which is fully functional without. It just seems like a simpler option, and again, I get it’s not a real problem, and I implied I would chance it if I were to use it anyways.

                  1. 3

                    meaning that it requires root to run

                    It requires root to drop privileges and chroot. I wouldn’t have it any other way.

                    Installing a third party daemon to avoid using the built-in features of the base system does not seem simpler really.

                    1. 1

                      Installing a third party daemon to avoid using the built-in features of the base system does not seem simpler really.

                      Well it does on a non-OpenBSD system (although one should remember that darkhttpd isn’t a daemon, but a single C file server, comparable to python -m http.server), which was the perspective I was looking at it. But from the authors point of view, his choice is reasonable.

                      1. 1

                        For systems with Ruby, the httpd function from the un package is the best choice. To require un and evaluate the httpd function:

                        $ ruby -run -e httpd
                        

                        Cute right?

                        Advantages over Python:

                        • multi-threaded
                        • supports SSL / TLS
                        • suitable for production deployment
                        • automatic indexes show modification time and file size
                        • don’t have to remember if it’s -m http.server or -m SimpleHTTPServer

                        More info: ruby -run -e help httpd

                        1. 2

                          Folks, you are right, I’d use Python or Ruby on macOS, for example. Installing them on OpenBSD is not an option for me, they are huge.

                          One of the reasons to replace Jekyll is having Ruby as a dependency. With my script all I need to build my site is: lowdown(1) and rsync(1). Also both can be replaced, but they are the fastest options I’ve tried so far.

                2. 2

                  Being written by the OpenBSD team implies a certain amount of simplicity, correctness, security, and documentation. Ive never heard of darkhttpd. If those are only choices, I’d default on OpenBSD’s implementation unless I had good reason to do otherwise.

                  1. 2

                    I would agree that httpd would be a better choice, when it comes to actually hosting the content on the web, and one could only choose between the two. But

                    1. Neither is that the case (one has more options) and even then it would be bad design if necessitated the use of a server on a remote system, for static content
                    2. If I didn’t misunderand anything, httpd is used to locally debug and preview ones site. I would personally argue that in this case, security would be less important if the choice means the script isn’t usable on systems, httpd doesn’t run on (which I assume is something the author wants, since the script is written with POSIX compliance in mind)

                    So there seems to be a good reason, at least not to use httpd as the default server to locally preview ones site. And all I want, is to suggest darkhttpd since it actually is quite nice. suckless even recommends it, if that is a standard one can use. But in general it would be nice if one could use shell variables or something comparable to it

                    1. 2

                      I havent researched enough on either solution to say much more. Someone online just yold me it was their web server to replace more bloated or messy stuff. I was just saying some will default on stuff written by people that normally focus on quality or security. We got quite a few here that like the suckless stuff, too.

                      Your reasons for checking out the other one sound good, too. I have no issue with it. Just saying why some default on OpenBSD code.

                      1. 2

                        I’ll probably remove serve from my script in the future to keep it simple. Here is my use case right now:

                        Run httpd(8) as a daemon once:

                        # rcctl enable httpd
                        # printf 'server "localhost" {\n  listen on * port 80\n}\n' > /etc/httpd.conf
                        # doas rcctl start httpd
                        

                        Then run ssg watch for previewing:

                        $ cd src/romanzolotarev.com
                        $ DOCS=/var/www/htdocs ssg watch
                        
                1. 3

                  …but the majority of posts get zero comments.

                  1. 10

                    Honestly, who in their right mind works in any business creating purely digital works and doesn’t keep a personal copy?

                    I mean, one of the primary benefits of working with digital content is the extreme ease with which you can make copies. If you’re a writer, either of prose or code, why wouldn’t you keep copies of things you made and are proud of?

                    For example, I keep copies of my code in a personal repository, so I can look through it and see how I solved a particular issue in the past. It’s also a convenient way of keeping track of what you’ve done in the past, in case you want to summarize your experience. It’s so easy to forget things without having some catalog to go through.

                    The temporary nature of online hosting or publishing is not a new thing. This story has been told many times before. People in industries exposed to this problem should be aware of the issue by now, and act accordingly.

                    1. 11

                      I’m starting to think I want a browser that stores my complete history (without videos or the like). I’m already downloading every page I visit, and the storage requirements for text are tiny by modern standards. All too often I want to cite or link to some page, months or years later, only to discover that it’s gone.

                      1. 5

                        This is one of the great things about the Pinboard bookmarking service; if you pay for the $25/year (IIRC) premium service, it will crawl and store every page you bookmark.

                        1. 5

                          Is there a way to export those saved crawls and keep a copy locally? I’d feel a bit more comfortable with ultimate control over my archive, even if I currently have faith that pinboard will stick around.

                        2. 1

                          You could make your browser cache $LARGE_MB so it hopefully doesn’t clear too often or at all and then make a backup of it every week. Unfortunately it would not have an index, but you could grep it or sort by size to find your largest image/audio/video files. I used to do more cache diving when I had dial up.

                          1. 1

                            The Perma.cc site provides this sort of service - it’s aimed at maintaining information about online sources that are cited.

                            1. 1

                              I wanted this and tried to write a Chrome extension to do it but got stuck on https://code.google.com/p/chromium/issues/detail?id=104058 (fetching the response body)

                              1. 1

                                I’ve always thought that URL shortening services should give an option to serve a cached copy instead.

                              2. 8

                                I’d love to keep copies of code I wrote for former employers, but in the current legal environment, I’m afraid to. If I go back to some of that code to remind myself of how I solved a particular problem once, and then I solve a similar problem in a similar way for my current employer, am I opening myself (or even my current employer) up to a lawsuit? Or, worse, a criminal charge of some kind? Sergey Aleynikov went to jail for doing pretty much exactly this on some bullshit trade secrets charge.

                                1. 6

                                  Yes. I’m aware of this issue. I just chose to ignore it. I never take large or recognizable parts of my previous commercial work, and I never copy code to a new project verbatim (if the source is not owned by me).

                                  I think this is fair, and I’m fairly convinced this is not provable by any previous employer to be infringing on any of their copyright.

                                  I also make sure I don’t copy the work in a way they can legally intercept (where I live, an employer cannot easily & legally eavesdrop on your personal communications, not even on your company email).

                                2. 6

                                  What you say makes sense, but it also ignores the larger issue — so, you do save all your work locally, but how exactly do you save (and present) the proof that it’s been published? Being published online also means that the work has generated some comments; if you’re doing work-for-hire as in the original article, you can’t exactly own the comments and all the jazz, too, right?

                                  This is also one of the issues with the closed-source software — outside of the org it’s written in, you most likely have no proof that you ever wrote any of that (well, outside of the being familiar with it, and making some seemingly intelligent discussions about it, with a future interviewer etc). Sadly, since most people take close-source for granted, in my experience I’ve found that it seems like having the actual proof and source-code being available online is actually the outlier, and is deemed kind of weird, without getting much extra points. (Although, to their credit, some progressive open-source-related companies do require published source code right in their job descriptions nowadays.)


                                  BTW, I found this link on DragonFly digest, but a couple of days ago I have also witnessed another shutdown by AOL, noticed through DMOZ’s OpenBSD/Companieswww.aolserver.com/, which started redirecting directly to aol.com now. It has been marked as a redirect automatically, and now has been removed by an editor. (Which is kinda funny, because AOL actually owns DMOZ!)

                                  Frankly, I don’t understand why companies shut these things down — don’t they realise that people will shortly remove links to these shutdown properties (e.g. aolserver.com is gone from dmoz now), it’ll expire from the search engines etc. No matter how stale the content is, it’s likely still generating at least some revenue, and hosting being as cheap as it is, I find it hard to believe that they’re loosing money on simply keeping stuff like that online, all at the same time that we hear the stories about cybersquatters being able to afford to register random domains just for the sake of showing ads to the misplaced users, having their whole business model made just around showing ads on random domains with no real content whatsoever!

                                  1. 3

                                    This is also one of the issues with the closed-source software — outside of the org it’s written in, you most likely have no proof that you ever wrote any of that

                                    I don’t think that is really an issue with closed source software, and I don’t think it’s really that big of a deal. In fact, it’s a “problem” that has existed in almost all professions, for a very long time, and employers solved it by doing background checks and contacting references.

                                  1. [Comment removed by author]

                                    1. 4

                                      Because especially among desktop users, people simply don’t care about their init system. Those that do care and do not like systemd will simply install BSD or something.

                                      Plus at least from what I can see, most of the people who care about this are developers, not end users. An OS without a good userbase will probably not gain traction.

                                      I may be wrong about all this, but that is just my gut feeling.

                                      1. 3

                                        As an application developer I feel like I don’t really have a dog in this race. I rarely interact directly with the init system. Even when I do interact with it its through “/etc/init.d/whatever start” or “service whatever start”. Nothing I would get passionate about. Maybe somebody can explain what all the fuss is about? As an application developer, what’s a concrete reason I should care?

                                        1. 3

                                          As a desktop user, I experienced a small blip of caring when systemd broke my ability to boot, but otherwise I think youre right. :)

                                        2. 1

                                          I would venture that there are two crises here: the goals and design of systemd, and then debian’s subsequent takeover by the freedesktop crowd. Of the two, the second is very problematic for a fork, because it reveals that poor decision making is now majority endemic in the debian voters – so where does the rot stop. Given their public statements (http://0pointer.net/public/gnomeasia2014.pdf), It’s likely that the freedesktop people will now do everything in their power to attempt to both cement their position and make it difficult for those who disagree, so debian is likely to be a bad base for further work.

                                          Personally, I’m both moving everything to freebsd and advising all my clients to do the same.

                                          1. 1

                                            But how does this affect anyone besides Debian developers?

                                      1. 0

                                        Every “Veteran Unix Admin” should be able to install a recent Debian without systemd…

                                        1. 5

                                          Today, yes. My understanding is that the fear is about creeping dependencies.