1. 42
  1. 5

    The new hotness is a single binary blog, with all the posts embedded inside.

    mutters kids these days <closes tab>

    Seriously though, why? Generating a static site from a bunch of files or some content in a DB is a Solved Problem™. I guess it’s the age old truth that it’s way more fun to design and code a blog engine than to… blog.

    1. 13

      That’s what I’d expect, one is programming and the other is writing. Most of us find programming easier.

      1. 8

        to me, part of the idea is that the “blog” can be extended to be much more than a blog. if you use hugo or a similar tool, how would you implement a tool like my age encryption thing or even a public ip fetcher?

        you noted that building a blogging engine is fun - it is! a lot of fun! my take is that people should focus on making a website that’s not just a blog - it’s a fun, personal engine that happens to contain a blog. focusing on “blogs are a solved problem” is missing the point, imo.

        1. 6

          I use Hugo, and to provide the tools you do I’d simply use a separate subdomain and program for each tool. Why should my blog be concerned with the details of age encryption or even echoing a user’s IP? A blog, in my mind, is a collection of articles. And a website can be backed by several programs.

          In fact, I provide a variety of small sub-services to myself, and they’re simply separate programs. This has the added benefit that I could shuffle them between machines independently of one another.

          1. 4

            Right. It feels like we’re basically reinventing CGI scripts, but worse.

            1. 1

              why should my blog concern itself with…

              if you enjoy what you’re currently doing, i’m not here to persuade you. i optimize my personal projects for fun, reliability, and maintainability. :3 building separate services and making subdomains for trivial functions isn’t a good time imho. i also don’t like that i’d have to think pretty hard if i wanted to integrate those services on the blog (say, dynamically load a users IP in a post). with one binary, everything is already all connected. but honestly to defend the “validity” of “my way” feels meh. i like parades - the more the merrier!

            2. 1

              I use software to generate HTML from Markdown for my blog, but that’s only part of my site. I have some other static content, some CGI service, a gemsite, etc.

              As far as I can see from my limited understanding of the linked post, it’s basically akin to serving a single HTML page, albeit compressed, from a compiled executable. You still need to generate the HTML before zipping it. So you’ve just shifted one step of the integration pipeline.

              By using tried and tested technology, I can focus on producing the stuff I want. I’ve already run into some issues with gemini server software and it reminded me why I don’t want to deal with that kind of stuff.

              https://portal.mozz.us/gemini/gerikson.com/gemlog/tek/gemserv-update-woes.gmi

              In summary and conclusion, serving my site as a single executable would give me nothing other than bragging rights, and like I stated above, I’m too old for that particular flavor of ordure.

            3. 5

              Same reason why it’s so much more fun to write a game engine than a game 😅

            4. 4

              just because you do something doesn’t mean you need to blog about it. there are too many blog posts about blog engines and static site generators.

              1. 7

                just because you read yet another post about blog engines and static site generators doesn’t mean you need to comment about it. there are too many comments even in this thread complaining about posts about blog engines and static site generators.

                1. 3

                  i disagree. there should only be posts complaining about it. ;)

                1. 5

                  There’s a more specific issue: https://github.com/golang/go/issues/51971

                  From the article:

                  The use of Seek doesn’t even seem necessary. The fs.File interface implements Stat, which can be used to obtain the size.

                  This is wrong. Seek is necessary because the whole point of using http.FileServer is to get range request support. If you don’t want/need range requests, you can use the trivial handler listed in the post much more simply than the hacks to get zip files to work with http.FileServer here.

                  1. 4

                    That seems silly. A FS can contain several files. It’s plausible that some are seekable and some are not. HTTP is entirely capable of mixing resources that support Range and resources that don’t on the same server. http.FileServer should simply do the right thing on non-seekable files; erroring on them is trash design.

                    1. 2

                      http.FileServer predates fs.FS and made the sensible requirement that all files support seeking. The problem is that now it’s being retrofitted to use with fs.FS, which sensibly enough doesn’t require seeking. Really there should be some new handler that can work within the limitations of fs.FS. That’s more or less what issue #51971 is about.

                    2. 2

                      You can hack range support though even if the underlying file does not support seeking… although not in a great way, but you can simply read + discard what the client doesn’t want and respond with the ranges the client does want.

                      1. 1

                        Seems like this could be used to DOS the server.

                  2. 3

                    The “single binary” blog thing has mystified me. (I saw a couple recent posts but didn’t comment.)

                    I guess this is because rsync has pitfalls? The trailing slash issue was mentioned recently and that is indeed annoying.

                    Otherwise there’s no difference between a single file and a tree of files. And in fact the tree of files is better because you can deploy incrementally. This matters for a site like https://www.oilshell.org (which is pretty big now).

                    When I update a single blog post it takes about 1 second to deploy because of rsync (and 1 second to rebuild the site because of GNU make, which I want to replace with Ninja)

                    Using git would also be better than copying the entire blog, since it implicitly has differential compression.


                    Another thing I’m mystified by is people writing their own web servers, setting up their own nginx, managing certificates, etc.

                    I just used shared hosting (Dreamhost, and I tried Nearly Free Speech, but didn’t think it was as good). But you can also use Github pages?

                    I think shared hosting dropped the ball a lot in terms of marketing and user friendliness. They didn’t figure out a good deployment and tooling story. They left everyone alone with shell and I guess most people are not fluent with shell. To be honest I wasn’t as fluent with it when I first got the account! (way back in 2009)

                    But now I think it’s by far the best solution because it gives me a standard / commodity interface. I can use git / rsync / scp, and I don’t have to manage any servers. It’s “serverless”.

                    Maybe another problem is that shared hosting was associated with PHP. I think it fell out of favor because it couldn’t run Rails or Django well.

                    But then people started writing static site generators, and doing more in client side JavaScript, and it perfect for those kinds of sites. So IMO it’s a more stable Netlify or Github pages.

                    I’ve heard people complain that with Github pages you are never sure when the changes will be visible. Did you do something wrong, or is the site just slow? Not a problem when you have shell access.

                    I guess the other downside is that it costs some money, like $5 or $10/month, but I view that as a positive because the business model is self-sustaining. Dreamhost has been stable since 2009 for me. There are many fewer dark patterns. The only annoyance is that they advertise a lower rate first year rate for domains, so the second year you’re surprised. But pretty much everyone does that now :-/


                    Another trick I use is to bundle 10,000 static files together in a zip file and serve it with a FastCGI script .wwz

                    https://github.com/oilshell/wwz

                    This is basically for immutable archives and cuts down on the metadata. I have automated scripts that generate 10,000 files and rsync will slow down on all the stats(), so I just put them in a big zip file. And again the nice thing is that I am not maintaining any web servers or kernels.

                    1. 4

                      Some people prefer to use things they understand, and part of that might be standing up a web server for self-hosting. It’s not rocket science to configure nginx + Let’s Encrypt… And part of it, I suspect, is that shared hosting sites come and go, so not having to be reliant upon a host which may be out of business tomorrow is also a benefit.

                      I had mostly negative experiences using early shared hosting interfaces (ugh cpanel..), wordpress was a little better (and you could self host it) but it is waaaay overkill for simple static sites. Not to mention it’s a beast to set up.

                      Of course there are risks and additional costs with self-hosting stuff. But I’d expect the person who created a new shell to understand the trade-offs/benefits, generally speaking, between using what already exists and making something brand new :D

                      1. 3

                        Yeah I think part of it is that early shared hosting sites did suck.

                        I remember I used 1and1 and it was full of sharp edges, and the control panel was also confusing.

                        Specifically I think Dreamhost is quite good. I have been using it since 2009. Nearly Free Speech is pretty good, but it seems to use some kind of network-attached disk which makes it slow in my limited experience. (Also, they don’t seem to advertise/emphasize that the shell account is BSD! Important for a Linux user.)

                        I also maintain nginx and LetsEncrypt on a VPS for a dynamic site I’ve been using for over a decade. (Used to be Apache and a certificate I bought, etc.)

                        I think shared hosting is by far superior, although as mentioned I recognize there are a few sharp edges and non-obvious choices. I’m not responsible for the kernel either.

                        I would call using shared hosting “using what already exists” … And standing up your own VPS as “making something new”. It will be your own special snowflake :)


                        One interesting thing is that nginx doesn’t actually support multi-tenant sites with separate user dirs and .htaccess. So all shared hosting uses Apache as far as I know. That isn’t a problem since I never touch Apache or nginx on shared hosting – I just drop files in a directroy and it’s done.

                        But I would say there’s strictly less to understand in shared hosting. I rarely log into the control panel. I just drop files in a directory and that’s about it. My interface is ssh / rsync / git. Occassionally I do have to look up how to change an .htaccess, but that’s been maybe 2-3 times in ~12 years.

                      2. 2

                        Another thing I’m mystified by is people writing their own web servers, setting up their own nginx, managing certificates, etc.

                        I can give you my reason for this one. I run my site and other stuff on a simple vps with nginx. That was not my first ides for a solution, but I discovered that there are hardly any offerings for what I want and what I want is dead simple: I have a bunch of html pages and I want them to be served on my domain. You can not do this with the well known platforms. They all require you to put the files in a git repository.

                        I am not going to set up some scripts that automatically commit new versions to a git repository and then push that. There is no need to, the html gets generated from other projects that are already version managed. sudo apt install nginx actually the easiest solution here. I want to be able to do one rsync command because that is all what is needed.

                        If you look a bit further, there are some companies that offer this, but the costs are always equal or higher than renting a vps and they will always limit you flexibilty. There are probably some giant tech companies that have a way of getting this done for free or for very cheap, but it will inevitably involve accounts, convoluted dashboards/configuration and paying constant attention not to accidentally use a resource that costs a lot of money.

                        Perhaps it sounds complicated for someone who has never seen a shell, but managing a site with nginx, certbot and rsync is to about as simple as you can get for this.

                        1. 1

                          Paying $5/mo for a VPS has improved my computing QoL immensely.

                          • It hosts my sites
                          • I use it to host tmux to have IRC access
                          • I can run random net stuff from it

                          All in all a great tool and service.

                        2. 2

                          I guess this is because rsync has pitfalls?

                          it’s pretty simple for me: i don’t like using rsync. i don’t like the awkward syntax (trailing slash or no?), i don’t like managing files, putting my SSH keys everywhere, etc. i understand rsync just fine - i use it professionally.

                          but when it comes to my free time, i optimize for maintainability, reliability, and fun. i can’t explain the titillating feeling i get when i see a binary compile & launch my entire contained website - and i don’t have to :D it’s just good subjective fun.

                          it’s nice to look at my website’s code and know that it’s self-contained and doesn’t rely on this-being-here or that-being-there. it launches exactly the same way on my local machine as it does on my remote machine. also, my website isn’t just about writing posts - it’s also about providing tools. for example, i include an rss reader, an rss fetcher, an age decryption utility, an IP fetcher, etc - all without any javascript! and it’s all right there & testable locally! it’s fun to treat my website like an extension of my ideas instead of “yet another blog” - and my one-binary approach really lends itself to that.

                          I can use git / rsync / scp, and I don’t have to manage any servers.

                          very fair. i have a server sitting around that i like poking once in awhile

                          But you can also use Github pages?

                          in my blog post, i have a section dedicated to why i don’t want to use some-other-hosting-platform for my personal website. the idea that my website is almost entirely under my own control is very important to me. eventually, i plan on porting it to a small SBC running in a van! a future post will describe this process :D

                          not knocking anyone who chooses something like github pages btw, it’s just not for everyone - it depends on the persons values.

                          i hope this helps make things less mystifying.

                          1. 2

                            trailing slash or no?

                            You should always use trailing slashes with rsync, that’s it. rsync -a src/ dst/

                            1. 2

                              i think that this is misleading given that trailing slashes on the source can change rsync’s behavior

                              A trailing slash on a source path means “copy the contents of this directory”. Without a trailing slash it means “copy the directory”.

                            2. 1

                              Yes this makes sense for the dynamic parts of the site. I can see why you would want to have it all in a single language and binary.

                              I’m more skeptical about the static content, but it all depends on what your site does. If you don’t have much static content then the things I mentioned won’t come up … They may come up in later in which case it’s easy enough to point the static binary at a directory tree.

                              (Although it looks like Go has some issues there; Python does too. That’s one reason I like “borrowing” someone else’s Apache and using it directly! It’s very time-tested.)

                              1. 1

                                totally! if i had like 10k pages of content i’m sure things would look different. :3

                            3. 2

                              Why? Because Outsource All The Things! limits what you can actually do. I wrote my own gopher server [1][2] because the ones that exist don’t do what I want (mainly, to serve up my blog via gopher). And while I didn’t write my own web server, I do run Apache, because I then have access to configure it as I want [3] and by now, you can pretty much do anything with it [4]. For instance my blog. I have several methods of updating it. Yes, I have the obligatory web interface, but I’ve only used that about a handful of times over the past two decades. I can add an entry directly (as a file), but because I also run my own email server (yes, I’m crazy like that) I can also add blog entries via email (my preferred method).

                              Is it easy to move? Eh … the data storage is just the file system—the hard part is getting the web server configured and my blogging engine [5] compiled and running. But I like the control I have.

                              Now, I’m completely mystified and baffled as to why anyone in their right mind would write a program in shell. To me, that’s just insane.

                              [1] gopher://gopher.conman.org/

                              [2] https://github.com/spc476/port70

                              [3] You would never know that my blog is CGI based.

                              [4] Even if it means using a custom written Apache module, which I did back in the day: https://github.com/spc476/mod_litbook, still available at http://literature.conman.org/bible/.

                              1. 1

                                I find the limitation in outsourcing to often be more about learning than features per se. I could well write a gopher server myself, but I’d expect it to be worse than existing gopher servers. Still a worthwhile endeavor, if I didn’t know how to write servers.

                                In a similar vein, I created a wiki engine that works over gemini and is edited with sed commands. It’s probably the worst wiki UX in existence, but last I checked it was the only wiki running natively on gemini (only). I learned a lot in the process, and some other people also found the approach interesting (even if not useful). Maybe it even inspired some other gemini software.

                                So yes, there are many good reasons for writing software even when the problem has already been solved.

                                1. 1

                                  I think either extreme is bad – outsourcing everything or writing everything yourself.

                                  Also not all “outsourcing” is the same. I prefer to interface with others using standard interfaces like a www/ dir, rather than vendor-specific interfaces (which are subject to churn and lock-in).

                                  Shell helps with that – it’s reuse on a coarse-grained level. I am OK with dependencies as long as I can hide them and contain them in a separate process :)

                                  I hinted at similar ideas on the blog:

                                  http://www.oilshell.org/blog/2021/07/cloud-review.html

                                  More possible concept names: Distributed Shell Scripts and Parasitic Shell Scripts. These names are meant to get at the idea that shell can be an independent control plane, while cloud providers are the dumb data plane.

                                  A prominent example is that we reuse multiple CI providers but they are used as “dumb” resources. We have shell to paper over the differences and make programs that are portable between clouds.

                                  This also relates to my recent “narrow waist” posts – a static website is a narrow waist because it can be produced by N different tools (my Python+shell scripts, Jekyll, Hugo, etc.) and it is understood by multiple cloud providers (shared hosting, github pages, even your browser).

                                  So it’s a good interface to serve as the boundary between code you own and code that other people own.

                                  Here is a comment thread with some more links (login required):

                                  https://oilshell.zulipchat.com/#narrow/stream/266575-blog-ideas/topic/Comments.3A.20FastCGI.20and.20shared.20hosting

                              2. 1

                                better to just have the zip and binary together in one folder