Threads for spc476

  1. 23

    Good self-hosted software really needs to have leak-proof abstractions. Too many leaks means too much admin intervention, which is in short supply for hobbyists.

    Gitea is one that does this well IMO. A single binary, a single config file, and a single data directory are all key. Contrast this with my MediaWiki instance that needs a dozen packages installed, config is split between httpd.conf and LocalSettings.php, and data is split between the static files and database files. Not as bad as some, but still not ideal.

    1. 3

      Configuration woes are exactly why I’m considering writing my own web server instead of using Apache or Nginx. My needs are simple:

      • Static files only.
      • TLS (that kind of sucks but I don’t have a choice).
      • Content negotiation for languages (some of my posts are dual English/French).
      • Nice to have: cache everything in RAM (my website is that small).

      Then maybe one day I’ll stare into the SMTP abyss.

      1. 24

        You sound like a person who is yet to discover the warm convenience of https://caddyserver.com/

        1. 2

          I am indeed. Thanks for the tip.

        2. 4

          Using libretls makes using TLS painless.

          1. 3

            Nice to have: cache everything in RAM (my website is that small).

            Since you have a static site I’d assume that this is mostly handled by file system anyways, minus compression. I wonder how much one really gains from that, especially when using the right syscalls.

            Then maybe one day I’ll stare into the SMTP abyss.

            If you want a simple SMTP config OpenSMTPD is the way to go. See the examples section of their man page.

            Of course that doesn’t cover delivery (IMAP, etc.) or anti-spam measures. The good thing here is that it doesn’t change much.

            1. 1

              Then I’d advise going full OpenBSD and use opensmtpd, https and relays for the config simplicity and practical usage.

          2. 1

            Making self hosting easy would be very possible, but I think the amount of work it would take is just too much. For an ideal system, everything needs to be standardized. There needs to be some way to just browse a UI and click “install”.

            Yes I know there are many projects that advertise this, but none of them actually work well. It’s a monumental amount of work for the projects to patch and configure every service so it fits a set of standards for backups, SSO, webserver access, certificates, etc. And then last I checked these projects were not containerized so there were major issues doing things like OS updates because PHP/etc would update which would present major issues for running services.

            And then there is just no money in it to make it worth the effort.

          1. 8

            Text files checked into the source repository. Unix is my IDE.

            1. 12

              Starting October 1st, we will start to randomly select tenants and disable basic authentication access for… IMAP, …

              IMAP, at least, isn’t going away, just “basic authentication.” You’ll still be able to use it, but you won’t be logging in with a username and password. Instead, you’d need to use a predictably non-standard MS extension to oauth2 called xoauth. It relies on Azure and O365/Exchange and I didn’t find it to be difficult, but whether or not clients will adopt it is another matter.

              I ran into this last month. I had to work on some IMAP integration recently with O365, and about a week after getting something working MS switched off legacy auth in my employer’s tenant. I had to switch over to xoauth2. The only difficult part was finding any information about how to do it. Microsoft’s documentation, given how much of it there is, is usually scattered and often hard to piece together.

              1. 3

                My workplace did the same. Seems like the easiest way is to pretend to be Thunderbird, and use its client ID and secret.

                I don’t know what this adds other than friction, but maybe that’s the point.

                Using OAuth like this just doesn’t work. I thought corporations learned that during the rise of smartphones, when people were plucking OAuth credentials out of apps for fun.

                1. 1

                  I used client modules in go and node. Roughly, it consisted of setting up a new azure app, granting that app permission to use imap in the tenant, enabling oauth flows for the app, creating a principle (user)/client I’d/secret, granting O365 permission for the user. Then it was a matter of enabling imap access on the specific O365 user account/mailboxes, and finally granting the principle access to the boxes. There were a few powershell scripts mixed in.

                  It wasn’t obvious or documented and took some trial and error. This has been my experience for anything having to do with azure and integration with virtually any of Microsoft’s other cloud products: business central, o365, etc. It’s a time suck.

                  After all that, my understanding based solely on my admin’s findings is that IMAP is sticking around, at least for paying (enterprise) customers. However, vanilla, plaintext, basic auth goes away.

                2. 2

                  IMAP, at least, isn’t going away, just “basic authentication.”

                  That’s not exactly how the mail is phrased:

                  [The Enterprise] only supports the use of the approved email client Outlook, Outlook 365 available in our [The Enterprise] tenant, or Outlook for iOS and Android.

                  That makes it sound like you can only use Outlook.

                  Also: do they even read what they’re writing? This is so user-hostile it isn’t even funny:

                  “If you use an alternate email program that relies on POP3, IMAP or SMTP, like native mail programs in iOS and Android, expect your [The Enterprise] email connection for that app to stop functioning when Microsoft chooses to disable these protocols.”

                  1. 2

                    [The Enterprise] only supports the use of the approved email client Outlook, Outlook 365 available in our [The Enterprise] tenant, or Outlook for iOS and Android.

                    That’s the policy of the tenant. A lot of orgs completely disable POP/IMAP in Exchange, and just use OWA/EWS/MAPI. As stated, IMAP is staying if your org uses it, just with OAuth.

                    (The funny thing is Outlook doesn’t support the OAuth extensions to IMAP, so it’ll just be using EWS/MAPI.)

                    “If you use an alternate email program that relies on POP3, IMAP or SMTP, like native mail programs in iOS and Android, expect your [The Enterprise] email connection for that app to stop functioning when Microsoft chooses to disable these protocols.”

                    Mobile apps are usually using EAS instead of IMAP.

                    1. 1

                      I agree totally agree regarding the user hostility bit. Just to be clear, my employer /is/ using O365 in an enterprise tenant.

                      In my case the integration service relied on IMAP with regular authentication. My admin read that they were going to disable IMAP on some tenants in the future, and the following week they did for ours. He re-enabled it, but the integration wasn’t able to authenticate.

                      After a day or two of spelunking though the MSDN nightmare we were able to piece together a process to get the oauth client credentials flow, suitable for integration because it doesn’t require manual consent, working against their tenant/O365. Works like a charm. Still IMAP, different auth scheme.

                      So you’re right, IMAP may be going away in general, but it’s alive and well in their enterprise environment. “Basic authentication” no longer works, at least for us.

                      Edit: I’m not using an outlook /client/. I’m using imap clients in go, node, and at least initially c#.

                      1. 1

                        Wow! I didn’t expect this to get posted here. Neat.

                        Anyway, yes, user-hostile. But by now, I’m used to this attitude from The Enterprise. They brought the company I was working for and by God, we’re going to do things their way (for development, they’re doing The Enterprise Agile method, which is mostly the stuff on the right hand-side of the Agile Manifesto and ignoring the left hand-side).

                    1. 1

                      Decision tables seem nice, but I don’t think they scale much. I’m thinking of work, where we have 13 feature bits for an account, so that’s at 8,192 rows. Yes, I know, too much. But it’s legacy code that’s grown over the years.

                      1. 3

                        They don’t scale all that much, true! But it’s usually obvious in the first couple minutes that they won’t scale to your problem, meaning you’ve only lost a couple minutes applying the wrong tool. That’s why I love decision tables: they’re a useful tool with extremely low cost.

                        1. 1

                          We have over 50 feature bits. It’s insane.

                          You don’t need one row per combination in a decision table though, pretty much always there are redundant combinations that you can communicate with -

                        1. 4

                          The first part of the post up to Maud reads to me like more condemnation of Markdown as a format unsuitable for any real content. If you want definition lists, and not some garbage front matter that isn’t standardized across implementations or codified in the spec and doesn’t render properly by the basic parsing tools, then don’t use Markdown—and you’ll get more features like figures, admonitions, detail/summary/collapsible syntax, blockquotes with attribution, file imports, section numbers, table of contents, ya know, stuff for actually posting rich text content for books and blogs. A lot of the hurdles encountered could have been circumvented for a long time with Asciidoctor alone. Slap on a standalone static binary tool like Soupault and you can do a lot of ‘dynamic’-looking things by pre/post-processing and augmenting that output by parsing metadata that already exists in elements on the page without resorting to all of this custom frontstuff, custom elements, shortcodes, et. al. syntax hacks that lock you into specific static site generators and breaking the lightweight markup spec.

                          I’m not calling out the author specifically for having gone down this route—I had too—but we as a community need to start choosing the right tool for the job and Markdown ain’t it.

                          But in the end the post endorses Nix in lieu of more complex tools (Kubernetes, Hereby/Dokku), so it’s okay.

                          1. 3

                            I mean, markdown is suitable for “real content”, but I needed to extend it as things got more complicated. I assume I’d have to do this with any other template language too.

                            1. 4

                              I do think it’s a valid criticism of Markdown that it doesn’t have clean, well-defined extension points. E.g. reStructuredText / LaTeX / HTML all allow you to add some additional tag-like things without leaving the format, to an extent that Markdown doesn’t.

                              1. 1

                                It always needs extensions once you try to write a blog or book… or even a semifancy README. The base language does not give you enough for these tasks and the extensions collide with other implementations pretty quickly because of how Spartan the basics are.

                              2. 2

                                I’ve been flailing a bit on it as a ~project, but I’ve been trying to write about this rough part of the map lately. I may poke some of the people in this subthread when I get the next post sorted…

                                Resisting the urge to vomit much context here, but a little: over the past ~decade, I’ve had multiple projects drag me through the pain of trying to automate things that deal with soft/idiomatic use of presentational markup.

                                Last year I started prototyping a ~toolchain for documentation single-sourcing on top of a very flexible markup language I’d wanted to try out for a while, D★Mark.

                                Both building it and figuring out how to write about it have made me wrestle with markup a lot. I have a nagging self-doubt that they’re all trite observations that everyone else knows, but it’s helped me make sense of a lot of things that felt unrelated. TLDR: everything sucks (at least, outside of narrow use-cases) but I have more empathy as I get a better handle on why :)

                                1. 1

                                  I use a custom markup language that is an unholy mix of Markdown, Org Mode and my own custom quirks but the messages aren’t stored in that format, but in the rendered HTML. That way, I’m not stuck with any one Markdown (or other) format. And it’s from the HTML format that I re-render the output for gopher and Gemini.

                                  1. 2

                                    To each his own at the end of the day, but sticking to a spec takes a lot of burden off of yourself and lets you participate in a larger community. I’m curious what was missing from Org Mod for you? It seems it lacks some of the features of AsciiDoc or reStructuredText.

                                    1. 2

                                      My main point was not to write custom markup languages, but rather to use whatever markup language you want, but don’t store the blog post in said markup language, but the final HTML output.

                                      As to your question, my custom markup was inspired by OrgMode [1] (for the block level elements) and Markdown [2] (more for the inline elements). And I wasn’t trying to use a format to replace HTML, just make it easier to write HTML [3].

                                      I am also able to gear my custom markup to what I write. I know of no other markup language (admittedly, I don’t know many) that allow one to mark up acronyms. Or email. The table format I came up with is (in my opinion) lightweight enough to be tolerable to generate an HTML table (again, just to support what I need). I’ve also included some bits from TeX (using two grave accents (ASCII 96) to get real double quotes for example), as well as other characters (like ‘(C)’ will turn into the Unicode Copyright character). And a way to redact information (as can be seen here).

                                      Standards are good but they fall short in supporting what I need. And there is no way I would want to force my blogging quirks onto other people.

                                      [1] I don’t use Emacs, nor any other tool that support it.

                                      [2] There are too many different Markdown standards to pick just one.

                                      [3] Which is an element I think many people miss about Markdown—it wasn’t developed to replace HTML, but to make it easier for John Gruber to write his blog posts, which is why you can type HTML in his version of Markdown.

                                    2. 1

                                      Interesting! Do you edit the HTML directly or re-render it after changes?

                                      1. 1

                                        If I need to update an entry after it’s posted, I will edit the HTML directly.

                                  1. 2

                                    I’ve been looking at this stuff as a way to drive traffic to my new blog but my sensibilities are at odds with that goal. I’ve tweaked my Markdown pipeline so much that I think at least 3 (out of less than 10) of my posts so far would fail to render properly anywhere else. I don’t like the idea of the full text of my stuff showing up on other sites. I should make a Twitter account and automatically cross post to that, but I deleted my Twitter account in 2016 and I can’t make myself sign back up. I do manually post links to my new stuff on my Mastodon account, but that’s even less discoverable, and somewhat on purpose.

                                    1. 4

                                      Why not just post the title and the link to your blog posts? It’s what I do for Facebook and Linked In (manually).

                                      1. 3

                                        Markdown tends to get flavored like that because it lacks so many features for actually writing anything beyond a comment or the simplest of READMEs. It’s a shame developers haven’t embraced more lightweight markup languages. It’s not just you, but you can’t even take the same Markdown document from GitLab and assume it renders fine on Gitea.

                                        1. 2

                                          I don’t like the idea of the full text of my stuff showing up on other sites.

                                          The linked page says you don’t have to:

                                          then publishing copies or sharing links to third parties

                                          Just a link and a summary would probably be okay as well, right?

                                        1. 1

                                          A long time ago when I was learning Perl I asked how I could easily share my idea. Someone mentioned perl2exe which packaged the Perl interpreter with my program. Of course the binary was huge. When I see sudden claims of compile to binary benefits on a language that usually does not do this, I look at the filesize and ask: is this a perl2exe kind of thing?

                                          I don’t know what’s in the CI artifact of doc2dash but it’s 225MB. I’m guessing yes, this is a perl2exe kind of thing. Which is fine but has tradeoffs like static compilation. Generally, I like this method of software distribution instead of please install a dev environment, it’s just that compiling to binary (or some kind of app) is not the strong suit of Python or similar languages.

                                          On the other side, you have things like GraaVM. You can tell that this is doing something different than shipping the whole platform because it might explode spectacularly on your project. :)

                                          1. 3

                                            Oh yeah it totally is a “Docker for endusers a of Python apps”.

                                            It does it in a somewhat sophisticated way (Greg’s blog posts are super interesting and usually super over my head), but effectively it bundles the Python runtime along with the app. Interestingly the sizes vary among the platforms quite a bit.

                                            ‘Tis the price of static linking for dynamic runtimes, for better or worse.

                                            1. 2

                                              This sort of bundling up is something that Tcl seems to have gotten mostly right with starpacks (or, at least, the binary sizes of doing so are in the 1-10MB range, instead of 100MB+), and Janet’s JPM build system also does a good job of bundling things into an executable like this.

                                              Granted, many scripting languages don’t have this as a focus.

                                              1. 2

                                                This is a reason why I like Lua. I’m able to compile my gopher server in Lua and compile it into a 300K executable. And if I compile with default options and debugging information, the executable is only 925k in size.

                                            1. 1

                                              It seems like the suggested API would be a little more logical if (a) everything were in the memory_block and (b) try_expand() allowed shrinking, as well.

                                              struct memory_block {
                                                  void* ptr;
                                                  size_t size;
                                                  size_t alignment;
                                              };
                                              
                                              memory_block allocate(size_t size, size_t alignment);
                                              void deallocate(memory_block block);
                                              bool try_change_size(memory_block block, size_t new_size);
                                              

                                              Having to pass the length / alignment as separate parameters is just an opportunity to get things wrong.

                                              The allocator can maintain a map of addresses to CRC16s of the other parameters to ensure that old / wrong memory_block structures are not being passed in.

                                              1. 1

                                                I might suggest you try this out. Take a program you wrote using malloc()/realloc()/free() and convert it to use this API (and the default implementation could just wrap malloc()/realloc()/free() without using the extra data for a “proof-of-concept”). This way, you can get a feel for how it would be to use this API in an actual program.

                                              1. 7

                                                My perspective, from working on a high-performance memory allocator:

                                                Alignment is actually pretty difficult to support in memory allocators. We align everything to 16 bytes but for anything larger than that we just round the requested size up to that power of two (alignment for power-of-two-sized allocations is easy to support). C11 / C++11’s aligned_alloc or POSIX’s posix_memalign don’t convey the fact that aligned allocations might get a lot more padding than you’d expect. Windows has a very interesting aligned allocation API that allocates such that a specific offset is aligned. This should be useful for OO languages that provide some kind of object header (e.g. a refcount) and so might want a 32-bit field in the header and then for the next field to be 16-byte aligned. Unfortunately, it actually does this by allocating something 16-byte aligned and giving you a pointer into the middle of the allocation, so you don’t actually save memory.

                                                Metadata storage overhead is pretty low for sizeclass-based allocators. In snmalloc, there are two metadata structures:

                                                • The pagemap, which is an array with one entry per chunk (by default, the chunk size is 16 KiB), reserved in the address space and lazily backed by real memory. This uses two pointers worth of space per 16 KiB.
                                                • The slab metadata. This is a structure allocated per chunk for managing free lists. It is quite small and there’s one per 16 KiB chunk (for small allocations) or one shared across multiple 16 KiB chunks (for large allocations).

                                                The sizeclass for a chunk is actually stored in the low bits of one of the pointers in the pagemap, so we’re storing <1 B for every 16 KiB of allocations. Calling this wasteful is nonsense. Storing a size_t outside of the allocator to track the size is far more wasteful.

                                                Oh, and the size that the user knows is not always the useful one. There are two sizes: the size of the underlying allocation and the size that the user requested. The former is greater than or equal to the size of the latter. If the user gives us the latter, it’s about as expensive for us to get the former as it is for us to get it if they give us nothing. The only thing that we actually use this information for is sanity checking: if the user thinks they’re freeing a size that doesn’t match the sizeclass of the allocation, then we can abort.

                                                For the third problem, malloc_usable_size is non-standard, but it (or some different spelling) is available basically everywhere. Jemalloc has an API that returns the allocated size (snmalloc has a compat implementation of this). I’d like to see this in the standard.

                                                I completely disagree on the fourth point. The best thing to do with realloc is throw it away. Allocators that subdivide or merge allocations are useful only on very resource-constrained systems. For high-performance allocators, realloc is no more efficient than doing a malloc and a copy because that’s what it’s going to be doing almost all of the time. In snmalloc, if the new size is the same sizeclass as the old, realloc is a no-op, if it’s different then we do an alloc and a copy. It is a headache for performance because it’s either a very cheap or a very expensive operation depending on the two sizes. I would love to see realloc deprecated in C2y. It’s also very easy to get into UB-land accidentally with realloc. Because it is logically a free and a malloc, it is UB to use any pointer that you had before the allocation. The only safe way for it to work in C++ would be if the signature looked something like std::unique_ptr<T> realloc(std::unique_ptr<T> old, size_t newSize).

                                                On the solutions, I agree that allocate_at_least is nice. Jemalloc has had this for years and snmalloc also has it, I believe other allocators probably do as well.

                                                1. 1

                                                  Disclaimer: I coming from this as a C programmer.

                                                  I do agree that realloc() is a problematic function. The “allocate/resize/free” nature of it is really only true for C89, which includes this sentence missing in the C99 and C11 standards:

                                                  if size is zero and ptr is not a null pointer, the object it points to is free.

                                                  C99 and C11 only state that realloc() resizes the memory block, so a call to realloc(ptr,0) could still return a valid pointer to a block of 0 bytes. But even this is further confounded by this (which does appear in the C89, C99 and C11 standards):

                                                  Whether the calloc, malloc, and realloc functions return a null pointer or a pointer to an allocated object when the size requested is zero.

                                                  It could be cleared up by mandating that allocating 0 bytes returns a valid pointer to 0 bytes, and that while it can’t be used (because there’s no memory actually allocated) it still exists until free() is called up on. Then realloc() only allocates or resizes memory, never frees it.

                                                  The issue with removing realloc() entirely is that programmers will just code their own version, because it is convenient when creating a dynamically sized array, like:

                                                  foo *p = NULL;
                                                  size_t i = 0;
                                                  size_t max = 0;
                                                  size_t delta = 16;
                                                  
                                                  while(moredata)
                                                  {
                                                    if (i == max)
                                                    {
                                                      foo *n = realloc(p,delta * sizeof(foo));
                                                      if (n == NULL)
                                                        handle_the_error();
                                                      p = n;
                                                      max = delta;
                                                      if (delta < SIZE_MAX)
                                                        delta *= 2;
                                                      else
                                                        handle_no_more_memory();
                                                   }
                                                    ...
                                                  }
                                                  
                                                  foo *n = realloc(p,i * sizeof(foo));
                                                  if (n != NULL)
                                                    p = n;
                                                  

                                                  I could get behind separate functions to allocate/grow a meory block, and to shrink a memory block, and even adding an additional parameter like calloc() to let the library function handle any overflow. Removing it I think is a mistake.

                                                  1. 1

                                                    I do agree that realloc() is a problematic function. The “allocate/resize/free” nature of it is really only true for C89, which includes this sentence missing in the C99 and C11 standards:

                                                    That’s not the problem. The problem is that it is logically equivalent (in the abstract machine) to malloc then memcpy then free. This means that using the original pointer is always UB in C, including for comparisons to see if it was relocated. You can do a cast to intptr_t or similar and add that check, but no one does and occasionally they are bitten by this (compilers will now sometimes optimise a comparison of the old pointer to any other pointer away because the abstract machine says that it is UB).

                                                    The issue with removing realloc() entirely is that programmers will just code their own version, because it is convenient when creating a dynamically sized array, like:

                                                    That’s a feature, not a bug. It would make it absolutely clear to programmers that there is a free operation happening and so they’d be aware that they needed to ensure that they were not doing this in places where the pointer might be aliased. They’d also have greater control over when to do this.

                                                    I could get behind separate functions to allocate/grow a meory block, and to shrink a memory block

                                                    The problem is that growing and shrinking never happen with modern allocators. If you want O(1) allocation, then you want a sizeclass-based allocator. This is the model used for pretty much everything written in the last decade. If you don’t do this, you have O(log(n)) operations on your fast path. To put that in perspective, the xalanc benchmark from SPEC is about 30% faster with a sizeclass allocator than with a range-based allocator such as the one in glibc.

                                                    With a sizeclass-based allocator, no operation can change the size of an underlying allocation. Shrinking may choose to leave it in place if the copy would be expensive, growing will either leave it in place if the requested memory is less than the size class or grow it if the requested memory is greater. You get no benefit from the grow / shrink APIs, but the interface makes it appear as if you do.

                                                    The one exception to this is if you are on Linux and your allocation is really big. Then you can use mremap to update the mappings to shuffle pages around in your virtual address space. Once you’ve got to the size where this makes sense; however, you are definitely using the wrong data structure. You’re also relying on behaviour that is specific to a combination of OS and memory allocator and so you’re well outside of the space of stuff that should be in a standard API.

                                                1. 3

                                                  I used to program under an OS that had a free() that required the size parameter (only it was called FreeMem() [1]). Later on, when I moved to programming on Unix, I copied that API and use it for a few years. I don’t recall why exactly I gave up on it, maybe it was just too much trouble tracking the size of all allocations, or having to call strlen() + 1 all the time when freeing memory [2].

                                                  Not mentioned is that realloc() can also shrink a memory allocation. It’s useful when you are reading dynamic data in and keep bumping up the allocated memory, then fix the final size once you finished reading the data in. But these days, over commit isn’t that bad as all you really allocate is memory addressing, not memory itself.

                                                  The functions I would like to see are:

                                                  extern void *pool_default(void); /* default memory pool to allocate from */
                                                  extern void *pool_init   (void *ptr,size_t size); /* initialize pool of memory */
                                                  extern void *pool_malloc (void *pool,size_t nmemb,size_t size);
                                                  extern void *pool_calloc (void *pool,size_t nmemb,size_t size);
                                                  extern void *pool_realloc(void *pool,void *ptr,size_t oldnmemb,size_t newnmemb,size_t size);
                                                  extern void  pool_free   (void *pool,void *ptr);
                                                  extern void  pool_freeall(void *pool); /* free all memory from pool */
                                                  

                                                  I’m on the fence about letting pool_free() accept a NULL data pointer (like free() does). And what should pool_malloc(pool,0,x) would return (NULL? A pointer of 0 bytes?). I like the nmemb parameter to avoid any multiplication overflow that might happen now with malloc() and realloc(). And it would be a god send if there was a call to free all currently allocated memory.

                                                  [1] Bonus points if you know the OS.

                                                  [2] Yes, I know, I should use a better abstraction for strings other than NUL terminated character array.

                                                  1. 2

                                                    If designed properly from the start:

                                                    allocate(size=0) should return a valid pointer to a zero byte region, which may be the same pointer as every other zero byte region at the implementation’s discretion.

                                                    1. 4

                                                      It’s kind of implicitly assumed by a lot of code that two live heap allocations will never have the same address. For instance, any code that uses pointer comparisons to test equality. Violating this for zero-size blocks would probably cause trouble.

                                                      1. 3

                                                        This is also an assumption made by the compiler: malloc returns a pointer that is either null, or one that does not alias with any other pointer in the system. I’m not sure that this matters for 0-sized allocations though.

                                                        1. 2

                                                          Two pointers compare equal if … one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space.

                                                          I think this allows for zero-sized allocations to alias. If you consider a pointer to the beginning of a zero-sized allocation to also be a pointer to one past its end.

                                                          1. 1

                                                            No, that’s one of the annoying corner cases. The standard requires it to be either null or a new allocation and it requires new allocations to not alias. Zero-sized objects are not permitted (zero-sized fields are allowed in C++, as of a recent version, if you add an explicit attribute saying that you promise not to care that the address of the field will be the same as the address of the next field).

                                                            In snmalloc, we return a new 16-byte allocation. I wanted us to return null, but there’s some code (FreeBSD’s pkg and sort were the first that hit this) that wrap realloc in something that aborts if it returns null and don’t check that the size > 0 before doing this, so they end up aborting as a result of a spurious check. This is another place where the realloc API is bad: it uses in-band signalling for error conditions and so you can’t differentiate between ‘I got null because I asked for the object to be freed’ and ‘I got null because I reallocation failed’.

                                                        2. 1

                                                          Rust doesn’t have this assumption (zero-sized types can have dangling pointers), and it works out well. It enables compiler to optimize-out zero-sized types entirely, which is useful in generic contexts. For example Result<(), Err> doesn’t waste space on the void OK value.

                                                        3. 2

                                                          That has foot guns. Or at least needs careful specification. Freeing NULL is valid, but if the same address is returned multiple times, is that guaranteed ok?

                                                      1. 2

                                                        This looks like a logic error:

                                                        if (mask & (WGPUColorWriteMask_Alpha|WGPUColorWriteMask_Blue)) {
                                                            // alpha and blue are set..
                                                        }
                                                        

                                                        Shouldn’t it be this?

                                                        if ((mask & WGPUColorWriteMask_Alpha) && (mask & WGPUColorWriteMask_Blue)) {
                                                            // alpha and blue are set..
                                                        }
                                                        
                                                        1. 2

                                                          No, because that requires that both flags are set. The equivalent condition to s & (A|B), which works no different, would be (s & A) || (s & B).

                                                          Notice the commonality in how the bitmasks are joined: always with some form of the disjunctive (“or”) operator, either bitwise | or boolean ||. In any case, the bitmask A or B must be applied to the operand s using the & bitwise “and” operator: s & A, s & B, s & (A|B).

                                                          The equivalent operation to your “correction” (s & A) && (s & B), using the technique of joining the bitmasks first, would be s & (A|B) == A|B. This checks that all of the bits are set, rather than that any of the bits are set.

                                                          Edit: I got confused 😅 You are right: the original code tests whether either alpha or blue is set. My initial comment above would have been applicable if the commented-out text had read, “// alpha OR blue is set..”. I think that’s as good a case as any for “tagged” bit-fields over “untagged” bit-masks.

                                                          Note for any lurkers who have read this far and are rather confused: You may want to read up on how bitmasks are used.

                                                          1. 2

                                                            Which makes either the comment or the code wrong. The comment says “and” not “or”. @smaddox was matching the code to the comment

                                                          2. 2

                                                            As others have pointed out, the comment is a bit misleading. But if you want to check if both are set, this would work:

                                                            if (mask & (WGPUColorWriteMask_Alpha | WGPUColorWriteMask_Blue) == (WGPUColorWriteMaskAlpha | WGPUColorWriteMask_Blue))
                                                            {
                                                                // alpha and blue are set ...
                                                            }
                                                            
                                                            1. 1

                                                              Wouldn’t the | operator join the two bit masks together to create a new intermediate with both bits set? It’s a matter of A == (B|C) versus (A==B) && (A==C) at this point.

                                                              1. 3

                                                                It does, but you get “or” instead of “and”. If either bit is set, the result is not zero.

                                                                1. 2

                                                                  Correction: (A == B) && (A == C) is always false (0) when B != C, due to the transitive property of strict equality. You probably meant (A & B) && (A & C). See my other comment.

                                                              1. 8

                                                                I have been trying to get my students into the habit of explaining their code for my intro class. I dont see this as that much different than them cobbling together stack overflow posts to get to a working answer they dont understand. Its just built into the editor now

                                                                1. 7

                                                                  One of my professors did this. We submitted both the code as well as a paper explaining how it worked. That worked okayish on a large scale with low amount of cheating.

                                                                  Another professor had the perfect small scale solution. You went to his office with your program, opened it in an editor and he deleted ~5 lines somewhere in the middle. If you could not recreate them, and make the program run, you failed.

                                                                  1. 6

                                                                    That’s horrible. I can’t do that and I’ve been programming for over 20 years :P

                                                                    1. 4

                                                                      Hey, remember, we’re talking about undergrad level programming and not rocket surgery. Think merge sorts and binary trees.

                                                                  2. 5

                                                                    Well, apparently Copilot is really good at writing comments, so it can “explain” the code too. :-)

                                                                    1. 3

                                                                      Thought experiment: If a student uses Copilot to generate both the code and the explanation, then memorizes both and can reproduce them on command, have they learned the material?

                                                                    2. 1

                                                                      In a way I’m not sure copilot makes anything much different than when I was in school. The people that didn’t want to learn how to write an algorithm would just “borrow” the solution from someone else, or the internet. (And hopefully change a few variable names and indentation). If people want to learn they will, and if they don’t then they won’t. It’s not really a teacher’s job to force anyone to learn.

                                                                    1. 7

                                                                      Apologies in advance if this comes off harsh or overly critical. I’m open to feedback on how I can raise criticisms without being confrontational.

                                                                      Not only am I pretty unconvinced that IDEs (or other modern tools that lower the barrier to entry for technical tasks) constitute “crutches,” I think it’s a bit questionable how the article equates the need for a “crutch” with a departure from being “normal” or “robust” on the part of the user. “Normal” can mean entirely different things to people with different perspectives. I think there are better ways to get at the idea that knowing and using simpler tools can be a useful skill. I also recognize that this article was written back in 2009, but this seems relevant to discuss with it being reposted.

                                                                      One comment quoted in the article resonated with me:

                                                                      […]

                                                                      The revolution we are experiencing is for every one who chooses to participate, not just those who poke bits well.

                                                                      What may be a “crutch” from one persons perspective is simply how someone else accomplishes their goal. I don’t think it should bother you just because someone solves a problem differently than you do … even if your way is better.

                                                                      I also think the issue brought up at the very end, of everyone “who know[s] how everything really works get[ting] out of the field” seems overblown. Plenty of young folks have the skills and curiosity necessary to learn the underpinnings of the modern computing stack, and I believe many are actively learning “how everything really works.”

                                                                      1. 2

                                                                        This was a followup to a previous post where I said “the culture is totally alien to my way of thinking.”

                                                                        At that point, I had spent several years as a Unix admin/network admin/Unix programmer (wore many hats at a very small company) and I regularly had to fight the Unix “control panels” the owner of the company was using. An issue would come up, I would know how to fix the issue (or configure Apache to do the weird customer request) yet my hands were tied because of the “control panel” that the owner used. I was being paid to help run Unix systems, not play second fiddle to an overly restrictive program.

                                                                        I’ve also experienced those who rely entirely too much on GPS (and nearly got killed because of it).

                                                                        Plenty of young folks have the skills and curiosity necessary to learn the underpinnings of the modern computing stack, and I believe many are actively learning “how everything really works.”

                                                                        And I’ve encountered where the young folk have no interest in learning how things are currently done and want the new shiny. There’s only so many times I can explain how syslog works to both fellow developers and operations people. Or the semantics of files on POSIX systems. Hell, the concern these days are students who have no idea what a file even is. And speaking of students, would you consider it cheating if a student uses Copilot for assignments?

                                                                        So again, when does technology move from being a tool to being a crutch?

                                                                        (Sorry for my tone—I seriously think work is getting to me and I need to quit.)

                                                                        1. 3

                                                                          So again, when does technology move from being a tool to being a crutch?

                                                                          hopefully as often as possible, if we are defining a crutch as “a tool that works so well that anyone can use it without even having to learn how they could do the same thing manually”! the existence of “crutches” does not mean people who want to know how things work are unable to, it just means you don’t have to go through the trouble of learning in order to benefit.

                                                                      1. 3

                                                                        A third example of an ELF library that can be executed on Linux is libc. All it does is print out information about libc itself.

                                                                        1. 2

                                                                          Glibc*

                                                                          1. 2

                                                                            Yes. Here is how that looks like:

                                                                            /lib64/libc.so.6
                                                                            GNU C Library (GNU libc) stable release version 2.31 (git 0a8262a1b2).
                                                                            Copyright (C) 2020 Free Software Foundation, Inc.
                                                                            This is free software; see the source for copying conditions.
                                                                            There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
                                                                            PARTICULAR PURPOSE.
                                                                            Configured for x86_64-suse-linux.
                                                                            Compiled by GNU CC version 7.5.0.
                                                                            libc ABIs: UNIQUE IFUNC ABSOLUTE
                                                                            For bug reporting instructions, please see:
                                                                            <http://bugs.opensuse.org>.
                                                                            
                                                                          2. 1

                                                                            I was wrong. It also happens on musl.

                                                                            $ /usr/lib/libc.so
                                                                            musl libc (x86_64)
                                                                            Version 1.1.24
                                                                            Dynamic Program Loader
                                                                            Usage: /usr/lib/libc.so [options] [--] pathname [args]
                                                                            
                                                                          1. 15

                                                                            Wasn’t Server Push added to the spec by Google? It’s one of those micro-optimizations that only make sense when you have google-level traffic.

                                                                            1. 23

                                                                              I think that describes all of HTTP/2.

                                                                              1. 13

                                                                                Pretty much. The HTTP specs have been more or less taken over by Google and are adding features/functionality according to what Google wants/needs. Which is sad to me because the wire protocol has become far less debuggable and explorable than it used to be – I remember the days of doing telnet 80 and typing in a raw HTTP request to learn how it worked (and doing the same to learn how email worked by putting together the HELO, etc.). With later HTTP versions you need tooling to generate even basic requests for you since it’s no longer a plain-text protocol.

                                                                                1. 8

                                                                                  The HTTP specs have been more or less taken over by Google and are adding features/functionality according to what Google wants/needs.

                                                                                  This isn’t true at least for HTTP/3 & QUIC, both of which have been worked on by far more than just Google. (QUIC has actually morphed significantly from the original Google version.)

                                                                                  1. 5

                                                                                    Hey I used a telnet mail 143 earlier this week!

                                                                                    I had the same reservations with HTTP/2. Implemented it anyway because Google said it was good for speed and SEO, discovered the speed gains were dubious, pre-load never actually helped, and it didn’t seem to improve SEO.

                                                                                    Ask me about AMP.

                                                                                    1. 3

                                                                                      AMP and Dart are proof that things don’t just succeed because Google pushes them. It needs to also clear some minimum bar of quality or else it will be rejected by the internet no matter how much Google pushes it.

                                                                                      1. 2

                                                                                        The nice thing about AMP is that I can just ignore it. However, my browser will be lugging around a useless layer of HTTP/2 support for a couple decades because it was a “standard”.

                                                                                        1. 2

                                                                                          HTTP/2 will likely remain in use for decades.

                                                                                          HTTP/3 is great but won’t replace HTTP/2, simply because it’s not always feasible to use anything non-TCP. Some network admins block UDP, there will always be hosting environments that can’t do anything other than TCP for reasons, and so on…

                                                                                    2. 4

                                                                                      For https: over HTTP/1.0 or HTTP/1.1, you can always do openssl s_client -connect www.google.com:443 and it works just like telnet did for port 80. For HTTP/2+, yes, you need specialized clients.

                                                                                      Edit: for clarity

                                                                                    3. 6

                                                                                      HTTP/2 has at least one very useful consequence for everybody: we don’t have to optimize for a number of simultaneous connections anymore. Removing code that was trying to cleverly bundle extra data to unrelated requests has made a huge impact to maintainability of our code at $WORK.

                                                                                      HTTP/3, though, is exactly what you say. Google making everyone’s life more complicated because they settled on a stupid business model, and drowned clients in ad code.

                                                                                      1. 6

                                                                                        HTTP3 is great for video delivery performance.

                                                                                        Actually it improves load times for just about everything, but losing TCP’s head-of-line blocking and replacing TCP’s slow-start and loss-recovery mechanisms with more suitable ones makes a real difference to the quality/latency/buffering-probability tradeoff for DASH and HLS, especially on mobile or otherwise unreliable connections.

                                                                                  1. 6

                                                                                    Honestly a lot this is like the tech worker equivalent of a gamer pc. Maybe some of it helps, most of it is just looks, and some of it is demonstrably worse.

                                                                                    There’s also a not insignificant element of elitist bs that goes with it. You know the sort: “oh you use an IDE? That means you’re not a real programmer”, etc

                                                                                    To be clear I’m not saying “never customize anything”, just that I think it’s important to acknowledge that most of this stuff isn’t inherently superior to anything/everything else.

                                                                                    1. 6

                                                                                      You use an IDE? Cool. They’re not for me. Why? A few reasons.

                                                                                      1. I’ve yet to find an IDE that will work with my code (or not outright crash in ten minutes). I’ve got stories going back 30+ years on this.

                                                                                      2. I’ve been using the same editor for nearly 30 years now, and it has largely disappeared into the background and I no longer have to consciously think about using it. Again, going by what I’ve seen of IDEs over the years, is that by the time an IDE has gone invisible, it’s obsolete and you have to basically start over with a new one.

                                                                                      1. 17

                                                                                        I’ve yet to find an IDE that will work with my code (or not outright crash in ten minutes). I’ve got stories going back 30+ years on this.

                                                                                        Then I legitimately do not know what you’re doing, as I and clearly millions of others don’t have this problem.

                                                                                        I’ve been using the same editor for nearly 30 years now, and it has largely disappeared into the background and I no longer have to consciously think about using it.

                                                                                        And that’s fine, no one said it isn’t.

                                                                                        But let’s have some serious talk. I don’t know why you felt the need to respond to my comment like this, as I didn’t say anything that implied you had to use an IDE. What I said is that this behavior of denigrating people who do use IDEs is BS, and your comment here is skating awfully close to that line.

                                                                                        1. 4

                                                                                          I also have difficulties with IDEs. Now you’ve heard of two people. Maybe there’s a third person somewhere out there.

                                                                                          In all seriousness, you chose IDEs as an example, and invited the response. By making IDEs about elitism, rather than about velocity and convenience, you’re suggesting that folks who have difficulties with IDEs are trying to make some sort of moral statement when an IDE doesn’t work for them.

                                                                                          1. 8

                                                                                            No, I said, very clearly, that acting like you’re superior to other people for using an IDE is BS. At no point, at all, did I say that not using an IDE is bad.

                                                                                            I think you should be asking yourself why my saying simply that using an IDE doesn’t make you an inferior engineer has resulted in you claiming that I’ve said non-IDE users are inferior.

                                                                                            My reason for asking you to think about this is super hard to explain as the completely normal piece of human behavior that leads to this usually comes up in situations that are vastly more important and/or sensitive than something as trivial as IDE vs non-IDE, or similarly emacs vs vim, grey vs gray, etc

                                                                                            [tw: this behavior/response often comes up in relation to actually important things like racism, etc. I want to be clear that I don’t consider these remotely equivalent, but just providing a heads up]

                                                                                            The particular, again normal human, behavior is interpreting “treating someone else as equal” when you’ve internalized that person - for whatever reason, no matter how trivial - as being inferior is interpreted as favoritism or claiming the “inferior” person is superior.

                                                                                            Given that common response I hope you can see how I might see your response as being related: All I did was say

                                                                                            You know the sort: “oh you use an IDE? That means you’re not a real programmer”

                                                                                            and I even followed it with

                                                                                            To be clear I’m not saying “never customize anything”, just that I think it’s important to acknowledge that most of this stuff isn’t inherently superior to anything/everything else.

                                                                                            So again, I never said “Using an IDE makes you better than people who don’t”. I’ve now even had other people (including at least a couple I know use emacs or other Command-line editors) make sure I didn’t say anything that might imply it as these responses made me legitimately concerned.

                                                                                        2. 4

                                                                                          Before this conversation went off the rails and turned into an argument, I’m pretty sure the only behaviour that olliej meant to criticize was “denigrating people for using an IDE”. Not not using an IDE. Everyone here thinks your continued use of vim or joe or notepad++ or whatever does not make you a bad person.

                                                                                          1. 3

                                                                                            Have you used any software released in the past ~8 years?

                                                                                          2. 4

                                                                                            Maybe some of it helps, most of it is just looks, and some of it is demonstrably worse.

                                                                                            Have you heard of unixporn? https://old.reddit.com/r/unixporn/

                                                                                            1. 4

                                                                                              Code completion is rather cool, but not having it means you eventually read the docs from the top to bottom. That is sometimes a win for productivity as well.

                                                                                              Similar to how people rush into things instead of investigation the tools that might help them finish an order of magnitude faster. Same with carpentry, spreadsheets and software development.

                                                                                              1. 3

                                                                                                I don’t necessarily agree here, for a few reasons. I’ll note that in the context of the “IDE vs non-IDE” argument code completion isn’t the only thing that they do - in fact it’s relatively recent. The first thing that was integrated in IDEs was debugging, where I would argue it is simply a superior experience to raw gdb/lldb with the actual code in a separate window. Many people use plugins/scripting to setup vim, emacs, etc as an IDE for that reason. Honestly I’d argue that the debugging is really the thing that makes something Integrated - certainly it predates more or less every other feature.

                                                                                                Code completion itself isn’t a feature solely of IDEs - plenty of code editors support it these days, especially with the introduction of LSP which means supporting a language no longer means “re-implement that language in each new editor”. As far as the learning the API, I think it’s 50/50 in helping vs not-helping knowledge. When I was a tutor/TA (in NZ tutor would mean a person who runs/supports labs, etc whereas TA means person who helps/does lectures) I definitely saw people who would write code by code completion. But even without inline completion I’ve also seen people code via compiler errors.

                                                                                                But I also think code completion helps in learning an API as it is fundamentally a contextually aware interface to the API, so a user can type in some approximation of what they think an API would be called, and the suggestions let them know which part of the API they should be searching for (rather than trying to search for “I want to do X” which these days especially will take you take a bunch of stack overflow questions with variable quality of answer - honestly it was easier back when the only place talking about “do X in MyLanguage” were the actual docs and so that was what google/webcrawler/lycos returned). These days it’s even common for code completion to directly include the documentation for the API so you can get contextually relevant documentation.

                                                                                                That is increasingly important, as I do think that your comment about “reading the documentation from top to bottom” is simply not feasible, let alone reasonable, for many modern APIs. Hell I know many people who work on and design library APIs who would need to check documentation for some uncommon parts. Obviously you need to learn the parts of the API you would use regularly (which even the coding through completion/compiler errors folk manage, albeit [very] slowly), but completion gives you convenience for what parts you do know, and access for parts you don’t.

                                                                                              2. 1

                                                                                                There’s also a not insignificant element of elitist bs that goes with it.

                                                                                                Maybe that’s true for some people, but for the record I personally think it’s fine to use whatever you prefer. This post is more about getting to know better the stack you prefer to use.

                                                                                              1. 3

                                                                                                One approach I sometimes use under Linux could be used here. First open up a few extra terminals, and note which device they are using (use w for this). So now you have two terminals, say on /dev/pts/3 and /dev/pts/4`. Then run your test program from a different terminal, assigning stderr for each program under test to the other terminals. You get live output, and it should be easy to see what each program is writing to stderr.

                                                                                                As a test, I just opened up two terminals, where one ended up on /dev/pts/0 and the other one on /dev/pts/1. On the first one (assigned to /dev/pts/0) I ran ls -l >/dev/pts/1 and the output appeared on the other terminal window.

                                                                                                1. 34

                                                                                                  … until you do.

                                                                                                  I firmly believe a suite of interacting microservices represents one of the worst technical designs in a world where there are many particularly bad options. Microservices are harder to observe, harder to operate, impose rigid boundaries that could make change management harder, and almost always perform worse than the monolith they replace. You should almost never chose them for any perceived technical reasons!

                                                                                                  Where microservices might just suck less than monoliths is organizationally, something that this article doesn’t even touch on – which is honestly infuriating, because it’s easy to build a straw argument for/against on technical grounds, but IMO those barely matter.

                                                                                                  Your monolith probably was great when it had n people hacking on it, because they could all communicate with each other and sync changes relatively painlessly. Now you’ve grown (congrats!) and you have 3n, or god help you 10n engineers. Merging code is a nightmare. Your QA team is growing exponentially to keep up, but regression testing is still taking forever. Velocity is grinding to a halt and you can’t ship new features, so corners get cut. Bugs are going into production, and you introduce a SRE team whose job is to hold the pager and try to insulate your engineers from their own pain. Product folks are starting to ask you for newfangled things that make you absolutely cringe when you think about implementing them.

                                                                                                  You could try to solve this by modularizing your monolith, and that might be the right decision. You land with tight coupling between domains and a shared testsuite/datastore/deployment process, which could be OK for your use case. Might not, though, in which case you have to deal with tech that’s slower and harder to observe but might actually let your dev teams start shipping again.

                                                                                                  1. 11

                                                                                                    Yeah, services in general (not just “microservices”) are an organizational thing. It’s basically leaning into Conway’s Law and saying that the chart of components in the software was going to end up reflecting the org chart anyway, so why not just be explicit about doing that?

                                                                                                    1. 10

                                                                                                      You could try to solve this by modularizing your monolith, and that might be the right decision. You land with tight coupling between domains and a shared testsuite/datastore/deployment process, which could be OK for your use case. Might not, though, in which case you have to deal with tech that’s slower and harder to observe but might actually let your dev teams start shipping again.

                                                                                                      There’s a third option here: several monoliths. Or just regular, non-micro services. Not as great as one monolith, but scales better than one, technically better than microservices, and transitions more easily into them if you really need them.

                                                                                                      1. 13

                                                                                                        This is my first port of call after “plain ol’ PostgreSQL/Django” stop cutting it. Some people say services should be “small enough and no smaller”, but I think “big enough and no bigger” is closer to the right way to think about it.

                                                                                                      2. 5

                                                                                                        I wonder if that is true. I used to be a firm believer of what you are saying and have said the same thing in my own words.

                                                                                                        However, I am not so sure anymore. The reason is that a lot of the organizational efforts people make are only made when people run microservices, when I don’t see anything preventing from the same (in terms of goals) efforts would be taken with monoliths.

                                                                                                        I’ve seen companies shift from monoliths to microservices and they usually end up initially having the same problems and then work around it. There’s usually huge process changes, because switching to microservices alone seemed to make things even worse for a while. Another part of those switches tends to be people being granted time to have good interfaces and here is where I wonder if people are partially mislead.

                                                                                                        While microservices to some degree enforce sane interfaces - unless they don’t and your organization still creates a mess - on a technical level nothing hinders you from creating those sane interfaces and boundaries within a monolith. From a technical perspective whether you call a function, a REST API, use RPC, etc. doesn’t make a difference only that that the standard function call is usually the most reliable.

                                                                                                        Of course this is anecdotal, but it happened more than once that “preparing a monolith for migrating to microservices” resulted in all the benefits being reaped. Of course it’s not likely that anyone stops there. The usual mindset is “our goal was to use microservices, we won’t stop one step before that when all the hard stuff is already done”.

                                                                                                        But there’s a lot more complexity added than just going over HTTP. Adding more moving parts as you mention bring a lot of issues with it.

                                                                                                        In other words. I am on the fence. I agree with you, it matches what I have seen, but given that I see first companies at least partly converting microservices to monoliths again for various reasons and simply keeping concerns separate - something that good software engineering also should do - I wonder if it wouldn’t make sense to find a way to organize monoliths like microservices to lower the complexity. Maybe this could be implemented like a pattern, maybe code analysis could help, maybe new programming paradigms, maybe a new or modern way of modularization.

                                                                                                        Or in other words, when even people who making a living off Kubernetes and microservices say that Monoliths are the Future I’d at least step back and consider.

                                                                                                        But then again I think it might simply depend on company culture or even the particular team implementing a project. People work differently, use different tools, frameworks, languages, different ways of time and project management work best for different people. So maybe that’s what it boils down to. So maybe just don’t listen to people telling you that you NEED to use one or the other. There’s enough of highly successful projects and companies out there going completely opposite directions.

                                                                                                        1. 6

                                                                                                          While microservices to some degree enforce sane interfaces - unless they don’t and your organization still creates a mess - on a technical level nothing hinders you from creating those sane interfaces and boundaries within a monolith. From a technical perspective whether you call a function, a REST API, use RPC, etc. doesn’t make a difference only that that the standard function call is usually the most reliable.

                                                                                                          Don’t forget that the other problems - shared persistence layer, shared deployment, etc. are still there.

                                                                                                          As a junior I got to see up close some of the problems a big monolith can present: it was about mid-year and we had to do a pretty big launch that crossed huge parts of that monolith by Q4 to win a (fairly huge for us!) contract. It was going to take many deployments, tons of DB migrations, and an effort spanning multiple teams. We all knew our initial “good plan” wasn’t going to work; we just didn’t know how badly it’d be off. The architects and leads all argued over the path, but we basically realized that we were trying to condense 3 quarters of work into 2.

                                                                                                          We pulled it off, but it sucked:

                                                                                                          • All the other work had to be paused: didn’t matter if it was in a completely unrelated area, we didn’t have the QA bandwidth to be sure the changes were good, and we could not risk a low priority change causing a rollback & kicking out other work

                                                                                                          • We deployed as much as we could behind feature flags, but QA was consistently short of time to test the new work, so we shipped tons of bugs

                                                                                                          • We had to pay customers credits because we gave up on our availability SLAs to eke out a few more release windows

                                                                                                          • We had to relax a ton of DB consistency – I can’t remember how many ALTER TABLE DROP CONSTRAINTs our DBAs ran, but it was a lot. This + the above lead to data quality issues …

                                                                                                          • … which lead to us hitting our target, but with broken software; we basically hit pause on the next two months of work for the DBAs and devs to go back and pick up the broken pieces

                                                                                                          Much of our problem came about because we had one giant ball of mud on top of one ball of mud database; if we’d been working on an environment that had been decomposed along business domains that had been well thought out and not evolved, we might’ve been fine.

                                                                                                          Or we might’ve still been screwed because even with clean separation between teams, and the ability to independently work on changes, we still were deploying a single monolith - which meant all our DB changes / releases had to go together. Dunno.

                                                                                                          But then again I think it might simply depend on company culture or even the particular team implementing a project. People work differently, use different tools, frameworks, languages, different ways of time and project management work best for different people. So maybe that’s what it boils down to. So maybe just don’t listen to people telling you that you NEED to use one or the other. There’s enough of highly successful projects and companies out there going completely opposite directions.

                                                                                                          ^^ – the best two words any programmer can say are “it depends”, and that goes double for big architectural questions.

                                                                                                        2. 3

                                                                                                          i like microservices because when one fails you can debug that individual component while leaving the others running. sometimes you can do this with monolith designs, but not typically, in my experience.

                                                                                                          1. 14

                                                                                                            That’s usually a very small advantage compared to the loss of coherent stack traces, transactions and easy local debugging. Once you have a Microservices you have a distributed system and debugging interactions between Microservices is orders of magnitude harder than debugging function calls in monoliths

                                                                                                            1. 5

                                                                                                              Yes, this is what I jokingly call the law of conservation of complexity. Your app has got to what it has got to do and this by itself brings a certain amount of interaction and intertwining. That does does not magically go away if you cut up the monolith into pieces that do the same thing. You just move it to another layer. For some problems this makes things easier, for others it does not.

                                                                                                              1. 3

                                                                                                                See also “law of requisite variety” in cybernetics.

                                                                                                                1. 1

                                                                                                                  I’m a huge fan of the concept of conserved complexity. I find that it particularly shines when evaluating large changes. I’ll often get a proposal listing all the ways some project will reduce complexity. However, if they can’t tell me where the complexity is going, it’s clear they haven’t thought things through enough. It always has to go somewhere.

                                                                                                            2. 3

                                                                                                              I’m genuinely curious if people who claim microservices solve an organisational problem have actually worked somewhere where they have been used for a few years.

                                                                                                              It was so painful to try and get three or four separate teams to work on their services in order to get a feature out. All changes need to be in backwards compatible steps to avoid downtime (no atomic deployment) and anything that needed an interface change was extremely painful. Lets not even get into anything that needed a data migration.

                                                                                                              A lot of places get around this pain by always creating new services instead of modifying old ones, and there is a lot of duplication. It’s not a ball of mud, it’s much worse.

                                                                                                              The idea that you have to communicate or work together less because you’re using microservices is… I’ll be kind here… flawed.

                                                                                                              IME everything slows down to a crawl after a few years.

                                                                                                              1. 2

                                                                                                                I’ve been through this and where I see things slowing to a crawl, it’s where the teams and their connections with the other teams are weak - and the organisational priorities are conflicting.

                                                                                                                This happens with multiple teams working on different parts of a monolith.

                                                                                                                With microservices, we get to avoid everyone _ else_ being affected as much as they would have. This is Conway again. We can fix the teams. We can grow the teams (in maturity and capability). We can fix the organisational boundaries that interfere with communications. We can align priorities.

                                                                                                                Al of the above needed to happen anyway with a monolith, but we used to have hundreds - sometimes thousands - of people being stuck or firefighting because there were some teams unable to collaborate on a feature.

                                                                                                                Feature teams are a great answer to this general problem, but they are hard to make happen where there are huge pieces of tech that require esoteric skillsets and non-transferable skills (‘I only want to write C#’).

                                                                                                                I’m seeing developers enjoying picking up new languages, tools, and concepts, and I’m seeing testers become developers and architects, and us actually getting some speed to market with exceptional quality.

                                                                                                                This isn’t because of microservices. It’s because the organisation needed to be refactored.

                                                                                                                Microservices aren’t what we need. They are slightly wrong in many ways, technically, but we now build with functions, topics, queues (FIFO where we need to avoid race conditions: not all distributed systems problems are hard to solve), step functions, block storage (with a querying layer!) - and other brilliant tools that we wouldn’t have been able to refactor towards if we hadn’t moved to microservices - or something else - first.

                                                                                                              2. 3

                                                                                                                I’ve spent the past 12 years working on a service implemented as a bunch of microservices. At first, the Corporation started a project that required interfacing to an SS7 network, and not having the talent in-house, outsourced the development to write a service that just accepted requests via the SS7 network, and forward them to another component to handle the business logic. The computers running the SS7 network required not only specialized hardware, but proprietary software as well. Very expensive, but since the work this program did was quite small, hardware was minimized, compared to the hardware to run the business logic (and the outsourced team was eventually hired as full time employees).

                                                                                                                A few years down the road, and now we need to support SIP. Since we already had a service interfacing with SS7, it was just easier to implement a service to interface with SIP and have it talk to the same backend that the SS7 service talked to.

                                                                                                                Mind you, it’s the same team (the team I’m on) that is responsible for all three components. Benefits: changes to the business logic don’t require changes to the incoming interfaces (for the most part—we haven’t had to mess with the SS7 interface for several years now for example). Also, we don’t need to create two different versions of our business logic (one for SS7, which requires proprietary libraries, and one for SIP). It has worked out quite well for us. I think it also helps in that we have only one customer (one of the Oligarchic Cell Phone Companies) we have to support.

                                                                                                                1. 2

                                                                                                                  Where microservices might just suck less than monoliths is organizationally

                                                                                                                  that depends on your perspective. I remain convinced that the primary function of microservices is to isolate the members of a laboring force such that they cannot form a strong union. That is, take Conway’s Law and reverse it; create a policy to specifically -introduce- separation between workers and they won’t have a reason to talk, which makes it less likely that they’ll unionize. In that framing, the primary function of microservices to prevent programmers from unionizing.

                                                                                                                  1. 2

                                                                                                                    I chuckled.

                                                                                                                    Truly, people around me (as far as I can notice, including public figures covered by media) tend not to think about communicating effectively with others and instead tend to vilify them and otherwise avoid having the conversations necessary for further progress.

                                                                                                                    Perhaps it’s just a simple fact that most people are not trained in communication and IT people specifically have not had that much hands-on experience to compensate. Not that rest of the population were that much better at it (on average).

                                                                                                                    In short, I wouldn’t attribute the phenomena to malice. I think that IT people are not unionizing simply because that means talking to people, which is (on average) exhausting and hard.

                                                                                                                    1. 1

                                                                                                                      Would be interesting to know then if microservices are less common in this country where more or less everyone is in a union already.

                                                                                                                    2. 1

                                                                                                                      if increasing the number of developers meant you can’t ship new features and corners got cut then you need to reduce the number of developers. the shapes of organizations can change, they must. it’s important for us to fight for changing them for material reasons like reducing complexity and friction.

                                                                                                                      making the monolith modular is a good example of when we realize solving the real problem is very hard so we solve a different problem instead. problem is we didn’t have that second problem. and in fact maybe it makes our real problem (reality) harder to solve in the future.

                                                                                                                    1. 4

                                                                                                                      How to get me to be quite, umm, unhappy, with you….

                                                                                                                      Pass an error code up and up and up the call graph and…. cast it to void.

                                                                                                                      I have solved more unsolvable bugs than I can count by the simple algorithm of searching for all cast return codes to void and logging the code if not success, printing a backtrace (yes you can do that in C/C++) and aborting.

                                                                                                                      Weehee! Bugs come crawling out of the wood work and you go whack whack whack and they go away for ever.

                                                                                                                      1. 5

                                                                                                                        I have solved more unsolvable bugs than I can count by the simple algorithm of searching for all cast return codes to void and logging the code if not success, printing a backtrace (yes you can do that in C/C++) and aborting.

                                                                                                                        You are going to love this. I’ve seen code like this in at least one library that is widely available in Linux distributions etc. I’m not going to name and shame, because I still need to open an issue.

                                                                                                                        void add_handler(HandlerFunc mfh) {
                                                                                                                          struct entry *e = malloc(sizeof(struct entry));
                                                                                                                          if (e == NULL) {
                                                                                                                            return;
                                                                                                                          }
                                                                                                                          e->func = mfh;
                                                                                                                          e->next = handlers;
                                                                                                                          handlers = e;
                                                                                                                        }
                                                                                                                        

                                                                                                                        So we don’t actually know if our handler ever got registered.

                                                                                                                        That’s the basic outline, anyhow. It’s yet another take on the ostrich method. I’d call the ostrich method the most widely used in C, just because nobody I know of actually checks the return value of printf.

                                                                                                                        Error handling in C is annoying, and a lot of programmers – including myself – are optimists.

                                                                                                                        1. 3

                                                                                                                          Sigh. I do not love. No sir, not at all, at all.

                                                                                                                          I hate code that is written to “might do something”.

                                                                                                                          It might work, it might not work, but nobody will know it didn’t unless a user is paying close attention and complains.

                                                                                                                          If the user asked the program to do something, do it or tell him why you couldn’t, don’t silently do nothing.

                                                                                                                          Sure there are “best effort” use cases, but they are few and far between and should be made clear from the function name.

                                                                                                                          If you call a function “do_x” (or “add_handler) I expect it to do x or tell me why not, not “might do x, feeling cute, might not do”.

                                                                                                                          Grrumble grrowl mutter mutter mutter.

                                                                                                                          1. 2

                                                                                                                            aaaaaaaaa

                                                                                                                            at least do exit(4206969); // lmao or something instead of just returning!

                                                                                                                            1. 1

                                                                                                                              While exit() does take an integer value, the C standard explicitly states three values you can give to it—0 (which indicates success to the operating system), EXIT_SUCCESS (which does the same thing, but nothing states it has to be 0) and EXIT_FAILURE. POSIX limits the value to 8-bits. MS-DOS (and probably Windows) has the same limit.

                                                                                                                              1. 1

                                                                                                                                Given the context, I genuinely do not know how to respond to this. XD Thanks for the info though.

                                                                                                                        1. 3

                                                                                                                          I’m surprised this is so new, as Lua is now on version 5.4.

                                                                                                                          1. 1

                                                                                                                            Lua 5.1 is actively used, for example LuaJIT still supports 5.1+ partially. See also results of survey - https://docs.google.com/forms/d/1yacI6dj0d9OD8P5nhNp3647amjeVHauCvgVw9K3otFM/viewanalytics

                                                                                                                            Feel free to create manual pages missed for newer Lua versions.