It’s worth noting that there are two variants of the GNU nested functions extension. Neither is supported by clang but one is fairly benign. I believe glibc is using the benign variant where you’re just defining a function inside another but the inner function does not capture any variables in the outer. This is just a way of making the name of the inner function local to the outer function but aside from that it generates code that is identical to a static function. The extension also allows the inner function to capture variables in the outer. This is still fairly benign in the case where the function is called directly (it will be inlined and it’s really just a way of avoiding some copying and pasting, the sort of thing you’d do with a lambda in modern C++). If you take the address of the function, you get a thunk: a pointer to a stack-allocated function that adds the address of the enclosing frame to the arguments and then calls the inner function. This requires executable stacks and is spectacularly dangerous: there is no way in the type system for the recipient to know that this is a function pointer that it may not capture (and in standard C it is always safe to capture any function pointer) and if you do capture it then you introduce an arbitrary code execution vulnerability into your code.
Clang chose not to support this extension at all because full support is a terrible idea and the safe subset can be implemented better with other language features. Even benign form is surprisingly complicated to implement (it adds a lot of complexity to name lookup rules and, especially in C++, has a lot of corner cases).
Clang supports Apple’s blocks extension, which requires the caller to know that it’s calling a block, which allows the compiler to carry the pointers to the captures and the function pointer separately. You can’t capture a block without calling _Block_copy, and if you don’t link a blocks runtime then you can’t capture it at all, so you get linker failures if you try.
I remember refactoring some code to remove nested functions around the clang 2.8 release time. The version without nested functions was much cleaner.
Is there any interest in GCC for supporting Apple’s block extension?
Apple GCC supported an older version of it but it never got merged. Iain Sandoe was interested in doing it back when he was at CodeSourcery because the Apple system headers use blocks and so folks wanted to be able to use them for C code targeting macOS. I don’t think he managed it.