Worth noting some of the techniques to do this: all the data structures go through opaque pointers (aka handles), always accessed through named functions. So changing internal layout doesn’t break the user code. Then make sure the constants defined never change and functions aren’t removed and you’re in a good position to make this reality.
Especially notable that these things are contrary to the way a lot of languages work… even private member variables in a C++ class can affect the abi since it changes the layout and size of the class. Any virtual function in C++ or a COM interface or similar added or removed can break others since it looks them up by index in the table instead of by name.
The C-style pointers to structs is easier to bind from other languages too exactly because it doesn’t have to be all complete, matching in even private details.
There are a bunch of techniques in C++ that let you do this kind of thing. The oldest in common use is the pointer to implementation (pImpl) pattern, where you provide a public class that has a single field that is a pointer to the real one.
In more modern C++, it’s common to provide type-erased interfaces from your library. Things in your header are always inline and use templated types, the things exposed from the shared library are functions that take wrapped types. This lets the library add new entry points if the wrapped things need to do more and maintain the old ones. Most things in the standard library work in this way.
Aye, there’s a lot of ways to do it, just it does take some effort. Of course, you can also just write C++ in the same opaque struct pointer style as C.
Curl is not unique in providing very strong backwards compatibility guarantees, but it does seem that this is far less common in newer projects than it used to be.
I feel like ABIs are falling out of fashion for a few reasons:
OSes and other such execution platforms are becoming much more trivial than they used to be, they’re not much more than an implementation detail now.
Consumer-grade computers are powerful enough to statically compile truly massive codebases in a reasonable time.
Static compilation is far simpler than dynamic linking, both for end-users and software distributors.
Given all that, and if we actually do get a future where all languages and libraries target WASM as the universal ISA, the entire concept of an ABI will be steamrolled into not being much more relevant than, say, knowing your machine’s endianness.
Many newer languages are also generally less amenable to dynamic linking. Parametric polymorphism (generics / templates) have become common, and generally compiling them requires having both the definition and use together so you know what monomorphized instances to generate.
Only if you want to do monomorphisation across a dynamic linkage binary. There are two approaches that work well here:
Use a JIT or install-time compiler. .NET does this.
Have generics decay to dynamic dispatch across shared library boundaries. Swift does this.
There are also some half-way points, where you expose some specialisations across the boundary. Or you put the code for the specialised routines somewhere where it can be found by the compiler for the caller.
C++ enables most of the non-JIT bit of the design space. It’s quite common to put templates in a header that then call shared-library functions that take a wrapper that does dynamic dispatch to the underlying type, and use extern template definitions to allow monomorphisation for common cases.
ABI stability is less important (outside of desktop applications, where you may have dozens of apps that all need the same large libraries), but API stability still very much matters. If every downstream user needs to modify their code to be able to pull in the new version, that makes it very hard to rapidly deploy security updates.
It also leads to fun problems in diamond dependency cases. You depend on libraries A and B, but both of those depend on C. A new release of C breaks compatibility, and this means that you can’t update A to a new version (which depends on a new version of C) until B has also been updated. If there is a security vulnerability in A, you can’t ship an update until B releases a new version. Given that typical programs now often have hundreds of dependencies with complex dependency graphs (hopefully DAGs, rarely trees), this is a huge problem.
HarfBuzz is another example of a critical library project that may deprecate things over time but the ABI is stable enough that upgrading is almost a universal no-brainer.
Current project version: 10.0.1
Number of releases to date: 177
Current SO version: 0
Worth noting some of the techniques to do this: all the data structures go through opaque pointers (aka handles), always accessed through named functions. So changing internal layout doesn’t break the user code. Then make sure the constants defined never change and functions aren’t removed and you’re in a good position to make this reality.
Especially notable that these things are contrary to the way a lot of languages work… even private member variables in a C++ class can affect the abi since it changes the layout and size of the class. Any virtual function in C++ or a COM interface or similar added or removed can break others since it looks them up by index in the table instead of by name.
The C-style pointers to structs is easier to bind from other languages too exactly because it doesn’t have to be all complete, matching in even private details.
There are a bunch of techniques in C++ that let you do this kind of thing. The oldest in common use is the pointer to implementation (pImpl) pattern, where you provide a public class that has a single field that is a pointer to the real one.
In more modern C++, it’s common to provide type-erased interfaces from your library. Things in your header are always inline and use templated types, the things exposed from the shared library are functions that take wrapped types. This lets the library add new entry points if the wrapped things need to do more and maintain the old ones. Most things in the standard library work in this way.
Aye, there’s a lot of ways to do it, just it does take some effort. Of course, you can also just write C++ in the same opaque struct pointer style as C.
Curl is not unique in providing very strong backwards compatibility guarantees, but it does seem that this is far less common in newer projects than it used to be.
I feel like ABIs are falling out of fashion for a few reasons:
Given all that, and if we actually do get a future where all languages and libraries target WASM as the universal ISA, the entire concept of an ABI will be steamrolled into not being much more relevant than, say, knowing your machine’s endianness.
Many newer languages are also generally less amenable to dynamic linking. Parametric polymorphism (generics / templates) have become common, and generally compiling them requires having both the definition and use together so you know what monomorphized instances to generate.
Only if you want to do monomorphisation across a dynamic linkage binary. There are two approaches that work well here:
There are also some half-way points, where you expose some specialisations across the boundary. Or you put the code for the specialised routines somewhere where it can be found by the compiler for the caller.
C++ enables most of the non-JIT bit of the design space. It’s quite common to put templates in a header that then call shared-library functions that take a wrapper that does dynamic dispatch to the underlying type, and use extern template definitions to allow monomorphisation for common cases.
ABI stability is less important (outside of desktop applications, where you may have dozens of apps that all need the same large libraries), but API stability still very much matters. If every downstream user needs to modify their code to be able to pull in the new version, that makes it very hard to rapidly deploy security updates.
It also leads to fun problems in diamond dependency cases. You depend on libraries A and B, but both of those depend on C. A new release of C breaks compatibility, and this means that you can’t update A to a new version (which depends on a new version of C) until B has also been updated. If there is a security vulnerability in A, you can’t ship an update until B releases a new version. Given that typical programs now often have hundreds of dependencies with complex dependency graphs (hopefully DAGs, rarely trees), this is a huge problem.
The abi-laboratory site is a great way to track ABI stability for various C and C++ libraries. Here is curl (libcurl) stats: https://abi-laboratory.pro/index.php?view=timeline&l=curl
Edit: sadly this site isn’t up to date past 2020. ☹️
HarfBuzz is another example of a critical library project that may deprecate things over time but the ABI is stable enough that upgrading is almost a universal no-brainer.
Current project version: 10.0.1
Number of releases to date: 177
Current SO version: 0