The subject of (m|re)alloc with a zero size came up in the discussion under the changes to the new POSIX standard, and this seems like a good historical overview of where the incompatible behaviours came from.
Some implementations have returned non-null values for allocation requests of zero bytes. Although this strategy has the theoretical advantage of distinguishing between “nothing” and “zero” (an unallocated pointer vs. a pointer to zero-length space), it has the more compelling theoretical disadvantage of requiring the concept of a zero-length object. Since such objects cannot be declared, the only way they could come into existence would be through such allocation requests.
The C89 Committee has decided not to accept the idea of zero-length objects. The allocation functions may therefore return a null pointer for an allocation request of zero bytes.
QUIET CHANGE IN C89
A program which relies on size-zero allocation requests returning a non-null pointer will behave differently.
The standards for C89, C99, C11 and C23 for malloc() and free() are identical. It’s realloc() that differs. C89 has: “If size is zero and ptr is not a null pointer, the object it points to is freed.” C99 and C11 never mention a zero size, and now C23 has made realloc(ptr,0) undefined. Sigh.
Honestly, the right thing for C23 to do was probably deprecate realloc:
The semantics of realloc(ptr, 0) are implementation defined and inconsistent.
To determine if reallocation happened, you have to compare the new and old pointers. But that’s undefined behaviour, so you have to convert them to integers in a fragile way.
The API signals failure by returning null, so idioms like a = realloc(a, size) will leak memory on failure.
On a modern sizeclass allocator, it’s always either a no-op or a copy, so it is incredibly hard to reason about its performance.
I generally regard any use of realloc as a bug these days. I’d love to have seen malloc_usable_size (or one of the other spellings) standardised because any use of realloc that is almost justifiable would be better done with malloc_usable_size.
The semantics of realloc(ptr, 0) are implementation defined and inconsistent.
This is because malloc(0) was never defined by the committee. The C Standard (at least around the time of C89) was to standardize existing behavior, and to minimize the amount of code broken by said standard. Some compilers at the time returned a ptr, some returned NULL. And the compiler vendors want to minimize the work required to get that “C STANDARD” stamp. That’s probably still the case today.
To determine if reallocation happened
Why is that needed? The documentation for realloc() makes clear that the original pointer, on success, has no meaning.
The API signals failure by returning null, so idioms like a = realloc(a, size) will leak memory on failure.
That’s a failure to read the standard and, unfortunately, on the programmer. I feel like a compiler could warn on this situation (or a linter), because the nature of realloc() is known and this idiom will leak memory. But what else could be done?
The two main uses of realloc() is to increase an allocated block, or decrease an allocated block. The idiom used is to grow a block when reading some form of dynamic data. Grow the size while collecting data, then resize to the actual amount when done. Any replacement for realloc() will end up doing this.
Your summary of what the ANSI C committee did with malloc(0) is contradicted by the history in this article and by the QUIET CHANGE paragraph in the C89 rationale.
I don’t see any contradiction. Standardization work for C started in the mid-80s (1984? 1985?) with the standard coming out in 1989. Their mandate was to standardize as much existing practice as possible. And you have two competing ways of interpreting malloc(0).
Also, could you give a link to the “QUIET CHANGE” in the C89 rationale, or at least a link to the C89 rationale? I’m having a hard time finding it (I found the C99 one, not the C89 one).
Because realloc is either a no-op, or a malloc, memcpy, free sequence. If it’s the latter, you need to update other pointers.
That’s a failure to read the standard and, unfortunately, on the programmer
No, if you design an API that is entirely sharp corners, that’s not the users’ fault. The excuse for realloc is that it’s forty years old, no one new any better, and mincomputers had less RAM than a modern microcontroller and a more primitive allocator,
The two main uses of realloc() is to increase an allocated block, or decrease an allocated block
Neither of those operations happen on a modern high-performance sizeclass-based allocator. It always triggers a copy and a reallocation. If you do those three operations separately, it’s obvious why it probably isn’t actually the performance win that you think it is. If you’re using a range-based allocation, you can get a much bigger performance win by switching to a sizeclass allocator than any use of realloc.
A no-op realloc() also updates the compiler’s model of the size of the allocation, which might not match the allocator’s idea of the size of the allocation. malloc_usable_size() gets the allocator’s idea of the size, but that might not actually be usable (despite the name) because the compiler might think it knows better. You have to do something like,
Though it’s probably less horrible to avoid realloc() entirely and use something more like jemalloc’s nallocx() to round up within the size class before allocating, or maybe xallocx() for non-copying resizes.
Microcontrollers are the one place where you don’t want a sizeclass allocator, because they consume too much address space (though even there the benefits of realloc are small because memcpy is insanely fast: microcontroller performance has grown far faster than memory capacity). Realloc is very rare in embedded code because, when memory is constrained, realloc failure is common and handling those failures gracefully is very hard. We didn’t provide realloc in CHERIoT RTOS and I’ve yet to find an embedded codebase where that’s a problem (I’m sure they exist, but they’re not common: even using malloc is fairly rare, but realloc is uncommon even in places that do heap allocation).
For anything on a system with an MMU, you want a sizeclass allocator. I’m biased towards snmalloc, but any modern high-performance allocator has the same property here. Realloc with a sizeclass allocator will never change the size of the allocation, it will return the original pointer if the old and new sizes map to the same size class or will do a malloc, memcpy, free sequence if it does not.
Somehow, I don’t get the conclusion. All sources seem to aggree that malloc(0) != NULL, and yet the verdict is “not real”, where the question was whether malloc behaves like this. Am I missing a joke?
We originally implemented malloc(0) to return null in snmalloc. It was allowed by C11 and POSIX 2018. We changed it because the glibc test suite expected malloc(0) to return a valid object. I don’t really like that, but we eventually came up with some bit twiddling that let us avoid an extra branch, so it doesn’t slow things down.
Our realloc returned null on zero, but we found that some BSD code (which had been copied into a bunch of places) that wrapped realloc in something that called abort on null returns, but didn’t check that the size was non-zero. This code is now UB in C23.
The subject of (m|re)alloc with a zero size came up in the discussion under the changes to the new POSIX standard, and this seems like a good historical overview of where the incompatible behaviours came from.
Surely something so minor and meaningless but yet so far-reaching would have had a simple solution agreed upon by now, if only by flipping a coin.
Surely.
I’ve been looking into some of the history in the wg14 document archive and there’s some major foreshadowing in the C89 rationale:
The standards for C89, C99, C11 and C23 for
malloc()andfree()are identical. It’srealloc()that differs. C89 has: “Ifsizeis zero andptris not a null pointer, the object it points to is freed.” C99 and C11 never mention a zero size, and now C23 has maderealloc(ptr,0)undefined. Sigh.Honestly, the right thing for C23 to do was probably deprecate realloc:
realloc(ptr, 0)are implementation defined and inconsistent.a = realloc(a, size)will leak memory on failure.I generally regard any use of
reallocas a bug these days. I’d love to have seenmalloc_usable_size(or one of the other spellings) standardised because any use ofreallocthat is almost justifiable would be better done withmalloc_usable_size.Yeah, though in the discussion about all this a few months ago we found out that the compilers have been weakening the semantics of malloc_usable_size, making it much more difficult to use. Zig has an interesting alternative take on realloc where it fails if it can’t extend in place, which neatly deals with the pointer provenance issues.
If you deprecate
realloc(), what replaces it?This is because
malloc(0)was never defined by the committee. The C Standard (at least around the time of C89) was to standardize existing behavior, and to minimize the amount of code broken by said standard. Some compilers at the time returned a ptr, some returned NULL. And the compiler vendors want to minimize the work required to get that “C STANDARD” stamp. That’s probably still the case today.Why is that needed? The documentation for
realloc()makes clear that the original pointer, on success, has no meaning.That’s a failure to read the standard and, unfortunately, on the programmer. I feel like a compiler could warn on this situation (or a linter), because the nature of
realloc()is known and this idiom will leak memory. But what else could be done?The two main uses of
realloc()is to increase an allocated block, or decrease an allocated block. The idiom used is to grow a block when reading some form of dynamic data. Grow the size while collecting data, then resize to the actual amount when done. Any replacement forrealloc()will end up doing this.Your summary of what the ANSI C committee did with malloc(0) is contradicted by the history in this article and by the QUIET CHANGE paragraph in the C89 rationale.
I don’t see any contradiction. Standardization work for C started in the mid-80s (1984? 1985?) with the standard coming out in 1989. Their mandate was to standardize as much existing practice as possible. And you have two competing ways of interpreting
malloc(0).Also, could you give a link to the “QUIET CHANGE” in the C89 rationale, or at least a link to the C89 rationale? I’m having a hard time finding it (I found the C99 one, not the C89 one).
https://lobste.rs/s/dtm0vj/malloc_0_realloc_0_0#c_fn6y8a
Thank you. I just blanked that you linked and quoted the document in this thread. Sorry about that.
Nothing. It is always the wrong tool for the job.
Because realloc is either a no-op, or a malloc, memcpy, free sequence. If it’s the latter, you need to update other pointers.
No, if you design an API that is entirely sharp corners, that’s not the users’ fault. The excuse for realloc is that it’s forty years old, no one new any better, and mincomputers had less RAM than a modern microcontroller and a more primitive allocator,
Neither of those operations happen on a modern high-performance sizeclass-based allocator. It always triggers a copy and a reallocation. If you do those three operations separately, it’s obvious why it probably isn’t actually the performance win that you think it is. If you’re using a range-based allocation, you can get a much bigger performance win by switching to a sizeclass allocator than any use of realloc.
A no-op realloc() also updates the compiler’s model of the size of the allocation, which might not match the allocator’s idea of the size of the allocation. malloc_usable_size() gets the allocator’s idea of the size, but that might not actually be usable (despite the name) because the compiler might think it knows better. You have to do something like,
Though it’s probably less horrible to avoid realloc() entirely and use something more like jemalloc’s nallocx() to round up within the size class before allocating, or maybe xallocx() for non-copying resizes.
David, not everyone has the privilege of working on modern microcontrollers with cutting-edge research. Please keep that in mind.
Microcontrollers are the one place where you don’t want a sizeclass allocator, because they consume too much address space (though even there the benefits of realloc are small because memcpy is insanely fast: microcontroller performance has grown far faster than memory capacity). Realloc is very rare in embedded code because, when memory is constrained, realloc failure is common and handling those failures gracefully is very hard. We didn’t provide realloc in CHERIoT RTOS and I’ve yet to find an embedded codebase where that’s a problem (I’m sure they exist, but they’re not common: even using malloc is fairly rare, but realloc is uncommon even in places that do heap allocation).
For anything on a system with an MMU, you want a sizeclass allocator. I’m biased towards snmalloc, but any modern high-performance allocator has the same property here. Realloc with a sizeclass allocator will never change the size of the allocation, it will return the original pointer if the old and new sizes map to the same size class or will do a malloc, memcpy, free sequence if it does not.
Somehow, I don’t get the conclusion. All sources seem to aggree that
malloc(0)!=NULL, and yet the verdict is “not real”, where the question was whethermallocbehaves like this. Am I missing a joke?We originally implemented
malloc(0)to return null in snmalloc. It was allowed by C11 and POSIX 2018. We changed it because the glibc test suite expectedmalloc(0)to return a valid object. I don’t really like that, but we eventually came up with some bit twiddling that let us avoid an extra branch, so it doesn’t slow things down.Our realloc returned null on zero, but we found that some BSD code (which had been copied into a bunch of places) that wrapped realloc in something that called abort on null returns, but didn’t check that the size was non-zero. This code is now UB in C23.