Specifically, this is -fsanitize=function, and it has slowly been making these ripples throughout the OSS ecosystem in the past year or two. I don’t know whether the sanitizer itself is new or only its inclusion under the -fsanitize=undefined or -fsanitize=address switch is new, but I’ve also been fixing some issues it flags at $WORK in preparation for upgrading the compiler.
The spec is clear that calling through the wrong function pointer type is UB, and code getting hit by this has essentially been relying on the underlying function call ABI not being affected by the discrepancies.
There’s an additional interesting implementation detail for this sanitizer, in that it requires that 8 bytes before the address stored in each function pointer being used in a sanitized call site be dereferenceable. This caused issues with function pointers to JIT code. See https://github.com/llvm/llvm-project/issues/65253#issuecomment-171094085.
If the function is defined with a type that is not compatible with the type (of the expression) pointed to by the expression that denotes the called function, the behavior is undefined.
Specifically, this is
-fsanitize=function, and it has slowly been making these ripples throughout the OSS ecosystem in the past year or two. I don’t know whether the sanitizer itself is new or only its inclusion under the-fsanitize=undefinedor-fsanitize=addressswitch is new, but I’ve also been fixing some issues it flags at $WORK in preparation for upgrading the compiler.The spec is clear that calling through the wrong function pointer type is UB, and code getting hit by this has essentially been relying on the underlying function call ABI not being affected by the discrepancies.
There’s an additional interesting implementation detail for this sanitizer, in that it requires that 8 bytes before the address stored in each function pointer being used in a sanitized call site be dereferenceable. This caused issues with function pointers to JIT code. See https://github.com/llvm/llvm-project/issues/65253#issuecomment-171094085.
I have some notes about how -fsanitize=function (part of -fsanitize=undefined) now applies to C https://maskray.me/blog/2022-12-18-control-flow-integrity#fsanitizefunction
C11 6.5.2.2p9 says
If the function is defined with a type that is not compatible with the type (of the expression) pointed to by the expression that denotes the called function, the behavior is undefined.