Not sure the author read the relocation logic in Spack too deeply. That, or maybe we should put a less dissuasive comment at the top of the file (I think the comment mentioned is older than the code 😬).
Spack does use patchelf and install_name_tool to relocate RPATHs in binaries. Those will lengthen paths in binaries if they are short, which is nice.
Spack also:
Spiders the installation directory and replaces instances of the install prefix in text files (shebangs, config files, etc.)
Modifies strings in the strings section of binaries. Spack will warn you if the target path is too long to relocate to.
That last one is the hard one. To solve it in the general case, where we could lengthen the path as we do with RPATHs, you’d have to figure out all the usages of the string in the binary. So you’d have to rewrite address computations and find all the places they’re used. It’s impossible to do generally because the address could be computed in theoretically any way. It’s likely that static string addresses are only computed a few ways in practice, so it might be possible to make a binary rewriting tool that does simpler pattern matches.
So far we have gone with padding the path, which at least allows us to find the path and shorten it. You can set up a Spack build pipeline and put the following in your spack.yaml:
That will pad the install path (like paths in /nix/store, but they can be wherever) out to 512 characters. You can also just omit the :512 and Spack will try to do something based on the max path allowed on the OS (that can get you in trouble as OS limits can vary, so we picked 512). Packages will now be built and installed to this very long synthetic path.
Binary packages made from these installations will now be relocatable to shorter paths, including strings in binaries. There are likely other weird special cases that we’d need to handle to cover all packages generally, but this has worked well for packages tested so far. We’ll likely find out more once we start making binaries available in a public build cache.
Hey @tgamblin thanks for the lengthy response, and apologies I clearly didn’t read the Spack implementation closely (I’ve also edited the post with a link to this comment).
Is my understanding correct that “/home/software/$padding:512” pads the remaining parts of the path so that the total path is 512 in length? And from there can you freely patch all binaries and text files with simple replacements?
Part of the reason I wanted to use path padding is because I want to more rigorously validate build output by hashing it. By padding out to a known long path length you could replace those links on the fly to get a hash of the output that is unrelated to the build location.
Also nix allows the package name to exist in the build path, I’m slightly worried this might exacerbate the length issue, does Spack deal with that or avoid it somehow?
Great to hear that padded paths and naive replacement have worked well so far.
Is my understanding correct that “/home/software/$padding:512” pads the remaining parts of the path so that the total path is 512 in length?
Yep, that’s the idea.
And from there can you freely patch all binaries and text files with simple replacements?
Yeah, as long as your replacement is a null-terminated string that’s shorter than 512 chars, you should be able to do this.
Part of the reason I wanted to use path padding is because I want to more rigorously validate build output by hashing it. By padding out to a known long path length you could replace those links on the fly to get a hash of the output that is unrelated to the build location.
Yep – this is definitely an advantage. It makes me wonder what nix does with RPATHs though, as I thought nix used patchelf to change them, and patchelf can lengthen paths. Maybe nix only hashes after patching, and the store path is authoritative? We just ignore this and assume the package can be built (more typical case right now) or relocated anywhere.
Also nix allows the package name to exist in the build path, I’m slightly worried this might exacerbate the length issue, does Spack deal with that or avoid it somehow?
Not sure exactly what this means. The build path in Spack is a temporary directory that we get rid of after the build is done. I’m actually not sure how this works in nix. What’s the issue with the package name being there?
Not sure the author read the relocation logic in Spack too deeply. That, or maybe we should put a less dissuasive comment at the top of the file (I think the comment mentioned is older than the code 😬).
Spack does use
patchelfandinstall_name_toolto relocateRPATHs in binaries. Those will lengthen paths in binaries if they are short, which is nice.Spack also:
stringssection of binaries. Spack will warn you if the target path is too long to relocate to.That last one is the hard one. To solve it in the general case, where we could lengthen the path as we do with RPATHs, you’d have to figure out all the usages of the string in the binary. So you’d have to rewrite address computations and find all the places they’re used. It’s impossible to do generally because the address could be computed in theoretically any way. It’s likely that static string addresses are only computed a few ways in practice, so it might be possible to make a binary rewriting tool that does simpler pattern matches.
So far we have gone with padding the path, which at least allows us to find the path and shorten it. You can set up a Spack build pipeline and put the following in your
spack.yaml:That will pad the install path (like paths in
/nix/store, but they can be wherever) out to 512 characters. You can also just omit the:512and Spack will try to do something based on the max path allowed on the OS (that can get you in trouble as OS limits can vary, so we picked 512). Packages will now be built and installed to this very long synthetic path.Binary packages made from these installations will now be relocatable to shorter paths, including strings in binaries. There are likely other weird special cases that we’d need to handle to cover all packages generally, but this has worked well for packages tested so far. We’ll likely find out more once we start making binaries available in a public build cache.
Hey @tgamblin thanks for the lengthy response, and apologies I clearly didn’t read the Spack implementation closely (I’ve also edited the post with a link to this comment).
Is my understanding correct that “/home/software/$padding:512” pads the remaining parts of the path so that the total path is 512 in length? And from there can you freely patch all binaries and text files with simple replacements?
Part of the reason I wanted to use path padding is because I want to more rigorously validate build output by hashing it. By padding out to a known long path length you could replace those links on the fly to get a hash of the output that is unrelated to the build location.
Also nix allows the package name to exist in the build path, I’m slightly worried this might exacerbate the length issue, does Spack deal with that or avoid it somehow?
Great to hear that padded paths and naive replacement have worked well so far.
Yep, that’s the idea.
Yeah, as long as your replacement is a null-terminated string that’s shorter than 512 chars, you should be able to do this.
Yep – this is definitely an advantage. It makes me wonder what
nixdoes with RPATHs though, as I thoughtnixusedpatchelfto change them, and patchelf can lengthen paths. Maybenixonly hashes after patching, and the store path is authoritative? We just ignore this and assume the package can be built (more typical case right now) or relocated anywhere.Not sure exactly what this means. The build path in Spack is a temporary directory that we get rid of after the build is done. I’m actually not sure how this works in
nix. What’s the issue with the package name being there?