I think the use of Pin for this is an overkill.
Pin has some rather complex requirements for its usage, and makes safety promises based on that, but it can’t uphold them for you (apart from gating access behind unsafe where it’s your responsibility to understand the guarantees and uphold them).
It’s important to understand that Pin doesn’t do the pinning. It wraps types that are already pinned. If a value is stable enough to be put in Pin, it’s stable enough to be used without it. You can gate access to it using your own private fields and accessors, without paying the complexity cost of generalizing it.
(use of PhantomPinned marker is fine tho. Even without Pin, it may be used in the future by rustc to recognize self-referential structs and not cry UB when it finds out aliasing violations).