If you have a subdirectory in your monorepo with a third party project, how do you
- Keep pace with upstream
- Maintain your own changes on top
- Submit changes back to upstream
I’ve never worked in a monorepo, but I do have some experience doing radical surgery on git repositories with git-subtree and git filter-branch. These techniques seem like they’d be difficult to automate.
When looking for information about monorepos online, there’s plenty of PRO vs CON discussion, but not a lot of details.
I’ve seen people work directly in the monorepo with both git submodules and
git-subtree(depending on the organizational approach). The tooling is painful to use, but it’s possible to push and pull changes entirely inside the monorepo.I didn’t understand what you meant by “automate”: in the same way that I don’t expect someone to automate their normal contributions to the monorepo, I wouldn’t expect them to automate contributions to the upstream repos either.
If possible, you can consider inserting an intermediate forked repo between the upstream repo and the monorepo, and consume the intermediate repo instead of the upstream repo directly. Then you can do your work in a normal repo with normal tooling and the monorepo can consume it as usual.
There are some projects like https://github.com/josh-project/josh which could compose a “monorepo” in a dynamic fashion, but I haven’t seen it used in production.
About josh, I did a proof-of-concept at work; wrote some notes here:
https://github.com/josh-project/josh/discussions/1386#discussion-7177501
I didn’t use it enough to endorse it, but I think it’s a very promising approach. Also note that while the documentation (at least when I last read it) focuses on using it as a daemon, the Miri project uses it “daemonless”.
The secret is that one says third party projects need to live in your monorepo. If you use a sane language with support for pinning, the only time you have third party source code you need to include in your repo is when you’ve forked it - i.e. when it’s no longer third party by definition.
Things like hosting your own module/package/nuget/cargo/whatever cache servers in-house (optionally restricted only to package/version combos you’ve vetted for license and security compliance) takes you the rest of the way there.