I’m a little confused, is this a real problem? Or am I misunderstanding the problem?
Been a while since I did proper python software for work but I don’t remember ever running into that problem. We deployed to pretty barebones Ubuntu VMs and sometimes packaged the applications as PEX. If we’d require something that wasn’t there it would’ve instantly failed the build step in CI.
It’s true that if you follow best practices for development, then this should be covered by CI, and thus not be a problem at all. The fact remains that there is a considerable gap between best practice and common practice.
For example, one area where Python has seen a lot of growth lately is in data science, but it appears that many data science projects struggle at the most basic level of declaring dependencies correctly. A recent study (not affiliated with FawltyDeps) shows that 42% of notebooks executed failed due to missing dependencies: https://arxiv.org/abs/2308.07333
Ah thanks, yeah that makes more sense. I suppose notebooks might also get shared quite differently (or at all!), but what I thought was: Develop a (probably even closed source) application, have CI, package and roll out yourself.
I’m a little confused, is this a real problem? Or am I misunderstanding the problem?
Been a while since I did proper python software for work but I don’t remember ever running into that problem. We deployed to pretty barebones Ubuntu VMs and sometimes packaged the applications as PEX. If we’d require something that wasn’t there it would’ve instantly failed the build step in CI.
(Hi, FawltyDeps author here)
It’s true that if you follow best practices for development, then this should be covered by CI, and thus not be a problem at all. The fact remains that there is a considerable gap between best practice and common practice.
For example, one area where Python has seen a lot of growth lately is in data science, but it appears that many data science projects struggle at the most basic level of declaring dependencies correctly. A recent study (not affiliated with FawltyDeps) shows that 42% of notebooks executed failed due to missing dependencies: https://arxiv.org/abs/2308.07333
Ah thanks, yeah that makes more sense. I suppose notebooks might also get shared quite differently (or at all!), but what I thought was: Develop a (probably even closed source) application, have CI, package and roll out yourself.