I’m a little confused, is this a real problem? Or am I misunderstanding the problem?
Been a while since I did proper python software for work but I don’t remember ever running into that problem. We deployed to pretty barebones Ubuntu VMs and sometimes packaged the applications as PEX. If we’d require something that wasn’t there it would’ve instantly failed the build step in CI.
It’s true that if you follow best practices for development, then this should be covered by CI, and thus not be a problem at all. The fact remains that there is a considerable gap between best practice and common practice.
For example, one area where Python has seen a lot of growth lately is in data science, but it appears that many data science projects struggle at the most basic level of declaring dependencies correctly. A recent study (not affiliated with FawltyDeps) shows that 42% of notebooks executed failed due to missing dependencies: https://arxiv.org/abs/2308.07333
Ah thanks, yeah that makes more sense. I suppose notebooks might also get shared quite differently (or at all!), but what I thought was: Develop a (probably even closed source) application, have CI, package and roll out yourself.
Love ShellCheck! We use it in our CI to lint all our shell scripts at $dayjob. Adding ShellCheck to our build process and fixing all the initial issues was at times boring, but also immensely educational about sh/bash as a language (and why you should never use it for anything more than the simplest stuff…). I’m positive that running ShellCheck on any non-trivial collection of shell scripts will help you find and fix a lot of security issues as well as other bugs. I’m not sure whether that is praise for ShellCheck, or an indictment of shell as a programming language… ;-)
Line 1:
at $dayjob
^-- SC2154: dayjob is referenced but not assigned.
^-- SC2086: Double quote to prevent globbing and word splitting.
Did you mean: (apply this, apply all SC2086)
at "$dayjob"
I’m a little confused, is this a real problem? Or am I misunderstanding the problem?
Been a while since I did proper python software for work but I don’t remember ever running into that problem. We deployed to pretty barebones Ubuntu VMs and sometimes packaged the applications as PEX. If we’d require something that wasn’t there it would’ve instantly failed the build step in CI.
(Hi, FawltyDeps author here)
It’s true that if you follow best practices for development, then this should be covered by CI, and thus not be a problem at all. The fact remains that there is a considerable gap between best practice and common practice.
For example, one area where Python has seen a lot of growth lately is in data science, but it appears that many data science projects struggle at the most basic level of declaring dependencies correctly. A recent study (not affiliated with FawltyDeps) shows that 42% of notebooks executed failed due to missing dependencies: https://arxiv.org/abs/2308.07333
Ah thanks, yeah that makes more sense. I suppose notebooks might also get shared quite differently (or at all!), but what I thought was: Develop a (probably even closed source) application, have CI, package and roll out yourself.
Love ShellCheck! We use it in our CI to lint all our shell scripts at $dayjob. Adding ShellCheck to our build process and fixing all the initial issues was at times boring, but also immensely educational about sh/bash as a language (and why you should never use it for anything more than the simplest stuff…). I’m positive that running ShellCheck on any non-trivial collection of shell scripts will help you find and fix a lot of security issues as well as other bugs. I’m not sure whether that is praise for ShellCheck, or an indictment of shell as a programming language… ;-)