Indeed. I’ve chosen Perl for my static blog engine (App::PFT), and I never regretted it.
Perl is a really good language, if you know how to do it. It is very expressive, and has a very thorough documentation. It has a bad reputation due to the fact it allows a lot of freedom, but I think most of people who blame Perl don’t really know how to write good Perl. Trust me, it is perfectly possible to achieve a good code base in Perl, if one applies some discipline.
And just to say it out loud, I’m somewhat bored/annoyed by this “let’s rewrite it in X” meme. This makes sense only if the software is broken and/or unmaintainable.
Even a lot of Perl 4 stuff runs either unmodified or with small changes on perl 5. It’s only in the past few years that a few of the backwards-compatibility things that were most harmful to adding new features or to maintainability finally got sacrificed. Odds are if some 25 or 30 year old code forms an important part of your system, you’re not upgrading to the latest version of the perl interpreter anyway :)
Interestingly, macos still includes Perl as the most prominent scripting language (in terms of number of executables in the base install). On a just installed catalina there’s many more perl scripts than shell and python scripts altogether:
# count number of programs of each type
for j in perl pyton shell mach; do
echo $j $(for i in `echo $PATH|tr : \ `; do file $i/*|grep -i $j; done|wc -l)
done
perl 224
python 21
shell 104
mach 1106
I just noticed this when a colleague was bugging me than his mac was a really slick modern unix, with no bizarre legacy like perl scripts and whatnot.
Exactly, it’s still in base because there are base utilities which require perl. And changing that would be quite the hassle, i.e. rewriting the package management tools (pkg_*) and some other scripts.
Alas, this overcounts on systems where one or more bin directories are symlinks to other directories. By adding (yet another) sort/uniq stage, we can deduplicate everything
( for dir in $(echo "$PATH" | tr : '\n'); do ls -d "$dir"/*; done ) | xargs realpath | sort | uniq | xargs -L 1 file -bn | cut -d , -f 1 | sort | uniq -c | sort -n
On my (arch linux) system, this yields the output
1 Algol 68 source
1 a /usr/bin/env ash script
1 a /usr/bin/env csh script
1 a /usr/bin/env dash script
1 a /usr/bin/env fish script
1 a /usr/bin/env pdksh script
1 a /usr/bin/env php script
1 a /usr/bin/env tcsh script
1 a /usr/bin/guile1.8 \ script
1 a /usr/bin/guile1.8 -s script
1 DOS batch file
1 GNU awk script
1 gzip compressed data
1 Java source
1 Paul Falstad's zsh script
1 setuid ELF 64-bit LSB executable
1 sticky ELF 64-bit LSB pie executable
1 TeX document
1 XML 1.0 document
2 a /usr/bin/env ksh script
2 a /usr/bin/env texlua script
2 a /usr/bin/env /usr/bin/python script
2 awk script
2 POSIX shell script executable (binary data)
2 Tcl/Tk script
3 a /usr/bin/fontforge -lang=ff script
3 ELF 32-bit LSB executable
3 regular file
3 setgid ELF 64-bit LSB pie executable
3 setuid
4 Node.js script
4 setuid executable
5 Perl script
7 Tcl script
8 ReStructuredText file
11 ASCII text
11 ELF 32-bit LSB pie executable
12 a /usr/bin/env sh script
17 directory
17 setuid ELF 64-bit LSB pie executable
19 a /usr/bin/env texlua script
21 Ruby script
23 a /usr/bin/ocamlrun script executable (binary data)
36 ELF 64-bit LSB shared object
163 Bourne-Again shell script
286 ELF 64-bit LSB executable
330 Perl script text executable
491 Python script
664 POSIX shell script
4736 ELF 64-bit LSB pie executable
There is an additional part to this though: for some reason, a bunch of scripts are provided as two or even three versions of the same thing, running via “perl”, “perl 5.18” and “perl 5.28”.
of the ~233 results on my system, 11 are from my own install of auto tools (so not part of the base system); 71 target perl 5.18 specifically, 63 target perl 5.28 specifically. So that leaves me with ~99 that just target ‘perl’, or 88 from the base system.
How many of the perl scripts are part of the actual perl installation (eg. pod, dbi, lwp, cpan, etc)? heh
It would certainly be interesting to know how many of those scripts were actually things that were actual system components or utilities in their own right.
This post is the perfect example of why more engineers (like the one who asked the question this post answers) should try to open their minds and broaden their views on the incredible diversity of use cases out there.
Who CARES that the scripting lanauge du-jour won’t build without dynamic library support? OpenBSD does because it runs on VAX.
Taint checking comes to mind, and Perl has it. I think OpenBSD folks prefer tech where it’s easier to do the right thing; doing the right thing in shell or php can require more effort, with painstaking, error-prone effort to avoid pitfalls.
Ruby code ages poorly compared to perl, though. I’ve been on too many projects where code written in ruby a year or two earlier at most had already been broken by language or dependency changes.
To be fair, OpenBSD controls base, so they could keep a check on the dependency changes. Language changes are rarely breaking with Ruby, when was the last one?
We may be talking past each other here. I said that they could keep a check on the dependency changes, by which I meant that they would author code in such a way that it does not require external dependencies (or at least not few enough that they couldn’t vendor them), which wouldn’t be any different from what they’re doing with Perl already. This means that this downside of the Ruby ecosystem could be mitigated. And language changes they’d just have to accept and roll with, but I hold that Ruby rarely introduces breaking changes.
OpenBSD will have to vendor $language_of_choice in any case because that’s how the BSDs’ whole-OS approach works.
What would bring any security then, as opposed to just helping catch bugs? Barring “legitimate” cases of correct algorithms outliving their usefulness (e.g. the math behind crypto algorithms getting “cracked” to the point where it’s feasible to mount attacks on reasonably-priced hardware) virtually all security issues are bugs. Things like shell injections are pretty easy to miss when writing shell code, no matter how careful you are.
But that’s exactly what taint checking does: it helps you catch bugs that occur due to executing stuff that’s under the user’s control. Some of these can be exploited for security reasons, but security isn’t the only reason why you want this – it’s just as good at preventing a user enumeration attack as it is at preventing accidental “rm -rf /”
I thought the same. I figure the OpenBSD people know what they are talking about but I am still not really clear on what Perl has over Tcl, for example. Hopefully a Perl monk will show up and clarify.
Because there are scripts in base written in it, and it’s inclusion in base means that scripts continue to be written in it. Barring a complete rewrite of (at the very least) OpenBSD’s packaging/ports tooling and infrastructure, I can’t see it being removed ever.
pkg_add is written in perl. Having looking at it, I find it terrible. A rewrite is necessary. But to get that done, somebody would have to step up and put in the effort.
I always wished that they could adopt a xbps-like package manager, but that’s maybe because I had more experience with XBPS than with OpenBSD (on workstations).
I dream that some BSD-family system will actually make the package manager a first class citizen.
This includes getting rid of the distribution tarballs (base.tgz etc) and replacing them with packages, ideally more fine grained. Thus also replacing the untarring done in the install process with a tool similar to debootstrap or pacstrap.
I believe it is one of the many problems holding the BSDs back.
Tangentially related, I also dream that they’ll eventually drop CVS for Git or a competing DVCS.
Between the two, the latter I feel more likely, thanks to the existence of those pushing for it. (ESR)
There isn’t one. In FreeBSD the base system contains everything that the FreeBSD project themselves want to take responsibility for directly. Boot loader, kernel, libc, at least one shell, the login daemon, sshd, a C compiler. Everything you need to get the system running and at least minimally useful and self hosting.
Everything in the base system is checked into a single repo. The base system has its own mechanisms for updating (used to be a thing called freebsd-update, dunno if it changed since).
Third party software (which is most useful software, since the set of things you might want to do is much bigger than what the FreeBSD project would ever want to put in their own tree) is installed & updated via a separate mechanism called packages. FreeBSD Ports is a repo of instructions for how to build packages.
As a rough approximation, the base system takes care of stuff in / and /usr and.packages all install into /usr/local and /opt.
Unlike in Linux where the kernel (Linux) and libc (glibc, musl, etc) are written by different sets of people who don’t necessarily even like each other, in FreeBSD the kernel and userland are all worked on together. It’s possible to land atomic patches that simultaneously add a new syscall and add support for it to libc and man pages for it, for example.
You could call the minimal set of packages that come with a default Debian install its “base” but it’s still not the same I think because they all upgrade separately.
Yeah, Debian has two or three related concepts of a base system.
There is the set of packages with Essential: yes — basically, the package manager and the utilities the package manager depends on for its operation. The package manager won’t even let you remove these, because if you did, you wouldn’t have anything left to reinstall them with. This amounts to around 20 packages (including perl-base, as a matter of fact, since debconf uses it).
Then there is the set of packages with Priority: required. This is another 50 or so packages that the package manager can live without, but the system administrator probably can’t. This includes libc6 (nice to have, right?), mount, passwd, procps, and e2fsprogs (provides fsck). The essential and required packages together form minbase.
Then there’s the set of packages with Priorty: important, which adds about another 100. These include bzip2 and xz (minbase only gets you gzip), the tools for loading and unloading kernel modules, less, vim-tiny, sudo, ping, and… unavoidably on a modern version, systemd and dbus. It also includes python3, making it the other language you’re pretty much guaranteed to find. minbase plus important equals base, which is the minimum thing you can install from install media (but you can debootstrap to minbase for containers or embedded or whatever, if you know what you’re doing).
Anyway, as you say, most of that stuff is third-party, not created by the Debian maintainers themselves. Although more than the usual amount of scrutiny goes into anything that’s marked as important enough to get into base.
This is one of the top reasons i use perl as the base language in my current project: its long-lasting stability and availability.
A script written for perl 5.00 would still run today without issues.
Indeed. I’ve chosen Perl for my static blog engine (App::PFT), and I never regretted it.
Perl is a really good language, if you know how to do it. It is very expressive, and has a very thorough documentation. It has a bad reputation due to the fact it allows a lot of freedom, but I think most of people who blame Perl don’t really know how to write good Perl. Trust me, it is perfectly possible to achieve a good code base in Perl, if one applies some discipline.
And just to say it out loud, I’m somewhat bored/annoyed by this “let’s rewrite it in X” meme. This makes sense only if the software is broken and/or unmaintainable.
Even a lot of Perl 4 stuff runs either unmodified or with small changes on perl 5. It’s only in the past few years that a few of the backwards-compatibility things that were most harmful to adding new features or to maintainability finally got sacrificed. Odds are if some 25 or 30 year old code forms an important part of your system, you’re not upgrading to the latest version of the perl interpreter anyway :)
Interestingly, macos still includes Perl as the most prominent scripting language (in terms of number of executables in the base install). On a just installed catalina there’s many more perl scripts than shell and python scripts altogether:
I just noticed this when a colleague was bugging me than his mac was a really slick modern unix, with no bizarre legacy like perl scripts and whatnot.
Thanks for these stats, very interesting.
Is “mach” the compiled system executables? I’m not familiar with these parts of macOS.
Yes, Mach-O is the native executable format of the Mach microkernel, which is part of the macOS kernel.
Apparently. I just found a string that is unique, the complete “file” output is this:
EDIT: there’s also a few other types:
From OpenBSD 6.7 (
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin):Exactly, it’s still in base because there are base utilities which require perl. And changing that would be quite the hassle, i.e. rewriting the package management tools (pkg_*) and some other scripts.
Really clean! I do not dare to do this for a full ubuntu install, it would be horrifying.
On my Raspberry Pi:
Ahh, this is fun.
Ok, but at least do not replicate my ugly script. A shorter and more efficient code can treat all types on a single pass:
Alas, this overcounts on systems where one or more bin directories are symlinks to other directories. By adding (yet another) sort/uniq stage, we can deduplicate everything
On my (arch linux) system, this yields the output
There is an additional part to this though: for some reason, a bunch of scripts are provided as two or even three versions of the same thing, running via “perl”, “perl 5.18” and “perl 5.28”.
of the ~233 results on my system, 11 are from my own install of auto tools (so not part of the base system); 71 target perl 5.18 specifically, 63 target perl 5.28 specifically. So that leaves me with ~99 that just target ‘perl’, or 88 from the base system.
How many of the perl scripts are part of the actual perl installation (eg. pod, dbi, lwp, cpan, etc)? heh
It would certainly be interesting to know how many of those scripts were actually things that were actual system components or utilities in their own right.
This post is the perfect example of why more engineers (like the one who asked the question this post answers) should try to open their minds and broaden their views on the incredible diversity of use cases out there.
Who CARES that the scripting lanauge du-jour won’t build without dynamic library support? OpenBSD does because it runs on VAX.
What’s wrong with python’s license? This is the first time I’ve heard anyone say there’s issues with it.
Also, I think he forgot to mention Rust. Must definitely rewrite everything in Rust. /s
Marc Espie elaborates a bit on this in another post on the openbsd-misc mailing list:
https://www.python.org/download/releases/2.1/fsf/ ?
That was fixed in Python 2.0.1, released in June 2001…
Interesting read, but I don’t understand one detail of the argument: what makes Perl more secure than the other scripting languages mentioned?
Taint checking comes to mind, and Perl has it. I think OpenBSD folks prefer tech where it’s easier to do the right thing; doing the right thing in shell or php can require more effort, with painstaking, error-prone effort to avoid pitfalls.
ruby might be an interesting alternative, but I would assume it doesn’t support nearly as many platforms or architectures as perl.
EDIT: huh. Apparently ruby removed taint checking in 2.7.
Ruby code ages poorly compared to perl, though. I’ve been on too many projects where code written in ruby a year or two earlier at most had already been broken by language or dependency changes.
To be fair, OpenBSD controls base, so they could keep a check on the dependency changes. Language changes are rarely breaking with Ruby, when was the last one?
Now, you’ve got to start auditing upstream for bug and security fixes, and backporting them, rather than just updating when needed.
Forking is a bunch of work – why do it when there’s a suitable alternative?
We may be talking past each other here. I said that they could keep a check on the dependency changes, by which I meant that they would author code in such a way that it does not require external dependencies (or at least not few enough that they couldn’t vendor them), which wouldn’t be any different from what they’re doing with Perl already. This means that this downside of the Ruby ecosystem could be mitigated. And language changes they’d just have to accept and roll with, but I hold that Ruby rarely introduces breaking changes.
OpenBSD will have to vendor $language_of_choice in any case because that’s how the BSDs’ whole-OS approach works.
Yes. I thought you meant essentially forking the shifting dependencies instead of fully avoiding them.
In any case, perl is there and in use, so switching would be a bunch of work to solve a non-problem.
Yeah, you’re not wrong. Excellent point.
Maybe the “use warnings” and “use strict”?
That doesn’t bring any security though: it may give you a bit of safety, catching bugs earlier than in some other script languages.
What would bring any security then, as opposed to just helping catch bugs? Barring “legitimate” cases of correct algorithms outliving their usefulness (e.g. the math behind crypto algorithms getting “cracked” to the point where it’s feasible to mount attacks on reasonably-priced hardware) virtually all security issues are bugs. Things like shell injections are pretty easy to miss when writing shell code, no matter how careful you are.
Probably the taint mode that the other commenter mentioned
But that’s exactly what taint checking does: it helps you catch bugs that occur due to executing stuff that’s under the user’s control. Some of these can be exploited for security reasons, but security isn’t the only reason why you want this – it’s just as good at preventing a user enumeration attack as it is at preventing accidental “rm -rf /”
I thought the same. I figure the OpenBSD people know what they are talking about but I am still not really clear on what Perl has over Tcl, for example. Hopefully a Perl monk will show up and clarify.
Because there are scripts in base written in it, and it’s inclusion in base means that scripts continue to be written in it. Barring a complete rewrite of (at the very least) OpenBSD’s packaging/ports tooling and infrastructure, I can’t see it being removed ever.
What is wrong with TCL’s security? If anything, it is very easy to sandbox TCL — or at least I thought so.
pkg_add is written in perl. Having looking at it, I find it terrible. A rewrite is necessary. But to get that done, somebody would have to step up and put in the effort.
Thus perl needs to remain in base.
I always wished that they could adopt a xbps-like package manager, but that’s maybe because I had more experience with XBPS than with OpenBSD (on workstations).
I dream that some BSD-family system will actually make the package manager a first class citizen.
This includes getting rid of the distribution tarballs (base.tgz etc) and replacing them with packages, ideally more fine grained. Thus also replacing the untarring done in the install process with a tool similar to debootstrap or pacstrap.
I believe it is one of the many problems holding the BSDs back.
Tangentially related, I also dream that they’ll eventually drop CVS for Git or a competing DVCS.
Between the two, the latter I feel more likely, thanks to the existence of those pushing for it. (ESR)
Dream nearly granted: https://wiki.freebsd.org/PkgBase
Nearly, but it might go the way of many such efforts in FreeBSD. Down the drain.
Statistically speaking, the proponent will get tired and leave.
Among the BSDs, I’m specifically not a fan of FreeBSD. It had great talent and energy at a time. They left. It’s now called Dragonfly.
I still wish FreeBSD the best and hope the PkgBase effort does succeed. If anything, because it might motivate the other BSDs to do the same.
Re. DCVS, there is got.
I have a running bet that it will replace cvs for openbsd.
nix was recently ported! Virtually nothing works, but it’s a start!
Confused. What would the Linux equivalent to “base installation” be? Perl is included in pretty much every relevant Linux distro out there.
There isn’t one. In FreeBSD the base system contains everything that the FreeBSD project themselves want to take responsibility for directly. Boot loader, kernel, libc, at least one shell, the login daemon, sshd, a C compiler. Everything you need to get the system running and at least minimally useful and self hosting.
Everything in the base system is checked into a single repo. The base system has its own mechanisms for updating (used to be a thing called
freebsd-update, dunno if it changed since).Third party software (which is most useful software, since the set of things you might want to do is much bigger than what the FreeBSD project would ever want to put in their own tree) is installed & updated via a separate mechanism called packages. FreeBSD Ports is a repo of instructions for how to build packages.
As a rough approximation, the base system takes care of stuff in
/and/usrand.packages all install into/usr/localand/opt.Unlike in Linux where the kernel (Linux) and libc (glibc, musl, etc) are written by different sets of people who don’t necessarily even like each other, in FreeBSD the kernel and userland are all worked on together. It’s possible to land atomic patches that simultaneously add a new syscall and add support for it to libc and man pages for it, for example.
You could call the minimal set of packages that come with a default Debian install its “base” but it’s still not the same I think because they all upgrade separately.
Yeah, Debian has two or three related concepts of a base system.
There is the set of packages with
Essential: yes— basically, the package manager and the utilities the package manager depends on for its operation. The package manager won’t even let you remove these, because if you did, you wouldn’t have anything left to reinstall them with. This amounts to around 20 packages (includingperl-base, as a matter of fact, since debconf uses it).Then there is the set of packages with
Priority: required. This is another 50 or so packages that the package manager can live without, but the system administrator probably can’t. This includes libc6 (nice to have, right?),mount,passwd,procps, ande2fsprogs(providesfsck). The essential and required packages together formminbase.Then there’s the set of packages with
Priorty: important, which adds about another 100. These includebzip2andxz(minbase only gets you gzip), the tools for loading and unloading kernel modules,less,vim-tiny,sudo,ping, and… unavoidably on a modern version,systemdanddbus. It also includespython3, making it the other language you’re pretty much guaranteed to find.minbaseplusimportantequalsbase, which is the minimum thing you can install from install media (but you candebootstrapto minbase for containers or embedded or whatever, if you know what you’re doing).Anyway, as you say, most of that stuff is third-party, not created by the Debian maintainers themselves. Although more than the usual amount of scrutiny goes into anything that’s marked as important enough to get into base.
“- you want a modicum of security, so shell and tcl and php are out.”
yet there are a lot of scripts written in pdksh, like sysupgrade and all…
Considering sysupgrade is (mostly) just a wrapper around ftp(1) and autoinstall(8), a shell script is a good choice IMO.
NYTProf is pretty great though