-
Two foxes behind. :(
-
Apparently, that must be a typo; 2016Q2 comes with Firefox 47.0, not 45.1.0.
http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/www/firefox/Makefile#rev1.258
It looks like even the firefox45 package (the latest ESR one) is 45.2.0:
http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/www/firefox45/Makefile#rev1.4
No idea where 45.1.0 came from.
-
Sorry, my fault. For the list of new packages I did a naive grep for “Added” in the changes file, which missed the fact that the existing www/firefox package was updated to 47.x and that after www/firefox45 was added it was subsequently updated. The full list of available firefox revisions in 2016Q2 is
firefox-47.0nb1 firefox10-10.0.12nb21 firefox17-17.0.11nb12 firefox24-24.8.1nb8 firefox31-31.8.0nb4 firefox36-3.6.28nb24 firefox38-38.8.0 firefox45-45.2.0
-
-
-
I’ve been using Homebrew for a few years–long enough to not realize what may be its limitations. How is Pkgsrc different/better/worse?
-
One of the main differences is that it’s binary packages rather than all source. Whenever I finally updated homebrew my Mac was nearly useless for a few hours compiling some rather large packages and their dependencies. Updating pkgsrc takes only a few minutes to update in nearly every case.
Pkgsrc also has over 14k packages whereas homebrew only has about 3500. And pkgsrc is cross-platform. I use it on OS X, SmartOS, OmniOS, and Linux to get a consistent set of packages/versions/configuration style across all platforms. Pkgsrc directly supports 18 different platforms and I’ve seen unofficial builds on several more.
Pkgsrc packages are fairly easy to update if you want a newer version of something, and the maintainers, in my experience, are always willing to work with you to get patches/updates into trunk.
Homebrew cute when you’re in college and beer is the central focus of your life. But really, I just need to get stuff done. For the most part pkgsrc does a better job of that than homebrew.
-
-
But really, I just need to get stuff done.
Well, I guess Google doesn’t get stuff done then ;).
https://twitter.com/mxcl/status/608682016205344768?lang=en&lang=en
One of the main differences is that it’s binary packages rather than all source.
That’s years ago. Nowadays, many Homebrew formulae are precompiled, so Homebrew rarely compiles stuff anymore:
https://bintray.com/homebrew/bottles
Pkgsrc also has over 14k packages whereas homebrew only has about 3500.
This is a bit disingenuous, because pkgsrc keeps a lot of old versions. E.g.:
http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/multimedia/
There multiple versions versions of VLC, ffmpeg, etc. Then surprisingly, quite a few packages that I regularly use are absent from the Darwin packages (they are in Homebrew). For instance: Qt 5, bazel (for building tensorflow), Armadillo, ghc, cabal-install, pandoc, libsvm, rust (though I’ve switched to rustup.rs), and SWI Prolog is at an old version and only with all packages disabled (lite).
https://pkgsrc.joyent.com/packages/Darwin/trunk/x86_64/All/
Although I like pkgsrc, there are also some nice advantages to Homebrew. E.g., if you want to compile and install your own software, you just use
/usr/local/Cellar/<mypackage>/<myversion>as the prefix and then the usually Homebrew functionality works (brew link <mypackage>to link under/usr/local,brew uninstall <mypackage>to remove, etc.).-
Qt 5, bazel (for building tensorflow), Armadillo, ghc, cabal-install, pandoc, libsvm, rust (though I’ve switched to rustup.rs), and SWI Prolog
Thanks, this is useful, I’ll take a look at adding or fixing these. If there are any other packages that people would find useful that are missing, please feel free to raise an issue.
-
-
-
-
Fun fact: wall time of building rust is longer than building the other 17,000+ packages in pkgsrc put together.
Source for that?
I’d be surprised because aren’t a lot of those packages GNU packages which use the (slow) autoconf?
Or does pkgsrc somehow skip the autoconf?
If they each take 1 second, that’s 17,000 seconds, or 283 minutes, or nearly 5 hours. Rust takes longer than that, or the average package takes less than 1 second (impressive if true.)
My SmartOS bulk builds, e.g. https://twitter.com/jperkin/status/644209661516906496. The last rust build (1.21) took 24317 seconds by itself. Note that we distribute our builds across 48 or so concurrent builds, and I concede that “put together” wasn’t the best wording, hence the apparent confusion.
Oh that is interesting. Do you have any details on this setup? How do you deal with dependencies and so forth?
I didn’t know that Manta was being used for builds. Do you work for Joyent? I actually went to a talk at the Joyent office in SF about Manta around the time of launch (I think 3 or 4 years ago).
It seems like a really cool technology. As far as I remember, it’s basically a “polyglot” MapReduce with tasks run in jails. I’ve worked with MapReduce quite a bit, and I agree it needs the polyglot/shell treatment (and so do its successors like Flume and Google Cloud Dataflow). It runs in containers at Google, but it’s still at its heart a C++ framework.
Eventually I would like fold this paradigm in my shell [1], but that is still a long ways off.
As for the build part, I’d like to do a similar thing for Alpine Linux in the nearer term. I wrote about experimenting with it here [2]. Alpine still runs their builds on a handful machines and they get backed up a lot! The non-x86 builds appear to be in poor shape in particular.
I would probably run it on a single machine, but do some tricks to avoid reinstalling dependencies for every build, like using bind mounts.
So you’re saying that using 48 machines to build the entire SmartOS repo is faster than building Rust on a single machine? Yes it’s a little misleading, although not exactly unfair if the Rust build doesn’t parallelize. I imagine it must parallelize at least a little bit though. Although I think Rust does do more global compilation than C/C++ do because it doesn’t have the notion of translation units.
[1] http://www.oilshell.org/blog/2017/01/19.html
[2] http://www.oilshell.org/blog/2017/10/06.html
Manta is only used for storing the results. Obviously we love Manta but it’s not really suited for this type of work. The builds themselves are standard pkgsrc bulk builds using pbulk. I wrote up a bit about it here: https://www.perkin.org.uk/posts/building-packages-at-scale.html
Great writeup! Just read the whole thing. A few thoughts:
I’m jealous of Dtrace! I quite don’t understand the
ptimeand thendtrace -ninvocations in the post. Is thedtrace -nlimited to whatever is a child ofptime? How does that work?I’m surprised that so much win was due to shell inefficiency, but that’s good to know I would have expected more of the win to be due to scheduling. But that’s good, because it supports my thesis that “shell is a problem”, hence the reason I’m working on a shell!
How does your scheduling work? You start jobs on VMs with a fixed number of chroots and a fixed package list? Or is it more dynamic than that? That is, if you have 48 nodes, is it one job per package build (17K jobs total), or one job per node (48 jobs total), etc.? I couldn’t quite tell from the blog post. I’m not familiar with Joyent’s cloud or pkgsrc and its parameters. Is the number of VMs fixed or dynamic?
I would have expected a lot of the work to go into scheduling (i.e June 2013 section), but it looks like a lot of it was optimizing shell stuff in pkgsrc.
I know that Debian’s build system is very inefficient with regard to shell too. One of the reasons I was interested in Alpine Linux is that the build scripts were more sane. But I have yet to measure the performance there. Everything seems a lot faster in general.
But yes I would like a Linux distro where building every package under the sun in 5 hours is possible! That is pretty amazing! I’m not sure any Linux distro can do that, but I’d love to be corrected.