1. 4

    Preparing for the yearly relaunch of https://24pullrequests.com on 1st December

    1. 2

      Wow, this is exactly what I’ve been looking for! I had even started to make notes of how I’d build something like this myself. Very impressed!

      1. 1

        @andrewnez, are there any plans to also track events that don’t come through the GH notifications API? For example, when an issue is cross-referenced elsewhere, when CI status updates, etc. I’d like to know when these events have happened as well.

        1. 1

          We’re already getting those extra events via the GitHub App webhook at the moment, fancy opening an issue with more details of how you’d like them displayed? https://github.com/octobox/octobox/issues/new

      1. 1

        One of my current issue on top of limited and noisy Github notification systems is that I use Gitlab in addition to Github, doubling my amount of effort required to track all my stuff. Any plan to support more than Github in the future?

        1. 2

          I had a look to see if it was possible with the GitLab API about a year ago and there were some pretty big pieces missing, must revisit it soon to see if they’ve added anything similar to the notifications API

        1. 11

          This week I’m preparing to launch Octobox on the GitHub Marketplace and start building a sustainable business around the project.

          1. 3

            Just wanted to say that - Octobox is fantastic!

            1. 3

              Very exciting! Good luck :)

              1. 1
              1. 4

                This project, like others I’ve found, doesn’t seem handle to Python dependencies properly. For example, Flask 0.12 is listed as having no dependencies, but actually it depends on Werkzeug, Jinja2, click, and itsdangerous. [1]

                I think this is because these sites are grabbing package JSON data from Pypi, but many (most?) packages don’t declare their dependencies there. As far as I know the only way to accurately resolve dependencies for Python packages is

                1. Grab the Wheel (if there is one) and inspect the metadata.json file, otherwise
                2. Download the source distribution and actually install the package. This is very often necessary, as many projects (or older releases) don’t have a Wheel.

                I’ve been working on a project that actually has the correct dependency graph for Python libraries. I’ve had to follow the approach above. It’s not quite ready to show the world, but I’m hoping it’ll be interesting for people.

                [1] https://libraries.io/pypi/Flask/0.12

                1. 2

                  Yeah python dependencies are not easily machine readable, and for the moment I’m trying to avoid executing setup.py files downloaded from the internet on the Libraries.io servers, any help contributing better python support would be great.

                  1. 3

                    I’m trying to avoid executing setup.py files downloaded from the internet on the Libraries.io servers

                    That’s wise - I’ve seen all sorts of shenanigans in those files. Just importing some of them causes attempted sudo operations.

                    1. 2

                      I’m not particularly familiar with the python world, but this sounds like the perfect use-case for containers, no?

                      Note I said containers not “docker”. I believe what you want is a quick “spin up $distro, install $package, analyse installed deps” flow, which imo would suit lxc/lxd perfectly.

                      1. 2

                        I’ve hacked something similar together over here: https://github.com/librariesio/pydeps

                        1. 2

                          You’re right - I solved this issue by using docker

                          1. 2

                            Yup. The place I work at (shameless plug: https://fossa.io) does this using ephemeral Docker containers.

                            We scan projects to check if they’re compliant with the licenses of their open source libraries. To do this, we need to compute the dependency graph of a project. For most build systems (the exceptions are usually NPM and Golang tools), this means running a full build to execute any arbitrary build scripts.

                            If you only do static analysis of package manifests, you tend to overreport and underreport – you’ll miss packages brought in by build scripts, and you’ll have extra packages (or extra versions of packages) that are included in the manifest but might be unused/optimised out by the build system/brought in by version constraint solver weirdness.

                      1. 3

                        I’m assuming you’re going to try to get wycats on at some point? Considering he’s been involved with at least 3 different package managers (bundler, cargo, and yarn are the 3 I know of), it seems like he would be a great interview.

                        1. 3

                          Yep, he’s already on the list of people to interview ?

                          1. 3

                            Also, the SolusOS guys recently did a lot of work with the tool they use to make packages for their package manager, so they might be worth talking especially wrt the producer-side of package managers

                            1. 3

                              Will add them to the list, thanks!

                        1. 12

                          Nice! Would you like to do one about Nix / NixOS / Nix on macOS / etc?

                          1. 4

                            Absolutely, would you be able to email andrew@manifest.fm to schedule?

                          1. 1

                            On the mac I use https://selfcontrolapp.com to block access to news, email and social media site/apps for a few hours, works pretty well, also tend to leave my phone in another room to avoid temptation.

                            1. 4

                              I’m working on mining dependency activity (dependencies added/modified/removed per commit) from git repositories for https://libraries.io and streaming it all via twitch, code so far is over here: https://github.com/librariesio/repo_miner

                              1. 6

                                One of the maintainers of Libraries.io here :wave:

                                Today we’ve released around 200 million lines of open source metadata in csv form that we’ve been indexing over the past two years. This includes dependency information for 25 million open source repositories from GitHub, GitLab and Bitbucket, mapping out a huge dependency graph for open source software.

                                The actual csv download is available on Zenodo: https://zenodo.org/record/808273

                                Further documentation on the site itself: https://libraries.io/data

                                1. 1

                                  All we need now is for somebody to analyze the data ;-)

                                  1. 1

                                    Have you considered putting the dataset on Kaggle?

                                    1. 2

                                      Looks like they have a max upload size limit of 500mb of uncompressed csv, the Libraries.io release is ~25GB uncompressed!

                                  1. 8

                                    No open source? Booooooo.

                                    I wrote lowdown specifically to do PDFs via groff (-ms, or -man with groff and/or mandoc). It’s not an easy problem, I guess: Markdown parses easily into HTML; but with roff, you need to have an AST to handle spacing issues. Or at least look-ahead parsing. No word on the site of how these are generated.

                                    I can see that the linked page (again, no source code?) doesn’t handle links. For example, lowdown/README. Then as rendered with lowdown here. If you don’t do any span-level markup (i.e., only using \fI instead of invoking a macro) it’s easier—maybe that’s what they did. (Note that the links, as is, have a trailing space, which means it might’ve been generated with groff and that’s the extra space after a macro. Who knows.)

                                    I do like the PDF’s font, by the way.

                                    (Does anybody know why OpenBSD’s firefox PDF viewer renders so ass-tastically, and without utf8?)

                                    1. 3

                                      lowdown looks nice!

                                      afaik heirloom troff is now maintained by someone else and now lives at http://n-t-roff.github.io/heirloom/doctools.html

                                        1. 2

                                          Missed that. Seems it just pipes internally to markdown-pdf, which internally just pipes HTML through html-pdf. Eh.

                                      1. 2

                                        Working on adding support for Bitbucket and GitLab repositories to https://libraries.io

                                        1. 5

                                          It’d be great if this was hosted somewhere online with a dummy user for me to try out before investing time into setting up the environment for it.

                                          1. 2

                                            Hosted version coming later this week

                                            1. 1

                                              When looking at C projects, the ‘New C Projects’ are really Go projects from 2014. Is that deliberate?

                                              1. 1

                                                At the moment the C support is very poor, mostly because there aren’t really any application level package managers like rubygems or pip for C, I’m planning on adding support for system level packages (apt, yum etc) that will improve it

                                              1. 5

                                                I’ve been working on Dependency CI along with https://libraries.io the service that powers it, in my spare time for the past few months, it’s great to finally get it out into the real world.

                                                Dependency CI works like Travis CI but for the dependencies of your application, checking them for license and status issues every time you push to GitHub.

                                                I’ve written a up a post on medium with more details: https://medium.com/@teabass/introducing-dependency-ci-e859fa138eb6

                                                It’s 100% free for open source projects and there’s a 14 day free trial for checking private github repositories too.

                                                1. 1

                                                  Very cool! You should add the softwaret tag, though, since this is a release.

                                                  Also, your link to libraries.io has a trailing space in it and is broken.

                                                  1. 1

                                                    Thanks, fixed the link now

                                                1. 2

                                                  Not python compatible ?

                                                  1. 1

                                                    I’ve not got round to pulling in the dependency graph of pypi yet as it wasn’t easily accessible via their API, having to manually parse requirements.txt for every package which is taking a bit longer than expected.

                                                    1. 1

                                                      ok thanks

                                                  1. 2

                                                    I’m working on a public api for Libraries.io using https://github.com/cerebris/jsonapi-resources to follow the JSON API spec in an attempt to produce a standard interface for querying package manager data.

                                                    1. 2

                                                      No response for aacs or libaacs, does that just mean no one’s packaging it at the moment?

                                                      1. 2

                                                        At the moment Libraries doesn’t have great coverage of C projects, I’m hoping to start indexing launchpad at some point which should help with that.

                                                        1. 1

                                                          Yay!

                                                      1. 1

                                                        Nice idea. Can’t help thinking that some element of curation is needed?

                                                        Example - I can search for “date” or “time” under PyPI and there are plethora of options, but what I really need is someone pointing out the arrow library which IMHO is the only one you need.

                                                        1. 2

                                                          Thanks, I’m working on ways that people can curate collections of libraries to make the best ones easier to find. The Pypi area of the site is slightly lacking at the moment as I’ve not been able to get as much info as I’d like from their JSON API, as I get more information on which libraries depend on arrow it should naturally rise up the rankings.

                                                        1. 7

                                                          I been working on Libraries.io in my spare time for the past couple months, I wrote up a little more about how Libraries.io it works on medium: https://medium.com/@teabass/solving-open-source-discovery-db43a04cd9e7

                                                          1. 7

                                                            This looks great! Search is fast!

                                                            My only comment about this project is your use of the term “Open Source”. Yes, I am one of those people. The PC term is “FLOSS” which stands for “Free, Libre, and Open Source Software”.

                                                            I personally call myself a Free Software developer, not an Open Source developer. There are all sorts of wacky people like me who would get offended.

                                                            Or do you curate this to only have Open Source software?

                                                            1. 4

                                                              It’s not curated as such, libraries scans all the package manager registries for meta data about packages, where possible including the license. Not every project has a license but the ones that do are grouped together here: https://libraries.io/licenses

                                                              I use the term “Open Source” in a similar way to GitHub because it’s easy to understand at a glance, although for the unlicensed projects is really just “source available”, apart from some of the jars on maven where even the source isn’t available!

                                                              1. 5

                                                                I’m going to be a pain in the ass again. The way GitHub uses the term is wrong and deliberately wrong. GitHub has an interest in calling Free Software “Open Source” because they don’t believe in the Free Software principles. There is the famous Open Source (Almost) Everything from Tom.

                                                                The reason is obvious. GitHub makes it’s money from companies who want to keep their source proprietary. Proprietary software is anathema to Free Software.

                                                                I don’t see why your project would align with GitHub’s methodology in this case, unless you have similar plans to make money.