1. 3

    Napkin calculation: it’s 624,197,820,790 bytes, spread across 4,073,468 files. That’s about 153,235 bytes per file. Assuming the average length of a line is around ~40 characters, and knowing that Windows takes two bytes to store each character (because they use UTF-16 for legacy reasons) and it’s a bit extra in needing two characters to represent a line ending, we can assume about 100 bytes per line. That would mean the average code file has ~1,532 lines, and the whole monorepo has about 6,241,978,207 lines (that’s 6 billion lines of code).

    4 million files is a lot. The linux kernel supports far more architectures and possibly has more drivers built in, and it only has 61,725 files with an average length of ~414 lines. So, I think there’s a chance that there’s something wrong with the way they counted the files, for example, if it’s a git repository that they counted files in .git/ as well? Not quite sure.

    1. 14

      [this includes] source code, test files, build tools, everything The build tools and test fixtures probably make up a sizable portion, so that would definitely skew the total

      1. 1

        Does the repo include graphical assets too?

      2. 6

        more than a half million folders containing the code for every component making up the OS workstation and server products and all their editions, tools, and associated developement kits

        You cannot compare their entire stack with another family’s vanilla kernel alone. Pull in all of GNU userland, GCC, an IDE, some browser, and so on… maybe that will help the numbers starting to compare.

      1. 4

        This week I’m implementing two-factor authentication for Hex.pm!

        1. 1

          In my opinion, using new pieces of technology in your stack can be a good idea; in small doses.

          Trying out a new library or framework in a project is great, and if tested and trialled enough, can greatly help both you (the user) and the creator / developer.

          It gets silly, however, when someone decides to experiment with a new piece of technology in a major part of their stack (e.g. databases or alpha-grade programming languages). Of course, ever piece of software needs testers, but it’s definitely not a good idea to test these kinds of things in products that need to be reliable for users, etc.

          1. 2

            This week, I’m not working on much in the way of programming for once!

            Instead, I’m designing and building a new system for live video production, including switching, standards conversion and recording, as well as various camera equipment.

            1. 4

              I have noticed a large amount of controversy regarding the single USB Type C port.

              Although it may be regarded as an inconvenience and described as some as simply a way for Apple to make more money, I personally believe it is a great advancement in computing. Although the specification is relatively new, the plan is to make the USB Type C connector standard among computers, Apple is simply pioneering the effort to bring it into the hands of consumers.

              Another bonus with the USB specification is the openness behind the USB specification. This will allow third party manufacturers from all around the world to produce USB Type C accessories, which will no doubt diversify the market.

              I really look forward to seeing how this plays out, and how it is integrated into new products in the near future!

              1. 4

                I’ve been working on hex again this week.

                Primarily, I’ve been working on password resets, package signing and rebar3 support.

                We’ve just passed 190k downloads, so 200k isn’t too far away now!

                1. 8

                  “Take care when publishing a crate, because a publish is permanent. The version can never be overwritten, and the code cannot be deleted. There is no limit to the number of versions which can be published, however.”

                  This is good news!

                  1. 5

                    I don’t know… What if you accidentally published private information? I feel like there should at least be some sort of window within which you could undo the push.

                    1. 5

                      You can contact us and we’ll pull it in cases like this. There’ll be actual policy before 1.0: this is like a ‘beta prerelease’ for everyone to try out the infrastructure now.

                      But setting the expectation that you can’t just yank versions whenever you want is good.

                      1. 3

                        I know hex supports this with a one hour grace period. After that, it’s locked in for good. Maybe crates.io would benefit from a feature like this.

                        1. 1

                          Then it’s been published. Time to dust off that harm minimization document.

                          Deleting the version isn’t going to take it back. Especially as rust becomes more popular, and there are mirrors all over the net.

                          1. 5

                            But it may erroneously contain third-party IP. Continuing to distribute it would put you at legal risk.

                            1. 2

                              I won’t put any words in their mouths. There will almost certainly be a way to takedown specific URLs. (DMCA)

                              But, as a serious issue, if you have that as a risk then implement a delay / sign-off process. You can’t assume a central replication system is going to implement an “I take it back” system.

                        2. 5

                          It is good news! A big beef I have had with maintaining node related ports on OpenBSD is people publishing updates to existing versions! Totally breaks the checksums in the port tree!

                          Squeee - Testing out my nifty hat!

                          1. 2

                            I thought it was a boat. :)

                        1. 2

                          I think a oss or foss tag would be great.

                          1. 3

                            This week, I’m working on implementing password resets on https://hex.pm. I also working on improving the tests, a new tarball metadata format and better S3 log parsing.

                            If anyone is interested on working on Hex with some Elixir/Erlang, you can find Hex on GitHub here.

                            1. 3

                              Finally moving my MVP to our new stack! I can’t say too much, but it’s a combination of Elixir (OTP) and the JVM with a lot of influence from the Raft consensus algorithm.

                              In terms of Hex, I’ve been working on email verification for new accounts, email password reset, a new registry format (ets table + log table) and rate limiting for the API. The next step, after these are merged, is to focus on multiple registry support!

                              For anyone interested, the source for hex is hosted here and the source for hex_web is hosted here

                              1. 5

                                I’ve been working on https://hex.pm again, and I’ve almost got some pull requests ready to get it closer to a production ready state, including email verification, two factor auth, multiple package repos and package signing, hoping to have this all done my Elixir v1.0.0’s release!

                                1. 3

                                  I’ve been working on Hex yet again, putting the finishing touches on pull requests to make it both more seure, as well as more stable and usable in self run environments.

                                  1. 1

                                    Im glad to hear about progress! Im not familiar with erlang, but how do you handle versioning?

                                  1. 9

                                    In the OSS world, I’ve begun to contribute to the Hex package manager, I’ll be assisting in the web interface redesign, API documentation, trying to close all current issues and bring it into a state where it can be run on an isolated system, as well as many small improvements , such as email verification, password resets, and better support links.

                                    I’ve also pushed some bug fixes to my Lobsters browser for iOS which was accepted today!

                                    In !OSS, I’ve been working on my MVP application which is starting to take shape, as well as reading many papers on targeted advertising for the service!

                                    Until next week! :)

                                    1. 1

                                      Ive always been curious what sort of use erlang gets in the real world. It seems odd (from my perspective of course) that there is enough demand for it to need a package manager.

                                      1. 1

                                        Erlang is really good in applications that can’t go down, and where jobs can be picked up from where they left off in disposable processes, although, the syntax is quite a change, and in some cases inefficient. Elixir fixes a lot of these problems and makes writing OTP applications a lot easier and productive (and sometimes even fun), whilst still maintaining the power and VM of Erlang, hence why many are starting to experiment with it, and even use it in production systems. And from now on, there is a breaking change freeze until 1.0, so now is a perfect time for people to start using it. And people want packages, and it needs to be done well cough npm cough

                                    1. 10

                                      I’m putting the finishing touches on the initial release of Manifold, which is meant to sit at the intersection of the n-many synchronous and asynchronous JVM stream representations. Anyone who has an opinion on Clojure or stream abstractions is welcome to give feedback.

                                      1. 2

                                        This looks like a great way to deal with asynchronous operations in Clojure! Will definitely be using this in upcoming projects.

                                      1. 2

                                        This week I’ll be continuing porting my MVP to our new stack (Elixir and Scala).

                                        This week, I’ll be finishing up our RethinkDB driver, ID generator and Service Registry. Once they’ve been completed and open sourced with decent documentation, work will start on the application specific portions of the service, firstly by building out our core API and making sure we’re meeting our REST API specification.

                                        I hope I’ll have some finished libraries and services to show you all this time next week!

                                        1. 6

                                          This week I’ll begin porting my current MVP to our new stack based on Erlang (Elixir) and the JVM (Scala)!

                                          The first thing we will be building is a new Erlang RethinkDB Driver based on their new JSON API, of which we will base much of our platform on. We will also be building services for ID generation, service registry, and more, all of which will be open sourced in coming months.

                                          1. 1

                                            Curious why you are using both erlang and jvm in a new stack. What tradeoffs went into the decision to use a mixed architecture like that?

                                            1. 4

                                              The original prototype was built in Node.js, which worked well for a prototype, but not much else.

                                              We chose Erlang and Scala for several reasons:

                                              The kind of application we are developing (a realtime social application) preforms much better on Erlang’s process model, especially when it comes to concurrency and error handling, among many other benefits. The original plan was to build the entire stack on Erlang, but after some investigation with our test data, it was quickly shown that the JVM well out-preformed Erlang when it came to jobs like map reduce and machine learning. If we were building the service as a monolithic app, we would have stuck with Erlang for everything, but as the service is being built as a service-based architecture, we decided to build our processing nodes in Scala (chosen over Java mainly due to syntax and actor support).

                                          1. 3

                                            I started by looking at Github projects, and seeing how they work. Not only the code, but also how the issues and pull requests work.

                                            Then find an issue that people are having, and have a crack at fixing it. Even if it’s a small documentation change, or a typo in the code, every contribution helps, especially while seeing how it all works.

                                            That’s my two cents anyway :)

                                            1. 1

                                              Looks great! It would be nice to be able to login though - settings is just a blank white screen on my Samsung gs3

                                              1. 3

                                                Currently the API for Lobsters doesn’t allow anything other than stories over JSON, so things like login and voting are not yet possible via the API :/

                                              1. 5

                                                I’m writing this message from a Lobste.rs client that I wrote for iOS. I’m currently in the final stages before I open source it. Is there any licensing issues with creating an app for Lobste.rs?

                                                1. 2

                                                  I’ve heard that Apple explicitly do not allow GPL applications on their App Store, but maybe someone who knows more about app dev may be able to confirm.