1. 25

  2. 20

    As somebody who work with Gitlab on a day to day basis for the last 3 years: This really shows that Gitlab is getting desperate.

    Since before the IPO, I think the way they run their products line up: to favor breadth over depth, is already a non-sustainable approach. Even after the IPO, I still see they try to expand their product line up toward craps that they have no talents strategy to back: MLOps, Monitoring, Remote Development environment, Code Search… While their core product offering such as Merge Request, CI, Runner, Gitaly barely get staffed to fix critical bugs.

    As Github started to ramp up in features delivery in the last few years, I think very soon (if not already) they would surpass Gitlab in core features’ depth. I have strong doubt on Gitlab as a company.

    1. 3

      Since before the IPO, I think the way they run their products line up: to favor breadth over depth, is already a non-sustainable approach

      Pretty much this right here is why I’ve been so resistant to give SourceHut a real go, even though the product matches more closely what I want in terms of usage. I can’t imagine how it won’t all come crashing down under the weight of all that breadth, and the numerous side projects (and presumably consulting) that they engage in to keep it afloat. (Yes, I recognize that giving them my money will help here…)

      I mean, if GitLab can’t do it, how can a team of 3?

      1. 8

        I mean, if GitLab can’t do it, how can a team of 3?

        In recent months, I have been trying to figure out why the hell does Gitlab take so long to address some of the obvious Issues in their backlog that affected 30-40s paying enterprise customers. I spent the time to filter throught their backlog, watch their PM’s update video on the Unfiltered channel.

        Side note: One thing Gitlab has it right: a super transparency approach in running the company. Thanks to this, you could just dive in and get these information yourself.

        Turn out, the reason why the feature is keep getting delayed is… the entire team who own that feature has 3 engineers + 1 shared(?) PM + 1 shared(?) UX designer working full time on it. They have 1 huge backlog of high priority ‘new’ features to further expand their product breadth and critical bug fixes, pretty much enough work to drown the team of this size for another year.

        They would never be able to get to the feature request I have been watching because: it would only have impact on big enterprise customers. Gitlab’s product development pipeline are not optimized for this audience, they are trying to optimized for the startup of 5 people who are (1) smaller than Gitlab itself and (2) require something quick to bootstrap everything. There is no feature depths required for that targeted audience.

        I think such a product strategy makes sense pre-IPO: they spent 3 years hinting IPO via different media channels while trying to make their product offering to cover a massive range of things. This makes the company looks very nice in the eyes of non-technical investors, the growth potential is infinite. But a result of this is that their core product offering rot away:

        • Merge Request UI got slow, terrible bug that would let mergability check spinning forever until user manually refresh the page.

        • CI Merge Train was so ahead of everything else, but got no love because… they directed the talents who worked on it to lead new product lines. Now Github is coming out with a better feature offering.

        • Gitlab CI yaml was so rich in feature, all it requires is a saner YAML syntax. They did… nothing there. Github Actions came out with proper yaml syntax versioning and a much better plugin ecosystem. BuildKite, CircleCI are also getting a lot better.

        Here are the things I think Gitlab should have never get themselves into producing:

        • Kubernetes buzz: They used to advertise support for Serverless and WAF. Both of which are deprecated today, waste of effort.

        • Package registry: They underestimated the serious amount of talents required to build something to compete with Jfrog’s Artifactory and ‘s package registry solution. Their product have very little competitive edge except for “it’s already on Gitlab and we have Gitlab so let’s just use it”.

        • The things they are trying to get into today:

          • MLOps: It does not work, don’t make it a thing
          • Symantic CodeSearch: Trying to compete with SourceGraph using the search indexing engine that SourceGraph is maintaining?
          • Remote Dev Container: Competing with Github’s Codespace and GitPod and JetBrain’s Space and Replit. If I understand this correctly, Gitlab is starting from behind with 2-3 times less the headcount dedicated to this vs any of the players in the space.
          • Security Scanning: This one is actually sound product strategy, but instead of building things in-house, they should have aimed to enable better integration with 3rd party solutions.
        1. 7

          I mean, if GitLab can’t do it, how can a team of 3?

          Some of it has got to be the inefficiencies innate in a large organization, or having people who are not enthusiastic about just making a good product. Other parts are the inefficiencies in having engineering time doled out by those who don’t share the priorities of those doing the work.

          There are many things a small company can do, and well, that a large company can’t hope to do better. Inertia is a factor in engineering organizations.

          1. 1

            Yeah, as an employee of a many-hundred-person company it doesn’t surprise me at all that a team of 3 can outmaneuver a VC-backed corporation. Sometimes when I compare what I can do within the structures that are enforced at work vs what I can do in my free time I feel like I’m a hundred times more productive on my own doing what I want to do. I can think of a few factors:

            • allowing myself the room to do something right the first time VS cutting corners to hit a management-enforced deadline and claiming we’ll go back later to clean it up but never actually doing it
            • decisions about what to do next are made by people who use the software every day and possess a deep knowledge of how it works and why
            • not being affected by political maneuverings of managers trying to advance their own career (obviously larger OSS orgs can have plenty of political drama, but usually this doesn’t manifest until you reach a certain size)
            • not having to ever use Jira
          2. 5

            Worth noting that SourceHut publishes yearly finance reports. In particular, the 2021 report mentions:

            We have three full-time staff paid under the standard compensation model, and about $700/mo in recurring expenses. Currently we make about $9200/mo from the platform, putting our monthly profit at about $1000, without factoring in consulting revenue. Thus, the platform provides for a sustainable business independently of our other lines of business.

            1. 4

              I should have chosen a better word than “afloat.” They charge money to use the service, presumably enough to keep things paid.

              However, my concern is more with keeping the service actually running given how much it does, and it scaling. There was this a few years ago, that made me wonder if this is the scaling strategy, or if there’s another plan. The site has a large number of services and a very small team. That’s generally a risky bus factor, but I am probably being over cautious.

              1. 5

                Bro, they have $1000/month in profit. They’re good. Nothing could possibly go wrong with that massive warchest in reserve. /s

                For real: one decent lawsuit would bankrupt them.

                1. 2

                  What would you sue them for? Daring to compete with our dear overlord Microsoft?

                  1. 1

                    Literally anything, make up your own lawsuit. It just needs a sheen of merit. You don’t need to win, just run down the clock so they can’t afford to defend themselves.

                    Lack of accessibility under ADA. DMCA or copyright violations. Some OSS licensing or patent garbage.

                    1. 4

                      Drew has moved to the Netherlands; and it seems the Sourcehut company registry moved with it, so these kind of US-style trivial lawsuits with huge bills are probably less of an issue.

                      I get your point, but what other options are there if you’re just $some_guy looking to make a small tech business? Starting any business is always a matter of risk.

                      1. 1

                        what other options are there if you’re just $some_guy looking to make a small tech business?

                        For sure, initially you’re running on fumes but no one sues a party with no cash and no influence. But sourcehut has influence now. That means they need to raise prices and create a savings reserve. Ideally, make enough profit (e.g. $10k/mo) to pay for a lawyer to handle any legal cases. $1k/mo isn’t enough.

                  2. 1

                    one decent lawsuit

                    which would force Dr. Evil to out himself.

              2. 2

                I can’t imagine how it won’t all come crashing down under the weight of all that breadth

                Yeah, but the difference here is that sourcehut is fully open source. If it crashes and burns (and I hope it doesn’t) you can just host it yourself. That’s one of the reasons I switched to sourcehut from Gitlab a while ago, despite being quite happy with Gitlab.

                I mean, if GitLab can’t do it, how can a team of 3?

                I don’t mean any disrespect to the folks at Gitlab, but a small team actually gives me confidence. Per Kelly’s rules for the Lockheed Skunk Works:

                1. The number of people having any connection with the project must be restricted in an almost vicious manner. Use a small number of good people (10% to 25% compared to the so-called normal systems).
                1. 2

                  That’s one of the reasons I switched to sourcehut from Gitlab a while ago, despite being quite happy with Gitlab.

                  You mean like GitLab? I’m sure neither SourceHut, nor GitLab is trivial to host, but yeah, both are possible, at least.

                  1. 3

                    “GitLab’s open core is published under an MIT open source license. The rest is source-available.”

                    Not all of GitLab is possible to self-host, at least not at the moment.

                    1. 1


                2. 1

                  looks gitlab is confident they can, (just) if they charge.

              3. 3

                I honestly think this is a fair move, especially because of the new limit being significantly high for those only storing code on GitLab.

                The truth is that it’s expensive to subsidize free users if they incur significant costs. Furthermore, introducing a storage limit can make financial projections much safer as it will cap a free user’s cost.

                Also, considering GitLab itself is open-source, free users could simply host their own instance.

                1. 2

                  Now this sounds like a considerably better approach to control space usage by the free tier, because nothing gets deleted.

                  1. 1

                    at the end of the day there is but one single unpaid reliable operator of your infrastructure. You.

                    1. 4

                      there is but one single unpaid reliable operator of your infrastructure. You.

                      I think you are severely overestimating me.

                    2. 1

                      I think the scale and timeframe is fascinating.

                      Mid October the threshold is 45 TERABYTES for unpaid accounts. Terabytes. agreed the scheduletef is aggressive but I think we can all agree this is taking the piss for a free tier service. Help my ebook collection of 20 years is still under 3GiB. All my repos together take up at most 10GiB which includes 7GiB of BSD project clones.

                      45TiB is somebody taking the piss. Even 5 TiB is salty tears still. How w many “real users” fall into this category? Maybe open office is a legit example if they store every build artefact for every platform

                      1. 1

                        5GB seems very low! I wonder if they’re worried specifically about the size of storage or if it’s a proxy for general workload. Or maybe all the bigger projects are not things they want; people storing porn or whatever.

                        1. 9

                          It seems very low, but it’s still quite a lot, assuming you’re using repositories for source code. I’m currently using ~2.2Gb of storage for my git repos (on my own server mind you), 1.7Gb of which is my brother’s website with huge photos. The other ~500mb are ~170 repositories of various sizes (including a mirror of Guix, accounting for about half of that). My repositories, even those with a long history and a fair amount of commits clock in below 100mb, but most of them are even smaller than that. For reference, Gitea, a 8+ year old project with over 13k commits and 140 tags is only about 250mb. You’d need to have ~20 Gitea-scale (or ~15 Guix-scale) projects to exceed the 5Gb storage limit.

                          You can fit a lot of source code into 5Gb, so that quota seems quite reasonable to me.

                          1. 5

                            I am a little amused by their progressive size steps, starting at 45,000 GB and working their way (rapidly) downward. I have to wonder what proportion of their free tier users are consuming >45,000 GB of space!

                            1. 3

                              Yeah, the step sizes are a bit baffling indeed, and I’d love to see some numbers, how many repos/accounts are affected by each, and so on. I’m not entirely sure I understand why those steps are necessary, seeing as the delay between 45gb and 5gb is only 3 weeks, with the biggest drop (45000g->7500gb) is only a single day. Might aswell start at 7500g then, or as repos are going to be locked only, and ample preparation time was given, just flag-day the 5g.

                              They could just notify people now, and introduce the 5g limit on October 19th. That’d give people more time than when they start getting in-app notifications on October 22nd. But maybe I misunderstood when they’ll start notifying people.

                              1. 5

                                I mean, I can understand the gradual roll-out; first to see if it works at all, and second to give their infrastructure and customer support people both a chance to ramp up slowly. It’d be cool to see numbers though; I would have divided it up such that 10 repos get affected by the first change, 100 on the second, 1000 on the third, and so on, but there’s other valid ways of doing it they may want instead.

                            2. 3

                              The storage counts build artifacts as well, not just the git repo.

                              1. 1

                                To be specific:

                                Storage types that add to the total namespace storage are:

                                • Git repository
                                • Git LFS
                                • Artifacts
                                • Container registry
                                • Package registry
                                • Dependecy proxy
                                • Wiki
                                • Snippets


                              2. 2

                                You can fit a lot of source code into 5Gb, so that quota seems quite reasonable to me.

                                My ~/code directory is currently 872M. This includes projects going back almost 20 years (some of which aren’t even on GitHub but were on Sourceforge, Google Code, or BitBucket back in the day), a bunch of cache/data files that aren’t in git, projects I worked on but haven’t pushed (yet), generated/cached stuff that’s not in git, etc. All of that combined is probably at least 100-200M less, but I didn’t bother to really check. There are 398 directories/projects in total.

                                This doesn’t include binary uploads though; for example all of uni‘s git history is ~16M (it’s comparatively large as it includes a unicode database), but there are also ten releases each has 10 binaries of ~1.5M, so that’s ~150M extra.

                                Still, I have a lot more projects than most, and thus far I probably would have had enough within the 5G limit. As you mentioned, even large projects tend to be on the order of several hundred M.

                                I feel it’s regrettable that the only way to upgrade is $19/month; I pay $5/month to FastMail and get 30G of storage with that. I guess it makes business sense to focus on the business customers.

                                1. 1

                                  It seems very low, but it’s still quite a lot

                                  Just don’t work on Linux (or Chromium or Firefox)

                                  1. 3

                                    How does GitLab account for forks? I believe GitHub internally uses a content-addressable store and so multiple clones of mostly-identical repos use the same storage. If one person pushes Chromium then they’re adding a lot of data, but then every subsequent person who pushes the same repo just adds a reference count. This is why they don’t care too much about large popular repositories: the cost is amortised over the users who have forks. It’s only large private or unpopular repositories that have a large per-user cost to the platform.

                                    1. 1

                                      Neither of those are developed on GitLab to begin with, and I see no point in having a personal mirror there, either.

                                      1. 1

                                        There’s a few reasons. First, you may want to share a branch with other developers. This can be because it is a work in progress (and thus not suitable for submission yet). Pushing a branch which is a compilation of multiple separate patch series can allow other users/developers to test your work easier. You may want to run third-pary CI (Azure, Travis (although I suppose that is less common now), etc) against your branch.

                                    2. 0

                                      1.7Gb of which is my brother’s website with huge photos

                                      Do not store large binary files in Git.

                                  2. 0

                                    Title is editorialised to highlight the specific doc part.

                                    1. 4

                                      It’s a more useful title than the original.