1. 11

    That’s weird. My XPS 13 has been extremely well behaved - suspends / resumes flawlessly, wifi & bluetooth just work. Touchpad is fine. etc etc.

    It’s kind of refreshing to not have to work round some random piece of hardware that just doesn’t work for obscure reasons.

    Maybe it’s the dual GPU XPS laptops that are particularly bad? Or have they simply got worse again since I bought mine?

    1. 5

      An XPS 13 was the inspiration for this rant. I wasted 5 hours on it before washing my hands of the damn thing. Integrated GPU system.

      1. 9

        Which model? I have an XPS 13 9360 and haven’t had any problems running stock Fedora on it. Also curious what the problem is, I’m currently looking to buy a second laptop. My pinebook pro isn’t really working quite well enough to be that yet, and the XPS 13 is currently the top contender.

        1. 4

          yikes.. any more info on this? i have an 8550 with win10 and I’ll be moving to Linux soon, but now is a good time to switch it if I’m gonna have issues

          1. 2

            I have the 9360 and an older 9343. I’ve had an ubuntu release on each since I received ’em. I think the 9343 had some functionality challenges out of the box, but there were BIOS fixes available to be applied by the time I purchased it.

            The integrated GPU is underpowered for games but otherwise I haven’t encountered any issues with it or anything else with the laptop, really.

            1. 3

              Currently I’m running a 9360 & used a 9343 at work before that. Both were / are perfectly well behaved.

              (I should have bought a 16Gb model though - 8Gb is a bit tight for dev work in the modern world sadly. Compiling anything involving LLVM is an exercise in patience.)

              1. 3

                8Gb is a bit tight for dev work in the modern world sadly. Compiling anything involving LLVM is an exercise in patience.)

                Yeah, amen! My 9360 has 8GB and I use it for Chrome and gnome-terminal so I can remotely access a desktop with 32GB. I think I tried building clang+llvm once on the XPS, but never again.

                1. 4

                  I think I tried building clang+llvm once on the XPS, but never again.

                  It’s doable but you have to radially reduce the parallelism of the build, otherwise it eats all your memory & goes into swap hell. As a result it takes quite a while.

          2. 3

            How much of this was an issue with the hardware itself vs. trying to use linux on it?

            1. 5

              I have no idea. I only tried Linux and that should always be enough.

              1. 6

                Exactly, especially for a so called “developer” edition. You can’t say it’s a developer machine and then ignore about half the target audience. In the nineties/early naughts, I expected to be marginalized as a Linux user, but nowadays it’s the Windows developers that are often regarded as quaint (assuming web or mobile development; game or desktop development is a whole different ballpark of course).

          3. 5

            I only run Linux so I can’t vouch for BSD/Plan 9/Haiku compatibility but I’ve been using higher-end Dell laptops for the past couple of years (from the Latitude and now Precision lines) and have been mostly happy with them. My only real complaints revolve around they keyboards but nearly all laptop keyboards are awful in their own way.

            Some Lobsters have a seething hatred for all things Intel for whatever reason but when I spec a laptop, I look for one with an Intel CPU, Intel GPU (I do very little gaming), and Intel wifi because whatever their other faults, Intel is awesome at writing and maintaining Linux kernel drivers.

            I would like to check out some of the newer Thinkpad models but Lenovo’s website is such garbage that I can’t even tell which models they still sell these days.

            1. 4

              Intel is awesome at writing and maintaining Linux kernel drivers.

              It is very good, but not perfect. My dell precision was having random freezes for several due to a faulty intel GPU driver. This is a well-known bug in the driver that has been going for months, without an available solution yet. I had to switch on my nvidia graphics (which I had never used on my lab-issued laptop) because the intel GPU was unusable.

              1. 4

                I’ve solved a lot of flickering and other weirdness on my old Toshiba laptop by uninstalling the intel driver and letting the system fall back to a generic modesetting driver. Debian and others have made this the default since.

            2. 2

              That’s weird. My XPS 13 has been a bastard of a thing to deal with. Different USB-C ports seem to have different capabilities each time it resumes (or is that reboots?) and I never know if my external display is going to appear as DP-1 or DP-2. Admittedly, I have (mostly) working suspend/resume, don’t use bluetooth and never buy dual-GPU laptops to avoid that rats’ nest of trouble.

              1. 3

                The 9360 only has one USB-C port, so I didn’t have this problem :)

            1. 4

              Agreed. I prefer desktops but having a nice 12” laptop for travel is great to have. Problem is there isn’t a worthy 12” in the market at the moment and hasn’t been for a long time.

              Question- Why not use a desktop?

              1. 5

                My daily driver is a desktop.

                1. 4

                  Depending on workload, an x201 in 2020 may still be viable, I use an x201 as my daily driver and honestly it’s perfect, the only downside is youtube, which eats processor and causes the fan to come on. But I mean the computer cost me £100, can’t exactly complain!

                  1. 2

                    For me personally, desktops just don’t offer enough advantages to outweigh the fact that they don’t really fit the way I work.

                    I’m hoping that the next computer I buy will be some descendant of the Pinebook; portability and battery beat power (almost) every time.

                  1. 70

                    He’s not wrong, but a quarter of a page about how almost all laptops are worse than 2008-era thinkpads is a little bit low-information for the lobste.rs frontpage isn’t it?

                    1. 14

                      I think you’re right, it’s low information. But, if others here feel the same as the article, I’m optimistic it will promote conversation on what can be used (I’m hoping I am wrong about what’s out there). I’d love an alternative to the Dell stack I currently use.

                      1. 11

                        It is a good justification for me to mention (once again) that there are a lot of barely-touched second hand 10 year old thinkpads on ebay for, like, less than $100. So, you can get like a dozen good laptops for the price of a bad one.

                      2. 19

                        Maybe it’s not the most “technical” post, but I think the point / heart of it is that there is a change that many would like to see in future laptops. Instead of razor thin laptops with horrible keyboards, many people want something that just works.

                        It’s an opinion piece, and I personally don’t see anything wrong with it being posted here (and the tags are correct).

                        1. 7

                          Sure. It’s just both an opinion piece & very short.

                          1. 23

                            The only reason it’s on the front page is because drew wrote it. There’s zero doubt about this.

                            This place loves to laud over him which annoys me specifically given I’ve seen him harass friends of mine because according to him, if you don’t use the “right software” (aka, software he believes is right), it makes you an absolute horrible person deserving of harrassment.

                            Wish I could filter out domains…

                            1. 7

                              Mostly, I wish there was more consistency. I post ranty things with more substance than this & get downvoted to hell (here & HN) – and my rants are rarely less than a five minute read. (On top of that, I stay away from ad hominems when arguing with strangers on the internet, which is not nothing.) If lobste.rs was rant-friendly I’d post more of them here.

                              Folks posting links to low-quality posts that are popular because people already agree with them is a much bigger part of the signal/noise ratio problem on lobste.rs than actual spammers, & because Drew is a name, he seems to get attention even when the same exact content in a comment would get one or two upvotes. (Like, this blog post is just a more severe version of comments I’ve literally made on this site.)

                            2. 7

                              I’d rather read this than yet another post about how Unix is so terrible because it’s based on plain text like that “Programmer’s critique of missing structure of operating systems” post, or some company talking about using AWS (literally nobody cares whether you use AWS), or yet another “I rewrote [unix standard command] in Rust” post.

                              1. 3

                                “Programmer’s critique of missing structure of operating systems” post,

                                There was a lot more in that post than you’re giving it credit for, the AWS thing can be interesting from a “This is how switching works/is motivated”. The Rust stuff is what it is.

                                This submission was basically a complaint about the state of consumer products without any relevant links or analysis/context (at least when I read it). It contains really inflammatory and frankly mean screeds like:

                                If you work at any of these companies and you’re proud of the garbage you’re shipping, then I’m disappointed in you.

                                On a factual basis, it’s a lot easier to get a multicore laptop right now with a better camera than it was 12 years ago. It is a lot easier to get one with USB 3. It is a lot easier to get one with a high-resolution screen. It is a lot easier to get one with a good graphics card. Drew is happy to ignore all of that progress because he finds them difficult to service and difficult to run Free as in Freedom software on, and so immediately condemns everything else as garbage.

                                I’d rather read somebody talking about their attempt at making wordcount in Rust than such a shallow and negative and just obviously narrow-minded outburst.

                                1. 1

                                  Fair enough. I like that first category so long as it says something new (which it often does), but the other two are at least as spammy & low-effort as this.

                            3. 19

                              It’s correctly tagged as rant, and @cfenollosa can’t help that Drew is the darling child of lobsters who can say no wrong.

                              EDIT (clarification, in case you don’t see the below): this is criticism of Lobsters, and defense of this blog post being posted, not criticism of any individuals.

                              1. 52

                                In reality, there also seem to be a fair number of people on Lobsters who think I can say no right. I don’t post my own writing to Lobsters anymore because of issues ranging from valid concerns about spam to obnoxious toxicity.

                                My take is, I write for myself, not for Lobsters. When a Lobster seems to think something I’ve written is worth discussing, then I leave it up to the community to vote on as they see fit. I’ll usually drop into the threads to answer questions, but I find it difficult to engage with Lobsters beyond this - and even that much is difficult. Because you know who I am and what I work on, it seems like any topic I weigh in on is interpreted as spam, be it via submissions or commenting in discussions, even if I go out of my way not to frame my comments in the context of anything I work on.

                                I don’t really like this community anymore, but I reckon some Lobsters would argue that’s by design.

                                1. 13

                                  obnoxious toxicity

                                  I’m not going to claim that Lobsters is perfect, but “obnoxious toxicity” doesn’t seem like an apt description either; at least not in my experience. How is it toxic, specifically?

                                  1. 28

                                    Obviously tone is mostly subjective, but I have noticed the same thing recently on some of Drew’s writings that get posted here. Even the comment by @mz could be seen as really negative

                                    @cfenollosa can’t help that Drew is the darling child of lobsters who can say no wrong

                                    I have a hard time not reading that with vitriol or almost malice (though as indicated by @mz this was supposed to be sarcastic). Saying, “hey I don’t think there isn’t a ton of technical information here” and not upvoting is one thing, but when Drew clearly didn’t post the article themselves and still gets comments that to me seem very unpleasant. It really would be hard to like a community.

                                    EDIT: Again in this same thread, I’m just going to start collecting the comments that if I were reading about me that I’d consider to be disheartening and drive me away from this community.

                                    I’ve seen much worse Drew Dewalt rants keep the top spot on lobste.rs for weeks tbh.

                                    1. 22

                                      Ya, that comment is clearly less-than-perfect, but I also wouldn’t describe it as “obnoxious toxicity” personally, but YMMV.

                                      It’s not like Drew’s communication style is always … gentle. I mean, in this post he’s literally telling people they should die. Obviously I know it’s supposed to be a joke and intended to be hyperbolic, but sjeez man; that’s pretty harsh… Reasonably sure such a comment would be downvoted to hell on Lobsters or HN.

                                      If you dish out harsh words, then don’t be surprised if people reply with harsh words, too.

                                      1. 4

                                        This is definitely a case of a missing /s or otherwise tone being lost in text form. Please see my clarification and comment follow-up.

                                        1. 5

                                          I really don’t mean this as a cut to you either, it is just how I read it tonally. Text communication is hard.

                                      2. 9

                                        In my experience the toxicity has various specific loci, and if you happen to avoid those in your interactions with the community then you’re fine.

                                        Woe betide you however it you step on a hot button.

                                        Still love the place. I donned my asbestos underwear for the first time long before at least some of you were born :)

                                      3. 5

                                        I absolutely know what you’re saying, and I commend you for writing for yourself rather than for clout or for internet points. I think a lot of the blog spam we see on lobsters is stuff trying to be topical enough to be unoffensive, but content-less enough to be spam.

                                        I in no way meant this as a criticism of you @ddevault. I meant it as a criticism of lobsters, a community which seems to be very addicted to being polarized over your content.

                                        I totally get why you don’t post your stuff here. In general, a blog post not being appropriate for lobsters is not a criticism of the blog post (to think otherwise is some serious tech elitism).

                                        Thanks for engaging with the comment. It seems you interpreted it the way I intended it, as I had no intention of offense.

                                        1. 5

                                          Point of fact: you took advantage of us as a marketing channel.

                                          1. 44

                                            Here is my self-evaluation of my posting history. Green = unambiguously not spam. Blue = cool stuff I made, but which doesn’t make me money. Together these make up almost half of my posts. Pink = dubious, I could benefit indirectly from these, but would argue that they’re on topic and relevant. That covers all but 3 of my posts. Of the remainder, which are definitely more ads than not, one is the most highly upvoted story of all time in each of its tags.

                                            I think that this balance was reasonable enough, but in the BSD submission there was some discussion about whether or not it was too spammy, so I curbed my posting on Lobsters. The vast majority of submissions from my blog are not submitted by me, but it seems like I’m being held accountable for all of them as spam.

                                            In any case, I don’t post my stuff here anymore. Take it up with the rest of the community if you don’t like seeing my posts here.

                                            1. 31

                                              I’m sad and sorry that the community has responded this way to your posts, especially as you have stopped linking them yourself. You can’t control what other people do, yet you bear the brunt of their unhappiness.

                                              I like seeing your posts here, as there is often good discussion that is prompted by them. Even if some think of them as “ads”, at least it’s for good, open source software (and not things much worse like Google and Facebook).

                                              I’m glad you’re writing for yourself- I (and others) will continue to enjoy reading what you write.

                                            2. 7

                                              Drew’s content may not always be earth-shattering, but a typical post of his likely opens my eyes as anything else on the site. Lobsters is not a place that discourages people from posting their own content.

                                              ☑️ I am the author of the story at this URL (or this text)

                                              1. 7

                                                The link doesn’t seem to prove the “took advantage of” part, or am I missing something?

                                            3. -1

                                              Fair enough! And yes, I’ve seen much worse Drew Dewalt rants keep the top spot on lobste.rs for weeks tbh. I guess I can’t argue with success.

                                            4. 3

                                              Yes, it could have been contained in a comment. But maybe that’s why the article gets upvoted a lot: It’s low-effort to read, everyone basically agrees, and it’s a relief that apparently everyone feels the same (and it’s not their fault for not being able to find a proper laptop).

                                              1. 0

                                                How’s it a quarter of a page? It’s an entire page on my fairly high-resolution screen (1440p with no scaling). How zoomed out is your web browser?

                                                1. 1

                                                  I have a portrait display & my zoom setting is only 100% – basically because I don’t want a ‘screenful’ to be thirty seconds of reading.

                                              1. 9

                                                isn’t it a bit unfair to compare pine64 with the other ones?

                                                they at least try to be better :) https://forum.pine64.org/showthread.php?tid=8884&pid=57753#pid57753

                                                1. 11

                                                  I’m very happy with Pine64 and I think they’re trying to be better. But, the fact remains: they’re not there yet.

                                                  1. 4

                                                    Mostly as a factor of ARM devices being horrible booters. I wonder how RISC-V intends to boot, or if the walled bioses of X86 really are our best.

                                                    1. 8

                                                      The RISC-V boot process varies but generally goes like

                                                      1. On-chip ROM reads next stage from flash (or sets up XIP)
                                                      2. OpenSBI (or BBL) switches to S-Mode and jumps to the next stage (usually loaded by the previous stage along with OpenSBI)
                                                      3. This can be Linux itself, but is usually a bootloader like U-Boot or Coreboot. We may still be running with XIP, but after this everything runs from ram.
                                                      4. OS proper

                                                      The SBI handles some functionality which can only be done from M-mode. This lets the OS run in S-mode and not care about if it’s being virtualized or not. Steps 2 and 3 can also be swapped, depending on the platform.

                                                      1. 3

                                                        If core boot can be a single binary that can be distributed to non homogeneous fleets, then it seems to be as convenient as x86

                                                        1. 4

                                                          If coreboot uses the SBI as a baseline that could be possible. However, the current plan has it running in M-mode, so you would likely need a different coreboot for each board.

                                                1. 12

                                                  Screenshots would be helpful :P

                                                  1. 17

                                                    This project is being completed under a consulting contract with Migadu for their next-generation webmail. They’re also working on a theme:

                                                    https://sr.ht/mcvO.png https://sr.ht/ml1l.png https://sr.ht/Yk6A.png

                                                    If it’s not obvious yet from this, Amolith, and geocar’s comments: Koushin is themeable :)

                                                    1. 3

                                                      Thanks, that looks a lot more promising than the other 2 posts.

                                                      I don’t have high standards and it doesn’t have to be beautiful, but unthemed without even half an hour of css work is a bit too little effort.

                                                      1. 2

                                                        It looks really good! I like it. It reminds me of gmail, back in 2005, before it turned into a bloated piece of shit.

                                                        1. 1

                                                          I’m a bit confused.

                                                          For example is using $(()) which is a bashism, I’d assume

                                                          (almost) entirely written in POSIX shell

                                                          means using some external tools (like od), not relying on non-POSIX features.

                                                          1. 4
                                                            1. 1

                                                              You are right.

                                                              Thanks for the correction. I misread a table, clear sign I never used arithmetic expressions before.

                                                          1. 3

                                                            I’m always perplexed why people would leave GitHub for GitLab because of the Microsoft aquisition.

                                                            GitHub has many useful features (issue tracking, pull requests, integrated builds), so unless Microsoft began to blatantly embrace, extend, and extinguish, I see no reason to leave the platform just because a company that was anti-open source some 10 years ago bought it.

                                                            1. 6

                                                              It’s not all about the features for many people. Ideological alignment is important, too. Knowing your the customer, not the product, is also important. And even if you like the features - the purchase is a good impetus to take a new look at the competition, and maybe discover new features that GitHub’s behind the curve on.

                                                              1. 1

                                                                You mean vendor lock in?

                                                                I’ve been thinking about that for the past few days, but I probably won’t switch to sourcehut any time soon because I can’t find a good email provider (protonmail doesn’t support other clients without a bridge). I might try GitLab soon though.

                                                                1. 2

                                                                  I usually recommend Migadu for email.

                                                                  Full disclosure: sourcehut has an ongoing consulting arrangement with them, but I have advocated for Migadu since long prior.

                                                              2. 3

                                                                Just because someone did something repeatedly in the past, no reason to suspect they might do it again right?

                                                                1. 1

                                                                  If we were talking about an individual I’d agree with you. But a corporation doesn’t necessarily work like that.

                                                                  1. 1

                                                                    A corporation is generally less trustworthy than an individual, as it can be sold to anyone at any time, and is generally guided by whatever is believed to be most profitable, irrespective of current or past behaviour. Should you expect them to revert to their previous attitude? Maybe not. Can you trust them to remain aligned to your interests and needs? Absolutely not.

                                                                  2. 1

                                                                    Considering the change in Microsoft’s leadership, I don’t think they would.

                                                                  3. 2

                                                                    GitHub has many useful features (issue tracking, pull requests, integrated builds)

                                                                    These extras are precisely why it’s risky to stay on a platform where you don’t trust the owner. A git repo is fairly portable: you can easily transfer it to a different host, or even self host it. GitHub issues and pull requests aren’t portable: The more your project uses GitHub issues, pull requests and CI, the more painful it becomes to move to another platform in the event that GitHub starts serving malware (as SourceForge did) or charging for certain features (as many businesses with a free introductory tier do once users are locked into their platform).

                                                                    Even when GitHub wasn’t owned by Microsoft, there were risks to be assessed, and a balance to be sought between present convenience and future sustainability. For many, Microsoft’s past bad behaviour increases the risk enough that it is worth at least mirroring projects elsewhere, and possibly leaving GitHub altogether.

                                                                  1. 14

                                                                    Panfrost is a really impressive and much needed driver and a valuable part of Mesa, and Alyssa’s work on it has been superb. This group solved one of the biggest problems between ARM and free software, and the results speak for themselves.

                                                                      1. 4

                                                                        Very interesting read, many thanks to the author for writing it. The part about how his reputation has changed how he can contribute to other projects was unexpected and eye-opening. One thing I’m curious to know is how his busy-ness relates to his income. From what I can tell, he is currently community-funded via Patreon? If so, would it be possible for him to pare down, or abandon, some of his projects if he wanted to focus on his personal life, without affecting his livelihood.

                                                                        Personally, I think it would be great to be able to write FOSS full-time, and I would maybe be willing to take a pay cut to do it, but I would want to know that I could have a balanced personal life as well.

                                                                        1. 5

                                                                          Author here. Thanks for sharing your kind words of appreciation.

                                                                          Regarding my livelihood, I make additional income from sourcehut, which publishes financial reports you’re welcome to read, and through consulting. I plan my finances meticulously, live well within my means, and I’m far from being in dire financial straits. I’ve been poverse before, and the memory of what that’s like still weighs heavily on my heart - so I plan things very carefully and leave myself generous margins for error.

                                                                        1. 52

                                                                          nine-to-five-with-kids types

                                                                          God forbid people have children and actually want to spend time with them.

                                                                          As someone with kids, I’m reasonably sure I’m pretty good at what I do.

                                                                          1. 40

                                                                            The second statements includes a retraction of sorts:

                                                                            P.S. I have no problem with family people, and want to retract the offhand comment I made about them. I work with many awesome colleagues who happen to have children at home. What I really meant to say is that I don’t like people who see what we do as more of a job than a passion, and it feels like we have a lot of these people these days. Maybe everyone does, though, or maybe I’m just completely wrong.

                                                                            I disagree with them that it’s wrong to see your work as a job rather than a passion; in a system where it’s expected for everyone to have a job, it’s bullshit (though very beneficial for the capitalist class) to expect everyone to be passionate about what they’re forced to do everyday. Nonetheless, the retraction in the second statement is important context.

                                                                            1. 14

                                                                              Owning stock was supposed to fix this, rewarding employees for better work towards company growth. However the employees are somehow not oblivious to the fact that their work does not affect the stock price whatsoever, instead it being dependent mostly on news cycle about the company and their C-level execs (and large defense contracts in the case of Microsoft).

                                                                              1. 7

                                                                                Your appeal to our capitalist overlords here is incorrect. It’s a symptom of capitalism that you have people at work who are just punching in and don’t have a passion for it. If we weren’t bound by a capitalist system, people would work on whatever they’re passionate about instead of worrying about bringing home the bacon to buy more bacon with. Wanting people to be passionate about their work is a decidedly anti-capitalist sentiment, and wanting people to clock in, do their job, clock out and go home to maintain their nuclear family is a pretty capitalist idea.

                                                                                1. 2

                                                                                  I agree with you.

                                                                                  in a system where it’s expected for everyone to have a job, it’s bullshit (though very beneficial for the capitalist class) to expect everyone to be passionate about what they’re forced to do everyday.

                                                                                  That’s true I think, undoubtedly. You can “fix” it by accepting people who aren’t passionate, or by replacing the system where people need a job to survive. I’d definitely prefer to fix it by replacing the system, but in any case, blaming 9-to-5-ers is wrong - they’re victims more than anything, forced to work a job they’re not passionate about instead of going out on whatever unprofitable venture captures their heart.

                                                                                  1. 1

                                                                                    or by replacing the system where people need a job to survive

                                                                                    How do you propose we do that, without literally enslaving people? And without causing necessary but often necessarily low-passion jobs (like sorting freight in a courier company, at 1AM - one of my jobs as teenager) to simply not get done?

                                                                                    I mean, what you’re really proposing is that some people get to go on ‘whatever unprofitable venture captures their heart’ while … what? Someone else labours to pay their way?

                                                                                    I personally think that it’s entirely reasonable to accept people whose passion isn’t their job, provided they’re professional and productive. There’s nothing here to fix, really, beyond rejecting the (IMO) unrealistic idea that a good employee has to be passionate about their work.

                                                                                  2. 1

                                                                                    Every time we’ve tried an alternative at scale it’s led to mass murder, or mass starvation, or both.

                                                                                    You’re also ignoring the fact that some jobs are just plain unpleasant. I’ve worked some in my youth. It’s reasonable not to be passionate about those; they’re either high paying because they’re unpleasant but skilled, or the lowest paying job because they’re work you do because your labour isn’t valuable in any other way.

                                                                                2. 9

                                                                                  As another kind of beef against that line, in my experience, people working crazy long hours on a consistent basis generally aren’t doing it out of some kind of pure passion for the project, implementing a cool amazing design, etc. They’re usually pounding out ordinary boring features and business requirements, and are working long hours because of poor project management - unrealistic promises to customers, insufficient resources, having no idea what the requirements actually are but promising a delivery date anyways, etc. IMO, anyone with any wisdom should get out of that situation, whether or not they have kids.

                                                                                  Also IME, those who do have genuine passion to build some cool new thing often don’t work long hours at all, or only do so for a short time period.

                                                                                  1. 7

                                                                                    Someone I know could easily be working for Google or such, but he works for a smaller, non-tech, local company as a developer, so he can spend his free time building the projects he likes (or the luxury of not having to at all), instead of some absurd scale megacorp thing which will drain all the free time he has. (Or specifically, build the great thing you want to, and not be subject to what say, Facebook wants.) IMHO, it’s pretty smart…

                                                                                  2. 7

                                                                                    The anonymous author apologizes about this specifically in their follow-up.

                                                                                    1. 4

                                                                                      The apologies sound like something between a sincere apology for just blindly ranting at it all and an apology for apologies’ sake. It looks to me like the author actually feels (felt) closer to the way he described in the first message then the second. Not all the way, but somewhat.

                                                                                      1. 1

                                                                                        The apology is also written in an entirely different style. It’s not clear there’s strong reason to assume it’s the same person.

                                                                                        edit: reading through the comments, the blog’s owner says: “I won’t reveal how the anonymous poster contacted me, but I am 99.9% sure that unless his communication channel was hacked, he is the same person.” So, ok, it’s the same person.

                                                                                    2. 11

                                                                                      I know what people the author means though–not all these people have kids though, they’re just the ones who punch a clock from 9-5 and want to remain comfortable and don’t want push themselves to make great things. It’s not that these people want to make bad things, or don’t want to make great things, or aren’t capable of doing great work, they just don’t receive any self-fulfillment from work. I don’t blame these people for acting this way, I’d just rather not work with them.

                                                                                      The best dev on my team (by a long shot) is a 9-5 worker who never finished college, but accomplishes more in one day than most devs accomplish in a week or more. He aspires to do incredible work during his time and he exceeds every day.

                                                                                      1. 9

                                                                                        Once organizations get to a certain threshold of complexity, though, you need to be thinking much more about the incentives that are offered rather than the micro-level of which people want to do excellent things. You have to make it easier to do the excellent thing than not, and that can be difficult, or, in an org the size of Microsoft’s Windows kernel groups, basically, impossible.

                                                                                        1. 5

                                                                                          The comment was directed at the author’s “9-5 with kids” comment since the author is referring to the contributions of those people. Organizational structure produces different effects on products based on the personalities and abilities of those people within the organization.

                                                                                          In general, the bigger the organizational, the less risk-tolerance, but most actions incur some sort of risk. Also, staying with the crowd and doing what you’re told is safest, so inaction or repeating previous actions often becomes the default.

                                                                                          1. 2

                                                                                            Good point. It never makes sense to innovate and take risks in a large organization. If your gamble pays off, the pay off will invisibly pump up some vaguely connected numbers that will look good for some manager higher up. If it doesn’t pay off, it’s your fault. So, regardless of how good a gamble it was based on risk-reward ratio, it’s never favorable for you personally. Don’t expect your immediate superior to appreciate it either, even if the gamble pays off, because that person is also facing the same odds.

                                                                                        2. 7

                                                                                          As an anecdote, the eventually worst workplace I’ve been had a deeply ingrained culture of being passionate about work, and about being part of “a family” through the employer. It’s not entirely a healthy stance.

                                                                                          1. 4

                                                                                            I don’t blame these people for acting this way, I’d just rather not work with them.

                                                                                            Why not?

                                                                                            You said they want to make good things, great things, and can do great work. Why does someone else’s opinion of their job matter to you?

                                                                                            1. 5

                                                                                              It’s when people’s opinions affect the work they do. My point is they can do great work, but many don’t. Sometimes it manifests as symptom fixing rather than problem solving because it’s “whatever gets me out today by X, screw consequences later.” Other times it manifests as “I wasn’t told to do this, so I won’t”. When I’ve seen it, it’s often pervasive short term thinking over long term thinking.

                                                                                        1. 59

                                                                                          Update: been working on a better approach to these problems that leave affected users feeling less put-out. I’ll be starting with a better email template in the future:

                                                                                          https://paste.sr.ht/~sircmpwn/3d32eb7bbc564170c3d30f041e5e8dc71aa5a1c6

                                                                                          In the future I’ll be working on better automated soft limits, so that users aren’t surprised by this.

                                                                                          @sjl: after thinking it over more, I was unprofessional and sarcastic with you. I apologise.

                                                                                          1. 41

                                                                                            I think it would be beneficial for you to take on the mindset that your users’ use cases are always valid, by definition, as a premise. Whether or not your service can handle their use cases, maybe not, but this idea that you know better what your users should be doing is not going to accomplish your goals.

                                                                                            As another example, I happen to need 2-3 more GiB RAM than the sr.ht freebsd build services offers at the moment, and have offered to up my monthly donation to account for the resource usage, and you’ve turned me down, on the grounds that I’m fundamentally abusing computer hardware in some moral way. As a result, Zig freebsd builds have many of the tests disabled, the ones where the bootstrapping compiler is a bit memory hungry. Zig’s FreeBSD users suffer because of this. And as a result, when someone else offers me a different FreeBSD CI service with more RAM, I’m ready to accept it, because my use case is valid.

                                                                                            1. 6

                                                                                              Could linking with the boehm conservative gc work as a stop gap? I think it won’t require any code changes.

                                                                                              1. 4

                                                                                                Something Andrew doesn’t mention here is why he needs 2-3 GiB more RAM: because, by design, his compiler never frees memory. Nearly all of that RAM is dead memory. In order to accomodate this use-case, I’d have to provision dedicated hardware just for Zig. Sometimes, use-cases are wrong, and you need to correct the problem at the source. Just because someone is willing to throw unspecified sums of money at you to get their “use-case” dealt with doesn’t mean it’s worth dealing with. I have finite time and resources and maybe I feel like my time is better spent implementing features which are on-topic for everyone else, even at the expense of losing some user with more money than sense.

                                                                                                1. 44

                                                                                                  even at the expense of losing some user with more money than sense.

                                                                                                  I really hope you change your tune here. Insulting users is pretty much the worst thing you could do.

                                                                                                  Another thread recently talked about the fact that compilers don’t free memory, because the goal of a compiler is to be as fast as possible, so they treat the heap as an arena that the OS frees. Compilers have done this for 50+ years—zig isn’t special here.

                                                                                                  1. 4

                                                                                                    I didn’t mean to imply that Andrew doesn’t have sense, but that the hypothetical customer-thats-always-right might not.

                                                                                                    As for compilers never freeing to be fast, utter bollocks. So the compiler should OOM if I have <8G of RAM to spare? Absolutely nuts. Freeing memory is not a performance bottleneck.

                                                                                                    1. 38

                                                                                                      Your reasoning is sound, your wording and phrasing choices are not. In what I’ve read you don’t come off as witty when you’re dealing with a paying customer and telling them they can’t do something which I also think is unreasonable, you come off as a dick. That’s how it appears. I don’t have any problems with you or your services and I think you working on this stuff is awesome… but I wouldn’t pay for insults in addition to whatever else you might provide.

                                                                                                      1. 5

                                                                                                        As long as I’ve known him Drew has pretty consistently been like this. It’s not a bad thing. It’s quite refreshing actually.

                                                                                                        1. 35

                                                                                                          It’s refreshing to have a business make fun of you?

                                                                                                          1. 9

                                                                                                            It’s quite refreshing to see someone willing to say ‘no you’re wrong’ instead of the typical corporate ‘the customer is always right’ bullshit so many people here have obviously come to expect.

                                                                                                            Sometimes the customer is wrong.

                                                                                                            1. 34

                                                                                                              It’s OK for both people to be right, and the customer to stop paying for the service and walk away. It’s then OK for the customer to go tell people about how they were treated. Hopefully that happens more.

                                                                                                          2. 24

                                                                                                            As a former moderator in prior communities, I politely disagree. Folks that are never not toxic are a serious liability and require special effort to handle well. I recall one memorable day when Drew dared me to ban him; I should have let the emotions flow through me and removed him from the community.

                                                                                                            Also, as a former business owner, I politely disagree that this is good business practice.

                                                                                                            1. 16

                                                                                                              I agree it’s good, now I know to avoid this business!

                                                                                                      2. 14

                                                                                                        CPU speed vs memory usage is a fundamental resource tradeoff that occurs all the time in computing. Just because you disagree with where on the spectrum someone has chosen to aim their design doesn’t mean they’re stupid. Especially when they too are a mostly-one-person project operating on limited resources.

                                                                                                        It’s PERFECTLY VALID to say “I don’t have time to accommodate this one special case, sorry”. It is NOT perfectly valid to say “you are stupid for needing this special case, go away”. Money vs. person-time is another fundamental resource tradeoff where different people have different priorities.

                                                                                                        1. 21

                                                                                                          Regardless of the use case, I’d really rather not have my SCM platform making discretionary decisions about what I’m working on. The users aren’t paying for you to audit them, they’re paying for the services provided by the software. If you want your service to come with the exemption that you get to unilaterally decide whose content is allowed and whose content isn’t allowed, you’re free to do that. Just expect the community to nearly unanimously respond with “we’ll go elsewhere”

                                                                                                          1. 7

                                                                                                            He’s not making ‘discretionary decisions about what [you’re] working on’. I don’t see Drew saying ‘you can’t use this service because I don’t like the way your compiler is designed’. He’s saying ‘provisioning dedicated hardware for specific projects is a lot of time and effort that I don’t have, so I’d need to have a really really good reason to do it, no matter how much money you’re willing to throw at me, and you haven’t given me one’.

                                                                                                            Every service out there gets to decide what is allowed and what isn’t. Look at the terms of service of any file or repository hosting service anywhere. GitHub, GitLab, Bitbucket, imgur, pastebin services… ALL of them make it clear in their terms of service that it’s entirely up to their whim whether they want to host your files or not.

                                                                                                            1. 32

                                                                                                              Drew is literally commenting on a particular users project, and how its design is a problem, so I have no idea what you’re talking about:

                                                                                                              Something Andrew doesn’t mention here is why he needs 2-3 GiB more RAM: because, by design, his compiler never frees memory. Nearly all of that RAM is dead memory.

                                                                                                              As for compilers never freeing to be fast, utter bollocks.

                                                                                                              @andrewrk can hopefully clarify, but I thought his offer to up monthly donations was to improve sr.ht’s FreeBSD offering, in general, not necessarily to only improve Zig builds (Zig builds would improve as a byproduct of improving the FreeBSD infrastructure). If the donations were only to be used to improve Zig-specific experiences, then I understand the argument that Drew doesn’t want to commit to that.

                                                                                                          2. 12

                                                                                                            It just seems weird to me that one of your criteria for whether or not to give a customer resources is based on a personal audit of their code. Are you going to do this for every customer?

                                                                                                            1. 3

                                                                                                              I completly understand the concern here, and take it very seriously. I usually don’t dig into the repo at all and just reach out to the user to clarify its purpose. In this case, though, the repo was someone’s personal website, and named as such, and connecting the dots did not require much.

                                                                                                              1. 2

                                                                                                                As explained downthread, it’s “Alert fires -> look for what’s caused the alert -> contact customer whose repo tripped the alert”.

                                                                                                            2. -2

                                                                                                              ‘The customer is always right’ is nonsense.

                                                                                                              1. 10

                                                                                                                Nobody’s suggesting otherwise.

                                                                                                                1. -8

                                                                                                                  Literally everyone else in this thread is acting like the customer is always right.

                                                                                                            3. 18

                                                                                                              You handled this very professionally and courteously, I plan to continue to use sh for many happy years to come.

                                                                                                              1. 6

                                                                                                                You are under no obligation to explain or justify what your business model is to anyone, or on a personal level what self sustainability, your own peace of mind, well being or definition of meaningful sustainable work is.

                                                                                                                There is a particular mode of doing business these days which people inside that paradigm often do not understand that they are inside and therefore apply force to get others to conform.

                                                                                                                You’re breaking old paradigms and inventing new ways of running organisations and that is brave, ground breaking and commendable and politically powerful.

                                                                                                                I hope issues like this does not deter you one bit from blazing your own trail through the fucked up world that is tech organisations in late stage capitalism and I hope you share as much as you can about how you’re doing personally and in ‘business’.

                                                                                                                1. 2

                                                                                                                  git-lfs implementations often don’t allow to reclaim unreachable blobs: once you push a binary blob, even on a branch that you deleted, it will take some space forever.

                                                                                                                  Maybe it is worth investigating git-annex while you’re on this topic.

                                                                                                                  1. 5

                                                                                                                    Yeah, git annex is also something I intend to study. I’m only just setting up large file servers for blob storage, figuring out how to apply them is the next step.

                                                                                                                1. 10

                                                                                                                  @ddevault Would it be possible to get a clear “Terms of Service” clarifying these sorts of use cases? 1.1 Gb seems like an excessive file size, but having a crystal clear & mutually agreed upon set of rules for platform use is essential for trust (more so for a paid service), and right now users don’t know what does and does not constitute as a reasonable use of the service .

                                                                                                                  1. 37

                                                                                                                    No, they’re intentionally vague so that we can exercise discretion. There are some large repositories which we overlook, such as Linux trees, pkgsrc, nixpkgs, even mozbase is overlooked despite being huge and expensive to host.

                                                                                                                    In this guy’s case, he had uploaded gigabytes of high-resolution personal photos (>1.1 Gb - it takes up more space and CPU time on our server than on your workstation because we generate clonebundles for large repos). It was the second largest repository on all of SourceHut. SourceHut is a code forge, not Instagram.

                                                                                                                    1. 40

                                                                                                                      No, they’re intentionally vague so that we can exercise discretion.

                                                                                                                      I like to call this “mystery meat TOS”. You never know what you’ll get until you take a bite!

                                                                                                                      1. 24

                                                                                                                        I mean, honestly, a small fraction of our users hit problems. I’ve had to talk to <10 people, and this guy is the only one who felt slighted. It’s an alpha-quality service, maybe it’ll be easier to publish objective limits once things settle down and the limitations are well defined. On the whole, I think more users benefit from having a human being making judgement calls in the process than not, because usually we err on the side of letting things slide.

                                                                                                                        Generally we also are less strict on paid accounts, but the conversation with this guy got hostile quick so there wasn’t really an opportunity to exercise discretion in his case.

                                                                                                                        1. 30

                                                                                                                          the conversation with this guy got hostile quick

                                                                                                                          Here’s the conversation, for folks who want to know what “the conversation got hostile” means to Source Hut: https://paste.stevelosh.com/18ddf23cb15679ac1ddca458b4f26c48b6a53f11

                                                                                                                          1. 32

                                                                                                                            i’m not a native speaker, but have the feeling that you got defensive quickly:

                                                                                                                            Okay. I guess I assumed a single 1.1 gigabyte repository wouldn’t be an unreasonable use of a $100/year service. I certainly didn’t see any mention of a ban on large binary files during the sign up or billing process, but I admit I may have missed it. I’ve deleted the repository. Feel free to delete any backups you’ve made of it to reclaim the space, I’ve backed it up myself.

                                                                                                                            it’s a pay-what-you-like alpha service, not backed by venture capital. you got a rather friendly mail, noticing you that you please shouldn’t put large files into hg, not requesting that you delete it immediately.

                                                                                                                            ddevaults reply was explaining the reasoning, not knowing that you are a mercurial contributor:

                                                                                                                            Hg was not designed to store large blobs, and it puts an unreasonable strain on our servers that most users don’t burden us with. I’m sorry, but hg is not suitable for large blobs. Neither is git. It’s just not the right place to put these kinds of files.

                                                                                                                            i’m not sure i’d label this as condescending. again I’m no native speaker, so maybe i’m missing nuances.

                                                                                                                            after that you’ve cancelled your account.

                                                                                                                            1. 13

                                                                                                                              As a native speaker, your analysis aligns with how I interpreted it.

                                                                                                                              1. 9

                                                                                                                                Native speaker here, I actually felt the conversation was fairly polite right up until the very end (Steve’s last message).

                                                                                                                              2. 28

                                                                                                                                On the whole, I think more users benefit from having a human being making judgement calls in the process than not, because usually we err on the side of letting things slide.

                                                                                                                                Judgement calls are great if you have a documented soft limit (X GB max repo size / Y MB max inner repo file size) and say “contact me about limit increases”. Your customers can decide ahead of time if they will meet the criteria, and you get the wiggle room you are interested in.

                                                                                                                                Judgement calls suck if they allow users to successfully use your platform until you decide it isn’t proper/valid.

                                                                                                                                1. 12

                                                                                                                                  That’s a fair compromise, and I’ll eventually have something like this. But it’s important to remember that SourceHut is an alpha service. I don’t think these kinds of details are a reasonable expectation to place on the service at this point. Right now we just have to monitor things and try to preempt any issues that come up. This informal process also helps to identify good limits for formalizing later. But, even then, it’ll still be important that we have an escape hatch to deal with outliers - the following is already in our terms of use:

                                                                                                                                  You must not deliberately use the services for the purpose of:

                                                                                                                                  • impacting service availability for other users

                                                                                                                                  It’s important that we make sure that any single user isn’t affecting service availability for everyone else.

                                                                                                                                  Edit: did a brief survey of competitor’s terms of service. They’re all equally vague, presumably for the same reasons

                                                                                                                                  GitHub:

                                                                                                                                  [under no circumstances will you] use our servers for any form of excessive automated bulk activity (for example, spamming or cryptocurrency mining), to place undue burden on our servers through automated means, or to relay any form of unsolicited advertising or solicitation through our servers, such as get-rich-quick schemes;

                                                                                                                                  The Service’s bandwidth limitations vary based on the features you use. If we determine your bandwidth usage to be significantly excessive in relation to other users of similar features, we reserve the right to suspend your Account, throttle your file hosting, or otherwise limit your activity until you can reduce your bandwidth consumption

                                                                                                                                  GitLab:

                                                                                                                                  [you agree not to use] your account in a way that is harmful to others [such as] taxing resources with activities such as cryptocurrency mining.

                                                                                                                                  At best they give examples, but always leave it open-ended. It would be irresponsible not to.

                                                                                                                                  1. 17

                                                                                                                                    The terms of service pages don’t mention the limits, but the limits are documented elsewhere.

                                                                                                                                    GitHub:

                                                                                                                                    We recommend repositories be kept under 1GB each. Repositories have a hard limit of 100GB. If you reach 75GB you’ll receive a warning from Git in your terminal when you push. This limit is easy to stay within if large files are kept out of the repository. If your repository exceeds 1GB, you might receive a polite email from GitHub Support requesting that you reduce the size of the repository to bring it back down.

                                                                                                                                    In addition, we place a strict limit of files exceeding 100 MB in size. For more information, see “Working with large files.”

                                                                                                                                    GitLab (unfortunately all I can find is a blog post):

                                                                                                                                    we’ve permanently raised our storage limit per repository on GitLab.com from 5GB to 10GB

                                                                                                                                    Bitbucket:

                                                                                                                                    The repository size limit is 2GB for all plans, Free, Standard, or Premium.

                                                                                                                                    1. 8

                                                                                                                                      I see. This would be a nice model for a future SourceHut to implement, but it requries engineering effort and prioritization like everything else. Right now the procedure is:

                                                                                                                                      1. High disk use alarm goes off
                                                                                                                                      2. Manually do an audit for large repos
                                                                                                                                      3. Send emails to their owners if they seem to qualify as excessive use

                                                                                                                                      Then discuss the matter with each affected user. If there are no repos which constitute excessive use, then more hardware is provisioned.

                                                                                                                                      1. 11

                                                                                                                                        Maybe this is something you should put on your TOS/FAQ somewhere.

                                                                                                                                    2. 8

                                                                                                                                      This informal process also helps to identify good limits for formalizing later.

                                                                                                                                      Sounds like you have some already:

                                                                                                                                      • Gigabyte-scale repos get special attention
                                                                                                                                      • Giant collections of source code, such as personal forks of large projects (Linux source, nix pkgtree) are usually okay
                                                                                                                                      • Giant collections of non-source-code are usually not okay, especially binary/media files
                                                                                                                                      • These guidelines are subject to judgement calls
                                                                                                                                      • These guidelines may be changed or refined in the future

                                                                                                                                      All you have to do is say this, then next time someone tries to do this (because there WILL be a next time) you can just point at the docs instead of having to take the time to explain the policy. That’s what the terms of service is for.

                                                                                                                                  2. 8

                                                                                                                                    Regardless of what this specific user was trying to do, I would exercise caution. There are valid use cases for large files in a code repository. For example: Game development, where you might have large textures, audio files, or 3D models. Or a repository for a static website that contains high-res images, audio, and perhaps video. The use of things like git-lfs as a way to solve these problems is common but not universal.

                                                                                                                                    To say something like, “SourceHut is a code forge, not Instagram” is to pretend these use cases are invalid, or don’t exist, or that they’re not “code”, or something.

                                                                                                                                    I’ve personally used competing services like GitHub for both the examples above and this whole discussion has completely put me off ever using Sourcehut despite my preference for Mercurial over Git.

                                                                                                                                    1. 3

                                                                                                                                      I agree that some use-cases like that are valid, but they require special consideration and engineering work that hg.sr.ht hasn’t received yet (namely largefiles, and in git’s case annex or git-lfs). For an alpha-quality service, sometimes we just can’t support those use-cases yet.

                                                                                                                                      The instragram comparison doesn’t generalize, in this case this specific repo was just full of a bunch of personal photos, not assets necessary for some software to work. Our systems aren’t well equipped to handle game assets either, but the analogy doesn’t carry over.

                                                                                                                                2. 4

                                                                                                                                  I don’t think the way you’re working is impossible to describe, I think it’s just hard and I think most people don’t understand the way you’re doing and building business. This means your clients may have an expectation that you will give a ToS or customer service level that you can not or will not provide

                                                                                                                                  To strive towards a fair description that honours how you are actually defining things for yourself and tries to make that more transparent without having to have specific use cases, perhaps there is a direction with wording such as:

                                                                                                                                  • To make a sustainable system we expect the distribution of computing resource usage and human work to follow a normal distribution. To preserve quality of service for all clients and to honour the sustainability of the business and wellbeing of our stuff and to attempt to provide a reasonably uniform and undestandable pricing model, we reserve the right to remove outliers who use an unusually large amount of any computing and/or human resource. If a client is identified as using a disproportionate amount of service, we will follow this process: (Describe fair process with notification, opportunity for communication/negotiation, fair time for resolution, clear actions if resolution is met or not).
                                                                                                                                  • This system is provided for the purposes of XYZ and in order to be able to design/optimise/support this system well we expect all users to use it predominatly for this purpose. It may be the case that using our system for other things is possible, however in the case we detect this we reserve the right to (cancel service) to ensure that we do not arrive at a situation where an established client is using our service for another prupose which may perform poorly for them in the future because it is not supported, or may become disproportionately hard for us to provide computing resource or human time for because it is not part of XYZ. This will be decided at our discretion and the process we will follow if we identify a case like this is (1,2,3)
                                                                                                                                  1. 1

                                                                                                                                    No, they’re intentionally vague so that we can exercise discretion.

                                                                                                                                    Funny way to say “so I can do whatever I want without having to explain myself”

                                                                                                                                    1. 14

                                                                                                                                      I think that’s unfair. He did in fact explain himself to the customer and it was the customer who decided to cancel the service. I’d agree if the data was deleted without sufficient warning, but that is not the case here.

                                                                                                                                    2. 1

                                                                                                                                      Would it be possible to get a clear “Terms of Service” clarifying these sorts of use cases?

                                                                                                                                      No, they’re intentionally vague so that we can exercise discretion. There

                                                                                                                                      May I suggest, perhaps: “ToS: regular repositories have a maximum file size X and repository size Y. We provide extra space to some projects that we consider important.”

                                                                                                                                  1. 3

                                                                                                                                    If you’re putting binary files into git you’re doing it wrong. One could argue about small files, but compiled code/executables, photos or “gifs for the readme” are definitely misplaced in a git repository.

                                                                                                                                    1. 12

                                                                                                                                      I do find that having image files in a resources/ directory for something like a website is often simpler than separating the two. Even then making sure that images are compressed and generally not bloating repo size / git history is essential.

                                                                                                                                      1. 18

                                                                                                                                        I do find that having image files in a resources/ directory for something like a website is often simpler than separating the two.

                                                                                                                                        Yeah, the is exactly the use case here. Mercurial (and git) aren’t designed for handling large binary files, but if you’re checking in static assets/resources that rarely change it still tends to work fine. This repo was fine on Bitbucket for many years, and is working fine on an hgweb instance I’ve spun up in the mean time.

                                                                                                                                        I specifically asked about limits because if it’s just the size of the repo being a technical problem for their infrastructure, I can understand. But they would not specify any limits, but just reiterated several times that Mercurial wasn’t designed for this. So I don’t know which of these was the actual problem:

                                                                                                                                        1. The repo is so taxing on their infrastructure it’s causing issues for other users.
                                                                                                                                        2. The repo is so large it’s costing more to store than some portion of the $100/year account price can cover.
                                                                                                                                        3. They are morally opposed to me using Mercurial in a way that it wasn’t designed for (but which still works fine in practice).

                                                                                                                                        Cases 1 and 2 are understandable. Setting some kind of limit would prevent those problems (you can still choose to “look the other way” for certain repos, or if it’s only code that’s being stored). Case 3 is something no limit would solve.

                                                                                                                                        1. 3

                                                                                                                                          If you want to store large files and you want to pay an amount proportional to the file sizes, perhaps AWS S3 or Backblaze B2 would be more appropriate than a code hosting website? I don’t mean to be obtuse, but the site is literally called source hut. Playing rules lawyer on it read like saying “Am I under arrest? So I’m free to go? Am I under arrest? So I’m free to go?” to a police officer.

                                                                                                                                          1. 5

                                                                                                                                            B2 or S3 would make things more complicated than necessary for this simple repo. I’ve spun up a $5/month Linode to run hgweb and it’s been working great. I’m all set.

                                                                                                                                      2. 6

                                                                                                                                        This case was hg, but the same limitations are present. Hg has a special extension for supporting this:

                                                                                                                                        https://www.mercurial-scm.org/wiki/LargefilesExtension

                                                                                                                                        And it’s considered “a feature of last resort”. It’s not designed to deal with these use-cases.

                                                                                                                                        LFS support requires dedicated engineering and operations efforts, which SourceHut has planned, but is not ready yet.

                                                                                                                                        1. 5

                                                                                                                                          I have a repository with mostly PNG files. Each PNG file is source code; a chunk of data inside each PNG file is machine-readable code for the graph visually encoded in that PNG’s pixels. What would you have me do?

                                                                                                                                          I suspect that you would rather see my repository as a tree of text files. While this would be just as machine-readable, it would be less person-readable, and a motivating goal for this project is to have source files be visually readable in the way that they currently are, if not more so.

                                                                                                                                          git would not support binary files if its authors did not think that binary-file support were not useful; that is the kind of people that they are and the kind of attitude that they have towards software design.

                                                                                                                                          With all that said, I know how git works, and I deliberately attempt to avoid checking in PNGs which I think that I will have to change in a later revision. It would be quite nice if git were able to bridge this gap itself, and allow me to check in plaintext files which are automatically presented as PNGs, but this is not what git was designed to do, and we all can imagine the Makefile which I’d end up writing instead.

                                                                                                                                          1. 1

                                                                                                                                            I like the project, but pardon my ignorance - aren’t the PNG files still binary assets produced by the “real” source code, which is the textual expression parsed to generate both the embedded bitstring and the dot graph? If they’re machine readable, that places them in the same category as compiled object files.

                                                                                                                                            1. 3

                                                                                                                                              The real source code is non-textual; it is the diagram (WP, nLab) which is being given as a poset (WP, nLab). To achieve optimal space usage, each poset is stored as a single integer which codes for the adjacency matrix. However, this compressed format is completely unreadable. There are several layers around it, but each layer is meant to do one thing and add a minimum of overhead; JSON (in the future, BSON or Capn) for versioning and framing, and PNG for display and transport. There isn’t really source code; there’s just a couple Python and Monte scripts that I use to do data entry, and I want them eventually automated away in favor of API-driven development.

                                                                                                                                              For example, the raw integer for this “big” poset is (at the time of writing) 11905710401280198804461645206862582864032733280538002552643783587742343463875542982826632679979531781130345962690055869140174557805164079451664493830119908249546448900393600362536375098236826502527472287219502587641866446344027189639396008435614121342172595257280100349850262710460607552082781379116891641029966906257269941782203148347435446319452110650150437819888183568953801710556668517927269819049826069754639635218001519121790080070299124681381391073905663214918834228377170513865681335718039072014942925734763447177695704726505508232677565207907808847361088533519190628768503935101450078436440078883570667613621377399190615990138641789867825632738232993306524474475686731263045976640892172841112492236837826524936991273493174493252277794719194724624788800854540425157965678492179958293592443502481921718293759598648627823849117026007852748145536301969541329010559576556167345793274146464743707377623052614506411610303673538441500857028082327094252838525283361694107747501060452083296779071329108952096981932329154808658134461352836962965680782547027111676034212381463001532108035024267617377788040931430694669554305150416269935699250945296649497910288856160812977577782420875349655110824367467382338222637344309284881261936350479660159974669827300003335652340304220699450056411068025062209368014080962770221004626200169073615123558458480350116668115018680372480286949148129488817476018620025866304409104277550106790930739825843129557280931640581742580657243659197320774352481739310337300453334832766294683618032459315377206656069384474626488794123815830298230349250261308484422476802951799392281959397902761456273759806713157666108792675886634397141328888098305747354465103699243937608547404520480305831393405718705181942963222123463560268031790155109126115213866048693391516959219000560878337219324622230146226960346469769371525338127604307953786112516810509019551617885907067412613823285538493443834790453576561810785102306389953804151473860800342221969666874213156376831068606096772785272984102609049257833898258081466729520326827598704376424140779421965233471588921765110820238036094910936640446304632443760482611408445010230964335747094869968021425396439555206085281953007985784739643408074475440039274314217788647485602069097474262381690379456154426900896918268563062231294937080146199930562645748389040251871291840481739518244706752426504146889097315360662429293711705265772337748378759001582638301784557163848933046038798381667545043026975297902178839764134784634179453671000024868722179355800776002690855305662785522771116635997791339179517016284742206819482196944663461005128697584753594559406283638837841370287286682993990297923202976404261911087739188860505577427942276773287168600954693735964671046522557013031834557159173262849132567983767216098382093390056878765856939614383049277441.

                                                                                                                                              1. 1

                                                                                                                                                Ah, okay, I see. Makes sense, thank you for explaining!

                                                                                                                                          2. 4

                                                                                                                                            I’ve seen this argument quite a number of times, and almost always without a coherent explanation of why is that wrong. What’s the rationale behind this argument?

                                                                                                                                            1. 4

                                                                                                                                              Shameless plug, I contributed heavily to this help topic back when I was the PM for Microsoft’s Git server: https://docs.microsoft.com/en-us/azure/devops/repos/git/manage-large-files?view=azure-devops

                                                                                                                                              FWIW I disagree with the comment up-thread which says that GIFs for READMEs don’t belong. If you’re going to check in a logo or meme or whatever, that’s perfectly fine. Just don’t do 1000 of them and don’t churn it every week.

                                                                                                                                              1. 2

                                                                                                                                                I think a big part is also “are my tools there for me or am I slave to my tools?”

                                                                                                                                                If I have a website and most content is under version control, it’s annoying and complicated to have (big) assets outside. Most people simply want one repo with everything inside, and it’s mostly additive, often once per week - it simply doesn’t matter if it’s the wrong tool.

                                                                                                                                          1. 5

                                                                                                                                            How is he counting syscalls? strace, or something else? I ask because I want to do a few more tests myself.

                                                                                                                                            1. 8

                                                                                                                                              strace, yes.

                                                                                                                                              1. 2

                                                                                                                                                Is this the real Drew? Or was your username something like sircmpwm…?

                                                                                                                                                1. 4

                                                                                                                                                  Yes, I am Drew DeVault.

                                                                                                                                            1. 5

                                                                                                                                              To repeat myself again: This ought to be only for CPython, and not for interpreters like PyPy which are not following the Python Software Foundation’s deprecation schedule. Quoting again from an old PyPy blog post:

                                                                                                                                              PyPy’s own position is that PyPy will support Python 2.7 forever—the RPython language in which PyPy is written is a subset of 2.7, and we have no plan to upgrade that.

                                                                                                                                              I continue to be deeply disappointed in the ethics of the Python Software Foundation for their decades-long snub of PyPy, Jython, and the rest of the alternative interpreter ecosystem.

                                                                                                                                              1. 8

                                                                                                                                                I continue to be deeply disappointed with PyPy, and the rest of the bloody ecosystem’s, perverse and unhealthy obsession with sticking to Python 2 forever.

                                                                                                                                                1. 5

                                                                                                                                                  Then use PyPy for Python 3.6. Additionally, you could roll up your sleeves and help maintain PyPy. The problem is not the deprecation of Python 2 as much as the continued shunning of PyPy.

                                                                                                                                                  In a prior life, I recall demonstrating that PyPy could keep pace with your handwritten C++ and C♯ code, in the context of homebrewed Minecraft servers. I wonder how much of your disappointment is sour grapes.

                                                                                                                                              1. 3

                                                                                                                                                See also @emersion’s mrsh:

                                                                                                                                                https://mrsh.sh

                                                                                                                                                A strict POSIX shell written in POSIX C99, which ships with a library which can be used for parsing & interpreting shell scripts, or writing your own interactive shells, from C.

                                                                                                                                                1. 2

                                                                                                                                                  I thought mine was interesting because I havent seen a POSIX shell in Go before, and from my experience Go projects have had better Windows support (with exception of color output). Contrast with C, where Windows support by default doesnt exist unless the author explicitly codes it in.

                                                                                                                                                  1. 10

                                                                                                                                                    BTW you can’t implement a POSIX shell in portable Go, because Go doesn’t export fork() and exec(). It only exposes ForkExec() portably. Windows and Unix have very different process models, and the Unix shell uses a LOT of stuff that doesn’t make sense in Windows (in contrast to say the file system which I think is more or less a 1:1 translation.)

                                                                                                                                                    There are of course Windows ports of Unix shells but I think they must be done at a “higher level” than libc or Go’s abstractions. They’re also pretty slow.

                                                                                                                                                    https://golang.org/pkg/syscall/

                                                                                                                                                    In a shell you have to fork and re-run the interpreter, without exec. Consider:

                                                                                                                                                    f() { echo one; echo two; }
                                                                                                                                                    g() { while read x; do echo [$x]; done; }
                                                                                                                                                    
                                                                                                                                                    f | g
                                                                                                                                                    

                                                                                                                                                    (there are much simpler examples but this illustrates the principle)

                                                                                                                                                    I’m not even sure how it’s done with non-portable extensions, given the runtime.


                                                                                                                                                    The issue is that the Go runtime uses threads to implement goroutines. Threads and fork don’t mix. Especially when the go runtime uses a lot of concurrency primitives. Generally Go programs only do the simple ForkExec, never the more complex patterns that a shell requires – i.e. just setting up a pipeline requires a bunch of stuff between fork() and exec().

                                                                                                                                                    There are probably better references, but here a couple:

                                                                                                                                                    https://rachelbythebay.com/w/2011/06/07/forked/

                                                                                                                                                    https://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them

                                                                                                                                                    And fork, threads, and signals – which a POSIX shell requires – are its own special brand of nightmare. All existing shells use no threads – just fork and signals (and file descriptor manipulation).


                                                                                                                                                    Maybe you can set GOMAXPROCS=1 and the Go runtime will only use one thread? And then you can make the Linux syscalls for fork and exec. But I’m not sure if BSDs have stable syscalls for fork and exec – there was another thread here about OpenBSD validating libc calls which is very very relevant to writing a shell in Go.

                                                                                                                                                    It’s not clear to me if that’s the case, or if shfmt uses it. I grepped the repo for GOMAXPROCS and didn’t find anything.

                                                                                                                                                    https://stackoverflow.com/questions/37391009/is-lock-necessary-when-gomaxprocs-is-1

                                                                                                                                                    1. 3

                                                                                                                                                      The relevant link:

                                                                                                                                                      https://lobste.rs/s/6a6zne/some_reasons_for_go_not_make_system_calls

                                                                                                                                                      tl;dr I’m not sure how you can make a portable POSIX shell in Go. I have been wondering about this for a few years actually, because there are several alternative shells written in Go. However most of them don’t purport to support the f | g pattern above, but a POSIX shell must.

                                                                                                                                                      1. 1

                                                                                                                                                        You can emulate subshells by virtualize the process environment (working directory, environment variables, etc.). There are obviously things you cannot virtualize like PID, but you can get pretty far in terms of compatibility with existing code that relies on subshells.

                                                                                                                                                        1. 2

                                                                                                                                                          I think the biggest thing is the file descriptor table. You can maybe hack around f | g above somehow, but how about this:

                                                                                                                                                          f() { echo one; ls /; echo two; }
                                                                                                                                                          
                                                                                                                                                          f | g | g | g
                                                                                                                                                          

                                                                                                                                                          Well I guess you can implement pipes in user space, which is what I guess Elvish does, which makes sense since it has structured data anyway.

                                                                                                                                                          I’m more talking about a POSIX- / bash- compatible shell in Go. I don’t see how you do it, but I never tried.

                                                                                                                                                          Also I guess it should be very simple to try f | g in the shfmt interpreter (though I don’t keep a Go toolchain up to date.)

                                                                                                                                                          I think it can be made it work on x86 Linux with some one-offs but it would not port to BSDs or Solaris very well. You would have to do work for each platform, whereas with C everything is portable.

                                                                                                                                                          1. 1

                                                                                                                                                            It is much less tricky than you think; Go’s os.StartProcess gives you full control on what the file descriptor table of the child process should look like.

                                                                                                                                                            To simplify things, let’s start with a pipe consisting entirely of external commands, like:

                                                                                                                                                            cat a.txt | grep a

                                                                                                                                                            To execute this pipeline, you make a pipe, tell StartProcess to use the write end as FD 1 of the cat process, and use the read end as FD 0 of the grep process. Still kernel-space pipes.

                                                                                                                                                            Now to bring builtin functions into the picture, what you need to do is to maintain some user-space FD tables, and program the builtin functions so that they never use the “real” FD table, only your user-space ones. So if we suppose echo is a builtin command, then in

                                                                                                                                                            echo aaa | grep a

                                                                                                                                                            You still make a pipe and the handling of grep is unchanged; but for echo, you make a “virtual” FD table where FD 1 is the write end of the pipe, and make your implementation of echo to use that virtual FD table.

                                                                                                                                                            The same technique can be used for environment variables and the working directory, although I have only done this for FD tables in Elvish.

                                                                                                                                                            Some notes on Elvish’s implementation:

                                                                                                                                                            1. 2

                                                                                                                                                              How does it preserve state? The function being put in the pipeline can make use of the whole state of the shell.

                                                                                                                                                              FOO=foo
                                                                                                                                                              BAR=bar
                                                                                                                                                              
                                                                                                                                                              g() { while read x; do echo $FOO $x $BAR; done; }
                                                                                                                                                              
                                                                                                                                                              f | g | g
                                                                                                                                                              

                                                                                                                                                              As far as I understand, StartProcess can only start a new sh image. It doesn’t fork the existing image. That’s what I assume based on taking argv. If it forked then it wouldn’t need to take argv. And that wouldn’t work with Go’s runtime (as long as there are multiple threads in Go’s runtime, which I thought you could disable, but I’m not sure.)


                                                                                                                                                              To summarize, I’m not saying you can’t set up a shell pipeline in portable Go. I’m saying you can’t put a shell function in a pipeline transparently in portable Go. Because you have to fork() and re-run the interpreter, without exec().

                                                                                                                                                              (FWIW this is not academic since I use this all the time, e.g. in my benchmarks/ dir, and I wrote a couple blog posts about it. It’s a powerful form of composition in shell.)

                                                                                                                                                              1. 2

                                                                                                                                                                I put it here to make sure the example is runnable:

                                                                                                                                                                https://github.com/oilshell/blog-code/blob/master/function-in-pipeline/demo.sh

                                                                                                                                                                I guess since we got this far in the conversation I will try to run it with the shfmt interpreter, if I can get it to build.

                                                                                                                                                                It may work on one platform if there is a way to disable threads in the Go runtime. And then it can use non-portable syscalls. However I will be surprised if it works on multiple platforms without non-portable code (i.e. I claim that’s impossible.)

                                                                                                                                                                ~/git/oilshell/blog-code/function-in-pipeline$ ./demo.sh 
                                                                                                                                                                [ [ BEGIN ] ]
                                                                                                                                                                [ [ alpine ] ]
                                                                                                                                                                [ [ bin ] ]
                                                                                                                                                                [ [ boot ] ]
                                                                                                                                                                [ [ END ] ]
                                                                                                                                                                

                                                                                                                                                                edit: I think the more likely way to do it is to not use os.StartProcess, but to manually copy the entire interpreter state (so the pipeline element can’t modify the parent shell state, and to avoid races), and communicate over a file descriptor connected to two threads/goroutines in the same process. But I don’t think anybody is doing that.

                                                                                                                                                                I think there may be some stuff in addition to the $PID that would make such an implementation non-POSIX though. You might be able to get pretty close.

                                                                                                                                                                1. 1

                                                                                                                                                                  Maybe something isn’t clear: to implement a pipeline in a threaded runtime environment (like that of Go), you don’t fork a new process for each component of the pipeline, you just start a new thread [1]. Each of those threads has their own user-space FD table and variable table. When you do have to launch an external command, you let the external command inherit the FD table and the exported part of the variable table from the thread, not the process.

                                                                                                                                                                  [1] In Go you actually start a goroutine, but that’s irrelevant implementation detail.

                                                                                                                                                                  1. 1

                                                                                                                                                                    Right, that’s what I was thinking with the last edit. But to get POSIX semantics you will have to copy the interpreter state – at least the globals and the stack.

                                                                                                                                                                    I tried with Elvish and it works nicely, but the semantics aren’t POSIX because f | sort will increment a.

                                                                                                                                                                    fn f { echo ---; ls / | head; echo ---; a = (+ $a 1) }
                                                                                                                                                                    

                                                                                                                                                                    Of course the claim isn’t about anything in Elvish – it’s about writing a POSIX shell in portable Go.

                                                                                                                                                                    Though in Elvish it also seems like you need some kind of GIL to protect variables like a and compound data structures from races.

                                                                                                                                                                    1. 1

                                                                                                                                                                      You are right that Elvish does not copy the globals - Elvish’s semantics is different. But it seems we are in agreement that it is quite straightforward to copy the global states if you want, and get pretty far in emulating the semantics of subshells in portable Go code.

                                                                                                                                                                      1. 1

                                                                                                                                                                        I wouldn’t say it’s straightforward – as mentioned there is other state besides the FD table, shell globals, and the shell stack. I’ve spent a lot of time running shell scripts in the wild so I know how they depend on every nook and cranny of the shell.

                                                                                                                                                                        This paper[1] is a great summary of all the difficulties with fork() from the other side – the kernel rather than the shell. And they claim that shell is the ONLY application that fork() is good for! (I’ve brainstormed making a fork()-less shell with many of these techniques to alleviate the kernel’s burden, but it’s not high priority. )

                                                                                                                                                                        [1] https://lobste.rs/s/nfxsou/fork_road


                                                                                                                                                                        I like a lot of things about Go, and I considered it for writing Oil (along with hand-written C++, Lua, femtolisp, and many others).

                                                                                                                                                                        The string slicing is great and something I’m kind of doing by hand now. The GC would have saved me a significant amount of time too.

                                                                                                                                                                        But I consider the runtime to be a dealbreaker for writing a POSIX shell, especially one that can be used as the default system shell. If you have control over the semantics like Elvish, then it’s a somewhat different story.

                                                                                                                                                                        I’ve been working on a compatible shell for a long time so I have more of an appreciation for how constrained a problem it is. There are not many languages besides C or C++ that are appropriate for the task. Rust is probably the #2 contender if you’re not resorting to code gen like Oil.

                                                                                                                                                                        I’ll make a concrete prediction: no Linux distro or major Unix system will ever use shell written in Go as their default system shell.

                                                                                                                                                                        Of course that’s probably not a goal of many projects, so writing in Go may be a fine choice. But it’s a goal of Oil to be usable as the default system shell, i.e. to replace bash.


                                                                                                                                                                        Also, even if it’s not hard to program, I would much rather let the kernel copy the data in pages than recursively walk and copy data structures to set up a pipeline.

                                                                                                                                                                        And of course it’s not a copy, the virtual memory manager enables COW. Shells don’t generally have large heaps, but the goal of Oil is to expand shell usage and maybe some programs will have large heaps.

                                                                                                                                                                        I also think the issue with mutable variables is not a small one, from a design perspective. It’s a big issue in the semantics of a shell language. The easiest solution is the GIL – all the other ones are hard or impose some burden on the user.

                                                                                                                                                                        I think the persistent data structures are an interesting angle on it, but I guess it doesn’t completely eliminate the problem.


                                                                                                                                                                        Anyway thanks for the info about the virtual FD table. That did answer a bunch of my questions!

                                                                                                                                                                        1. 2

                                                                                                                                                                          Of course that’s probably not a goal of many projects, so writing in Go may be a fine choice. But it’s a goal of Oil to be usable as the default system shell, i.e. to replace bash.

                                                                                                                                                                          Depending on your definition of “default system shell”. I agree that with Go it’s very hard (if not impossible) to implement a shell with a semantics close to POSIX enough to be the default /bin/sh; that is, the default interpreter for all these shell scripts out there. But you can totally use a shell implemented in Go (like Elvish!) as your personal default shell and use it to write new scripts.

                                                                                                                                                                          You will still run into POSIX shell stuff every now and then, either snippets on the web, or things like venv‘s activation script and output of ssh-agent -s. I’d like to revisit POSIX emulation in Elvish some day; but as far as these use cases go, they almost never involve advanced semantics that you cannot emulate in Go.

                                                                                                                                                                          I also have a feeling that the “system shell” in becoming less important in the current trend of computing, at least for “ordinary” users and system administrators. As a user I used to write some initscripts; but now I use systemd and just write INI-style unit files. As a system administrator I used to write scripts to glue my services together, but today I configure containers. Of course you still need to sprinkle some shell, telling systemd and container which commands to invoke, but these are almost always very “dumb” code that is just a sequence of command invocations, with some occasional variable references.

                                                                                                                                                                          And of course it’s not a copy, the virtual memory manager enables COW. Shells don’t generally have large heaps, but the goal of Oil is to expand shell usage and maybe some programs will have large heaps.

                                                                                                                                                                          Persistent data structure solves the “data forking” problem pretty well - there is zero cost in copying, and near constant cost in modifying “forked” data. As long as you can live with its overhead, of course.

                                                                                                                                                                          I also think the issue with mutable variables is not a small one, from a design perspective. It’s a big issue in the semantics of a shell language. The easiest solution is the GIL – all the other ones are hard or impose some burden on the user.

                                                                                                                                                                          I haven’t thought enough about what is the “best” concurrency semantics for a shell language, so I am deferring the decision to future, when I have some experience programming something concurrent in Elvish.

                                                                                                                                                        2. 2

                                                                                                                                                          mvdan/sh is also very cool :)

                                                                                                                                                      1. 8

                                                                                                                                                        Even as a staunch advocate of C, I have a much simpler argument against this design from OpenBSD: languages which are not C should not depend on C. From a philisophical point of view, I think that a language design is much stronger when its compiler, runtime, stdlib, and so on - are all implemented in that language. If we constrain ourselves to language designs which can route syscalls through the C stdlib then we constrain our ability to innovate in new languages.

                                                                                                                                                        1. 8

                                                                                                                                                          language designs which can route syscalls through the C stdlib

                                                                                                                                                          If your language design cannot call a function (with the platform’s standard ABI) from a shared library, something has gone terribly, horribly wrong. I haven’t seen any production language where FFI is impossible.

                                                                                                                                                          If your language design just really really does not want to use syscall wrappers because it uses a completely alien calling convention and weird stacks and calling a function with platform ABI would incur performance overhead… well, I would argue something has gone wrong in this case too. I blame Linux monoculture — Linux is about the only significant unix-like where raw syscall numbers (called via raw CPU trap instructions) are the public API of the platform.

                                                                                                                                                          So many advantages to defining syscalls as ELF symbols:

                                                                                                                                                          • massively easier portability. Adding FreeBSD/aarch64 support to Go took three or four people, many many months of occasional work, tons of lines of code. Doing the same with Rust, D (LDC), OCaml, Crystal, Pony and others was trivial (in the syscall department — of course there were fun thread-local storage bugs and so on :D)
                                                                                                                                                          • hooking via LD_PRELOAD, which is very useful for sandboxing existing programs (e.g. with libpreopen I can exec a gedit inside the incredibly strict Capsicum sandbox)
                                                                                                                                                          • the platform can replace any syscall that actually goes through the kernel with a fast vDSO without breaking anything
                                                                                                                                                          • with symbol versioning, old ABI can be kept around — sometimes as purely userspace wrappers around new ABI, and the kernel can be cleaned up
                                                                                                                                                          • flexibility in what the platform actually even is at runtime…

                                                                                                                                                          Sure, current unixes live with some legacy design decisions that actually tie everything closer to the C language — like the errno example — but this doesn’t mean you can’t design a good ABI with “C-ish” ELF symbols.

                                                                                                                                                          1. 4

                                                                                                                                                            You seem to be arguing with a strawman here - I never said I was opposed to a syscall wrapper library, though it has problems of its own. I’m arguing against blithely routing everything through libc.

                                                                                                                                                            1. 1

                                                                                                                                                              If your language design just really really does not want to use syscall wrappers because it uses a completely alien calling convention and weird stacks and calling a function with platform ABI would incur performance overhead…

                                                                                                                                                              So resizable stacks are a bad thing?

                                                                                                                                                              1. 1

                                                                                                                                                                So this is a bit late as a comment. But not necessarily! The kernel doesn’t use your stack, it uses its own stack. So you really don’t very much stack space for these calls, and you could document and stabilise as part of the ABI the amount of required stack space needed to call these functions.

                                                                                                                                                                1. 2

                                                                                                                                                                  So you really don’t very much stack space for these calls

                                                                                                                                                                  Yes you do.

                                                                                                                                                                  1. 1

                                                                                                                                                                    Any stack intensive functions can do a stack switch

                                                                                                                                                                    1. 1

                                                                                                                                                                      That’s what Go does.

                                                                                                                                                                      The problem is that this adds overhead, so in order for Go to remain competitive, it must avoid stack switches for hot functions like time().

                                                                                                                                                                      1. 1

                                                                                                                                                                        The problem is that the system calls and libc are designed around big C-style stacks, so they expect effectively infinite stack space to be available. If you designed a libc that was intended to be used from languages with unusual/little stack space available, I would imagine that it would be possible to design things you can provide an ABI guarantee of a maximum amount of stack usage per function in your API. I believe that in embedded systems programming it’s quite common to work out your maximum possible stack space, with compiler help, and then just manually allocate that much stack space for that thread at compile time.

                                                                                                                                                                        On 64-bit systems with virtual memory, just overcommitting ~10MiB of stack for every thread and letting virtual memory handle growing it in 4KiB chunks seems like the best, simplest approach. That’s what I’ve done in my implementation of fibres. Virtual address space is rather plentiful.

                                                                                                                                                                        1. 1

                                                                                                                                                                          This, of course, means that every thread always uses at least 4 KB just for the stack. Golang’s default stack size on Linux was, at least at one point, 2KB.

                                                                                                                                                                          1. 1

                                                                                                                                                                            That’s a very tiny overhead for the simplicity and performance advantages of trivial C interop.

                                                                                                                                                            2. 3

                                                                                                                                                              For better or for worse, C is the Unix ABI. It’s not its contemporaries where calling conventions were either ill defined (mainframe stuff, PCs) or very language neutral (VMS, i).

                                                                                                                                                              1. 2

                                                                                                                                                                I think you’re looking at this from the wrong perspective. The SYSV ABI was standardized by SCO, and includes the ELF standard. It also happens to standardize the calling convention, and the way parameters about arguments (argc/argv) and the system (auxv) are passed to a program. It standardizes a lot more than that. Because of it being the System V Application Binary Interface, Linux and the BSDs chose to implement and follow that.

                                                                                                                                                                For many reasons, the C standards committee chose the SYSV ABI as the calling convention for the C language.

                                                                                                                                                                The C language ABI isn’t being forced on other languages, the C language chose the SYSV ABI as a base.

                                                                                                                                                                Edit:

                                                                                                                                                                everyone’s favorite case of errno, which you consult to get the error value from a failed system call (and from some failed library calls).

                                                                                                                                                                Eh? Linux system calls are returned in rax, and I assume that’s the case for OpenBSD too?

                                                                                                                                                                1. 4

                                                                                                                                                                  Linux system calls are returned in RAX IF you use the raw system call ABI. If you use libc, then you need to check errno.

                                                                                                                                                                  BSD system calls are returned in RAX:RFLAGS[OF], with OF acting as an error bit.

                                                                                                                                                                  1. 2

                                                                                                                                                                    The ABI is a different concern. I think that it’s reasonable to make an arbitrary programming language capable of speaking the C ABI. I am specifically talking about routing syscalls through libc.

                                                                                                                                                                  2. 1

                                                                                                                                                                    From my point of view, syscalls are another calling convention that occasionally leads to the C code in the kernel anyway. If platform (i.e. OpenBSD) developers want us to use cdecl instead of int 80h, then it’s fine as long as the ABI is specified.

                                                                                                                                                                    The main difference from the C calling convention here is that usually the kernel won’t mess with your stack. Although, I think it’s possible for libc to wrap all function calls with switch to some internal stack, and use some dynamic linker tricks or just defines in C headers to actually call wrappee in programs that use traditional C ABI.

                                                                                                                                                                    language design is much stronger when its compiler, runtime, stdlib, and so on - are all implemented in that language

                                                                                                                                                                    This, however, is not a barrier to implementing FFI in that language.

                                                                                                                                                                    1. 1

                                                                                                                                                                      If platform (i.e. OpenBSD) developers want us to use cdecl instead of int 80h, then it’s fine as long as the ABI is specified.

                                                                                                                                                                      I mostly agree. Is the ABI specified and stable though?

                                                                                                                                                                  1. 4

                                                                                                                                                                    The eMMC is 16G and, side note, had seventeen partitions on it when I first got the phone

                                                                                                                                                                    Well, if it came with some version of Android, that shouldn’t be surprising. Modern Qualcomm Android phones (especially with A/B Treble) have like 64 partitions.

                                                                                                                                                                    I have been working on some components of a mobile-friendly Wayland compositor, based on Sway

                                                                                                                                                                    I understand the drive to reinvent everything very well, but like… Purism is building Phosh with wlroots already. And they have a layer-shell keyboard. Why not use their stuff?

                                                                                                                                                                    (btw: what’s the protocol to make soft keyboards pop up when you focus a text field? Is it input-method?)

                                                                                                                                                                    1. 10

                                                                                                                                                                      I am using some of their stuff - I actually use a personal fork of their older virboard OSK right now. The main issue with their stuff is that it’s heavily GNOME-centric, implicates lots of dbus, and so on - not the kind of design choices I’m interested in for my personal environment. And the more the merrier, eh?

                                                                                                                                                                    1. 8

                                                                                                                                                                      This phone has me really excited. The price and specs seem good enough and I am super hyped for mobile Linux. But I think I will wait until some more things are polished before I get one myself. It is stuff like this that kind of makes me wish I was better at low level programming so that I could help develop this into a more polished user experience.

                                                                                                                                                                      1. 18

                                                                                                                                                                        The best way to learn is by doing.

                                                                                                                                                                        1. 5

                                                                                                                                                                          That is a good point. I currently have some other side projects on the back-burner (learning Haskell and Toki Pona). After the holidays and after I make some more headway with those projects I will seriously consider getting a pinephone and hacking on it. Maybe see if I can write some simple applications in something interesting like Zig or Rust.

                                                                                                                                                                          Thanks for the article, I always enjoy your content!

                                                                                                                                                                          1. 6

                                                                                                                                                                            Potentially also check out Nim.

                                                                                                                                                                            1. 3

                                                                                                                                                                              Nim rules!

                                                                                                                                                                        2. 3

                                                                                                                                                                          You might not need to be doing particularly low level programming to add value. The ecosystem will need quality mobile-friendly apps; I suspect there will be a lot of opportunities without having to dig into systems programming.