1. 4

    Teeth have improved since the last such thing I saw. Still has trouble with hair, clothes, and sometimes ears and wrinkles.

    I’d personally find it handy to have a way to link to individual photos, to show features and flaws. Still, nice work.

    1. 3

      This is the first I heard about Pijul, has anyone used it for production/hobby? What are your thoughts?

      1. 18

        I’ve used Pijul casually a few times over the past year or so. My impressions are:

        • it is pretty dang cool
        • the part where merge conflicts and the resolution (not just the resolution) are kept as part of the history is really incredible, and avoids scenarios where you’re 10 patches deep in a rebase and realized you messed it up somewhere.
        • the whole ecosystem (pijul + nest) is a bit rough, and it is a bit hard to use. I’m not surprised: they’ve taken on a LOT of work for a small project and new team.
        • diffs not necessarily being line-based is a very cool concept, with incredible possibilities: imagine using a LSP language server’s guidance in creating a semantic diff.

        I’d love to experiment with importing big Git repositories (Nixpkgs, Linux) and torture it until it breaks. I wouldn’t hold it against them, it is a young project. I don’t think that it is reasonable to expect it to handle such monstrosities already.


        By the way, if you’re a Nix user, get Pijul from nixpkgs unstable:

        $ nix-shell -p pijul -I nixpkgs=channel:nixpkgs-unstable
        
        [nix-shell:~]$ pijul --version
        pijul 0.11.0
        
        1. 6

          the whole ecosystem (pijul + nest) is a bit rough, and it is a bit hard to use

          I feel like nest could only be improved by opening the source, honestly.

          1. 2

            This has been brought up to pmeunier very often. He has his reasons, which I don’t fully agree with, but at least understand. It may be released, in time, in the future. I certainly hope so.

            1. 1

              Yeah, and I get it, but it does seem to be a blocker for a great many things. Do you know if pmeunier would support a different project intended for pijul hosting? It shouldn’t be too complicated to fork darcsweb, for example, and go from there.

              Basically, how attached is pmeunier to Nest as a flagship pijul project?

              1. 3

                He was pretty happy when I started to write pi-hoole (see here) so I think he would be very happy to see contribution to the project in that form!

                (btw, I do not work on pi-hoole right now, and will probably wait for the next version before updating it)

          2. 3

            Okay, kind of a funny thing, but I was experimenting with https://nest.pijul.com/pijul_org/git-pijul and then https://nest.pijul.com/rohan/git-pijul which got me to cloning https://nest.pijul.com/rohan/pijul. Directly after cloning this fork of pijul, I get merge errors:

            $ pijul clone grahamc@nest.pijul.com:rohan/pijul
            [...]
            $ pijul status
            On branch master
            
            Unresolved conflicts:
              (fix conflicts and record the resolution with "pijul record ...")
            
                    Cargo.lock
                    libpijul/Cargo.toml
                    libpijul/src/fs_representation.rs
                    libpijul/src/graph.rs
                    pijul/src/commands/fs_operation.rs
                    pijul/src/commands/record.rs
                    pijul/src/commands/status.rs
            

            Presumably merge conflicts are a valid state to be in, and can be pushed to a remote? If true (and not a bug), that is a pretty funny side effect of tracked merges!

            1. 4

              Presumably merge conflicts are a valid state to be in, and can be pushed to a remote?

              That’s exactly correct. The repo is just a collection of patches; two can conflict and both be in the repo. The resolutions are also explicit objects, so you’d normally avoid pushing conflicting patches without a resolution patch, but there’s no reason you have to do that. (And there are times you don’t want to—for example, if you need help on a merge. And you can do that in Git and Hg, too, but it explicitly adds an extra head.)

            2. 1

              diffs not necessarily being line-based is a very cool concept

              Can’t you also do it with git? I know Unity recommends configuring git to use a custom tool for diffing and merging its .asset files.

            3. 12

              I used darcs and am actually the person who’s responsible for pijul being implemented in Rust :).

              I use it on and off. Here’s a couple of experiences, good and bad:

              • The interface models the interface of darcs, but doesn’t quite have its elegance. Which isn’t a bad thing, it’s just not quite there yet.
              • Still, it is a substantial improvement over darcs theory of patches, which had several issues.
              • Sometimes, it has quality issues, there were phases where you couldn’t download and build pijul.
              • Nest, while nice, suffers from not enough hands. I actually am privy to the nest code and would love to see some of that out in the open.
              • The model clicks for me better than the model of git. Always has and I love having an alternative around.
              • Certain operations that are hard to do in git are much easier in pijul: figuring out whether a codebase contains a a certain patch (as pijul operations don’t change the identity of the patch) and keeping signed patchs intact (same, patches don’t change identity). This is the core of why I like the model more: pijul has clearer semantics of what the bundles are that it is handling, while git is very focused on having minutia and that building a model on top of it.

              The problem that every of those alternative systems though is that you just can’t live without some way to interact with the current state of the world and it’s been a frequent issue to have good interactions with DAG-based systems the way git-svn/p4 enabled people to start using git with an upstream SVN/P4.

            1. 5

              The problem appears to be a cycle I’ve created: Sprint through a small project, sloth it up until I recover and move on to the next project to start the cycle again. Now how do I fix it?

              Seems like an obvious question, but: Do you actually want/need to fix it?

              It sounds like sprinting on exciting new side projects is filling a need for you. Will “finishing” a side project fill a need? Or is it something that sounds like the “right thing to do”, so you should do it regardless of enjoyment?

              Basically, if shipping isn’t enjoyable, and you’re working on side projects because it’s fun, you don’t need to ship. Don’t make a fun thing painful and stressful!

              If you’re trying to bulk a portfolio or resume, of course, it’s a different story.

              1. 3

                I don’t try to finish every project, but if I’m not finishing ANY project, then that’s a problem. By “finish” I mean “ship” something minimal – get it into the state that at least one other person can use it. Otherwise you’re depriving yourself of a very large learning opportunity – the feedback of others. That’s typically how a side project goes on to be something more rewarding.

                1. 3

                  If that’s a problem for you, then I’m all for it. If you’re framing it as a problem in some objective sense, though, that’s when I say don’t ruin your fun with burdensome expectations.

                2. 2

                  In support of that:

                  Getting software actually used has a lot of the same traits as running a software business: it needs to have a ‘market’, much of the work is not programming, there’s a slog of maintenance, you’ll be rewarded if you program in a disciplined way (minimize clever, remove cruft, have tests, etc.). That can feel totally reasonable if your goal is about providing something to some users, and not about the code or coding itself.

                  But to go play with Rust or error correcting codes or such, I don’t think I would want to have all that in mind.

                  The tension between the interest in playing with computing and the realities of shipping (even for free) has certainly caused me some stress, and it sounds like it causes it to some folks whose open-source projects end up popular as well.

                  1. 1

                    If I could upvote this again, I would. Great comment!

                  1. 5

                    I must admit I didn’t know about Citus before reading this article, nevertheless I’m surprised by Microsoft’s moves towards libre software friendliness in the last couple of years; the acquisition of Github, support for SQL Server over Linux (which I remember thinking as unnecessary since Postgres improves upon it by a big margin) and now Citus Data acquisition, strategic moves that look (at least on the surface) as steps on the right direction for a company that big, considering how hostile they were to the GNU/Linux community back at the beginning of 2001.

                    1. 7

                      Don’t forget about WSL, which was by far the biggest surprise to me.

                      1. 4

                        Thanks you. And I almost forgot when they joined LOT and OIN, to fight against patent trolls.

                        1. 6

                          We’d be wisd to remember Embrace, Extend, Extinguish as well…

                          1. 4

                            Plus, them extracting billions in patent royalties from Android. I don’t trust anything they say about patents. They have both lobbyists trying to change patent law to maintain/enable their lawsuits along with excellent lawyers who wield it profitably.

                            1. 1

                              I’m also thinking it might be another chapter of a practice MS is expert on, but I hope to be proven wrong. Time will tell eventually.

                          2. 2

                            Although it was a surprise, hindsight shows me we shouldn’t be surprised that they built something to run Linux’s apps on a paid product of theirs. They did it before with OS/2 subsystem for Windows NT. They were even also a UNIX shop at once with Xenix. SCO and OpenServer still exist despite IBM and Linux stomping them.

                            If anything, we should assume they’ll do more if it results in users either staying on or going to Windows.

                          3. 6

                            All right, I’m going to raise my hand and say after dealing with Oracle and SQL Server professionally, they’re both terrible at making things easier for a normal developer. SQL Server can at least fit in the Visual Studio/.NET toolchain/ecosystem comfortably, but using it outside of that can be awkward. Oracle documentation is made for people that must be getting paid to read it and tries to make its error messages provide no understanding of problems.

                            Yes, I’m probably biased because I’m working on ~20 yr old code and the DBA’s I work with don’t seem to know much besides moving archives around, but I shouldn’t have to create a single char column with a constraint of ‘Y’ or ‘N’ to fake a boolean column type (and that was according to the best practices that I’ve found thus far) - this makes me think that developer productivity isn’t relevant to Oracle. So Postgres is appealing to me as a pretty conventional Java/C# dev because they give me features (array types, json types/functions, and not requiring me to go to weeks of training) and let me forget about the database as much as possible.

                            1. 3

                              I’m probably biased because I’m working on ~20 yr old code

                              We are facing that same issue at my current job, there we have databases running for production systems using Oracle 8i (last supported release is from 1998), which is not adequate for current data demands, but still Oracle 11g is not that different from 8i, at least for the most used features we make use of at work.

                              1. 2

                                Database upgrades aren’t too bad (we’ve done them once or twice at work) as long as the features you’re using were deprecated in favor of something Oracle can charge more for (CDC Goldengate I’m looking at you).

                                1. 4

                                  This is maybe the only time I’ve ever seen people who actually use Oracle databases pop up in a thread. Even on HN, with it’s much bigger readership (though maybe all drawn from the same bubble) you’d swear it was 95% Postgres and a few people who think they ought to feel ashamed for using MySQL even though it’s working fine for them.

                                  So, if you don’t mind me asking, are there plus sides to an Oracle database? Does it do things that no other DB does, is it crazy reliable, or crazy fast, or is the on-call support so damn good that when there’s big money on the line you can’t go wrong with Oracle? Or is their revenue really due to legacy lock-in / switching costs, tech dinosaur inertia, and expert CTO shmoozing? Because Larry Ellison’s still a billionaire, so they’re presumably getting something right…

                                  1. 3

                                    There are many places that have Oracle but aren’t really Oracle shops per se, as Oracle is a big part of a lot of enterprise-y software packages that are sold to CTO types.

                                    Too, the contortions Oracle partisans wrap themselves into to justify the lack of a Boolean datatype are hilariously stupid.

                                    1. 2

                                      Upsides? Maybe corporate inertia? We don’t do anything special (mostly big CRUD, and reporting), but the database decision(s) got made over a decade ago and nobody seems to think evaluating the costs would be a worthwhile exercise (which is odd considering Oracle licensing costs…).

                                      I’m firmly convinced that lots (maybe even most) decisions about software on a executive level (especially outside of the valley) are based on shmoozing. The software that my company makes (SaaS) is mostly made on that level, but there’s a point in going after a customer where the salespeople give up and say “(CxO) is an SAP guy, so this has no chance.” Which has nothing to do with cost/features/support. So we use Oracle for the same sort of reason.

                                      C’est la vie

                                      1. 1

                                        This is maybe the only time I’ve ever seen people who actually use Oracle databases pop up in a thread.

                                        I really think this is just a subculture issue. MS SQL is a bit similar, but one of the advantages of that solution is that MS is relatively more open. My (conspiracy) theory is that Oracle consultants keep knowledge secret to increase revenues ;)

                                        But there’s always a cut-off point where the maintenance costs exceed the costs of moving to another solution. I think Oracle has a stable base of satisfied customers, but the growth potential is low - hence buying Sun etc.

                                2. 5

                                  which I remember thinking as unnecessary since Postgres improves upon it by a big margin

                                  I keep seeing people say stuff like this and it drives me bonkers. Yes, PostgreSQL has lots of convenient features, data types, and functions. But SQL Server’s query execution engine is much faster than PostgreSQL’s, and it’s query optimizer blows PostgreSQL’s out of the water. Yes, PostgreSQL has improved performance a lot in 9.x, 10, and 11. No, it’s still not anywhere close to SQL Server, especially for hugely complicated queries.

                                  It’s like saying C++ is unnecessary since Python improves on it by a big margin. Good for you if Python meets your performance and scale requirements, but that doesn’t make C++ unnecessary.

                                  1. 6

                                    sophisticated query optimizer

                                    That’s nice and all, but have you heard about our lord and saviour, clustered indexes on schema-bound views!? They’re the one MSSQL feature that I can now barely imagine living without. (Tiny inconvenience: they very-often aren’t used without a WITH (NOEXPAND) hint but oh well no biggie.) All whole bunch of aggregations suddenly magically become possible to make O(1-ish) per row selected, at the cost of adding O(1-ish) cost to data modifications.

                                    1. 1

                                      Yeah, I mean, people who have used Postgres – people who understand how much better it is at the basic job of preserving data, when compared to MySQL – often have never used a serious commercial database.

                                      ETA: But that said, Postgres is a wonderful tool, and significantly more comfortable for developers.

                                      1. 1

                                        I must admit I spoke without being specific on which points postgres (which from my point of view are mainly richer features) improves upon, nevertheless I trust your experience on this topic (since mine hasn’t being enough with SQL Server to speak properly, much of what I know about RDBMS is from Postgres, Oracle and Sybase [poor me]) and thanks for pointing this out. Where is a good source of unbiased information I can learn more about what you’ve mentioned here?

                                        1. 4

                                          Unfortunately, there aren’t really any good sources of information, or any way to acquire this knowledge other than industry experience.

                                          Trouble is, database vendors have every incentive to avoid publishing benchmarks or other performance data. It’s difficult to get right, usually easy to see where they’ve biased the sample scenario in their favor, it almost always appears juvenile, and some salty web hipster will get a gajillion upvotes on HN for vilifying the publication regardless of whether said web hipster has the slightest clue what they’re talking about.

                                          The best you can do is read vendor documentation that explains how capitalize on certain optimizations, or avoid certain caveats. If you really know what you’re looking at and can read between the lines, you can figure out who does what well. Even then, lots of the juiciest details are undocumented and given to customers on a need-to-know basis.

                                          I’ve seen SQL Server generate incredible plans that I couldn’t even begin to explain how it knew to choose. And I’ve seen PostgreSQL throw it’s hands up in the air and hash join several full table scans when a simple lateral join obviously would work better. It just isn’t a competition.

                                          Good commercial databases cost buckets of money for a reason. When you need that kind of power, there usually isn’t any other way to get it.

                                          1. 1

                                            any other way to get it

                                            Pay someone to improve Postgres for you? :)

                                            1. 1

                                              acquire this knowledge other than industry experience

                                              It seems to me like an appropriate scientific and fair method to know the internals and the advantages of each engine. Thanks for your objectivity.

                                          2. 1

                                            Many of my database-related tasks have more to do with SQL ergonomics than with performance. I currently work using Oracle 11 and I miss so many things that are natural using Postgres. They include SELECT FROM VALUES (instead of silly SELECT FROM dual), array types, jsonb_agg, SELECT a>b and probably some other features (I’m also missing LIMIT OFFSET, but newer Oracle has FETCH FIRST, so it doesn’t really count as an advantage over Oracle). Sure Oracle has great features that Postgres lacks, but those are what I would use every day if I could.

                                            1. 1

                                              [MSSQL’s] query optimizer blows PostgreSQL’s out of the water

                                              Is this recent? I’ve DBA’d postgres, and have (distant past) MSSQL experience, but back then, postgres’ query optimizer seemed far more advanced. I seem to recall MSSQL’s bad query planning being one of the reasons the stored proc became such an important part of MSSQL culture.

                                            2. 3

                                              That surprising cuz we did get a lot of submissions, some authored by @craigkerstiens. I thought we were one of goto sources for Citus news and write-ups.

                                            1. 4

                                              As a dyed-in-the-wool rustacean, I love it. “No don’t Rewrite It In Rust. It already works, what’s the gain? Now if you’re writing something new, maybe consider Rust as an option…”

                                              1. 9

                                                I’ll give this presentation as a counterpoint showing how they’re constantly adding more bugs with both new code and changes to existing code. We need to make sure these people making these changes are using a language immune-by-default to common problems they can’t seem to avoid. So, that justifies gradually rewriting existing code.

                                                Getting them doing it for new code first is a smart strategy, though. They’ll also see the benefits of the safe language as they deal with problems in modifications to unsafe, lower-level code more than modifications to safe, higher-level code.

                                                1. 3

                                                  That’s a very good point and a convincing presentation. That said, Rust and other safe-by-default languages are not a panacea. I would expect this to be especially true in things like operating systems where you will have a fair amount of unsafe code floating around, trying to have safe semantics while often not quite managing it. That said, while OS dev in Rust is definitely of interest to me it’s not something I have dug into too much yet, so I shouldn’t try to extrapolate too much.

                                                  1. 3

                                                    Good point. I’ll note that a conversion with unsafe’s means it will still be safer with the unsafes being, at worst, no less safe than the original. However, unsafe Rust still has some safety over unsafe C. The unsafe parts might improve. Finally, there are automated techniques for verifying a lot of those unsafe components that can be applied. It will be more feasible to do it once the part that needs such verification is a tiny part of the codebase.

                                                    So, there’s still some potential to reduce risk even in presence of unsafes. The unsafes will still have risk, though. Far as OS dev, you might find the book on embedded Rust useful since it targets lots of low-level interactions. That community is both trying to do them and figure out ways to leverage the type system when doing them.

                                                  2. 2

                                                    Is there any evidence that people are adding more bugs than people rewriting things in Rust are? Rust isn’t immune to bugs, far from it, nor is it immune to security bugs. Everyone knows rewriting code will result in introducing some bugs. Can you really be confident they’re fewer and less severe than the bugs already there? Can you be confident they won’t just reintroduce the same bugs into the Rust version?

                                                    1. 1

                                                      Yeah. Rust is immune by default to problems many codebases keep having. Many lead to code injection whereas Rust’s just lead to panics or something. The default going from hackers controlling our boxes to applications crashing would be an improvement.

                                                      I’ll also note that the temporal errors the borrow checker catches are the source of many heisenbugs. Those are errors that are just hard to find or reproduce. Even OpenBSD had a bunch of them despite their attention to code quality. So, a language and/or tooling that prevents them makes more sense than trying to hunt for them.

                                                      1. 2

                                                        It’s just not true that Rust is immune to security bugs. Isolating a particular set of security bugs and claiming they’re worse, conveniently also being the ones that Rust can’t have (if you follow a long set of restrictions that nobody follows, like not using unsafe code)? I think that’s intellectually dishonest.

                                                        Rust fixes some things, sure, but there are lots of issues it doesn’t fix and cannot fix, and many of them are just as bad or worse than the issues it does fix.

                                                        1. 1

                                                          “It’s just not true that Rust is immune to security bugs.”

                                                          You’re being intellectually dishonest by misquoting me, setting up a strawman like that, and knocking it down. I’ll restate what I said so you can reply to that instead:

                                                          “Rust is immune by default to problems many codebases keep having. Many lead to code injection whereas Rust’s just lead to panics or something”

                                                          I didn’t say all security problems: just many that are common. Like described here, Safe Rust blocks spatial errors (i.e. memory-safe) and common types of temporal errors (eg null dereference, use-after-free, some races). These are blocked by design where the compiler either adds checks or forces code to be structured to make detection automatic. C allows these problems by default. Most vulnerabilities people find are these kinds of vulnerabilities. Most of the really, clever attacks start with one of them before building a chain. So, making a language immune to the specific classes of vulnerability that turn up all the time in C code will reduce those classes of vulnerability. That’s what you need to argue against.

                                                          The next claim I make is there will be reliability and security failures left due to stuff the type system can’t cover. The benefit a safe, system language retains is you can spend your bug hunting time on those other things. You don’t have to check the code for the same stuff that keeps getting people in C. I’ll add that arrays, stacks, and so on are common primitives that people have to use constantly. Whereas, the esoteric errors will be in less common code. It’s easier to find them when one has more time with less code to look at. So, making the majority of code safe even helps one potentially catch the other errors for those reasons. Obviously, we’ll also develop more checkers for stuff like that on the side.

                                                          1. 2

                                                            Rust is immune to an arbitrary subset of security issues. We both agree with that.

                                                            What I am saying is that taking that fairly arbitrary subset and suggesting they’re the more important issues is just not true. ‘The issues that Rust happens to prevent’ is actually characterised by anything other than that Rust happens to prevent them.

                                                            The other thing I’m saying is that there’s no evidence, as far as I am aware, that taking away those issues reduces the overall prevalence of security issues in software, or reduces their severity. Maybe it does? But I haven’t seen any evidence of it. And even if such evidence exists, is there evidence that Rust is the best way of achieving that reduction? Could the same reduction in security issues be achieved by doing something much simpler and smaller like standardising some safer string and buffer operations and types in the next C standard and promoting their use?

                                                            1. 1

                                                              “arbitrary subset of security issues”

                                                              We do not agree on that. Rust followed the path of many safe languages to look at most common failures to block them first. You saying arbitrary implies it’s as if they picked stuff at random with unknown effects on the code out there. No, they picked memory-safety and temporal errors that were hitting people constantly, including experts at secure coding. The errors that are in CVE’s with code injections all the time. That’s not arbitrary: it’s evidence-based mitigation focusing on stopping the most attacks with the least language or security features.

                                                              “The other thing I’m saying is that there’s no evidence, as far as I am aware, that taking away those issues reduces the overall prevalence of security issues in software, or reduces their severity.”

                                                              Most of the reported vulnerabilities that lead to code injection are due to unsafe languages having no mitigations. Rust mitigates those by design. That’s evidence it reduces overall, code injections. I’ll add I keep mentioning code injections since a hacker taking over your box in secret is much worse than it crashing, optionally telling you where it crashed. Both Ada and Rust prioritized stopping the most common bugs and severe outcomes.

                                                              1. 2

                                                                We do not agree on that. Rust followed the path of many safe languages to look at most common failures to block them first.

                                                                That’s literally just not true. That’s not what the design process for Rust looked like.

                                                                Most of the reported vulnerabilities that lead to code injection are due to unsafe languages having no mitigations. Rust mitigates those by design. That’s evidence it reduces overall, code injections.

                                                                Again, what Rust mitigates by design is not a special class of security issue to anyone except Rust advocates who like to pretend that ‘memory safety’ is a special class of security issue that far surpasses any other.

                                                                A hacker taking over the box so they can DDOS someone is a far less serious security issue than personal data of customers being leaked, IMO. But because Rust allegedly prevents one and not the other, Rust advocates reorient their world view around the former being qualitatively worse.

                                                    2. 2

                                                      We need to make sure these people making these changes are using a language immune-by-default to common problems they can’t seem to avoid.

                                                      That sounds scary, to be honest.

                                                      1. 6

                                                        I bet. Even more scary to me was a cliche that said we’ll keep programming like we only have PDP-11’s no matter what changes or happens. Lots of damage followed that mindset. Whereas, hardly any damage followed alternative practice of giving people stuff that was safe-by-default. So, I’m less scared about pushing it given the better results.

                                                        1. -1

                                                          There are no positive results from attempts to do back-seat driving.

                                                          1. 1

                                                            You mean seatbelt laws and regulations improving crash safety? Definitely been positive results from those.

                                                            Then again for software in DO-178B cuz it mandates quality with high penalties for mistakes. Folks naturally started using better tooling.

                                                            1. 2

                                                              I mean you telling people to write operating systems in your language of choice.

                                                              1. 5

                                                                Really? The topic is about whether it’s worthwhile to write operating systems in Rust, and you’re trying to call someone out for saying “yeah, it makes sense to write operating systems in Rust”? When he’s even got a reasonable supporting argument?

                                                                1. 2

                                                                  The topic is about whether it’s worthwhile to write operating systems in Rust, and you’re trying to call someone out for saying “yeah, it makes sense to write operating systems in Rust”?

                                                                  No, I never objected to writing new software (specifically operating systems) in Rust.

                                                                2. 1

                                                                  Maybe. Let’s judge different contexts:

                                                                  1. If it’s a company paying for it, they can dictate the language to developers. We saw this with Microsoft’s .NET snd Sun’s Java. It’s happening selectively at Mozilla with Rust and some companies who do Ada/SPARK (eg for Muen).

                                                                  2. If it’s FOSS and mostly paid developers (eg Linux), they might be able to get more code in a specific language if offering to match rewrites or threaten to pull support. This has lower odds of success than No 1.

                                                                  3. If it’s FOSS and done by volunteers, then odds of pushing a switch is close to nothing. In that case, the route would be to fork it rewriting legacy and recently-added code incrementally.

                                                      2. 8

                                                        As a Rustacean, I agree with you and with Bryan. The economics of taking an existing piece of software that works and rewriting it in Rust aren’t good and the second system syndrome might make Rust look like a terrible choice. Rather, we should include Rust in the pool of options when building new software, and not be zealous when it turns out that Python is a better choice.

                                                      1. 1

                                                        My website (static, Hakyll), and my git site (gogs) (and its associated postgresql database, local access only, occasionally used for exploratory dev). I have plans to get some indieweb stuff up and running, but those plans haven’t actually happened in all the years I’ve had them.

                                                        I host my own SparkleShare. I keep a bunch of stuff in there, including my KeePass file.

                                                        All of the above is on a small virtual server with Rimu.

                                                        I’m setting up a FreeNAS within the next two months (probably just buying a FreeNAS mini), and will be hosting a media server there (probably Plex, since it will work with my Roku). That won’t be externally accessible, but it’s still a thing.

                                                        I hosted my own email for well over a decade, but holy cow, that’s a pain in the ass. Moving to runbox was absolutely worth it.

                                                        Any or all of the above may have been obfuscated to avoid utility for red teams.

                                                        1. 2

                                                          This piece is from 2013. Is anyone a regular Office 365 user today? The web-based Word seems like a different animal from the native application.

                                                          1. 5

                                                            I use o365 for work. Prior to that, we used google apps in the same company.

                                                            The local feedback seems to be, essentially: people who are used to MS on the desktop loved moving to o365, since it implemented all the niche corner cases for everything the way they were used to. Everybody else strongly disliked it. It’s slow to load, poorly responsive at first and intermittently, crashes more often, and doesn’t integrate multi-user editing nearly as well.

                                                            So, pertinent to this article, it’s probably safe to say that Word 365 fulfills the author’s claim “To parse a .doc file you virtually have to write a mini-implementation of Microsoft Word”. It’s full-featured in exactly the ways the author dislikes.

                                                            1. 3

                                                              We use O365 at my current job and I don’t hate it. The worst part is definitely Skype for Business.

                                                              1. 1

                                                                Not any more, but used it at my previous employer.

                                                                I don’t know exactly what the author’s complaining about (get to the point, already), but standalone office is better than Google Docs, Office 365 and OpenOffice in every conceivable way, IMO. My only complaint is that it had to run in Wine on Linux, but it was even worth doing that instead of using OpenOffice.

                                                                Office365 is clunky and slow in comparison and 90% of the features are missing.

                                                                The author mentions a few times that Word is no good for writing books, but there are dedicated tools for doing that, so why doesn’t he use one?

                                                                1. 1

                                                                  He does. See the last paragraph.

                                                              1. 7

                                                                I agree /u/feoh that “ethical” is a bad descriptor, but I’ll go so far as to say it doesn’t even work for comparing centralized to distributed.

                                                                The reason that Facebook and Twitter are unethical has to do with what Facebook and Twitter do with data, and with their users. Going distributed doesn’t eliminate that threat. Unless you’re running your own Mastodon instance, your admins could be engaging in similar unethical behavior.

                                                                And if you run your own instance, and host for your friends, you might engage in similar unethical behavior.

                                                                That’s true even if “privacy” is your #1 ethical concern.

                                                                Frankly, I also take issue with the “easy-to-use” label for the distributed services.

                                                                This could all be fixed by removing “ethical” and replacing it with more concrete descriptors, like “distributed”, or “non-ad-driven”, or “privacy-aware”. You wouldn’t even have to change the url.

                                                                1. 6

                                                                  I’m surprised nobody has mentioned Leo. It’s a tree-based text/code editor. The tree is very flexible, and leaf nodes contain code/text/whatnot. You can even reference leafs from more than one place.

                                                                  It’s not code-specific. That is, you can use it for any text-based programming language (though there will be a step to export linear-text files), or you can use it for documentation, or todo lists, or whatever. It’s very flexible.

                                                                  I used it a long time back to do some literate programming. It’s evolved quite a bit since then.

                                                                  1. 2

                                                                    Leo is maybe more an outliner than a tree editor – the nuance being that a tree editor should have a notion of schema, while an outliner can be more freeform. They’re definitely two branches of the same family tree, but I feel there are unique challenges when interactively editing structured data that needs to (eventually) conform to a given schema.

                                                                    1. 1

                                                                      That looks quite neat; reminds me of a similar terminal program called hnb, although hnb seems more minimal and more targeted towards personal information management. Leo seems much more poweful and worth exploring, thanks for bringing it to my attention!

                                                                    1. 11

                                                                      How do you figure something like this out, given you would ideally also like to align that with the path you want your career to take?

                                                                      Learn what’s interesting to you, and your career will end up bending towards that.

                                                                      1. 2

                                                                        I think there is a larger question. What if one stops being passionate about programming (though still enjoy doing it as requested at work)?

                                                                        1. 1

                                                                          I don’t think this advice requires you code 24/7 for fun, just that you have some preferences.

                                                                          1. 1

                                                                            Then learn to knit, or build model trains, or do political organizing, or whatever. You’re going to build skills whatever you learn, and your character will slowly change, too. Maybe those skills will be broadly translatable to your career (planning/estimation, design/architecture, communication/leadership), maybe not.

                                                                            But being happy in your non-work life will definitely impact your career. And new interests will, if nothing else, lead to new perspectives on both your existing problems and on future possible career paths.

                                                                          2. 1

                                                                            That also works in reverse: Avoid learning something where you don’t want your career to go. At least be quiet about it.

                                                                          1. 4

                                                                            Counterpoint: your software is never correct.

                                                                            Unless your software is:

                                                                            • proven using a formal prover (as /u/kghose points out)
                                                                            • written in a language that itself has a formal, proven, specification (very few languages, though SML is one of them)
                                                                            • built using a compiler that complies with that specification in all cases
                                                                            • run only on hardware with a proven specification (compile/build stage)
                                                                            • … and whose interactions with the language and compiler are properly documented and proven
                                                                            • in a physical environment which conforms at all times to the hardware’s requirements
                                                                            • with the end product run only on hardware with a proven specification
                                                                            • …..

                                                                            … then it is impossible for what you build to be correct.

                                                                            1. 4

                                                                              And even if you achieve all that, you only know that your software correctly implements the specification. There’s still plenty of room for the specification to be a poor solution to the original problem, etc..

                                                                              1. 3

                                                                                proven using a formal prover

                                                                                Not just that, but the implementation of the formal prover has to be proven, as well

                                                                                And then the machine code of the implementation of the formal prover has to be proven that it matches the implementation.

                                                                                And then the hardware it runs on has to be trusted to execute those in the right sequence, etc.

                                                                                And (as you said) you need to ensure the hardware isn’t defective or has surpassed tolerance limits…

                                                                                1. 1

                                                                                  Milawa and HOL Light on VAMP. Initial implementation and checks done on 8-bitter that is itself brute-forced and hand-checked. Can do the initial checkers in it. Verification done on untrusted, high-speed hardware that hash result. They feed the trusted hardware the proof steps which it checks a bit at a time slowly producing its own hash that must be equivalent.

                                                                                  The gate and below, esp analog-level stuff, is typically verified empirically with simulation. There is also work verifying some stuff like that. Maybe it be included in future. Easier if one uses oldest, process node possible so less complexity in meeting design rules constraints.

                                                                                  So, it’s all doable with existing tech in verification with gates and below being cutoff for now.

                                                                              1. 7

                                                                                I feel like the author’s primary complaint is against process. Jira is a tool. It’s flexible enough to implement many different processes. You don’t have to have mini-waterfalls. A ticket could have statuses of “To-do”, “In Progress” and “Done” and “In Progress” can be anything you want. Pair programming, test driven, etc. Or you can get detailed and enumerate all the specific steps.

                                                                                No tool replaces having a good, clear process that people actually stick to.

                                                                                I will agree that Jira is more focused to lower level, detailed tasking and higher level project goals can benefit from a different tool.

                                                                                1. 2

                                                                                  Yeah, this seems to be written by someone largely unfamiliar with Jira. Or familiar with only its use in a single company where it’s poorly managed.

                                                                                  I agree that you shouldn’t use Jira to define your process, but instead to describe and document your process. But I also think that Jira works fine with high-level project goals. Having a ticket type for “Project” or “Epic” that can be used for high-level prioritization seems to work reasonably well where I’ve seen it done. And you can even use (perhaps to the author’s shock) prose to describe the project and its status. Actually attaching work tickets to those project tickets is not required by Jira in any way.

                                                                                  Basically, tickets types can scale up and down, and have different workflows per ticket type (and even per Jira project). If you approach it like a generalized tracking tool, then it’s capable of modelling a huge number of processes, large and small.

                                                                                1. 8

                                                                                  yet in many respects, it is the most modern database management system there is

                                                                                  It’s not though. No disrespect to PostgreSQL, but it just isn’t. In the world of free and open source databases it’s quite advanced, but commercial databases blow it out of the water.

                                                                                  PostgreSQL shines by providing high quality implementations of relatively modest features, not highly advanced state of the art database tech. And it really does have loads of useful features, the author has only touched on a small fraction of them. Almost all those features exist in some other system. But not necessarily one single neatly integrated system.

                                                                                  PostgreSQL isn’t great because it’s the most advanced database, it’s great because if you don’t need anything state of the art or extremely specialized, you can just use PostgreSQL for everything and it’ll do a solid job.

                                                                                  1. 13

                                                                                    but commercial databases blow it out of the water

                                                                                    Can you provide some specific examples?

                                                                                    1. 16

                                                                                      Oracle has RAC, which is a basic install step for any Oracle DBA. Most Postgres users can’t implement something similar, and those that can appreciate it’s a significant undertaking that will lock you into a specific workflow so get it right.

                                                                                      Oracle and MS-SQL also have clustered indexes. Not what Postgres has, but where updates are clustered as well. Getting Pg to perform sensibly in this situation is so painful, it’s worth spending a few grand to simply not worry about it.

                                                                                      Ever run Postgres on a machine with over 100 cores? It’s not much faster than 2 cores without a lot of planning and partitioning, and even then, it’s got nothing on Oracle and MS-SQL: Open checkbook and it’s faster might sound like a lose, but programmers and sysadmins cost money too! Having them research how to get your “free” database to perform like a proper database isn’t cost effective for a lot of people.

                                                                                      How about big tables. Try to update just one column, and Postgres still copies the whole row. Madness. This turns something that’s got to be a 100GB of IO into 10s of TBs of IO. Restructuring this into separate partitions would’ve been the smart thing to do if you’d remembered to do it a few months ago, but this is a surprise coming from commercial databases which haven’t had this problem for twenty years. Seriously! And don’t even try to VACUUM anything.

                                                                                      MS-SQL also has some really great tools. Visual Studio actually understands the database, and its role in development and release. You can point it at two tables and it can build ALTER statements for you and help script up migrations that you can package up. Your autocomplete can recognise what version you’re pointing at. And so on.

                                                                                      …and so on, and so on…

                                                                                      1. 3

                                                                                        Thanks for the detailed response. Not everyone has money to throw at a “real” enterprise DB solution, but (having never worked with Oracle and having only administered small MSSQL setups) I did wonder what some of the specific benefits that make a DBA’s life easier were.

                                                                                        Of course, lots of the open source tools used for web development and such these days seem to prefer Postgres (and sometimes MySQL), and developers like Postgres’ APIs. With postgres-compatible databases like EnterpriseDB and redshift out there, my guess is we’ll see a Postgres-compatible Oracle offering at some point.

                                                                                        1. 7

                                                                                          Not everyone has money to throw at a “real” enterprise DB solution

                                                                                          I work for a commercial database company, so I expect I see a lot more company-databases than you and most other crustaceans: Most companies have a strong preference to rely on an expert who will give them a fixed cost (even if it’s “money”) to implement their database, instead of trying to hire and build a team to do it open-source. Because it’s cheaper. Usually a lot cheaper.

                                                                                          Part of the reason why: An expert can give them an SLA and has PI insurance, and the solution generally includes all costs. Building a engineering+sysadmin team is a big unknown for every company, and they usually need some kind of business analyst too (often a contractor anyway; more £££) to get the right schemas figured out.

                                                                                          Professional opinion: Business logic may actually be some of the least logical stuff in the world.

                                                                                          lots of the open source tools used for web development and such these days seem to prefer Postgres

                                                                                          This is true, and if you’re building an application, I’d say Postgres wins big. Optimising queries for dbmail’s postgres queries was hands down much easier than any other database (including commercial ones!).

                                                                                          But databases are used for a lot more than just applications, and companies who use databases don’t always (or even often) build all (or even much) of the software that interacts with the database. This should not be surprising.

                                                                                          With postgres-compatible databases like EnterpriseDB and redshift out there, my guess is we’ll see a Postgres-compatible Oracle offering at some point.

                                                                                          I’m not sure I disagree, but I don’t think this is a good thing. EnterpriseDB isn’t Postgres. Neither is redshift. Queries that work fine in a local Pg installation run like shit in redshift, and queries that are built for EnterpriseDB won’t work at all if you ever try and leave. These kinds of “hybrid open source” offerings are an anathema, often sold below a sustainable price (and much less than what a proper expert would charge), leaving uncertainty in the SLA, and with none of the benefits of owning your own stack that doing it on plain postgres would give you. I just don’t see the point.

                                                                                          1. 3

                                                                                            Professional opinion: Business logic may actually be some of the least logical stuff in the world.

                                                                                            No kidding. Nice summary also.

                                                                                            1. 0

                                                                                              Queries that work fine in a local Pg installation run like shit in redshift

                                                                                              Not necessarily true, when building your redshift schema you optimize for certain queries (like your old pg queries).

                                                                                          2. 4

                                                                                            And yet the cost of putting your data into a proprietary database format is enough to make people find other solutions when limitations are reached.

                                                                                            Don’t forget great database conversion stories like WI Circuit Courts system or Yandex where the conversion to Postgres from proprietary databases saved millions of dollars and improved performance…

                                                                                            1. 2

                                                                                              Links to those stories?

                                                                                              1. 1

                                                                                                That Yandex can implement clickhouse doesn’t mean everyone else can (or should). How many $100k developers do they employ to save a few $10k database cores?

                                                                                                1. 2

                                                                                                  ClickHouse has nothing to do with Postgres, it’s a custom column oriented database for analytics. Yandex Mail actually migrated to Postgres. Just Postgres.

                                                                                              2. 2

                                                                                                You’re right about RAC but over last couple of major releases Postgres has gotten alot better about using multiple cores and modifying big tables. Maybe not at the Oracle level yet bit its catching up quickly in my opinion.

                                                                                                1. 3

                                                                                                  Not Oracle-related, but a friend of mine tried to replace a disk-based kdb+ with Postgres, and it was something like 1000x slower. This isn’t even a RAC situation, this is one kdb+ core, versus a 32-core server with Postgresql on it (no failover even!).

                                                                                                  Postgres is getting better. It may even be closing the gap. But gosh, what a gap…

                                                                                                  1. 1

                                                                                                    Not to be that guy, but when tossing around claims of 1000x, please back that up with actual data/blogpost or something..

                                                                                                    1. 6

                                                                                                      You remember Mark’s benchmarks.

                                                                                                      kdb doing 0.051sec what postgres was taking 152sec to complete.

                                                                                                      1000x is nothing.

                                                                                                      Nobody should be surprised by that. It just means you’re asking the computer to do the wrong thing.

                                                                                                      Btw, starting a sentence with “not to be that guy” means you’re that guy. There’s a completely normal way to express curiosity in what my friend was doing (he’s also on lobsters), or to start a conversation about why it was so much easier to get right in kdb+. Both could be interesting, but I don’t owe you anything, and you owe me an apology.

                                                                                                      1. 2

                                                                                                        Thanks for sharing the source, that helps in understanding.

                                                                                                        That’s a benchmark comparing a server grade setup vs essentially laptop grade hardware (quad-core i5), running the default configuration right out of the sample file from the Git repo, with a query that reads a single small column out of a very wide dataset without using an index. I don’t doubt these numbers, but they aren’t terribly exciting/relevant to compare.

                                                                                                        Also, there was no disrespect intended, not being a native english speaker I may have come off clumsy though.

                                                                                                        1. 1

                                                                                                          kdb doing 0.051sec what postgres was taking 152sec to complete.

                                                                                                          That benchmarks summary points to https://tech.marksblogg.com/billion-nyc-taxi-rides-postgresql.html which was testing first a pre-9.6 master and then a PG 9.5 with cstore_fdw. Seems to me that neither was fair and I’d like to do it myself, but I don’t have the resources.

                                                                                                          1. 1

                                                                                                            If you think a substantially different disk layout of Pg, and/or substantially different queries would be more appropriate, I think I’d find that interesting.

                                                                                                            I wouldn’t like to see a tuning exercise including a post-query exercise looking for the best indexes to install for these queries though: The real world rarely has an opportunity to do that outside of applications (i.e. Enterprise).

                                                                                                      2. 1

                                                                                                        Isn’t kdb+ really good at stuff that postgres (and other RDBMS) is bad at? So not that surprising.

                                                                                                        1. 1

                                                                                                          Sort of? Kdb+ isn’t a big program, and most of what it does is the sort of thing you’d do in C anyway (if you liked writing databases in C): Got some tall skinny table? Try mmaping as much as possible. That’s basically what kdb does.

                                                                                                          What was surprising was just how difficult it was to get that in Pg. I think we expected, with more cores and more disks it’d be fast enough? But this was pretty demoralising! I think the fantasy was that by switching the application to Postgres it’d be possible to get access to the Pg tooling (which is much bigger than kdb!), and we massively underestimated how expensive Pg is/can be.

                                                                                                          1. 3

                                                                                                            Kdb+ isn’t a big program, and most of what it does is the sort of thing you’d do in C anyway (if you liked writing databases in C)

                                                                                                            Well, kdb+ is columnar, which is pretty different than how most people approach naive database implementation. That makes it very good for some things, but really rough for others. Notably, columnar storage is doesn’t deal with update statements very well at all (to the degree that some columnar DBs simply don’t allow them).

                                                                                                            Even on reads, though, I’ve definitely seen postgres beat it on a queries that work better on a row-based system.

                                                                                                            But, yes, if your primary use cases favor a columnar approach, kdb+ will outperform vanilla postgres (as will monetdb, clickhouse, and wrappers around parquet files).

                                                                                                            You can get the best of both worlds You can get decent chunks of both worlds by using either the cstore_fdw or imcs extensions to postgres.

                                                                                                            1. 1

                                                                                                              which is pretty different than how most people approach naive database implementation.

                                                                                                              I blame foolish CS professors emphasising linked lists and binary trees.

                                                                                                              If you simply count cycles, it’s exactly how you should approach database implementation.

                                                                                                              Notably, columnar storage is doesn’t deal with update statements very well at all (to the degree that some columnar DBs simply don’t allow them).

                                                                                                              So I haven’t done that kind of UPDATE in any production work, but I also don’t need it: Every customer always wants an audit trail which means my database builds are INSERT+some materialised view, so that’s exactly what kdb+ does. If you can build the view fast enough, you don’t need UPDATE.

                                                                                                              Even on reads, though, I’ve definitely seen postgres beat it on a queries that work better on a row-based system.

                                                                                                              If I have data that I need horizontal grabs from, I arrange it that way in memory. I don’t make my life harder by putting it on the disk in the wrong shape, and if I do run into an application like that, I don’t think gosh using postgres would really speed this part up.

                                                                                                  2. 3

                                                                                                    Spanner provides globally consistent transactions even across multiple data centers.

                                                                                                    Disclosure: I work for Google. I am speaking only for myself in this matter and my views do not represent the views of Google. I have tried my best to make this description factually accurate. It’s a short description because doing that is hard. The disclosure is long because disclaimers are easier to write than useful information is. ;)

                                                                                                    1. 2

                                                                                                      @geocar covered most of what I wanted to say. I also have worked for a commercial database company, and same as @geocar I expect I have seen a lot more database use cases deployed at various companies.

                                                                                                      The opinions stated here are my own, not those of my former or current company.

                                                                                                      To put it bluntly, if you’re building a Rails app, PostgreSQL is a solid choice. But if you’ve just bought a petabyte of PCIe SSDs for your 2000 core rack of servers, you might want to buy a commercial database that’s a bit more heavy duty.

                                                                                                      I worked at MemSQL, and nearly every deployment I worked with would have murdered PostgreSQL on performance requirements alone. Compared to PostgreSQL, MemSQL has more advanced query planning, query execution, replication, data storage, and so on and so forth. It has state of the art features like Pipelines. It has crucial-at-scale features like Workload Profiling. MemSQL’s competitors obviously have their own distinguishing features and qualities that make them worth money. @geocar mentioned some.

                                                                                                      PostgreSQL works great at smaller scale. It has loads useful features for small scale application development. The original post talks about how Arcentry uses NOTIFY to great effect, facilitating their realtime collaboration functionality. This already tells us something about their scale: PostgreSQL uses a fairly heavyweight process-per-connection model, meaning they can’t have a huge number of concurrent connections participating in this notification layer. We can conclude Arcentry deployments using this strategy probably don’t have a massive number of concurrent users. Thus they probably don’t need a state of the art commercial database.

                                                                                                      There are great counterexamples where specific applications need to scale in a very particular way, and some clever engineers made a free database work for them. One of my favorites is Expensify running 4 million queries per second on SQLite. SQLite can only perform nested loop joins using 1 index per table, making it a non-starter for applications that require any kind of sophisticated queries. But if you think about Expensify, its workload is mostly point look ups and simple joins on single indexes. Perfect for SQLite!

                                                                                                      1. 1

                                                                                                        But MemSQL is a distributed in-memory database? Aren’t you comparing apples and oranges?

                                                                                                        I also highly recommend reading the post about Expensify usage of SQLite: it’s a great example of thinking out of the box.

                                                                                                        1. 1

                                                                                                          No. The author’s claims “Postgres might just be the most advanced database yet.” MemSQL is a database. If you think they’re apples and oranges different, might that be because MemSQL is substantially more advanced? And I used MemSQL as one example of a commercial database. For a more apples-to-apples comparison, I also think MSSQL more advanced than PostgreSQL, which geocar covered.

                                                                                                          And MemSQL’s in-memory rowstore serves the same purpose as PostgreSQL’s native storage format. It stores rows. It’s persistent. It’s transactional. It’s indexed. It does all the same things PostgreSQL does.

                                                                                                          And MemSQL isn’t only in-memory, it also has an advanced on-disk column store.

                                                                                                  1. 2

                                                                                                    As a sysadmin I prefer MySQL since the high availability is way easier to setup. Either master master or master slave, and if more than two nodes, galera.

                                                                                                    Postgres is a box of Lego, some tools (barman, repmgr, pgpool, pgbouncer) and wal replication, but nothing cohesive or easy “just works”.

                                                                                                    Cannot say anything about its capabilities as a database but my developers like it, and as long as they’re happy I’m happy.

                                                                                                    1. 9

                                                                                                      As a DBA, I prefer postgres, because postgres hasn’t ever lost data for me, while mysql has. As a dev, I prefer postgres, since it has far better support for advanced datatypes and advanced querying.

                                                                                                      But, yes, MySQL is still slightly easier to set up out of the box, especially in HA setups.

                                                                                                      1. 0

                                                                                                        As someone who cares about the consistency and availability of their data, I wouldn’t wish MySQL on a dog I don’t like.

                                                                                                      1. 2

                                                                                                        Questions 5 and 11 both seem pretty much in line with my experience with programming languages.

                                                                                                        I only know python, or for that matter, had any interest in learning python, because of work. Outside of that I probably still wouldn’t know it; not well enough to write anything signifiant in it. For better or worse, python doesn’t really “do it” for me. I know it and write it purely to get paid. Don’t get me wrong, I quite frankly really enjoy my job, and I don’t have any regrets in learning python, but in a perfect world I wouldn’t have picked it.

                                                                                                        These days, it really takes something special for a language to peek my interest enough to learn purely for the pleasure of it. A case-and-point for me. I originally wrote balistica in Vala. Looking back on it, I think that was the wrong choice, and I would choose several other languages instead if I had to do it all over again. After going through that, picking which language I devote my free time to is a much stricter process.

                                                                                                        Long story short: I don’t know Haskell, is there a convincing argument to devoting my free time to learn it over say Lisp, Scheme, or Ocaml? The three languages that top my “want to learn” list presently.

                                                                                                        1. 5

                                                                                                          If you like stretching your brain to find new tools to solve problems, Haskell is good for that.

                                                                                                          You’ll get some of those same tools from lisp / scheme / ocaml, but they allow mutation where Haskell does not.

                                                                                                          Learn what you like, each new language makes it easier to learn more! Try Haskell sometime, there are lots of cool tools to discover.

                                                                                                          1. 3

                                                                                                            I’d recommend Haskell if you’d like to learn a functional language with an advanced type system, although OCaml is also interesting in that regard. For Lisp family I’d also recommend considering Clojure as it’s one of the more practical Lisps and actually gets use in the industry.

                                                                                                            1. 3

                                                                                                              Learning Lisp or Scheme will teach you different things than learning Haskell or an ML.

                                                                                                              Lisp and Scheme (and Clojure, etc) tend to have fairly weak type systems, very simple syntax, and (because of the latter) very robust macro support. For many people, writing a lisp program means, essentially, writing non-working pseudocode, then writing the macros to make that pseudocode work.

                                                                                                              Haskel, Ocaml, and SML are all strongly and robustly typed. Writing programs in these languages often means figuring out data structures first, then writing functions to manipulate them. Of these, Haskell is a pure language – you’re only writing functions which return values, never functions that manipulate global state. Haskell is also a lazy language, which is something you can get away with not thinking about, until it produces a result that makes you think.

                                                                                                              Haskell has the virtue of a surprisingly rich library ecosystem, but the language itself can also be quite complex. SML, on the other hand, has very few libraries available, but the language is simple enough to learn comprehensively quite quickly. Ocaml splits the difference between those two extremes.

                                                                                                              … Which is all a roundabout way of saying the only convincing argument in any direction is going to be from you, about what you’re interested in learning next.

                                                                                                            1. 2

                                                                                                              Any chance that means CentOS 8 is getting on the schedule?

                                                                                                              1. 9

                                                                                                                Disclaimer: I work alongside Kevin, who helped create the Commons Clause. I was not personally involved in the creation of the Commons Clause.

                                                                                                                I agree with the premise of this article, which is that projects licensed under the Commons Clause are not open source, and to call them open source is misleading.

                                                                                                                The subtext of this article seems to be “the Commons Clause is bad”, which I’m less sure I agree with:


                                                                                                                1. Commons Clause makes it possible to publish source code that would have originally been closed source.

                                                                                                                I’ve mentioned this before. Most of this code would have originally been closed source so the authors can make money off of it. The Commons Clause lets them still make money while making the program and source code available for as many users as possible.

                                                                                                                I think this is a net improvement.

                                                                                                                2. Using an open core model with existing OSI-approved licenses is unpleasant.

                                                                                                                Software developers must feed themselves. They must get paid somehow. This is a hard requirement.

                                                                                                                Open core has historically worked well for open source infrastructure companies at scale (e.g. Hashicorp, Docker, GitLab, Redis). These companies provide immense value for the software community by doing sustainable open source development. (The only successful large company I’m aware of that used the paid support model was Red Hat.)

                                                                                                                If you actually talk to many of the teams working on open source infrastructure, they’re frustrated by existing licensing tools. There is a big difference between the value that these teams generate and the value that they’re able to capture. A big contributor to this is the fact that service providers like AWS run hosted versions of open source software and capture disproportionately more value than they generate for the community. This is a strong disincentive to provide open core infrastructure.

                                                                                                                Permissive licenses like Apache allow this to happen. Highly restrictive licenses like AGPL are too broad and scare away proprietary users who would not be hurting the team’s sustainability. In practice, open source teams do not have an option for saying “I would like to be as permissive as possible, but still make money so I can fund sustainable development”.

                                                                                                                3. “Free software” and “open source” as concepts are more nuanced than the GNU and OSI definitions.

                                                                                                                The GNU and OSI definitions are very narrow, and generate a lot of confusion. This is exacerbated when people use “free software” and “open source” (as opposed to e.g. “Free Software” / “free-as-in-freedom” / “GNU free software” or “Open Source” / “OSI open source”) to refer to the GNU and OSI definitions.

                                                                                                                There are immediate layperson meanings for these terms: “software that I didn’t have to pay for” and “software where I can view the source code”.

                                                                                                                The terms “non-free” and “non-commercial” can mean too many things, and to use them to describe the Commons Clause is misleading:

                                                                                                                1. Often times, Commons Clause projects are provided at no cost.
                                                                                                                2. The entire point of the Commons Clause is to allow as much commercial usage as possible, and to only forbid the strictest possible subset of usage (reselling the software) to enable sustainable development.

                                                                                                                In practice, there are gradients of freedom. To say that software that is not strictly GNU Free Software is not free or projects that are not strictly OSI Open Source are not open source is misleading to the everyday user.

                                                                                                                I would love to see GNU and the OSI step up here with better education and clearer definitions around this.

                                                                                                                4. By insisting on their definitions of “free software” and “open source” and failing to recognise this nuance, GNU and the OSI are missing the forest for the trees and will ultimately push more projects towards closed source models.

                                                                                                                By winning this battle, they are losing the meta-game.

                                                                                                                At the end of the day, what matters is not whether there are more OSI Open Source projects, but how much value we provide to the community. When open source advocates say that Commons Clause projects are “not open source” (in the OSI sense) and discourage its use, they are correct but harmful.

                                                                                                                By taking away this middle ground, they push projects to be either free-as-in-freedom or proprietary. Given the pains of sustainable infrastructure development using OSI-approved licenses, I fear that this will ultimately drive projects that would rather be licensed under the Commons Clause (i.e. as permissive as possible) into becoming closed source and proprietary.


                                                                                                                I don’t believe that the Commons Clause is the best solution, but I believe that it’s important to start a discussion around:

                                                                                                                1. How can we best address the need for sustainable open source development?
                                                                                                                2. Should we create new licensing tools to enable sustainable development? If so, how?

                                                                                                                I think the fact that the Commons Clause has been able to get adoption is a strong signal that there is a real pain here (see also: MongoDB and its SSPL).

                                                                                                                @SirCmpwn, I would love to hear your thoughts on a better way to enable sustainable development for open source infrastructure.

                                                                                                                1. 5

                                                                                                                  Software developers must feed themselves. They must get paid somehow. This is a hard requirement.

                                                                                                                  In practice, open source teams do not have an option for saying “I would like to be as permissive as possible, but still make money so I can fund sustainable development”.

                                                                                                                  Redis Labs only became the official sponsor of Redis in 2015, four years after they were founded as a company. So if they weren’t the primary sponsor for four years, how did sustainable development get funded? Well, until Redis Labs hired him, antirez fed himself with a job at VMware. VMware and Pivotal sponsored Redis and funded that sustainable development.

                                                                                                                  Alright, so how do other developers feed themselves? Is antirez the exception? Did Gudio van Rossum work for some Python-centric company focused on support? Nope! He worked at research institutions, then Zope, then Elemental Security, then Google, then DropBox.

                                                                                                                  The creators of Apache Kafka worked at LinkedIn when they initially developed it. Three years after LinkedIn open sourced it, some of the contributors from within LinkedIn founded their own company to focus on Kafka. Until that point, those devs fed themselves with a job at LinkedIn.

                                                                                                                  Linus Torvalds fed himself with a job at Transmeta, then a series of non-profits that eventually evolved into the Linux Foundation (also a non-profit). Many kernel contributors are employed by hardware companies like AMD and Intel.

                                                                                                                  Developers must feed themselves, but they can totally do that by working at a company that doesn’t revolve around the software they write as its sole product.

                                                                                                                  I would argue that it’s more sustainable that way. Companies like Red Hat, Redis Labs, Canonical, Confluent, and a whole bunch more exist because companies want to buy a bunch of integration work ready-to-go rather than spending huge sums of money doing it themselves. If Canonical went bust tomorrow (as many Linux vendors before it have!) Linux would still get developed. Would Landscape, their closed-source management tool? Probably not. The open source component is the sustainer that allows new companies to spawn and die without fear of losing the core.

                                                                                                                  1. 3

                                                                                                                    You’re giving the exceptions to the rule, not the rule. The rule is most aren’t funded. The responses were really negative, too, with lots of stress and burnout. Whereas, the proprietary software that has users gets money automatically with it going up with product development and/or marketing. FOSS developers trying to catch up to that inherent advantage finding funding sources can itself seem like a 3rd job on top of their 2nd job of making the FOSS for free.

                                                                                                                    1. 2

                                                                                                                      You’re giving the exceptions to the rule, not the rule.

                                                                                                                      Do non-FOSS applications succeed at a higher rate than FOSS ones?

                                                                                                                      The rule is most aren’t funded.

                                                                                                                      Ok. Here’s the end of that article:

                                                                                                                      “P.P.S. [edit 1/18/16] The response to this post has been incredible. I’ve started a list of OSS projects needing support here. Please add your great examples there!”

                                                                                                                      That leads to a git repo that’s no longer maintained. Uh oh. But wait! There’s a link that leads to libraries.io with the tagline “Tidelift has over $1 million dollars available to pay open source maintainers. If you’re a maintainer: Find your package.” Here’s some info from Tidelift.

                                                                                                                      The responses were really negative, too, with lots of stress and burnout.

                                                                                                                      Do me a favor and read them again, only this time imagine they’re working at a Fortune 100 on a closed-source project. How many of those would sound out of place through that lens?

                                                                                                                      I have to say, if you asked me about the projects Ive been paid to work on full time (mostly internal stuff at various companies), I would have largely the same complaints at one time or another, including the one about “It is not fair to expect me to do even more work outside of my regular work, and then not get fairly compensated (time or money) for it.”

                                                                                                                      By the way, here’s what they’re doing now, 3 years on. EDIT: I just realized that some of these are highlighting users on here! I’d love to hear what you have to say now!

                                                                                                                      @pydanny, still going strong on GitHub and contributing to open source, in addition to writing books on Python.

                                                                                                                      @ryanbigg, quit maintaining his open source projects (at the time of the article) and handed some off to other teams and developers. Wrote several books on open source projects, continues to work on the i18n ruby gem.

                                                                                                                      @drmaciver, still working on Hypothesis and doing consulting and training around it.

                                                                                                                      @andrewgodwin, still a Django core member, employed by evenbrite.

                                                                                                                      @kantrn, continues to contribute to open source, did consulting. I didn’t see his employment listed on Twitter, but I found him on LinkedIn. He’s got a job as a principal engineer now.

                                                                                                                      @shazow, working on blockchain tech with his own project, Vipnode. Appears to be working full time on open source via grants and partnerships?

                                                                                                                      Even the person who said “I do not have the time or energy to invest in open source any more” is still contributing to open source. Some have started contributing to open source full time. Others are gainfully employed and either working on them as part of their core work.

                                                                                                                      Whereas, the proprietary software that has users gets money automatically with it going up with product development and/or marketing.

                                                                                                                      dwarffortress, WinRAR, WinAmp, Sublime text editor all release free versions hoping for donations or as a free trial. It is not automatic for proprietary software unless that software is so completely necessary that they can just bill you to even have a peek at it. Hell, even Windows offers a free trial despite decades of a virtual stranglehold on the market for desktop OSs.

                                                                                                                      FOSS developers trying to catch up to that inherent advantage finding funding sources can itself seem like a 3rd job on top of their 2nd job of making the FOSS for free.

                                                                                                                      See above for why it’s not an automatic inherent advantage. Also, that presumes the development needs external funding rather than external contributors who are already getting paid. I’d much rather convince my employer to let me contribute to open source for my regular salary and have other projects to work on for if the open source one goes bust or implodes or hires a toxic developer who everyone hates or I run out of things to do on it or I need a break from the grind.

                                                                                                                      1. 3

                                                                                                                        By the way, here’s what they’re doing now, 3 years on. EDIT: I just realized that some of these are highlighting users on here! I’d love to hear what you have to say now!

                                                                                                                        […]

                                                                                                                        @shazow, working on blockchain tech with his own project, Vipnode. Appears to be working full time on open source via grants and partnerships?

                                                                                                                        Can confirm. Typically I do consulting work for part of the year to make up for the year’s expenses, then I do self-guided OSS stuff the rest of the year. Vipnode turned out to be an exception as it was a self-guided open source project that happened to resonate with a generous community that wanted to support my work financially. (And give or take some sales effort on my part.)

                                                                                                                        Overall, I stand by my original quote in the cited article:

                                                                                                                        Publishing and contributing to open source is going to continue happening regardless whether I’m getting paid for it or not, but it will be slow and unfocused. Which is fine, it’s how open source work has always worked. But it doesn’t need to be this way. — @shazow

                                                                                                                        1. 1

                                                                                                                          Thanks for following up! Would you mind sharing some of how you did the conversion to working (mostly) full time on OSS stuff?

                                                                                                                          edit: boy did that sentence get muddled in the middle. Fixed.

                                                                                                                          1. 3

                                                                                                                            Any specific questions?

                                                                                                                            I have a yearly minimum earned income goal (as I’m generally self-employed). If I reach that goal, then I allow myself to switch to not-necessarily-paid OSS stuff. That income comes from a variety of sources (I have some SaaS projects like Briefmetrics, some ad income from high-traffic free projects like Tweepsect), but the majority comes from contracts with clients.

                                                                                                                            This year, a big chunk came from an Ethereum Foundation grant and the partnership with Infura.

                                                                                                                            I’ve considered doing a Patreon and funneling supporters through that, but I’m not convinced that the expected value would exceed the relative effort and anxiety that would come with it. Still mulling it over.

                                                                                                                            When I do paid work, I certainly prefer open source work, bonus points if it’s things I’m interested in independently. I have some recurring clients who allow me to do this kind of work sometimes, which is great. Sometimes I’ll take less-desirable yet high-paying work to reach the minimum income goal quicker so I have more time to do better things.

                                                                                                                            1. 1

                                                                                                                              Awesome summary! That’s really exciting about the partnership and the grant!

                                                                                                                              Any specific questions?

                                                                                                                              SO many. I’ll try to keep it brief though.

                                                                                                                              Where was the tipping point where you were like “You know what? I can work for myself.”

                                                                                                                              How do you find paid work on projects? Do people reach out to you for help, or do you hunt for places to help? Something else?

                                                                                                                              Have you found yourself to be more productive on projects in general now that you self direct?

                                                                                                                              What are some things that employers could do to make life better for folks doing open source contributions?

                                                                                                                              What’s a normal work-week like for you? What kind of hours do you keep?

                                                                                                                              1. 3

                                                                                                                                Where was the tipping point where you were like “You know what? I can work for myself.”

                                                                                                                                I’ve been doing it on and off for a long time (since I was in my teens), so it wasn’t a super scary idea. It helps to be good with tracking your own finances. Figure out how much income you need to at least break even with expenses, and see if you can find contract work to cover it. Then dial that number up year over year so you build up savings.

                                                                                                                                Hint: It’s much easier when your expenses are low. I suggest not living in SF, or even in the US if you can help it (where health insurance is expensive and risky). That said, I primarily work remotely with US customers (SF/NY).

                                                                                                                                How do you find paid work on projects? Do people reach out to you for help, or do you hunt for places to help? Something else?

                                                                                                                                Recurring customers that you have a good relationship with are best, but those take time to cultivate. Small/medium well-funded companies have worked best for me, especially if you know people there who have a good idea of the kinds of help they might need with. Bonus points if they rely on your OSS work.

                                                                                                                                Have you found yourself to be more productive on projects in general now that you self direct?

                                                                                                                                In some ways. I certainly do more of the kinds of things I think I should be doing, and less doing things that I think are a waste of time.

                                                                                                                                It can be hard to stay motivated day to day, especially if you’re working solo it can get lonely. That’s generally true.

                                                                                                                                It’s common to have self-doubt, but it’s easier to power through it when you’re being paid. If you’re just burning your own savings for your own idea and you’re having self-doubt, it can be very hard to get over that hump. It’s hard learning to trust yourself, but maybe it gets easier with time.

                                                                                                                                What are some things that employers could do to make life better for folks doing open source contributions?

                                                                                                                                Encourage employees to spend work-hours time contributing back to projects they use and rely on at work. At least a day or two per week

                                                                                                                                An open source donation budget would also be nice, maybe as a team you can sit down once every couple of quarters and make a top 5-10 projects that the team wants to have an impact on, and do it (whether financially or through effort).

                                                                                                                                What’s a normal work-week like for you? What kind of hours do you keep?

                                                                                                                                When I draft contracting agreements, I tend to limit it to 3 full-time days/wk of commitment. I feel like I get 80-90% of a “40 hour workweek” productivity in 3 days, and it cuts down on a lot of other things. Clients aren’t keen on excessive meetings when the time available is constrained like this.

                                                                                                                                Personally, I’ve found Mon-Wed works best for me, and brief written status updates on Tuesdays. I don’t charge by the hour, but I do “productive hours” tracking internally for my own analysis. Days with 6 “productive hours” (ie. actual time spent writing code and such) are considered good days. Some days aren’t good days, and that’s okay. If I’m having a particularly off day, I’ll sometimes write it off and try again another day (like today… I’ll probably end up working tomorrow instead).

                                                                                                                                If I wrapped up my primary commitment for the week and I have days left over, I’ll switch over to other projects. Occasionally, due to the nature of contracting (it’s impossible to maintain an equilibrium of clients around your own schedule), I’ll have multiple concurrent clients, but I try to avoid that when possible.

                                                                                                                                1. 1

                                                                                                                                  This feels like it could be an entire medium article or something on its own.

                                                                                                                                  It can be hard to stay motivated day to day, especially if you’re working solo it can get lonely. That’s generally true.

                                                                                                                                  We have a bunch of employees that are 100% remote work, and I’ve noticed that some will take advantage of video chat while others generally seem to just want to be left the hell alone, with little variation in their routines.

                                                                                                                                  I don’t charge by the hour, but I do “productive hours” tracking internally for my own analysis.

                                                                                                                                  Do you have a particular tool you use for this? I’ve been meaning to do the same thing at work, but have always been kind of lazy about it because it requires so much extra effort to track by hand. I have noticed that on a “bad” day I can generally tell how much distraction I’ve had by the number of times I’ve had to pause my music :)

                                                                                                                                  1. 1

                                                                                                                                    This feels like it could be an entire medium article or something on its own.

                                                                                                                                    What’s a good spicy title for it? :)

                                                                                                                                    Do you have a particular tool you use for this?

                                                                                                                                    Not really. Right now I just log it with Harvest which I also use for invoicing. My friend really likes Timing2 for macOS, but I’m on Linux these days. I’ve considered using something like Thyme but it does more than what I need.

                                                                                                                                    Really I should just write my own basic timer that lives in the tray and a quick start/stop shortcut.

                                                                                                                                    I have noticed that on a “bad” day I can generally tell how much distraction I’ve had by the number of times I’ve had to pause my music :)

                                                                                                                                    That is a good one!

                                                                                                                                    I often try to keep a policy to stop music anytime I’m not doing “work”, just to condition myself and avoid getting lost in distraction.

                                                                                                                                    1. 1

                                                                                                                                      What’s a good spicy title for it? :)

                                                                                                                                      Maybe:

                                                                                                                                      Abort to Orbit: How I went independent and what I do when I need more.

                                                                                                                                      The concept of Abort to Orbit is that you launch your ship and then if the crap hits the fan, you’ve got a still-relatively-good fallback option. More importantly, you look at possible failure modes and address them rather than letting them hold you back.

                                                                                                                                      I’ve considered using something like Thyme but it does more than what I need.

                                                                                                                                      Oooh, that looks very promising indeed.

                                                                                                                                      I often try to keep a policy to stop music anytime I’m not doing “work”, just to condition myself and avoid getting lost in distraction.

                                                                                                                                      I follow the rule of “if they have headphones on, leave them the hell alone.” Exceptions include the building being on fire or prod issues. I’m trying to encourage my colleagues to do the same. It’s… hit and miss.

                                                                                                                        2. 1

                                                                                                                          Re success rate. Yes, non-FOSS succeeds in getting their developers paid at a higher rate despite failure rate for new companies being high. They actually charge for software. So, they win by default.

                                                                                                                          Re $1 million. So, FOSS developers have to divide up a million while proprietary developers get a slice of tens of billions in recurring profit. Your example argues my point more than yours.

                                                                                                                          Re working big companies instead. Many already work for them before doing the FOSS on the side. You’re in luck though since Ive surveyed thousands of people on this issue when working general public. Most gripe, shrug, or roll their eyes before saying “at east Im working” (low pay) or “but it pays well” (high pay). The percentage with terrible jobs can attempt to switch companies. So, they’re not a counter to my position but I decided to answer anyway.

                                                                                                                          Re anecdotes. Good they’re still contributing and/or employed. Remember my model has them working one job in regular hours paid for their work. In most examples I see, many still do FOSS as extra thing they’re not paid for. And often for lower amounts than proprietary sector pulls in.

                                                                                                                          Re trials. That’s a marketing technique to generate sales. Nothing in my post precudes loss leaders if necessary. The Prosper license, a candidate for shared source, allows a 90-day evauation for free.

                                                                                                                          Re last stuff. My post was about people that would like to get paid for their contributions not working extra hours on them. There’s people like you and maybe those you mentioned that want something different. Ive even known people, been one, to turn down money to keep their intentions purely altruistic or just separate fun from business. My comments may not apply to you or them at least for selfish gain and/or employment.

                                                                                                                          All that said, you still havent refuted biggest point that money dictates laws governing software freedom, FOSS-lovers are losing ground due to lobbying, the need a fortune to counter it, and current models paying a pittance guarantees we loose more freedom over time. I gave examples involving money being seized via patent trolling. If they want to get dirty, copyright law has criminal penalties, too. Current or future expansions might use it. Who knows what they’ll do in future but we have less if staying on same trajectory. Doing nothing in this case is still doing something (for opponents).

                                                                                                                          1. 1

                                                                                                                            Re success rate. Yes, non-FOSS succeeds in getting their developers paid at a higher rate despite failure rate for new companies being high. They actually charge for software. So, they win by default.

                                                                                                                            Do you have any stats to back this statement?

                                                                                                                            Re $1 million. So, FOSS developers have to divide up a million while proprietary developers get a slice of tens of billions in recurring profit. Your example argues my point more than yours.

                                                                                                                            If FOSS developers had to divide up $1 million amongst all the projects, it might argue your point. However, since as I pointed out in my other examples of companies that sponsor particular projects or employe developers directly to work on FOSS projects and companies based on FOSS Projects, this represents an additional $1 million to try to fill in the edges rather than core development. It’s disingenuous to represent that $1 million as the sole source of money for FOSS developers when Red Hat alone has $4bn in assets.

                                                                                                                            Re working big companies instead. Many already work for them before doing the FOSS on the side. You’re in luck though since Ive surveyed thousands of people on this issue when working general public. Most gripe, shrug, or roll their eyes before saying “at east Im working” (low pay) or “but it pays well” (high pay). The percentage with terrible jobs can attempt to switch companies. So, they’re not a counter to my position but I decided to answer anyway.

                                                                                                                            Can we see those survey results?

                                                                                                                            Re anecdotes. Good they’re still contributing and/or employed. Remember my model has them working one job in regular hours paid for their work. In most examples I see, many still do FOSS as extra thing they’re not paid for. And often for lower amounts than proprietary sector pulls in.

                                                                                                                            You said the rule was that they’re not funded, and cited a series of anecdotes. OK, pop quiz: how many of the anecdotes you cited resulted in people contributing to FOSS as their full time job rather than as an extra? Does writing books about FOSS count?

                                                                                                                            Re trials. That’s a marketing technique to generate sales. Nothing in my post precudes loss leaders if necessary. The Prosper license, a candidate for shared source, allows a 90-day evauation for free.

                                                                                                                            You said they get money automatically, which is demonstrably not true if they do a 90-day evaluation that doesn’t result in purchase. While it doesn’t preclude loss-leaders, it does refute your statement that they automatically get money.

                                                                                                                            Re last stuff. My post was about people that would like to get paid for their contributions not working extra hours on them. There’s people like you and maybe those you mentioned that want something different. Ive even known people, been one, to turn down money to keep their intentions purely altruistic or just separate fun from business. My comments may not apply to you or them at least for selfish gain and/or employment.

                                                                                                                            Fair enough, but your post claimed that “most” don’t get funded without any backing evidence, proprietary software “automatically” gets money without any backing evidence, and then made assertions about FOSS developers (without bothering to qualify it like “some FOSS developers” or even “most FOSS developers”) trying to “catch up” might have an experience like working a 2nd or 3rd job. You started with broad generalizations and suddenly want to add disclaimers?

                                                                                                                            All that said, you still havent refuted biggest point that money dictates laws governing software freedom, FOSS-lovers are losing ground due to lobbying, the need a fortune to counter it, and current models paying a pittance guarantees we loose more freedom over time. I gave examples involving money being seized via patent trolling. If they want to get dirty, copyright law has criminal penalties, too. Current or future expansions might use it. Who knows what they’ll do in future but we have less if staying on same trajectory. Doing nothing in this case is still doing something (for opponents).

                                                                                                                            The comment I responded to, in it’s entirety, is the following:

                                                                                                                            You’re giving the exceptions to the rule, not the rule. The rule is most aren’t funded. The responses were really negative, too, with lots of stress and burnout. Whereas, the proprietary software that has users gets money automatically with it going up with product development and/or marketing. FOSS developers trying to catch up to that inherent advantage finding funding sources can itself seem like a 3rd job on top of their 2nd job of making the FOSS for free.

                                                                                                                            Please show me where you made that biggest point or gave any of the examples you just claimed in the comment I replied to.

                                                                                                                      2. 3

                                                                                                                        Developers must feed themselves, but they can totally do that by working at a company that doesn’t revolve around the software they write as its sole product.

                                                                                                                        This stance ensures that FLOSS contributors will only ever be people who have abundant energy and free time outside of their work week. That excludes a bunch of, e.g., parents, people with multiple jobs, and people with medical issues.

                                                                                                                        1. 1

                                                                                                                          My point was that they can be employed by a company to do that contribution as a function of their job even if the company itself isn’t focused on that particular project. Redis exists because of this model. Ditto for Kafka. If you work a full time job doing open source for a company that uses that project and have a second job doing other stuff, how would that interfere with the open source contribution?

                                                                                                                          1. 2

                                                                                                                            Then that’s a chicken/egg issue. How does one get a job doing open source contribution (of which there are very, very few) without having open source contributions on their resume?

                                                                                                                            If you’re saying simply that companies should all contribute to open source, remember that running an open source project is significantly more work than simply writing software. That’s a large extra investment over what they’re doing now, and doesn’t have a clear ROI.

                                                                                                                        2. 3

                                                                                                                          I think the wording of the questions in the parent post is misleading, though probably not intentionally so. If the question was really, “How can we develop open source software sustainably?”, then as you have described, one answer is simply, “the same way we have for the last few decades”. However, I suspect the question they really want to ask is something along the lines of, “How can we fit open source development into a for-profit business in a capitalist economy?”. I do not think this is a bad question to ask, as the motivation for asking it is almost certainly a desire to promote and improve open source software full time without starving, but it is not the same as asking how we can develop open source software sustainably.

                                                                                                                          Once the questions are rephrased in this way, it’s a lot less alarming: The issue is no longer dressed up as a desperate attempt to save free/open source software, it’s just some people looking for business opportunities.

                                                                                                                          If you can run a successful business centred around free software development, that’s great. If you can’t, then FLOSS software development will continue regardless. I personally do not believe it is worth the risk of watering down the ideals of software freedom by implying, even if only by the names, that a free software license + commons clause is somehow still free.

                                                                                                                          As far as the “Commons clause” is concerned, I see two possibilities

                                                                                                                          1. It’s just ‘openwashing’. I.e. pretending to promote software freedom without actually making free software.
                                                                                                                          2. It’s genuinely trying to find some middle ground where software that would otherwise be closed source can be “slightly freer” without radically changing the owner’s business model.

                                                                                                                          Personally, I am opposed to 1. and have no interest in 2., but maybe option 2. might benefit some people in some way.

                                                                                                                          1. 3

                                                                                                                            This is essentially the “real musicians have day jobs” argument, and I find it very weak. There are a number of things wrong with it. The “day job” distracts from the work of open source maintenance and can make it much harder to do deep work. Only the top of the organization can really get jobs of this nature, it can’t fund helpers. It perpetuates the traditional problems in open source of making it easier to concentrate on technology as opposed to product management. The funding organization has too much control over the direction of the project, as opposed to the users who derive the most value from it.

                                                                                                                            But yes, in the current landscape, it’s (sadly) one of the more viable approaches.

                                                                                                                            1. 1

                                                                                                                              This is essentially the “real musicians have day jobs” argument

                                                                                                                              I disagree. The “real musicians have day jobs” argument is that because they can’t get a job doing the music they love, they do something else, right? Even if it’s still in music, they don’t do their main wish.

                                                                                                                              My argument is that not all projects require a company dedicated to that project to succeed. In many cases success is found in projects that have a variety of contributors. Some of them are paid to work full time on the project by a company that - while not primarily focused on the software - still want or need to invest in its development. Examples of this are AMD and Intel engineers working on the Linux Kernel, or how Kafka and Redis were initially developed before the contributors either formed or joined a company dedicated to the project.

                                                                                                                              Those companies provided incubation for projects that weren’t their core business model, but were useful nonetheless. Paying people to work full time on something you use - but can’t or don’t want to sell - is something many companies do already, but they don’t share the code. By open sourcing the project, they can get contributions from other companies with full time engineers that would use their code, or attract new talent from folks who want to work on open source but can’t because their current employer won’t allow it.

                                                                                                                              1. 3

                                                                                                                                Then we agree to disagree. Again to use my own example, I think it’s likely that working on xi-editor fulltime would likely be the largest value I could create right now. I had a sweet gig very much like you describe, working for a big company where they let me put some of my time into it. But the goals of that project weren’t precisely aligned, and I found my concentration pretty fragmented.

                                                                                                                                If I could find a sustainable way to work on xi-editor fulltime, I probably would. My decision to work on the synthesizer game is partly because it’s fun and exciting, and partly because, if it flies, it’s a nice revenue stream that will let me work sustainably.

                                                                                                                                This is my experience, I don’t speak for anyone else.

                                                                                                                                1. 1

                                                                                                                                  Your example is a good one, and I totally get that disagreement. Your point is well taken. Additionally, the idea that the goals weren’t aligned is a pitfall that I imagine has lead to many a fork over the years in various projects :)

                                                                                                                                  If you do find a way to work on it full-time and still want to, I absolutely want to revisit!

                                                                                                                                2. 1

                                                                                                                                  Examples of this are AMD and Intel engineers working on the Linux Kernel, or how Kafka and Redis were initially developed before the contributors either formed or joined a company dedicated to the project.

                                                                                                                                  The examples you keep giving are rare. Most FOSS doesn’t get funding or dedicated employees by big companies. Most FOSS with lots of users doesn’t get AMD/Intel-type investments. If we’re talking general case, it’s more accurate to assume what happens in the general case now will happen in general case for other people unless incentives and environment driving it changes. They haven’t changed. If anything, companies appear to be using FOSS in SaaS dodging the need to contribute many changes back more often than before.

                                                                                                                                  And your only reply from people you quoted was a person that doing FOSS at a loss using revenue from other work. Like a 2nd or extra job worked for free while proprietary vendors of similar libraries get paid. Just like I said most would be doing if it’s FOSS.

                                                                                                                                  1. 1

                                                                                                                                    The examples you keep giving are rare. Most FOSS doesn’t get funding or dedicated employees by big companies. Most FOSS with lots of users doesn’t get AMD/Intel-type investments. If we’re talking general case, it’s more accurate to assume what happens in the general case now will happen in general case for other people unless incentives and environment driving it changes. They haven’t changed. If anything, companies appear to be using FOSS in SaaS dodging the need to contribute many changes back more often than before.

                                                                                                                                    The question is are they rarer than closed-source software getting that kind of investment? You haven’t addressed that statement beyond your assertion that it is fact despite an utter lack of backing evidence.

                                                                                                                                    And your only reply from people you quoted was a person that doing FOSS at a loss using revenue from other work. Like a 2nd or extra job worked for free while proprietary vendors of similar libraries get paid. Just like I said most would be doing if it’s FOSS.

                                                                                                                                    People you cited, I quoted. I extracted their quotes from the thing that you specifically cited as an example of work not getting funded, from an article that had at the core of its premise “FOSS doesn’t get funded.” I specifically cited the examples you gave claiming they contradicted my point.

                                                                                                                                    Even then, in the territory of something you cited as directly opposing my view, the response here was not a slam dunk for you point. On the contrary, you claim that the person is doing FOSS at a loss using revenue from other work despite them clearly stating that was not the case this year where a substantial chunk of their income came from exactly the funding you say is rare.

                                                                                                                                    You can still hide behind the fact that it’s an exception if you want, but it doesn’t prove that it’s any different in the proprietary world. You keep asserting it’s true without the backing. Show me the stats to compare the two and maybe we can draw a conclusion.

                                                                                                                            2. 2

                                                                                                                              Hi liftM, thanks for taking the time to write up your thoughts in such detail.

                                                                                                                              I agree with the premise of this article, which is that projects licensed under the Commons Clause are not open source, and to call them open source is misleading.

                                                                                                                              You’re right. This article is talking about two things:

                                                                                                                              • Using the Commons Clause and calling that software open source is dishonest and unethical
                                                                                                                              • The Commons Clause is bad

                                                                                                                              The reason these aren’t separate articles is:

                                                                                                                              • The two are related, because I think that the design of the Commons Clause actively encourages its users to lie
                                                                                                                              • I’m sick of writing articles about the Commons Clause.

                                                                                                                              Let me explain more about why I dislike the Commons Clause before I answer your questions directly.

                                                                                                                              First, I don’t think that FOSS is the end-all-be-all of software licensing. Like I said in the article, anyone is welcome to license their software any way they choose. I’m also a fan of source-available models as an alternative to closed-source software, I think it’s strictly better than closed-source. However, the Commons Clause is not a good way to license source-available software.

                                                                                                                              The Commons Clause should have been a new source-available license entirely. As an addendum to FOSS licenses, it’s totally bunk. The entire point of FOSS is to enshrine basic protections for the authors, users, and contributors. By removing these basic protections, the original license effectively becomes meaningless. It’s like a Jenga tower, with the removal of some freedoms it has a cascading effect which removes many of the others. Restricting commercial use has many grave side-effects: reusing the code elsewhere, forking the project, providing support after the maintainers abandon it, etc - all of this is now difficult-to-impossible, or at least egregiously under-specified by the Commons Clause.

                                                                                                                              Additionally, I take offense with the Commons Clause’s stated goal of turning previously open source projects into source-available projects. Though source-available is strictly better than closed source, it’s strictly worse than open source, and the change is a net loss for the commons - which is why I take offense with the name as well. It also fails to adequately explain the responsibilities maintainers hold to their contributors - without a CLA in place which assigns copyright to the maintainers, it’s in fact illegal to switch to the Commons Clause. Or at the least, the Commons Clause can only apply to new code, and the old license has to be kept in place and in many cases (notably Apache 2.0) the differently-licensed changes must be explicitly disclosed.

                                                                                                                              Now, to answer your questions directly…

                                                                                                                              Commons Clause makes it possible to publish source code that would have originally been closed source.

                                                                                                                              No, the stated goal of the Commons Clause is to transition code which was already open source to source-available. The code wouldn’t have originally been closed source because it was originally open source.

                                                                                                                              The Commons Clause lets them still make money while making the program and source code available for as many users as possible.

                                                                                                                              I don’t think it does. Without the rights afforded to users by the Apache 2.0 license, why would users have the good faith desire to use your software? Additionally, with so many grey areas, such an amateurish license, and only one option for support, any company who uses this kind of software is nuts.

                                                                                                                              If you actually talk to many of the teams working on open source infrastructure

                                                                                                                              Like me

                                                                                                                              A big contributor to this is the fact that service providers like AWS run hosted versions of open source software and capture disproportionately more value than they generate for the community.

                                                                                                                              So make a better hosted version than AWS has. Boo-hoo.

                                                                                                                              Permissive licenses like Apache allow this to happen. Highly restrictive licenses like AGPL are too broad and scare away proprietary users who would not be hurting the team’s sustainability. In practice, open source teams do not have an option for saying “I would like to be as permissive as possible, but still make money so I can fund sustainable development”.

                                                                                                                              Dual-licensing has allows this to work for ages. Use AGPL with a paid BSD option. Also note that many organizations have been successful with paid support, or with something like Patreon. You can also do sponsored blog posts, put their logos at your conference… get creative with it. You don’t need to capture all of the value of your software. Let AWS have some of it. By making it open source, it’s no longer yours. If you can make a living income on your software, demanding more is just greed. This is part of the commitment you make to the public by making your software open source. Companies like AWS have a vested interest in making sure you have a sustainable income - it’s your job to point this out to them, not to pull the rug out from beneath them.

                                                                                                                              “Free software” and “open source” as concepts are more nuanced than the GNU and OSI definitions.

                                                                                                                              No, they’re not. There’s no point in arguing this because I’m not going to concede it. Free software is defined by the GNU free software definition. Open source is defined by the OSI open source definition. If you do something which doesn’t fit in those niches, call it something else.

                                                                                                                              I would love to see GNU and the OSI step up here with better education and clearer definitions around this.

                                                                                                                              They already have. You just don’t like their definitions. Everyone else does, and in language the consensus is correct. If you misuse these terms you are a liar.

                                                                                                                              By insisting on their definitions of “free software” and “open source” and failing to recognise this nuance, GNU and the OSI are missing the forest for the trees and will ultimately push more projects towards closed source models.

                                                                                                                              Maybe they are. But the Commons Clause is doing a piss-poor job in their stead.

                                                                                                                              1. 0

                                                                                                                                I’ve mentioned this before. Most of this code would have originally been closed source so the authors can make money off of it. The Commons Clause lets them still make money while making the program and source code available for as many users as possible.

                                                                                                                                No it wouldn’t. You’re relicensing free software as ‘commons clause’ proprietary software.

                                                                                                                                Using an open core model with existing OSI-approved licenses is unpleasant.

                                                                                                                                Nobody is saying you have to use OSI-approved licenses, and nobody is saying you aren’t allowed to license things as source-available. What you aren’t allowed to do is to call those source-available models ‘open source’. That’s misleading.

                                                                                                                                “Free software” and “open source” as concepts are more nuanced than the GNU and OSI definitions.

                                                                                                                                No they aren’t. Literally it’s as simple as ‘no they aren’t’. They just aren’t more nuanced than that. Those definitions are what those mean to literally the entire software industry and all software hobbyists. They’re universally agreed upon terms with precise technical definitions. Muddying the waters then claiming that the waters are muddy is very poor behaviour.

                                                                                                                                ‘Software I don’t have to pay for’ is called freeware. ‘Software where I can view the source code’ is called ‘source available software’. Laymen are completely irrelevant. The terms exist, they’ve existed for decades.

                                                                                                                                Often times, Commons Clause projects are provided at no cost.

                                                                                                                                Commons Clause is a proprietary software license. It does not give users the four freedoms, so it is a non-free software license. What it costs has nothing to do with whether it’s free. You know this, I know this, everyone knows this. This was all sorted out long ago.

                                                                                                                                In practice, there are gradients of freedom. To say that software that is not strictly GNU Free Software is not free or projects that are not strictly OSI Open Source are not open source is misleading to the everyday user.

                                                                                                                                No, you are being misleading. And you know it. That ‘open source’ means ‘open source’ and ‘free’ means ‘free’ is long established. Very long established. The words work, they’re good.

                                                                                                                                By winning this battle, they are losing the meta-game.

                                                                                                                                What an awful article. Someone just learnt what an analogy was, I guess, and decided to practice them. Unfortunately, it makes the article impossible to skim-read.

                                                                                                                                At the end of the day, what matters is not whether there are more OSI Open Source projects, but how much value we provide to the community. When open source advocates say that Commons Clause projects are “not open source” (in the OSI sense) and discourage its use, they are correct but harmful.

                                                                                                                                You acknowledge that they are correct, but before you’re saying it’s misleading. Sorry, but that’s a contradiction. And what is harmful is misleading people, which is what you are doing.

                                                                                                                                By taking away this middle ground, they push projects to be either free-as-in-freedom or proprietary.

                                                                                                                                Proprietary software is, by definition, software that is not free. So pushing projects to either be free or not to be free, to either not be proprietary or be proprietary, is doing nothing. Every project is either proprietary or not, just like every project either includes Java or doesn’t, and every project either is lead by one person or isn’t lead by one person.

                                                                                                                                Given the pains of sustainable infrastructure development using OSI-approved licenses, I fear that this will ultimately drive projects that would rather be licensed under the Commons Clause (i.e. as permissive as possible) into becoming closed source and proprietary.

                                                                                                                                There are no pains. Free software has existed for decades, and will continue to exist for decades to come. It works fine as a model, as it is. Sell support. Sell proprietary licenses. Or don’t sell anything, and get a job doing something else. The world doesn’t owe you a paycheque.

                                                                                                                                How can we best address the need for sustainable open source development?

                                                                                                                                For a start, you need to stop misleading people and muddying the waters. You need to come up with your own terminology that is not misleading, or adopt the existing terminology (‘source available’). Until you have done so, you are actively harming the software industry and software community.

                                                                                                                                @SirCmpwn, I would love to hear your thoughts on a better way to enable sustainable development for open source infrastructure.

                                                                                                                                I’m not SirCmpwn, but my thoughts are thus: it’s fine as it is.

                                                                                                                              1. 1

                                                                                                                                This all sounds very much like the BSD vs. GPL arguments of twenty years ago.

                                                                                                                                • “GPL isn’t free because it has limitations on use!”
                                                                                                                                • “GPL is free because it enforces freedom!”

                                                                                                                                Now it’s

                                                                                                                                • “Commons Clause isn’t free because it has limitations on use!”
                                                                                                                                • “Commons Clause is free because it prevents commercial hijacking!”

                                                                                                                                Honestly, I lean in favor of Commons Clause. It’s open source in the very common understanding of it, even if it doesn’t comply with the OSI definition. And it’s a useful experiment in the evolution of OSS licensing. I don’t know that it will survive past a few years, but I predict whatever succeeds it will not only last, but become a first-class licensing option.

                                                                                                                                1. 3

                                                                                                                                  I don’t think anyone should ever touch a GUI framework that doesn’t come with a designer. I mean come on, we solved that problem in the 90s! Why are we designing GUIs via text files in 2018?

                                                                                                                                  1. 4

                                                                                                                                    Some people actually prefer to create GUIs in text files. This is because it’s possible to understand the structure of GUI in a better way. Also it allows to create more sophisticated UX than what’s possible with designers.

                                                                                                                                    The drawback is a lack of instant feedback on how the UI will actually look like. And learning curve is worse I suppose.

                                                                                                                                    So I’d say both of approaches have their advantages and disadvantages, and it’s not like one way is definitely better than the other way.

                                                                                                                                    1. 2

                                                                                                                                      This is because it’s possible to understand the structure of GUI in a betterdifferent way

                                                                                                                                      Corrected this for you.

                                                                                                                                    2. 2

                                                                                                                                      First you need to do it via text files before creating the GUI to manage that. Surely it’s a first step in a new combination of technologies to develop UIs. It’s not trivial to build a designer and shouldn’t be considered on the start. Imho.

                                                                                                                                      1. 3

                                                                                                                                        I think what @vegai meant is that they could have used an existing format like QML and parsed that. Right now, it seems you need to write code to lay out your controls which is bonkers indeed.

                                                                                                                                        1. 2

                                                                                                                                          Writing code to lay out controls is not bonkers at all. All the same abstraction tools you have for abstracting over the repeated bits when you write code normally are available when doing so. Expressing things in code makes sense. Rendering a list is just folding over that list, etc.

                                                                                                                                          1. 2

                                                                                                                                            All rendering engines need to have a programmatic solution before they can have a gui designer. Suggesting otherwise is ridiculous. Not only do you sometimes need to create user interfaces dynamically, how exactly is the magic XML going to turn into pixels?

                                                                                                                                      2. 2

                                                                                                                                        Every time I’ve looked at one of those ‘do it for me’ GUI tools for making GUI tools, it’s looked quite hard to use, and always produces absolutely awful code.

                                                                                                                                        1. 1

                                                                                                                                          I guess it’s probably because creating a designer is a huge investment, and that it’s not incompatible with writing GUI layouts in text files. This kind of project, if maintained and used in the future, might have one at some point!

                                                                                                                                        1. 1

                                                                                                                                          I loved BeOS, and I’ve followed the Haiku project for years. But many of the author’s points in favor of Haiku are less about the OS than about working with young and/or groups with small numbers of contributors.

                                                                                                                                          There’s nothing wrong with liking that, and certainly the Haiku community fits his description. I just can’t shake the feeling that, if you removed the GUI API and messaging references, this article could have been written about any small project that’s not dictatorial.

                                                                                                                                          1. 3

                                                                                                                                            I loved BeOS, and I’ve followed the Haiku project for years. But many of the author’s points in favor of Haiku are less about the OS than about working with young and/or groups with small numbers of contributors.

                                                                                                                                            We have around ~40-ish major code contributors between the core system and HaikuPorts, and most of them are not especially young. And if you include the community-at-large, there are over a few hundred people who show up on the forums, mailing lists, etc. So it’s not that we are especially small, though of course we are not nearly as large as even the BSDs…

                                                                                                                                            1. 5

                                                                                                                                              A few, hundred users sound small or tiny. Forty contributors would normally sound huge. This is a graphical, desktop OS, though. That still seems small number for such a huge goal. So, it’s fair to say it’s a small project with some dedicated people making steady progress on their goal. Hopefully, you get even more contributors since it’s a very, interesting project.

                                                                                                                                              1. 2

                                                                                                                                                Please take my comment as a criticism of the author, not any kind of critique of Haiku, its team, or its community.

                                                                                                                                                1. 1

                                                                                                                                                  I did; I was just expressing an anecdote which I think backs up the author’s point.