1. 2

    This is very old and somewhat outdated, I recommend a recent presentation as a complement to it: https://www.postgresql.eu/events/pgconfeu2018/sessions/session/2058/slides/96/hackingpg-present.pdf

    1. 1

      Greatly appreciated! I got the initial one from the PostgreSQL FAQ - must say the amount of documentation and presentations regarding developer on-boarding in the project is amazing.

    1. 3

      Package repositories really need to add processes for verifying that the code on GitHub matches the code on the package site. npm publish / etc on the developer’s machine shouldn’t be the primary way to publish packages, it should be the “untrusted” way. Telling the package repo to just take a git tag from the git repo should be the “most trusted” way, resulting in a big checkmark on the package page that links to the git tag. If the tag was PGP signed, enlarge the checkmark even more. The second best way — for packages that need processing (codegen/transpile/minify/etc) — would be to ensure the package was built on a public CI service. That would display a smaller checkmark.

      (obviously the UI indication should be better than the size of the checkmark lol)

      And yeah, of course this doesn’t address everything, but it fixes the problem of “everyone is way too lazy to check the actual files on the package repo, everyone will just browse on github” which is IMO a very important problem

      1. 2

        “everyone is way too lazy to check the actual files on the package repo, everyone will just browse on github”

        We actually vendor-ed all node dependencies at work and did my best attempt at auditing it with the team. That’s almost 3k packages clocking well over 30k files.

        We found 3 binaries. Two bundled with a package, one with sources in an external repo (not the package repo), the other without sources and one downloaded off the web doing install.

        Ended up designing a vendoring process for npm dependencies. It makes the life of front-end developers slightly harder but significantly improves the security of the product.

        1. 1

          Any chance of a writeup? I’ve got a few friends who would absolutely use such a thing (and me).

          1. 1

            Without going into too much details.

            1. Have your front-end team provide a package.json and a package-json.lock
            2. The person responsible for vendoring runs npm ci --ignore-scripts to reproduce the package state on a new machine (preferably a jail or a virtual machine). The –ignore-scripts part is really important to prevent automatic builds and additional binary downloads.
            3. On first pass identify suspect files like binaries, you can run file on all the files and see what type of files are there (ie. uniq the output). Keep in mind that file itself is unsafe so do it in a jail or vm.
            4. Remove not needed binaries (like the ones in term-size) or drop dependencies on packages you can’t verify the source of each binary for. Never vendor a binary even if you think you know where it’s sources are from.
            5. At this point audit the code. If you did this before you have a vcs diff which ‘should’ make it easier but since the directory tree inside npm is unstable that is unfortunately rarely the case.
            6. Now some packages need post processing with their scripts. Identify those packages and run npm run install just for them. For node-sass that would be cd ./node_modules/nose-sass && npm run install. However before doing that disable the network connection on the jail/vm. In the example of node-sass it defaults to downloading the binary off the internet. You can force it to always build without trying to download a binary by setting the environment variable SKIP_SASS_BINARY_DOWNLOAD_FOR_CI. This step should be automated in your CI build phase and you should not vendor the binary artifacts in your repository.

            That’s pretty much it.

      1. 3

        Slides for an upcoming PostgreSQL meetup (dtracing postgresql fast update pending list rebuilds, dtracing postgresql internals in general), I do the PostgreSQL part @oshogbo does the dtrace part.

        Updating laptops at home & probably updating some of my ports, perhaps wrapping a new one that I just did for work - pglast which is a python binding to lib pg_query which links to PostgreSQL parser internals to parse SQL into AST. Tons better than doing regexes or rolling your own parser :)

        Most likely a bit of gaming, reading and for sure having fun with the dog.

        1. 1

          I just had a similar need of accessing Dropbox (to see what I have there before I remove the account) and booted my old work laptop to do this.

          It’s encrypted and not using ext4… I however ended up picking a different solution that allowed me to still run their client. I simply made a new 10 GB file, formatted it as ext4 and mounted over loopback and pointed Dropbox at the mountpoint.

          Just takes a few steps:

          dd if=/dev/zero of=~/dropbox.img bs=1M count=10240
          mkfs.ext4 ~/dropbox.img
          sudo mount -o loop ~/dropbox.img ~/Dropbox
          sudo chown -R $USER:$USER ~/Dropbox
          

          and just start up Dropbox, the changes are transparent for it.

          1. 9

            It’s a very well written write up . I am however disappointed with no performance metrics - even seeing a comparison of storage used would be interesting.

            1. 2

              I assumed from the lack of writeup around this that the answer was probably along the lines of “no clearly interesting difference one way or the other”. The declared motivation for moving was operational smoothness rather than performance.

              1. 2

                no clearly interesting difference one way or the other

                That is an interesting information in itself for anyone planning a database migration. Especially if you consider that they most likely (or did they?) didn’t yet optimize their application and tune PostgreSQL for their workloads.

            1. 2

              This is worrying. Do I have to start looking at fastmail alternatives? Any recommendations?

              1. 2

                How do you think this will affect fastmail, specifically? They do not provide encrypted email, and already will comply with Australian government search orders.

                1. 2

                  I’m not an expert, but I don’t see how it would change anything for Fastmail, they already give the govt. anything they ask for(that’s legal).

                2. 1

                  There is kolabnow.com (Kolab hosted) also Migadu.com, all hosted in the nice European countries that tend to care about privacy. There are Hosting providers for Zimbra, Horde, etc out there as well.

                  1. 1

                    I stumbled upon proton mail the other day. Not tried it though: https://en.wikipedia.org/wiki/ProtonMail

                    If you like pain, you can host your own mail server.

                    1. 1

                      I stumbled upon proton mail the other day. Not tried it though: https://en.wikipedia.org/wiki/ProtonMail

                      caveat emptor regarding protonmail.

                      1. 1

                        Who or what is Duke-Cohan?

                        I’m a happy customer of ProtonMail, but I don’t like the pain of self-hosting unless it’s solely my own shit, and Google is a no-go zone.

                        1. 1

                          Oopsy!

                    1. 7

                      Rationale for my off topic flagging of this post:

                      should Apple or Google get to own the computing and communication infrastructure of the world, and be able to set the rules as they see fit?

                      This article essentially has no technical content. Nothing actionable or informative. It doesn’t even touch on self-hosting. Three paragraph of quoted text and two statements with an open ended question.

                      1. 5

                        I’ve been trying to get a patchfix into OpenBSD with no luck. No response to my patch on tech@openbsd.org. This isn’t the first time. Can any OpenBSD contributor help me out?

                        1. 7

                          If you didn’t get any feedback, just keep asking the list for feedback every two weeks by replying to your own post. There’s a bit of luck to it because each patch has to catch someone’s interest in a moment when they have time to deal with it.

                          1. 4

                            Cool I can do that, thanks for the tip.

                            1. -1

                              just keep asking the list for feedback every two weeks by replying to your own post.

                              What a ridiculous response. Not even an apology. That’s no way to run a welcoming community or encourage people to contribute.

                              1. 9

                                Nothing to apologize for - what did you expect? Sending reminders is a common idiom on tech@ where a mail gets drown easily by other threads.

                                Making sure your submissions are well tested and reasoned helps getting a response, but you cannot demand anything.

                                1. 1

                                  what did you expect?

                                  Maybe this is how OpenBSD runs things, if that’s the culture there, that’s fine, but don’t expect it to attract very many contributors.

                                  1. 5

                                    It does attract contributors. In fact, this culture is one of the reasons joined the project.

                                    So I eventually started reviewing the diff but failed to do so because it was both malformed (did not apply) and broken (did not compile). That is, instead of focusing on the intented changes, reviewers get thrown back because they did not test it. Note how I explicitly mentioned this in my previous reply.

                                    Edit: I mixed you up with the OP/diff author, text adjusted.

                                    1. 4

                                      Thank you for the review kn, very much appreciated. I hastily reposted an old version of the patch. I’ll make sure the diff applies cleanly in my reply and fix up the SIGCHLD typo.

                                    2. 6

                                      Maybe this is how OpenBSD runs things, if that’s the culture there, that’s fine, but don’t expect it to attract very many contributors.

                                      Ah but whose job is it to reply to every mail? Whose job is it to apologize if whoever had the first job failed to deliver? What is this sentient entity called OpenBSD that supposedly runs things? Does it have the power to appoint an individual for such a role?

                                      1. -2

                                        What is this sentient entity called OpenBSD that supposedly runs things?

                                        It’s called the OpenBSD Foundation. You can read about it on its website. This year, it has about half a million to spend on answering your other questions.

                                        1. 8

                                          You gotta be joking. They provide funding for the project. They don’t run the project.

                                          1. 0

                                            I assumed that in order to provide funding for a project you need to decide what to fund and what not to fund, and that sort of decision-making is called “running the project”, but I guess I was mistaken, my bad.

                                            1. 7

                                              I just decided to fund you as my personal assistant. Your salary is $20 a month, you work 24/7, aren’t you so glad that I run you now? Hand over the keys to your house by the way, because with this decision, I run it…

                                              Actually the OpenBSD Foundation isn’t the OpenBSD Project. The OpenBSD Foundation doesn’t own OpenBSD, and there are things it cannot do because it does not own OpenBSD. It can’t hand out commit bits, it can’t change the website, it can’t turn people into mailing list admins.. it does not run OpenBSD. If someone or something really “runs” OpenBSD, I’d say it’s Theo… and no, Theo doesn’t run the Foundation. The Foundation doesn’t run Theo either. The Foundation doesn’t decide what Theo or the individual developers (volunteers mainly!) of the project do, though they can choose to support whatever it is by providing funding.

                                              1. -1

                                                What is this sentient entity called OpenBSD that supposedly runs things?

                                                If someone or something really “runs” OpenBSD, I’d say it’s Theo

                                                1. 5

                                                  Which leads to the follow up question.. you want him to force the volunteers to reply to every mail and apologize for every mail that wasn’t responded to? Or you want him to employ people for that purpose? Out of his own pocket?

                                                  Sorry, I just don’t see the issue of some messages directed at a volunteer-driven software group going unresponded to because the volunteers happened to be volunteering their time for something else at the time (or whatever the reason).

                                                  If people are so entitled to responses, I no longer wonder why some people get burned out on OSS development. I wouldn’t, at least not for that reason, because I have no trouble ignoring issues I don’t have time for. It is my own time.

                                                  IMHO kn is right, there is nothing to apologize for.

                                                  1. 0

                                                    I’ve seen small businesses provide better support to their users and developers on far less budget than OpenBSD has.

                                                    For the past 5 or so years they’ve received hundreds of thousands of dollars each year, and each year they had a surplus averaging ~$100k that they didn’t seem to use for anything.

                                                    Are you telling me they can’t afford to pay someone to say, “we’re looking into this”, or “we’re sorry the patch didn’t compile”, or even setup an automated patch submission system? Because if you are, according to their public finances page, that would be a lie.

                                                    1. 3

                                                      The OpenBSD Project isn’t a business. I think you’re just trolling here and it’s dumb.

                                                      1. -1

                                                        I’m not trolling, and I’m done with this conversation because it’s clear it’s going nowhere fast.

                                                        EDIT: and to be clear, from the OSS projects I’ve seen — even those that do not have a half-million dollar budget and a foundation — still somehow manage to reply to developers who’ve put in the time and effort into submitting a pull request. They also have pull requests. And automated build systems. And aren’t stuck in 1990 with their version control system.

                                                        1. 6

                                                          You are generalizing from one example and you don’t know our comunity well enough to judge it.

                                                          During almost 10 years now I have committed many patches from other contributors and never had my own patches go ignored, which is why I stuck around in OpenBSD in the first place.

                                                          1. 0

                                                            You are generalizing from one example and you don’t know our comunity well enough to judge it.

                                                            And how do you know how well I know the OpenBSD community? You have no clue.

                                                            Over on Mastodon I pointed out that OpenBSD “perpetuates false and negative stereotypes that security people don’t care about usability, or that security must come at a cost of usability”.

                                                            That’s a fact. And then OpenBSD developer @mulander jumped in to call me a troll, and on top of it, demand that I work for free to submit patches to the project. So I pointed out to him how the OpenBSD community treats those who work for free and submit patches.

                                                            I’ve observed this project for many years, and I think it gets a bit too much hype on Lobsters lately for delivering a terrible user experience. Sure, there are lots of things to praise about it, but I don’t see anyone criticizing it for its glaringly obvious faults, so the end result is a community that is delusional, and a harmful role model.

                                                            1. 4

                                                              Link the thread so people can judge by themselves.

                                                              Also link yourself trying to spin the thing around on Mastodon and on twitter.

                                                              1. 0

                                                                I did, see my reply below from before your comment. But sure I should have linked it here as well.

                                                              2. 1

                                                                Your opinions are not facts. I don’t think the “community” is what’s delusional here.

                                                                1. -1

                                                                  It’s not an opinion, it’s a fact, and one OpenBSD fanbois don’t dispute.

                                                            2. 1

                                                              Great. I hope you feel better now that you’ve got this all out of your system.

                                      2. 4

                                        There is nothing to apologize for. It is a volunteer project. Developers are people who live lives, not borg drones assimilating other people’s patches.

                                        1. 4

                                          All of your comments in this thread of inappropriate. They are inappropriate regardless of whether other folk’s comments are or are not appropriate and regardless of whether they do or do not contain true statements.

                                          Please drop the issue, do not bring it back up, and do not engage in this style of discussion again on lobste.rs.

                                      3. 3

                                        What stsp said, but also, can you link us to the thread?

                                          1. 2

                                            I just get

                                            I expected an e-mail address, but none was defined.

                                            1. 2

                                              Sorry I’m not entirely sure what the best way is to post a link to a thread on the OpenBSD listserv. If you log in you should be able to see the thread.

                                              EDIT: use this http://openbsd-archive.7691.n7.nabble.com/lib-libfuse-Handle-signals-that-get-sent-to-any-thread-tp352472p353099.html

                                              1. 2

                                                marc.info works pretty well. I’d say it’s the preferred interface for most people.

                                                1. 1

                                                  thanks for the pointer

                                        1. 2

                                          Not a contributor, but I figure it might help to point out what patch you sent.

                                        1. 4

                                          I just finished my first week at a new job for which we moved acros the country. Tomorrow is full of exploratory dog walks, followed by some OpenBSD hacking. I’m expecting some new furniture on Sunday so will spent most of the day putting it all together.

                                          This is also my first real free weekend after 4 years working at a startup… I’m not sure what to do with so much free time O_O

                                          1. 5

                                            Man I remember when Google+ was opened to the public and there was actually something going on there, and people were “hoping” it could win facebook users over (retroactively naive). Circles seemed superficially cool, there were interesting groups, some people really spent time to become popular…

                                            In a certain way it reminds me of Mastodon today, although I would say that due to quite relevant technical factors, Mastodon will be able to avoid G+’s fate.

                                            1. 4

                                              I had (like many others) similar hope for Google Wave. Nowadays I don’t trust any Google products to stick around and started self hosting my data whenever I can.

                                              1. 1

                                                Yeah, I had high hopes for it when it debuted. It really seemed like it had the chance to become a less obnoxious Twitter as well. It’s not entirely clear to me why it failed. I stopped using it after they redesigned it to look more like Twitter (lower information density), but I suspect that wasn’t what really did it in.

                                              1. 3

                                                Moving across the country, so lots of packing, logistics etc. This is a culmination point of a project I started 2 months ago with the goal of doing more BSD development on a daily basis. If someone wanted to sabotage me at this time one would only need to steal my bullet journal - I wouldn’t be able to pull all this off without it :D

                                                1. 2

                                                  Nice move and write-up by dbp. One nitpick:

                                                  “Postfix chosen because of security (Exim has a less than stellar history), and dovecot because it is simpler and faster than Courier (and more importantly, combined with Postfix frequently). Security is really important to me (as I want this system to be easy to mantain), so I decided to switch it. “

                                                  It makes Postfix look more security-focused than Dovecot. Might be the case since I haven’t looked at MTA’s in a while. I just recall that the Dovecot website highlights security as a primary focus with another statement offering a bounty for the “first” vulnerability found it in. If that’s BS self-promotion, then I’d have phrased it with Dovecot being the most secure component I could find with Postfix just having less vulnerabilities than others with similar functionality. Also, I’d move what use cases I could to Dovecot.

                                                  1. 6

                                                    Dovecot actually underwent a security audit passing with flying colors.

                                                    The Cure53 team were extremely impressed with the quality of the dovecot code. They wrote: “Despite much effort and thoroughly all-encompassing approach, the Cure53 testers only managed to assert the excellent security->standing of Dovecot. More specifically, only three minor security issues have been found in the codebase, thus translating to an exceptionally good outcome for Dovecot, and a true testament to the fact that keeping security promises is at the core of the Dovecot development and operations.”

                                                    https://wiki.mozilla.org/MOSS/Secure_Open_Source/Completed#dovecot

                                                    https://www.dovecot.org/list/dovecot/2017-January/106683.html

                                                    1. 1

                                                      Thanks for the links! Yeah, that’s impressive. I’ll save those.

                                                      Next question is what common thimgs does it lack that Postfix has? I figure carefully extending it a bit more with optional features might lead to something more secure than Postfix in those use cases.

                                                      1. 3

                                                        Postfix is a MTA whereas Dovecot is a POP3 and IMAP server. They’re really complementary.

                                                        1. 1

                                                          Thanks. I clearly will have to do some research on stuff I’ve forgotten in this area in the future. At this point, my memory fragments just say use some combo of Qmail, Dovecot, and Postfix. I only had Postfix in the list since a lot of people who favor quality software use it. Guess I forgot the rest. (shrugs)

                                                  1. 1

                                                    Lack of any feedback from the repo maintainers would be such a let down for me.

                                                    1. 2

                                                      The last time this project had significant activity was around 20 days ago. The pull request is aged at 19 days - I think it’s just unlucky timing. Perhaps the maintainer went on for holiday with his family? Perhaps he just finished a lot of work around the project and needs a rest from it? Perhaps he just doesn’t feel like looking at it now - it’s volunteer time.

                                                      Point is, don’t assume your work is unappreciated because you didn’t get a reply - open source is hard work but once you put things into the public time works for you, someone will see it sooner or later :)

                                                      1. 2

                                                        Rubygems.org is maintained by the RubyTogether non-profit, which pays devs hourly for maintenance. The money is nice but it likely means it will take a bit to get approved for work. I’m guessing this also isn’t high priority – Rails 5.1 is working just fine.

                                                    1. 1

                                                      What I’m curious to know is if “Mark” is a contributor or just someone use uses Redis (or doesn’t even do that). I my eyes this changes the debate from a discussion about the project and it’s values to a random interjection from outside.

                                                      1. 4

                                                        “Mark” is a pretty well known person in the database world and a quite prolific FOSS contributor that has been raising this concern multiple times.

                                                        1. 3

                                                          apparently it will be decided by a twitter poll which is an absurd place to settle this as people completely unrelated to redis (not even in the IT field) can manipulate the result. This is farce.

                                                        1. 3
                                                          • remotely attending as technical support a demo/testing event of a work product
                                                          • playing urban terror with #openbsd-gaming this evening (drop by on IRC!)
                                                          • reading https://nostarch.com/seriouscrypto
                                                          • trekking a mountain trail with wife and the dog (tomorrow)
                                                          1. 15

                                                            Congrats and great work. I would like to emphasise the pull request as it’s also a nice read!

                                                            edit: @pushcx do you have stats pre & post update? I’m curious what impact this had on system load and performance if you have metrics for that.

                                                            1. 2

                                                              Didn’t track any stats beyond what alynpost linked already, no.

                                                            1. 26

                                                              Fight for Pareto’s law, look for the 20% of effort that will give you the 80% of results.

                                                              Perfect is enemy of good, first do it, then do it right, then do it better.

                                                              It’s either worth doing right or not worth doing at all.

                                                              I don’t identify with this manifesto, it feels more like something written for startup cowboys slinging code out than people wanting to write minimal, clean code.

                                                              Here is my take on a minimalist software engineer manifesto.

                                                              1. Keep it correct - measure twice, code once
                                                              2. Keep it simple - aim to solve a single task well
                                                              3. Keep it secure - only allow things required to perform your task
                                                              4. Keep it documented - your tool is useless, if no one knows how to use it
                                                              1. 3

                                                                When something isn’t already a solved problem, being able to identify potential correctness pitfalls depends upon having a functional implementation. In other words: write a prototype and throw it away (i.e., first do it and then do it right).

                                                                Of course, most code written for businesses isn’t actually original or solving original problems – the ‘prototype’ is somebody else’s implementation, written years before.

                                                                1. 2

                                                                  Sure. This is engineering 101. Build a model in scale.

                                                                  However the way this manifest is written it sounds like he would like to build a bridge then patch it up when cracks start to show up while people start driving through it - that is not a prototype and that is not engineering.

                                                                  1. 1

                                                                    About half the crustaceans are interpreting this manifesto the way you are, and about half are interpreting it the way I am, which probably means it’s poorly written!

                                                                    I’ve got a tendency to steelman essays like this, so it’s absolutely possible that this guy is actually defending cowboy coding, but that’s absolutely not how I read it.

                                                              1. 1

                                                                oh this is a delight! had no idea there were so many re-implementation efforts. would be nice to have this outside of a pdf, hah!

                                                                1. 4

                                                                  The link is actually a regular site, not a pdf. I agree that it reminds the in-browser pdf renderers a lot in style :D

                                                                  1. 1

                                                                    hahahah oh my goodness. got me. a lovely list indeed!

                                                                1. -3

                                                                  “Considered Harmful” Essays Considered Harmful (I think “considered dangerous” falls in the same category)

                                                                  It’s not difficult to use C correctly. Don’t blame your vulnerabilities on C when the real culprit is your own sloth.

                                                                  I’ll concede that C (and it’s API) has quite a few foot guns, but I’ve learned how to avoid them pretty effectively, and I should be able to expect the same from kernel devs. The whole “rewrite everything in <insert promising new lang here>” mentality doesn’t work for large projects (like kernels). To rewrite the Linux kernel in Rust would take months (even if you had all hands on deck). And, who’s to say that Rust wouldn’t change incompatibly three times in the middle?

                                                                  1. 21

                                                                    It’s not difficult to use C correctly.

                                                                    [citation needed]

                                                                    There is no evidence to suggest that large codebases written in C can maintain memory safety in the face of that. The counter evidence, that writing code in C/C++ tends to produce large volumes of vulnerabilities, for reasons that are explained by language choice, is plentify. To whit, every major OS (Windows, Linux, macOS), every major browser (Chrome, Firefox, Edge, Safari), every major anti-virus program, every major image parsing library, I can keep going for a while.

                                                                    Denialism about the dangers of memory unsafety is not productive, we need to move on to discussing how we address this.

                                                                    1. 0

                                                                      There is no evidence to suggest that large codebases written in C can maintain memory safety in the face of that.

                                                                      Using C correctly means not making large codebases. C isn’t a language for programming in the large.

                                                                      1. -1

                                                                        there is no evidence that large codebases in any language produces anything better.

                                                                        1. 7

                                                                          Yes there is. The default, failure mode of safe languages doing common things is not potential code injection. The default for C language is. Given same bug count, using C will lead to more severe problems. The field results confirm that from fuzzing to CVE’s.

                                                                          1. 4

                                                                            Yes there is. The default, failure mode of safe languages doing common things is not potential code injection.

                                                                            I don’t think this is wrong, exactly, but there’s a 100 exploits related to python pickle, etc. as counterexamples. And java serialize, etc.

                                                                            1. 3

                                                                              Do the memory-safe parts have the memory errors of C (a) at all or (b) as much? And do libraries in concurrency safe languages show same or less races as equivalent in multithreaded C?

                                                                              You’re going to find vulnerabilities in all of them. My side are saying C amplifies that number by default or others greatly reduce it by default. That’s all we’re saying. I think the evidence is already supporting that.

                                                                              1. 1

                                                                                amplify requires some comparative numbers.

                                                                                1. 2

                                                                                  The numbers on using C are that the common operations lead to piles of vulnerabilities with code injection. This happens a lot on average. It happens less with veterans but still happens. That’s irrefutable. The numbers on safe languages show the problems mostly lead to compiler failures or DOS’s from runtime checks. The burden of proof is on your side given your side’s stuff is getting smashed the hardest all the time whether the app is small or big.

                                                                                  What numbers do you have showing C is safer for average developer than Ada, Rust and so on? And I’m especially interested in fuzzing results of software to see how many potentially lead to code injection among new, half-ass, or just time-constrained programmers in C vs the same in safe, systems languages.

                                                                                  1. 1

                                                                                    you don’t even have good examples of large scale systems built using some other language that are substantially safer. Until you do, it’s just folklore.

                                                                            2. 0

                                                                              I see a real shortage of example of large-scale systems constructed in any language that are secure and bug free but I am happy to look at references. Like what do we have comparable to Qmail written in something better that has fewer bugs? I know that C has numerous limitations, but in CS we tend to embrace projects that claim a win by hiding a problem by e.g. using pragmas to do the things that are the most buggy as if pushing the problem into the corner made it go away.

                                                                              And the code injection bugs I see are all example of bad engineering - not of bad programming.

                                                                              1. 3

                                                                                There’s bugs and there’s serious bugs that the language causes. The latter are what hackers hit the most. The latter are what we’re talking about, not just bugs in general. The size of the program also doesnt matter since the safe language is immune to the latter by design. Scaling code up just increases odds of severe vulnerabilities in the unsafe, control language.

                                                                                Java and .NET apps are what to look at if you want big ones. Very few CVE’s posted on the apps of the kind you see in C apps. The ones that are posted are usually in C/C++ runtimes or support libraries of such languages. That just illustrates the problem more. The languages whose runtimes arent C have fewer of those since they’re immune or contain them by design.

                                                                                1. 1

                                                                                  My impression is that a) the reasons that those c/c++ runtimes show up so much is that these language delegate the most dangerous code such as parsing of raw input or packets or complex interaction with the OS to the C/C++ runtimes where it is possible to do that work and b) the same errors show up in different form in different languages. The massive prevalence of scripting exploits is not due to C but to lazy interface construction where, for example, user inputs are treated as parts of database scripts etc etc. I do not think that “do all the hard stuff in pragmas or C libraries” actually does limit vulnerabilities.

                                                                                  1. 1

                                                                                    “where it is possible to do that work”

                                                                                    The first part is true. That part isn’t. They think lower-level language is better for speed, bit handling, or OS interface. The second part implies you need C to do that work. There’s systems languages which can do that work with more safety than C. So, it’s “possible to do that work” in them without C’s drawbacks. Many low-level programs and OS’s were written in PL/0, PL/S, Ada, Modula-2, Oberon, Modula-3, Clay, and so on. They’re safe by default turning it off only where you need to. C doesn’t do that since it’s designers didn’t care when they were hacking on a PDP-11 for personal use.

                                                                                    “b) the same errors show up in different form in different languages. The massive prevalence of scripting exploits is not due to C but to lazy interface construction where, for example, user inputs are treated as parts of database scripts etc etc.”

                                                                                    Aside from something language-specific, the logic errors that happen in scripting languages can happen in C, too. You get those errors plus C’s errors plus the catastrophic effect that comes with them being in C. Let’s say you wrote the interpreter in Ada or Rust with safety-checks on. Most of the errors in the interpreter won’t lead to hacks. The extensions would have same property if building on base language like how extensions to C-based programs are often in C having same problems. Platforms like Java that built libraries on C are hit heavily in those C dependencies.

                                                                                    Additionally, the extensions could leverage aspects of these languages, such as type or module systems, designed for knocking out integration errors. Finally, if it’s Ada 2012 and SPARK, they can eliminate runtime checks in performance-critical code by using the provers to show they’re not needed if specific pre-conditions pass early on. Unlike Frama-C, they get a good baseline on code they hurried and highest assurance of what they proved.

                                                                                    1. 1

                                                                                      Data would help. These arguments by what seems sensible to different people don’t go anywhere.

                                                                        2. 16

                                                                          To rewrite the Linux kernel in Rust would take months (even if you had all hands on deck).

                                                                          Months? It would take at least 10 years, regardless of headcount.

                                                                          I’ve learned how to avoid them pretty effectively, and I should be able to expect the same from kernel devs.

                                                                          I’m impressed with your abilities, but then something nags me about the order-of-magnitude mistake in your rewrite estimate. Hmm.

                                                                          1. 13

                                                                            It’s not difficult to use C correctly. Don’t blame your vulnerabilities on C when the real culprit is your own sloth. I’ll concede that C (and it’s API) has quite a few foot guns, but I’ve learned how to avoid them pretty effectively, and I should be able to expect the same from kernel devs. The whole “rewrite everything in ” mentality doesn’t work for large projects (like kernels). To rewrite the Linux kernel in Rust would take months (even if you had all hands on deck). And, who’s to say that Rust wouldn’t change incompatibly three times in the middle?

                                                                            I suggest you read the linked article first. The title is clickbait but the content is solid. No one even mentioned Rust or anything else… The guy talks on their effort to reduce the foot guns in the kernel code…

                                                                            Here is a quote for the lazy:

                                                                            Kees Cook gave a presentation on some of the dangers that come with programs written in C. In particular, of course, the Linux kernel is mostly written in C, which means that the security of our systems rests on a somewhat dangerous foundation. But there are things that can be done to help firm things up by “Making C Less Dangerous” as the title of his talk suggested.

                                                                            1. 4

                                                                              I suggest you read the linked article first.

                                                                              Ok, you got me, I only skimmed the article and I didn’t see any mention of rewrite until the comments (it was literally the first response to the second comment). Although I do hear that mentality about other large projects (such as Firefox) as well. I guess I should’ve said “Clickbait considered harmful” ;-)

                                                                              I’ve read some more of the article and he seems to know what he’s talking about but I would like to see the original talk.

                                                                              As far as reducing foot guns, I guess Linux did start out as just one guy so I can understand a lot of foot shooting, but it’s been years and I would’ve thought that things like VLAs would’ve been avoided in the kernel. Then agian, I’ve never worked on a project as large as Linux so i guess I’m not the best judge of such things.

                                                                              1. 4

                                                                                Ok, you got me, I only skimmed the article and I didn’t see any mention of rewrite until the comments (it was literally the first response to the second comment). Although I do hear that mentality about other large projects (such as Firefox) as well.

                                                                                Agreed. It’s annoying as hell, and the loud-mouths never do the work.

                                                                                I guess I should’ve said “Clickbait considered harmful” ;-)

                                                                                Funny because the talk is titled ‘Making C Less Dangerous’ - the lwn reporter is actually responsible for the horrible title that misrepresents the content and invites rewrite talks. I think this is the first time I’m using the lobste.rs ‘suggest’ a new title option to rename the link to ‘Making C Less Dangerous’ disrespecting the reporters chosen title. This is an abstract of the talk so keep the title close to the content.

                                                                            2. 7

                                                                              Literally 20+ years of unending computer security exploits disagree with you.

                                                                            1. 6

                                                                              He asked: why is there no argument to memcpy() to specify the maximum destination length?

                                                                              That’s the third one.

                                                                              If you really insist, #define safe_memcpy(d, s, dn, sn) memcpy(d, s, min(dn, sn))?

                                                                              1. 4

                                                                                Yeah, also, I don’t understand why would they want that.

                                                                                Imagine calling memcpy(d, 10, s, 15), and having your data not copied entirely, having your d buffer with cropped data. Garbage, essentially. How would that be better?

                                                                                edit: to be clear, I’m not complaining about your suggestion, but about the reasoning of the presenter on this.

                                                                                1. 4

                                                                                  Yeah, also, I don’t understand why would they want that.

                                                                                  Imagine calling memcpy(d, 10, s, 15), and having your data not copied entirely, having your d buffer with cropped data. Garbage, essentially. How would that be better?

                                                                                  Cropped data would be a logic error in your application. With standard memcpy the additional 5 bytes overwrite whatever is in memory after the d buffer. This can even enable an attacker to introduce execution of their own code. That’s why ie. Microsoft ships a memcpy_s.

                                                                                  Reading materials:

                                                                                  1. 8

                                                                                    But the unanswered question is why you’re calling memcpy(d, s, 15) instead of memcpy(d, s, 10)? At some level the problem is calling the function with the wrong argument, and adding more arguments maybe doesn’t help.

                                                                                    1. 4

                                                                                      Every security exploit can be drilled down to “why were you doing this!”. If there was an obvious answer, security exploit would have been a thing of the past. Meanwhile advocating harm reduction is as good as we can get because even if calling memcpy with a smaller destination is wrong to begin with, truncated data still has a more chance to end up with non-exploitable crash than plain old buffer overflow that often end up with reliable code exec.

                                                                                      1. 3

                                                                                        But why do we assume this extra parameter is better than the other parameter which we have assumed is incorrect? Why not add another extra parameter? memcpy_reallysafe(dest, src, destsize, srcsize, destsize_forserious, doublechecksize)

                                                                                        1. 3

                                                                                          Because in ten years a line of code can change and the assumptions that made one variable the right one will break. Suddenly you got the wrong variable in there. Personally, I think this is where asserts belong, to codify the assumptions over a long span of time and multiple developers.

                                                                                          1. 3

                                                                                            A common use case of memcpy is to copy a buffer over another. The way program are structure we often end up with srcsize and dstsize that matches their buffer. The error come from the implicit contract that srcsize is always at least bigger than dstsize. Sure, good code would ensure this is always true. Actual code had many instance where it is not. Adding dstsize to memcpy means that this contract is now explicit and can be asserted by the actual function that put this contract in place.

                                                                                            I mean, at this point we are not arguing of hypothetical scenario, we have a whole history of this bug class happening over and over again. Simply keeping track of the semantic (Copy one buffer to the other) and asking for all the properties required (Buffer and their size) is a low effort and easy way to prevent many of those bug.

                                                                                            1. 1

                                                                                              Yeah, keeping track of the buffer size is a very good idea. But if you want it to always be correct, it should be done without requiring the programmer to manually carry the buffer size along in a separate variable from the buffer pointer.

                                                                                              Either something like “Managed C++”, where the allocator data structures are queried to figure out the size of the buffer, or something like Rust slices:

                                                                                              typedef struct {
                                                                                                  char *ptr;
                                                                                                  size_t len;
                                                                                              } slice_t;
                                                                                              slice_t slice(slice_t slice, size_t start, size_t end) {
                                                                                                  assert(start <= end);
                                                                                                  assert((end - start) <= slice.len);
                                                                                                  slicet.ptr += start;
                                                                                                  slice.len = end - start;
                                                                                                  return slice;
                                                                                              }
                                                                                              slice_t slice_front(slice_t slice, size_t start) {
                                                                                                  assert(start <= slice.len);
                                                                                                  slice.ptr += start;
                                                                                                  slice.len -= start;
                                                                                                  return slice;
                                                                                              }
                                                                                              slice_t slice_back(slice_t slice, size_t end) {
                                                                                                  assert(end <= slice.len);
                                                                                                  slice.len = end;
                                                                                                  return slice;
                                                                                              }
                                                                                              void slicecpy(slice_t dest, slice_t src) {
                                                                                                  assert(dest.len == src.len);
                                                                                                  memcpy(dest, dest.len, src);
                                                                                              }
                                                                                              

                                                                                              The point being to make it harder to mix up which len goes with which ptr, plus providing a assert-assisted pointer manipulation in addition to the safe memcpy itself. A safe abstraction needs to account for the entire life cycle of its bounds check, not just the point of use.

                                                                                              Also, this would really, really benefit from templates.